Manifest Ingestion is failing during searching record on IBM Pre-Ship environment: R3 M11
We are trying to run the collection which converts SEGY to OpenVDS, however while running the manifest ingestion to create the pre-requisite records, getting the below error at Airflow: AIRFLOW_CTX_DAG_OWNER=*** AIRFLOW_CTX_DAG_ID=Osdu_ingest AIRFLOW_CTX_TASK_ID=provide_manifest_integrity_task AIRFLOW_CTX_EXECUTION_DATE=2022-06-14T19:57:44.719948+00:00 AIRFLOW_CTX_DAG_RUN_ID=d208bfa1-b496-4aae-a772-75cca57fd892 [2022-06-14 19:57:59,396] {logging_mixin.py:104} INFO - env_vars_enabled ************************ true [2022-06-14 19:57:59,396] {logging_mixin.py:104} INFO - cloud provider ******************* ibm [2022-06-14 19:57:59,396] {logging_mixin.py:104} INFO - Inside if [2022-06-14 19:58:00,465] {authorization.py:137} ERROR - {"code":500,"reason":"Search error","message":"Error processing search request"} [2022-06-14 19:58:00,465] {taskinstance.py:1501} ERROR - Task failed with exception Traceback (most recent call last): File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1157, in _run_raw_task self._prepare_and_execute_task_with_callbacks(context, task) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1331, in _prepare_and_execute_task_with_callbacks result = self._execute_task(context, task_copy) File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1361, in _execute_task result = task_copy.execute(context=context) File "/home/airflow/.local/lib/python3.8/site-packages/osdu_airflow/operators/ensure_manifest_integrity.py", line 71, in execute manifest, skipped_ids = manifest_integrity.ensure_integrity( File "/home/airflow/.local/lib/python3.8/site-packages/osdu_ingestion/libs/validation/validate_referential_integrity.py", line 265, in ensure_integrity manifest, missing_entities = self._ensure_manifest_integrity( File "/home/airflow/.local/lib/python3.8/site-packages/osdu_ingestion/libs/validation/validate_referential_integrity.py", line 215, in _ensure_manifest_integrity self._ensure_external_references_integrity(manifest_analyzer) File "/home/airflow/.local/lib/python3.8/site-packages/osdu_ingestion/libs/validation/validate_referential_integrity.py", line 190, in _ensure_external_references_integrity missing_external_ids.update(self._find_missing_external_ids(external_references_entity_id)) File "/home/airflow/.local/lib/python3.8/site-packages/osdu_ingestion/libs/validation/validate_referential_integrity.py", line 99, in _find_missing_external_ids found_ids = search_handler.search_records() File "/home/airflow/.local/lib/python3.8/site-packages/osdu_ingestion/libs/search_record_ids.py", line 154, in search_records response = self.search_client.query_records(query_request=search_query_request) File "/home/airflow/.local/lib/python3.8/site-packages/osdu_api/clients/search/search_client.py", line 29, in query_records return self.make_request(method=HttpMethod.POST, url='{}{}'.format(self.search_url, '/query'), File "/home/airflow/.local/lib/python3.8/site-packages/osdu_api/clients/base_client.py", line 222, in make_request response = self._send_request_with_token_refresher(headers, method, url, data, params) File "/home/airflow/.local/lib/python3.8/site-packages/osdu_api/auth/authorization.py", line 159, in _wrapper return send_request_with_auth_header(_token_refresher, File "/home/airflow/.local/lib/python3.8/site-packages/osdu_api/auth/authorization.py", line 138, in send_request_with_auth_header raise e File "/home/airflow/.local/lib/python3.8/site-packages/osdu_api/auth/authorization.py", line 135, in send_request_with_auth_header response.raise_for_status() File "/home/airflow/.local/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://os-search-ibm.osdu.svc.cluster.local:8080/api/search/v2/query [2022-06-14 19:58:00,471] {taskinstance.py:1544} INFO - Marking task as FAILED. dag_id=Osdu_ingest, task_id=provide_manifest_integrity_task, execution_date=20220614T195744, start_date=20220614T195758, end_date=20220614T195800 [2022-06-14 19:58:00,529] {local_task_job.py:151} INFO - Task exited with return code 1
From the above log, it is not clear what is not able to find and since it is getting 500 internal server error, there is something wrong with the search service, please look into it.