Azure R3M25 pre-ship, While inserting the reference data for BulkPyrolysisMethod, the ingestion is failing, and the airflow log does not provide any meaningful information that helps debug the issue

When executing the request "Ingest RefData BulkPyrolysisMethod" the from Postman collection "RAFS DDMS Reference Data Ingestion Manifests - MVP" Airflow Log RunId: 00a7c7f8-555c-445e-a7bd-4a14b396a114 Task: validate_manifest_schema_task

Click to expand

2025-04-02, 11:47:09 UTC] {local_task_job_runner.py:123} ▶️ Pre task execution logs [2025-04-02, 11:47:10 UTC] {logging_mixin.py:190} INFO - user_id in Context Initialization is b70184aa-e9b7-4577-b58c-ccde1c79d631 [2025-04-02, 11:47:11 UTC] {azure_credentials.py:39} INFO - Initializing Azure credentials object with auth_type: None and auth_properties: None [2025-04-02, 11:47:11 UTC] {azure_credentials.py:54} INFO - No auth type was provided. Resolving auth type... [2025-04-02, 11:47:11 UTC] {azure_credentials.py:72} INFO - No auth properties passed. Resolving auth properties.. [2025-04-02, 11:47:11 UTC] {azure_credentials.py:94} INFO - Client secret fetch type set to Plain Text [2025-04-02, 11:47:11 UTC] {azure_credentials.py:51} INFO - Auth type: AzureAuthType.CLIENT_SECRET [2025-04-02, 11:47:11 UTC] {taskinstance.py:3313} ERROR - Task failed with exception Traceback (most recent call last): File "/home/airflow/.local/lib/python3.11/site-packages/jsonschema/validators.py", line 1137, in resolve_fragment document = document[part] ~~~~~~~~^^^^^^ KeyError: 'definitions' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 768, in _execute_task result = _execute_callable(context=context, **execute_callable_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 734, in _execute_callable return ExecutionCallableRunner( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/utils/operator_helpers.py", line 252, in run return self.func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/baseoperator.py", line 424, in wrapper return func(self, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/osdu_airflow/operators/validate_manifest_schema.py", line 83, in execute valid_manifest_file, skipped_ids = schema_validator.ensure_manifest_validity( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/osdu_ingestion/libs/validation/validate_schema.py", line 444, in ensure_manifest_validity valid_entities, not_valid_entities = self._validate_manifest_section( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/osdu_ingestion/libs/validation/validate_schema.py", line 408, in _validate_manifest_section self._validate_entity(entity) File "/home/airflow/.local/lib/python3.11/site-packages/osdu_ingestion/libs/validation/validate_schema.py", line 316, in _validate_entity self._validate_against_schema(schema, entity) File "/home/airflow/.local/lib/python3.11/site-packages/osdu_ingestion/libs/validation/validate_schema.py", line 336, in _validate_against_schema validator.validate(data) File "/home/airflow/.local/lib/python3.11/site-packages/jsonschema/validators.py", line 420, in validate for error in self.iter_errors(*args, **kwargs): File "/home/airflow/.local/lib/python3.11/site-packages/jsonschema/validators.py", line 354, in iter_errors for error in errors: File "/home/airflow/.local/lib/python3.11/site-packages/jsonschema/_validators.py", line 305, in properties yield from validator.descend( File "/home/airflow/.local/lib/python3.11/site-packages/jsonschema/validators.py", line 402, in descend for error in errors: File "/home/airflow/.local/lib/python3.11/site-packages/jsonschema/_validators.py", line 335, in allOf yield from validator.descend(instance, subschema, schema_path=index) File "/home/airflow/.local/lib/python3.11/site-packages/jsonschema/validators.py", line 402, in descend for error in errors: File "/home/airflow/.local/lib/python3.11/site-packages/jsonschema/_validators.py", line 284, in ref yield from validator._validate_reference(ref=ref, instance=instance) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/jsonschema/validators.py", line 447, in _validate_reference scope, resolved = resolve(ref) ^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/jsonschema/validators.py", line 1067, in resolve match = self._find_in_subschemas(url) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/jsonschema/validators.py", line 1056, in _find_in_subschemas subschema = self.resolve_fragment(subschema, fragment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/osdu_ingestion/libs/validation/validate_schema.py", line 100, in resolve_fragment document = super().resolve_fragment(document, fragment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/airflow/.local/lib/python3.11/site-packages/jsonschema/validators.py", line 1139, in resolve_fragment raise exceptions._RefResolutionError( jsonschema.exceptions._RefResolutionError: Unresolvable JSON pointer: 'definitions/osdu:wks:AbstractReferenceType:1.0.0' [2025-04-02, 11:47:11 UTC] {taskinstance.py:1226} INFO - Marking task as FAILED. dag_id=Osdu_ingest, task_id=validate_manifest_schema_task, run_id=00a7c7f8-555c-445e-a7bd-4a14b396a114, execution_date=20250402T114655, start_date=20250402T114710, end_date=20250402T114711 [2025-04-02, 11:47:11 UTC] {taskinstance.py:341} ▶️ Post task execution logs