Azure - 3 issues with pre-ship environment
Issues with Azure pre-ship environment.
-
Azure is now using Entitlements v2 and because of this, the requests in the postman collection (from the Platform Validation project) need to be modified.
When Entitlements v1 was being used, the ACL being formed for Azure had group name owner and viewer instead of owners and viewers (singular vs plural). So special logic was added while creating the ACL e.g. cp = pm.environment.get("cloud_platform"); if (cp == "Azure") { pm.environment.set("New_OwnerDataGroup", "data.default.owner"); pm.environment.set("New_ViewerDataGroup", "data.default.viewer"); } else{ pm.environment.set("New_OwnerDataGroup", "data.default.owners"); pm.environment.set("New_ViewerDataGroup", "data.default.viewers"); }Now since Entitlements v2 is being used the special logic is not needed. The above logic just needs to be replaced with pm.environment.set("New_OwnerDataGroup", "data.default.owners"); pm.environment.set("New_ViewerDataGroup", "data.default.viewers");
-
The second issue is with Schema API. In the previous version and still for other CSP at this point in time the following request made was successful GET https://{{SCHEMA_HOST}}/api/schema-service/v1/schema/{{data-partition-id}}:wks:master-data--Well:1.0.0
wherein Azure pre-ship environment is failing. It requires the request to be modified as GET ttps://{{SCHEMA_HOST}}/api/schema-service/v1/schema?id={{data-partition-id}}:wks:master-data--Well:1.0.0
-
The issue with manifest ingestion. The Manifest ingestion is failing (Osdu_ingest), it is not able to find the schema definition (Do not know whether is it because of issue 2) Airflow Log: [2021-11-18 17:40:32,490] {taskinstance.py:901} INFO - Executing <Task(ValidateManifestSchemaOperator): validate_manifest_schema_task> on 2021-11-18T17:40:18.417189+00:00 [2021-11-18 17:40:32,492] {standard_task_runner.py:54} INFO - Started process 936 to run task [2021-11-18 17:40:32,514] {standard_task_runner.py:77} INFO - Running: ['airflow', 'run', 'Osdu_ingest', 'validate_manifest_schema_task', '2021-11-18T17:40:18.417189+00:00', '--job_id', '43250', '--pool', 'default_pool', '--raw', '-sd', 'DAGS_FOLDER/osdu-ingest-r3.py', '--cfg_path', '/tmp/tmpb80lgskv'] [2021-11-18 17:40:32,515] {standard_task_runner.py:78} INFO - Job 43250: Subtask validate_manifest_schema_task [2021-11-18 17:40:32,516] {cli_action_loggers.py:68} DEBUG - Calling callbacks: [<function default_action_log at 0x7fb91bcae730>] [2021-11-18 17:40:32,548] {settings.py:233} DEBUG - Setting up DB connection pool (PID 936) [2021-11-18 17:40:32,548] {settings.py:241} DEBUG - settings.configure_orm(): Using NullPool [2021-11-18 17:40:32,592] {logging_mixin.py:112} INFO - Running %s on host %s <TaskInstance: Osdu_ingest.validate_manifest_schema_task 2021-11-18T17:40:18.417189+00:00 [running]> airflow-worker-3.airflow-worker.airflow.svc.cluster.local [2021-11-18 17:40:32,646] {init.py:101} DEBUG - Preparing lineage inlets and outlets [2021-11-18 17:40:32,646] {init.py:137} DEBUG - inlets: [], outlets: [] [2021-11-18 17:40:32,745] {validate_manifest_schema.py:74} DEBUG - Manifest data: {'ReferenceData': [{'data': {'AttributionAuthority': 'OSDU', 'Descrption': 'Auto Test', 'Code': 'SpudCode_999207272833', 'Source': 'Auto Test Published/FacilityEventType.1.0.0.xlsx; commit SHA 38615b34.', 'Name': 'Spud_999207272833'}, 'kind': 'opendes:wks:reference-data--FacilityEventType:1.0.0', 'legal': {'legaltags': ['opendes-Well-Legal-Tag-Test3460168'], 'otherRelevantDataCountries': ['US']}, 'id': 'opendes:reference-data--FacilityEventType:SPUD_DATE_999207272833', 'acl': {'viewers': ['data.default.viewers@opendes.contoso.com'], 'owners': ['data.default.owners@opendes.contoso.com']}}, {'data': {'AttributionAuthority': 'OSDU', 'Descrption': 'Auto Test', 'Code': 'DEPTH_DATUM_ELEVCode_999207272833', 'Source': 'Auto test Published/VerticalMeasurementPath.1.0.0.xlsx; commit SHA 38615b34.', 'Name': 'DEPTH_DATUM_ELEV_999207272833'}, 'kind': 'opendes:wks:reference-data--VerticalMeasurementPath:1.0.0', 'legal': {'legaltags': ['opendes-Well-Legal-Tag-Test3460168'], 'otherRelevantDataCountries': ['US']}, 'id': 'opendes:reference-data--VerticalMeasurementPath:DEPTH_DATUM_ELEV_999207272833', 'acl': {'viewers': ['data.default.viewers@opendes.contoso.com'], 'owners': ['data.default.owners@opendes.contoso.com']}}, {'data': {'AttributionAuthority': 'OSDU', 'Descrption': 'Auto Test', 'Code': 'AliasForWell_999207272833', 'Source': 'Auto Test Published/FacilityEventType.1.0.0.xlsx; commit SHA 38615b34.', 'Name': 'Alias_Auto_Test_999207272833'}, 'kind': 'opendes:wks:reference-data--AliasNameType:1.0.0', 'legal': {'legaltags': ['opendes-Well-Legal-Tag-Test3460168'], 'otherRelevantDataCountries': ['US']}, 'id': 'opendes:reference-data--AliasNameType:WELL_NAME_999207272833', 'acl': {'viewers': ['data.default.viewers@opendes.contoso.com'], 'owners': ['data.default.owners@opendes.contoso.com']}}, {'data': {'AttributionAuthority': 'OSDU', 'Descrption': 'Auto Test', 'Code': 'FacilityNameForWell_999207272833', 'Source': 'Auto Test Published/FacilityEventType.1.0.0.xlsx; commit SHA 38615b34.', 'Name': 'FacilityName_Auto_Test_999207272833'}, 'kind': 'opendes:wks:reference-data--FacilityType:1.0.0', 'legal': {'legaltags': ['opendes-Well-Legal-Tag-Test3460168'], 'otherRelevantDataCountries': ['US']}, 'id': 'opendes:reference-data--FacilityType:WELL_999207272833', 'acl': {'viewers': ['data.default.viewers@opendes.contoso.com'], 'owners': ['data.default.owners@opendes.contoso.com']}}], 'MasterData': [{'data': {'OrganisationName': 'Auto_Test_999207272833', 'Source': 'AUTO test'}, 'kind': 'opendes:wks:master-data--Organisation:1.0.0', 'legal': {'legaltags': ['opendes-Well-Legal-Tag-Test3460168'], 'otherRelevantDataCountries': ['US']}, 'id': 'opendes:master-data--Organisation:Auto_Test_999207272833', 'acl': {'viewers': ['data.default.viewers@opendes.contoso.com'], 'owners': ['data.default.owners@opendes.contoso.com']}}, {'data': {'FacilityID': 'FaciltyIdAutoTest_999207272833', 'NameAliases': [], 'SpatialLocation': {'Wgs84Coordinates': {'features': [{'geometry': {'coordinates': [3.51906683, 55.68101428], 'type': 'Point'}, 'type': 'Feature', 'properties': {}}], 'type': 'FeatureCollection'}}, 'FacilityNameAlias': [{'AliasName': 'Alias_Auto_Test_999207272833', 'AliasNameTypeID': 'opendes:reference-data--AliasNameType:WELL_NAME_999207272833:'}], 'VerticalMeasurements': [{'VerticalMeasurementID': 'Kelly Bushing', 'VerticalMeasurementPathID': 'opendes:reference-data--VerticalMeasurementPath:DEPTH_DATUM_ELEV_999207272833:', 'VerticalMeasurement': 36.6}], 'GeoContexts': [], 'FacilityEvent': [{'EffectiveDateTime': '1999-06-03T00:00:00', 'FacilityEventTypeID': 'opendes:reference-data--FacilityEventType:SPUD_DATE_999207272833:'}], 'FacilityOperator': [{'FacilityOperatorID': 'FacilityOperatorIdAutoTest_999207272833', 'FacilityOperatorOrganisationID': 'opendes:master-data--Organisation:Auto_Test_999207272833:'}], 'Source': 'AUTO test NL_TNO', 'FacilityName': 'FacilityNameAutoTest_999207272833', 'FacilityTypeID': 'opendes:reference-data--FacilityType:WELL_999207272833:'}, 'kind': 'opendes:wks:master-data--Well:1.0.0', 'legal': {'legaltags': ['opendes-Well-Legal-Tag-Test3460168'], 'otherRelevantDataCountries': ['US']}, 'id': 'opendes:master-data--Well:999207272833', 'acl': {'viewers': ['data.default.viewers@opendes.contoso.com'], 'owners': ['data.default.owners@opendes.contoso.com']}}], 'kind': 'opendes:wks:Manifest:1.0.0'} [2021-11-18 17:40:32,790] {connectionpool.py:230} DEBUG - Starting new HTTP connection (1): schema.osdu-azure.svc.cluster.local:80 [2021-11-18 17:40:33,173] {connectionpool.py:442} DEBUG - http://schema.osdu-azure.svc.cluster.local:80 "GET /api/schema-service/v1/schema/opendes:wks:Manifest:1.0.0 HTTP/1.1" 404 None [2021-11-18 17:40:33,175] {authorization.py:137} ERROR - {"error":{"code":404,"message":"Schema is not present","errors":[{"domain":"global","reason":"notFound","message":"Schema is not present"}]}} [2021-11-18 17:40:34,179] {connectionpool.py:230} DEBUG - Starting new HTTP connection (1): schema.osdu-azure.svc.cluster.local:80 [2021-11-18 17:40:34,574] {connectionpool.py:442} DEBUG - http://schema.osdu-azure.svc.cluster.local:80 "GET /api/schema-service/v1/schema/opendes:wks:Manifest:1.0.0 HTTP/1.1" 404 None [2021-11-18 17:40:34,575] {authorization.py:137} ERROR - {"error":{"code":404,"message":"Schema is not present","errors":[{"domain":"global","reason":"notFound","message":"Schema is not present"}]}} [2021-11-18 17:40:35,580] {connectionpool.py:230} DEBUG - Starting new HTTP connection (1): schema.osdu-azure.svc.cluster.local:80 [2021-11-18 17:40:35,830] {connectionpool.py:442} DEBUG - http://schema.osdu-azure.svc.cluster.local:80 "GET /api/schema-service/v1/schema/opendes:wks:Manifest:1.0.0 HTTP/1.1" 404 None [2021-11-18 17:40:35,831] {authorization.py:137} ERROR - {"error":{"code":404,"message":"Schema is not present","errors":[{"domain":"global","reason":"notFound","message":"Schema is not present"}]}} [2021-11-18 17:40:35,831] {validate_schema.py:175} ERROR - Error on getting schema of kind 'opendes:wks:Manifest:1.0.0' [2021-11-18 17:40:35,832] {validate_schema.py:176} ERROR - 404 Client Error: Not Found for url: http://schema.osdu-azure.svc.cluster.local/api/schema-service/v1/schema/opendes:wks:Manifest:1.0.0 [2021-11-18 17:40:35,832] {taskinstance.py:1150} ERROR - There is no schema for Manifest kind opendes:wks:Manifest:1.0.0