Pre Shipping issueshttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues2024-01-29T15:15:37Zhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/660Manifest Ingestion - Schema validation error when persistable reference not p...2024-01-29T15:15:37ZChad LeongManifest Ingestion - Schema validation error when persistable reference not provided## Introduction:
In manifest ingestion, a [validation step](https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/blob/master/osdu_ingestion/libs/validation/validate_schema.py?ref_type=heads) is performe...## Introduction:
In manifest ingestion, a [validation step](https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/blob/master/osdu_ingestion/libs/validation/validate_schema.py?ref_type=heads) is performed to validate the schema according to the schema in https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/Generated/abstract/AbstractMetaItem.1.0.0.json?ref_type=heads#L148 , the required field is `"kind"` and `"persistableReference"`.
This [MR](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/796) has been introduced with [ADR 188](https://community.opengroup.org/osdu/platform/system/storage/-/issues/188) where the UnitofMeasureID can be provided instead of persistable reference.
This results in a schema validation error when persistable reference is not provided. Example payload, in the meta block where the persistableReference is not provided.
```json
{
"executionContext": {
"acl": {
"owners": ["data.default.owners@osdu.example.com"],
"viewers": ["data.default.viewers@osdu.example.com"]
},
"legal": { "legaltags": ["osdu-public-usa-dataset"], "otherRelevantDataCountries": ["US"] },
"Payload": { "AppKey": "test-app", "data-partition-id": "osdu" },
"manifest": {
"kind": "osdu:wks:Manifest:1.0.0",
"MasterData": [
{
"id": "osdu:master-data--Well:Well_CL_883_0",
"data": {
"Source": "TNO",
"NameAliases": [
{
"AliasNameTypeID": "osdu:reference-data--AliasNameType:WELL_NAME:",
"AliasName": "ACA-0"
},
{ "AliasNameTypeID": "osdu:reference-data--AliasNameType:UWI:", "AliasName": "0" }
],
"GeoContexts": [
{
"GeoPoliticalEntityID": "osdu:master-data--GeoPoliticalEntity:Netherlands_Country:",
"GeoTypeID": "osdu:reference-data--GeoPoliticalEntityType:Country:"
},
{
"GeoPoliticalEntityID": "osdu:master-data--GeoPoliticalEntity:Limburg_Province:",
"GeoTypeID": "osdu:reference-data--GeoPoliticalEntityType:State%2FProvinceID:"
},
{
"GeoPoliticalEntityID": "osdu:master-data--GeoPoliticalEntity:L_Quadrant:",
"GeoTypeID": "osdu:reference-data--GeoPoliticalEntityType:Quadrant:"
}
],
"SpatialLocation": {
"Wgs84Coordinates": {
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": { "type": "Point", "coordinates": [5.98136045, 51.43503877] },
"properties": {}
}
]
}
},
"FacilityTypeID": "osdu:reference-data--FacilityType:Well:",
"FacilityOperators": [
{ "FacilityOperatorOrganisationID": "osdu:master-data--Organisation:ROVD:" }
],
"VerticalMeasurements": [
{
"VerticalCRSID": "osdu:reference-data--CoordinateReferenceSystem:MSL:",
"VerticalMeasurement": 100,
"VerticalMeasurementID": "Rotary Table",
"VerticalMeasurementPathID": "osdu:reference-data--VerticalMeasurementPath:Elevation:",
"VerticalMeasurementUnitOfMeasureID": "osdu:reference-data--UnitOfMeasure:M:"
}
],
"FacilityID": "ACA-0"
},
"legal": { "legaltags": ["osdu-public-usa-dataset"], "otherRelevantDataCountries": ["US"] },
"kind": "osdu:wks:master-data--Well:1.0.0",
"acl": {
"owners": ["data.default.owners@osdu.example.com"],
"viewers": ["data.default.viewers@osdu.example.com"]
},
"meta": [
{
"kind": "Unit",
"name": "ft",
"unitOfMeasureID": "osdu:reference-data--UnitOfMeasure:ft:",
"propertyNames": ["VerticalMeasurements[].VerticalMeasurement"]
}
]
}
]
}
}
}
```
## Error:
```json
Manifest kind: osdu:wks:master-data--Well:1.0.0
[2024-01-15, 20:17:26 UTC] {validate_schema.py:321} ERROR - Error: {'name': 'ft', 'unitOfMeasureID': 'opendes:reference-data--UnitOfMeasure:ft:', 'kind': 'Unit', 'propertyNames': ['VerticalMeasurements[].VerticalMeasurement']} is not valid under any of the given schemas
Failed validating 'oneOf' in schema['properties']['meta']['items']:
...
```
AWS: https://a452580d-4da3-4e22-aa4c-edcf6286c5c2.c5.us-east-2.airflow.amazonaws.com/log?dag_id=Osdu_ingest&task_id=validate_manifest_schema_task&execution_date=2024-01-15T19%3A48%3A04.766160%2B00%3A00&map_index=-1
Azure: https://osdu-ship.msft-osdu-test.org/airflow2/log?dag_id=Osdu_ingest&task_id=validate_manifest_schema_task&execution_date=2024-01-15T20%3A17%3A12.621733%2B00%3A00
GC: https://548a04f7b3aa4e52a897b94772864397-dot-us-central1.composer.googleusercontent.com/log?dag_id=Osdu_ingest&task_id=validate_manifest_schema_task&execution_date=2024-01-15T20%3A17%3A22.072263%2B00%3A00
RI: https://airflow.bm22.gcp.gnrg-osdu.projects.epam.com/log?dag_id=Osdu_ingest&task_id=validate_manifest_schema_task&execution_date=2024-01-15T20%3A17%3A26.295718%2B00%3A00https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/650String array becomes String after index2024-01-08T08:56:09ZChad LeongString array becomes String after indexThe String array becomes String after it is indexed. Bug should be introduced by [MR 649](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/649)
See issue created under Indexer https://community.openg...The String array becomes String after it is indexed. Bug should be introduced by [MR 649](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/649)
See issue created under Indexer https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/137https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/457Need example of how to use the POST /query/records:batch Fetch multiple rec...2023-03-13T09:51:30ZKamlesh TodaiNeed example of how to use the POST /query/records:batch Fetch multiple recordsThe Storage API documentation mention about
POST /query/records/batch Fetch multiple records. Would like to get the sample of how is this feature expected to be used.
Need clarification on
Account ID is the active OSDU account (OSDU ...The Storage API documentation mention about
POST /query/records/batch Fetch multiple records. Would like to get the sample of how is this feature expected to be used.
Need clarification on
Account ID is the active OSDU account (OSDU account or customer's account) which the users choose to use with the Search API.
frame-of-reference: This value indicates whether normalization applies, should be either 'none' or 'units=SI;crs=wgs84;elevation=msl;azimuth=true north;dates=utc;'
@chad @debasiscM17 - Release 0.20https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/436EDS : Mutiple Work product component ingestion2023-03-20T13:44:41ZNisha ThakranEDS : Mutiple Work product component ingestionwork product component is failing due to a referential integrity issue for a data set id.
While ingesting mutiple work product component DataSet is not getting converted from File.Generic to ConnectSource.Genric.
**'WorkProductComponent...work product component is failing due to a referential integrity issue for a data set id.
While ingesting mutiple work product component DataSet is not getting converted from File.Generic to ConnectSource.Genric.
**'WorkProductComponents': [{'modifyUser': 'serviceprincipal@testing.com', 'modifyTime': '2023-01-13T09:23:26.349Z', 'data': {'Datasets': ['opendes:dataset--File.Generic:6d0a547cf0153202feeec449816f5ad26df21317ed48260a31ffb56be5cc10133:']**M16 - Release 0.19Nisha ThakranNisha Thakranhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/347IBM M13 WITSML Parser - Validation Schema Error2024-01-24T14:26:07ZMichaelIBM M13 WITSML Parser - Validation Schema ErrorI encountered a validation error when trying to use ingest a witsml file into IBM M13 instance.
From dag logs: http://airflow-web-osdu.odi-og-osdu-ba8e38d4c011d627379af1a4280c4e35-0000.us-south.containers.appdomain.cloud/log?dag_id=Ener...I encountered a validation error when trying to use ingest a witsml file into IBM M13 instance.
From dag logs: http://airflow-web-osdu.odi-og-osdu-ba8e38d4c011d627379af1a4280c4e35-0000.us-south.containers.appdomain.cloud/log?dag_id=Energistics_xml_ingest&task_id=validate_manifest_schema_task&execution_date=2022-08-29T18%3A58%3A19.343176%2B00%3A00
```
AIRFLOW_CTX_DAG_RUN_ID=c62097bf-3375-4661-9a39-3786700e9c35
[2022-08-29 18:59:17,616] {authorization.py:137} ERROR - {"error":{"code":404,"message":"Schema is not present","errors":[{"domain":"global","reason":"notFound","message":"Schema is not present"}]}}
[2022-08-29 18:59:17,616] {validate_schema.py:170} ERROR - Error on getting schema of kind 'opendes:wks:Manifest:1.0.0'
[2022-08-29 18:59:17,616] {validate_schema.py:171} ERROR - 404 Client Error: Not Found for url: http://os-schema-ibm:8080/api/schema-service/v1/schema/opendes:wks:Manifest:1.0.0
```
I have attached a document that details the steps I performed.[IBM_M13_Manifest_Ingestion_Validation_Issue.docx][IBM_M13_Manifest_Ingestion_Validation_Issue.docx](/uploads/eb5e95aa21d7ab36697af82fbb675e50/IBM_M13_Manifest_Ingestion_Validation_Issue.docx)M13 - Release 0.16https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/85Airflow regex issue - likely not CSP related (?)2021-10-01T12:15:53ZEsmira RafigayevaAirflow regex issue - likely not CSP related (?)
Encountered issue through manifest ingestion inside airflow DAG:
"UnitQuantityID": "opendes:reference-data--UnitQuantity:1:", The error here seems to be regex related, as airflow does the search for: "opendes:reference-data--UnitQuanti...
Encountered issue through manifest ingestion inside airflow DAG:
"UnitQuantityID": "opendes:reference-data--UnitQuantity:1:", The error here seems to be regex related, as airflow does the search for: "opendes:reference-data--UnitQuantity"
It seems that the number 1 at the end is identified as a version and removed. Who can fix this?
This issue is CSP independent and therefore a bug across all CSPs
Edited by ivar Soerheim just now @s0rhe1mivar SoerheimMANISH KUMARVivek Ojhaivar Soerheim