AWS R3M10 - Reference Data Issues - WITSML/Manifest Ingestion
The following error is shown in the xcom summary when WITSML parser is ran:
[{'id': 'osdu:work-product--WorkProduct:20C60DDC-D36D-4A3C-800F-504CE0B5605D', 'kind': 'osdu:wks:work-product--WorkProduct:1.0.0', 'reason': 'Missing parents: {SRN: osdu:work-product-component--WellboreTrajectory:20C60DDC-D36D-4A3C-800F-504CE0B5605D}'}, {'id': 'osdu:work-product-component--WellboreTrajectory:20C60DDC-D36D-4A3C-800F-504CE0B5605D', 'kind': 'osdu:wks:work-product-component--WellboreTrajectory:1.0.0', 'reason': 'Missing parents: {SRN: osdu:master-data--Wellbore:00a7482cc78d40c1aeec31956ddb0a39, SRN: osdu:master-data--Organisation:SLB, SRN: osdu:reference-data--CoordinateReferenceSystem:ED50, SRN: osdu:reference-data--AzimuthReferenceType:grid%20north, SRN: osdu:dataset--File.WITSML:20C60DDC-D36D-4A3C-800F-504CE0B5605D, SRN: osdu:reference-data--ResourceSecurityClassification:RESTRICTED}'}, {'id': 'osdu:dataset--File.WITSML:20C60DDC-D36D-4A3C-800F-504CE0B5605D', 'kind': 'osdu:wks:dataset--File.WITSML:1.0.0', 'reason': 'Missing parents: {SRN: osdu:reference-data--ResourceSecurityClassification:RESTRICTED}'}]
As the referential integrity shows, there are few missing reference data needed. One of them being:
osdu:reference-data--ResourceSecurityClassification:RESTRICTED
I've ran the reference data ingestion via manifest
And I got the results loaded according to the log:
[2022-03-07, 20:48:02 UTC] {process_manifest_r3.py:120} INFO - Sending records to Storage service
[2022-03-07, 20:48:03 UTC] {process_manifest_r3.py:128} INFO - Records 'osdu:reference-data--ResourceSecurityClassification:Secret' were saved using Storage service.
[2022-03-07, 20:48:03 UTC] {process_manifest_r3.py:120} INFO - Sending records to Storage service
[2022-03-07, 20:48:03 UTC] {process_manifest_r3.py:128} INFO - Records 'osdu:reference-data--ResourceSecurityClassification:Public' were saved using Storage service.
[2022-03-07, 20:48:03 UTC] {process_manifest_r3.py:120} INFO - Sending records to Storage service
[2022-03-07, 20:48:03 UTC] {process_manifest_r3.py:128} INFO - Records 'osdu:reference-data--ResourceSecurityClassification:CONFIDENTIAL' were saved using Storage service.
[2022-03-07, 20:48:03 UTC] {process_manifest_r3.py:120} INFO - Sending records to Storage service
[2022-03-07, 20:48:03 UTC] {process_manifest_r3.py:128} INFO - Records 'osdu:reference-data--ResourceSecurityClassification:CLASSIFIED' were saved using Storage service.
[2022-03-07, 20:48:03 UTC] {process_manifest_r3.py:120} INFO - Sending records to Storage service
[2022-03-07, 20:48:03 UTC] {process_manifest_r3.py:128} INFO - Records 'osdu:reference-data--ResourceSecurityClassification:Private' were saved using Storage service.
[2022-03-07, 20:48:03 UTC] {process_manifest_r3.py:120} INFO - Sending records to Storage service
[2022-03-07, 20:48:03 UTC] {process_manifest_r3.py:128} INFO - Records 'osdu:reference-data--ResourceSecurityClassification:RESTRICTED' were saved using Storage service.
[2022-03-07, 20:48:03 UTC] {process_manifest_r3.py:120} INFO - Sending records to Storage service
[2022-03-07, 20:48:04 UTC] {process_manifest_r3.py:128} INFO - Records 'osdu:reference-data--ResourceSecurityClassification:Confidential' were saved using Storage service.
[2022-03-07, 20:48:04 UTC] {process_manifest_r3.py:120} INFO - Sending records to Storage service
[2022-03-07, 20:48:04 UTC] {process_manifest_r3.py:128} INFO - Records 'osdu:reference-data--ResourceSecurityClassification:MOST-CONFIDENTIAL' were saved using Storage service.
But when I try reingesting the WITSML, I still get the same warning about the missing
osdu:reference-data--ResourceSecurityClassification:RESTRICTED
When I try to query this, it seems like the reference data has not been indexed. I cannot search for it.
POST {{osduonaws_base_url}}/api/search/v2/query
{
"kind": "osdu:wks:reference-data--ResourceSecurityClassification:1.0.0"
}
Results:
{
"results": [],
"aggregations": [],
"totalCount": 0
}
The missing reference data is causing manifest parser to fail.