Pre Shipping issueshttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues2023-08-16T10:04:39Zhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/563Energistics XML parser DAG not able to execute successfully. Fails while cr...2023-08-16T10:04:39ZKamlesh TodaiEnergistics XML parser DAG not able to execute successfully. Fails while creating new pod for the oepratorWhen one looks at the DAG runs there has not been any successful run. It is failing while creating new pod.
```plaintext
[2023-08-04 20:00:50,255] {pod_launcher.py:128} WARNING - Pod not yet started: witsml-parser-task.10c656bb27934c3c9...When one looks at the DAG runs there has not been any successful run. It is failing while creating new pod.
```plaintext
[2023-08-04 20:00:50,255] {pod_launcher.py:128} WARNING - Pod not yet started: witsml-parser-task.10c656bb27934c3c90951ea67d10e8a0
[2023-08-04 20:00:50,272] {taskinstance.py:1463} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 367, in execute
final_state, remote_pod, result = self.create_new_pod_for_operator(labels, launcher)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 520, in create_new_pod_for_operator
launcher.start_pod(self.pod, startup_timeout=self.startup_timeout_seconds)
File "/home/airflow/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 329, in wrapped_f
return self.call(f, *args, **kw)
File "/home/airflow/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 409, in call
do = self.iter(retry_state=retry_state)
File "/home/airflow/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 356, in iter
return fut.result()
File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 437, in result
return self.__get_result()
File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/home/airflow/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 412, in call
result = fn(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/utils/pod_launcher.py", line 131, in start_pod
raise AirflowException("Pod took too long to start")
airflow.exceptions.AirflowException: Pod took too long to start
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1165, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1283, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1313, in _execute_task
result = task_copy.execute(context=context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 374, in execute
raise AirflowException(f'Pod Launching failed: {ex}')
airflow.exceptions.AirflowException: Pod Launching failed: Pod took too long to start
[2023-08-04 20:00:50,273] {taskinstance.py:1506} INFO - Marking task as FAILED. dag_id=Energistics_xml_ingest, task_id=witsml_parser_task, execution_date=20230804T195547, start_date=20230804T195549, end_date=20230804T200050
[2023-08-04 20:00:50,318] {local_task_job.py:151} INFO - Task exited with return code 1
[2023-08-04 20:00:50,339] {local_task_job.py:261} INFO - 0 downstream tasks scheduled from follow-on schedule check
```Anuj Guptavikas ranaAnuj Guptahttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/562CSV parser DAG on IBM R3 M19 environment is failing. Issue seems to be config...2023-08-22T18:30:48ZKamlesh TodaiCSV parser DAG on IBM R3 M19 environment is failing. Issue seems to be configuration and or resource allocationWhen one looks at the DAG runs for CSV parser, there is not a single run that has been successful.
The log indicates that there is some issue in dict - undefined (may be setup configuration issue)
```
[2023-08-04 19:00:31,847] {standa...When one looks at the DAG runs for CSV parser, there is not a single run that has been successful.
The log indicates that there is some issue in dict - undefined (may be setup configuration issue)
```
[2023-08-04 19:00:31,847] {standard_task_runner.py:77} INFO - Job 986: Subtask csv-parser
[2023-08-04 19:00:32,162] {logging_mixin.py:109} INFO - Running <TaskInstance: csv-parser-dag.csv-parser 2023-08-04T18:55:28.858482+00:00 [running]> on host ***-worker-0.***-worker.osdu.svc.cluster.local
[2023-08-04 19:00:32,194] {taskinstance.py:1463} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1165, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1246, in _prepare_and_execute_task_with_callbacks
self.render_templates(context=context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1755, in render_templates
self.task.render_template_fields(context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 997, in render_template_fields
self._do_render_template_fields(self, self.template_fields, context, jinja_env, set())
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1010, in _do_render_template_fields
rendered_content = self.render_template(content, context, jinja_env, seen_oids)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1061, in render_template
return [self.render_template(element, context, jinja_env) for element in content]
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1061, in <listcomp>
return [self.render_template(element, context, jinja_env) for element in content]
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1047, in render_template
return jinja_env.from_string(content).render(**context)
File "/home/airflow/.local/lib/python3.8/site-packages/jinja2/environment.py", line 1090, in render
self.environment.handle_exception()
File "/home/airflow/.local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/home/airflow/.local/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
raise value.with_traceback(tb)
File "<template>", line 1, in top-level template code
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'correlationId'
[2023-08-04 19:00:32,197] {taskinstance.py:1506} INFO - Marking task as FAILED. dag_id=csv-parser-dag, task_id=csv-parser, execution_date=20230804T185528, start_date=20230804T190031, end_date=20230804T190032
```Anuj Guptavikas ranaAnuj Guptahttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/559Cannot trigger the DAG Osdu_ingest_by_reference to execute manifest ingestion...2023-08-08T09:43:28ZKamlesh TodaiCannot trigger the DAG Osdu_ingest_by_reference to execute manifest ingestion by Reference in IBM R3 M19 environmentCannot trigger the DAG Osdu_ingest_by_reference to execute manifest ingestion by Reference in IBM R3 M19 environment
Request
curl --location 'https://cpd-osdu.apps.ibmosdu-preship.lndu.p1.openshiftapps.com/osdu-workflow/api/workflow/v1...Cannot trigger the DAG Osdu_ingest_by_reference to execute manifest ingestion by Reference in IBM R3 M19 environment
Request
curl --location 'https://cpd-osdu.apps.ibmosdu-preship.lndu.p1.openshiftapps.com/osdu-workflow/api/workflow/v1/workflow/Osdu_ingest_by_reference/workflowRun' \
\--header 'data-partition-id: opendes' \
\--header 'Content-Type: application/json' \
\--header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsIn...JjQhFlbeJb0gg' \
\--data '{ "executionContext": { "Payload": { "AppKey": "test-app", "data-partition-id": "opendes" }, "manifest": "opendes:dataset--File.Generic:5bb42ca88eca49da90c48272601fca68" } }'
Response
{ "code": 500, "reason": "could not create addressopendes-status-event-default-topic", "message": "io.netty.util.internal.ReferenceCountUpdater.setInitialValue(Lio/netty/util/ReferenceCounted;)V" }
@anujgupta @chad @debasisc @davidglassvikas ranavikas ranahttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/558IBM M19 Policy engine incorrect settings2023-08-29T10:55:12ZDadong ZhouIBM M19 Policy engine incorrect settingsKamlesh reported the errors when calling the Policy Evaluate api:
```
{
"result": {
"records": [
{
"errors": [
{
"code": 404,
"id...Kamlesh reported the errors when calling the Policy Evaluate api:
```
{
"result": {
"records": [
{
"errors": [
{
"code": 404,
"id": "opendes:master-data--Well:test1111111111",
"message": "Entitlements response 404 Error 404 - Not Found ",
"reason": "Unauthorized"
},
{
"code": 404,
"id": "opendes:master-data--Well:test1111111111",
"message": "Legal response 404 Error 404 - Not Found ",
"reason": "Error from compliance service"
},
{
"code": "403",
"id": "opendes:master-data--Well:test1111111111",
"message": "The user is not authorized to perform this action",
"reason": "Access denied"
}
],
"id": "opendes:master-data--Well:test1111111111"
}
]
}
}
```
I confirmed the error in IBM M19 environment.
From the error messages, it seems the two environmental variables ENTITLEMENTS_BASE_URL and LEGAL_BASE_URL in Policy engine are not set correctly. Please check and correct.
Thanks.
cc @todaiksAnuj Guptavikas ranaAnuj Guptahttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/543Azure M18 - search results not returning all fields2023-08-09T06:41:53ZMichaelAzure M18 - search results not returning all fieldsWhen doing a search on some records, the "data" field/node is not returned.
For instance, the following search request returns a well without the data node:
```
curl --location 'https://osdu-ship.msft-osdu-test.org/api/search/v2/query'...When doing a search on some records, the "data" field/node is not returned.
For instance, the following search request returns a well without the data node:
```
curl --location 'https://osdu-ship.msft-osdu-test.org/api/search/v2/query' \
--header 'Content-Type: application/json' \
--header 'data-partition-id: opendes' \
--header 'Authorization: Bearer ...' \
--data '{
"kind": "osdu:wks:master-data--Well:*",
"query": "id:\"opendes:master-data--Well:perforationwell\""
}'
```
Response:
```
{
"results": [
{
"createTime": "2022-12-07T15:04:14.096Z",
"kind": "osdu:wks:master-data--Well:1.0.0",
"authority": "osdu",
"namespace": "osdu:wks",
"legal": {
"legaltags": [
"opendes-public-usa-dataset-open-test-data"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "preshipping@azureglobal1.onmicrosoft.com",
"source": "wks",
"acl": {
"viewers": [
"data.default.viewers@opendes.contoso.com"
],
"owners": [
"data.default.owners@opendes.contoso.com"
]
},
"id": "opendes:master-data--Well:perforationwell",
"type": "master-data--Well",
"version": 1670425452534671,
"tags": {
"normalizedKind": "osdu:wks:master-data--Well:1"
}
}
],
"aggregations": null,
"totalCount": 1
}
```
If you request for the storage record, the data node is present.
```
curl --location 'https://osdu-ship.msft-osdu-test.org/api/storage/v2/records/opendes:master-data--Well:perforationwell' \
--header 'data-partition-id: opendes' \
--header 'Authorization: Bearer ...'
```
Response:
```
{
"data": {
"DefaultVerticalMeasurementID": "RotaryTable",
"FacilityEvents": [
{
"EffectiveDateTime": "1983-10-28T00:00:00",
"FacilityEventTypeID": "opendes:reference-data--FacilityEventType:Spud:"
},
{
"EffectiveDateTime": "1983-12-19T00:00:00",
"FacilityEventTypeID": "opendes:reference-data--FacilityEventType:TDReached:"
}
],
"FacilityName": "Perforation Well",
"FacilityOperators": [
{
"FacilityOperatorOrganisationID": "opendes:master-data--Organisation:Mobil:"
}
],
"FacilityStates": [
{
"FacilityStateTypeID": "opendes:reference-data--FacilityStateType:Abandoned:"
}
],
"FacilityTypeID": "opendes:reference-data--FacilityType:Well:",
"GeoContexts": [
{
"GeoPoliticalEntityID": "opendes:master-data--GeoPoliticalEntity:Netherlands_Country:",
"GeoTypeID": "opendes:reference-data--GeoPoliticalEntityType:Country:"
},
{
"GeoPoliticalEntityID": "opendes:master-data--GeoPoliticalEntity:B14_BlockID:",
"GeoTypeID": "opendes:reference-data--GeoPoliticalEntityType:BlockID:"
}
],
"NameAliases": [
{
"AliasName": "Perforation Well",
"AliasNameTypeID": "opendes:reference-data--AliasNameType:Well:"
},
{
"AliasName": "7534",
"AliasNameTypeID": "opendes:reference-data--AliasNameType:UWI:"
}
],
"OperatingEnvironmentID": "opendes:reference-data--OperatingEnvironment:Off:",
"Source": "TNO",
"SpatialLocation": {
"Wgs84Coordinates": {
"features": [
{
"geometry": {
"coordinates": [
4.33958866,
55.27975681
],
"type": "Point"
},
"properties": {},
"type": "Feature"
}
],
"type": "FeatureCollection"
}
},
"VerticalMeasurements": [
{
"VerticalCRSID": "opendes:reference-data--CoordinateReferenceSystem:MSL:",
"VerticalMeasurement": 38.8,
"VerticalMeasurementID": "RotaryTable",
"VerticalMeasurementPathID": "opendes:reference-data--VerticalMeasurementPath:Elevation:",
"VerticalMeasurementTypeID": "opendes:reference-data--VerticalMeasurementType:RotaryTable:"
}
]
},
"meta": [
{
"kind": "Unit",
"name": "m",
"persistableReference": "{\"abcd\":{\"a\":0.0,\"b\":1.0,\"c\":1.0,\"d\":0.0},\"symbol\":\"m\",\"baseMeasurement\":{\"ancestry\":\"L\",\"type\":\"UM\"},\"type\":\"UAD\"}",
"propertyNames": [
"VerticalMeasurements[].VerticalMeasurement"
],
"unitOfMeasureID": "opendes:reference-data--UnitOfMeasure:m:"
}
],
"id": "opendes:master-data--Well:perforationwell",
"version": 1670425452534671,
"kind": "osdu:wks:master-data--Well:1.0.0",
"acl": {
"viewers": [
"data.default.viewers@opendes.contoso.com"
],
"owners": [
"data.default.owners@opendes.contoso.com"
]
},
"legal": {
"legaltags": [
"opendes-public-usa-dataset-open-test-data"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "preshipping@azureglobal1.onmicrosoft.com",
"createTime": "2022-12-07T15:04:14.096Z"
}
```M20 - Release 0.23Chad LeongChad Leonghttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/535While Testing Augmented search feature in AWS R3 M18 Pre-ship environment - n...2023-06-30T00:44:47ZKamlesh TodaiWhile Testing Augmented search feature in AWS R3 M18 Pre-ship environment - not able to get the expected results.Following these steps to test Augmented search feature
1. Make sure that the schema of kind "osdu:wks:reference-data--IndexPropertyPathConfiguration:1.0.0" is deployed. It should be as it is part of M18 schema. (**Executed sucessfully**...Following these steps to test Augmented search feature
1. Make sure that the schema of kind "osdu:wks:reference-data--IndexPropertyPathConfiguration:1.0.0" is deployed. It should be as it is part of M18 schema. (**Executed sucessfully**)
2. Make sure the feature flag "index-augmenter-enabled" is turned in the tested data partition (**Do not have access to execute this step**)
3. Select a few kinds of data that users want to create extended properties from related objects (**Selected Well, Wellbore, WellLog, WellboreTrajectory,WellboreMarkerSet**)
4. Define the property extension configuration in the data block of the records with kind "osdu:wks:reference-data--IndexPropertyPathConfiguration:1.0.0".
5. Deploy the configuration records to the storage via storage API
<details><summary>Configuration records created</summary>
{
"recordCount": 5,
"recordIds": [
"osdu:reference-data--IndexPropertyPathConfiguration:work-product-component--WellLog:1.",
"osdu:reference-data--IndexPropertyPathConfiguration:work-product-component--WellboreTrajectory:1.",
"osdu:reference-data--IndexPropertyPathConfiguration:work-product-component--WellboreMarkerSet:1.",
"osdu:reference-data--IndexPropertyPathConfiguration:wks:master-data--Well:1.",
"osdu:reference-data--IndexPropertyPathConfiguration:wks:master-data--Wellbore:1."
],
"skippedRecordIds": [],
"recordIdVersions": [
"osdu:reference-data--IndexPropertyPathConfiguration:work-product-component--WellLog:1.:1687552840965025",
"osdu:reference-data--IndexPropertyPathConfiguration:work-product-component--WellboreTrajectory:1.:1687552840965025",
"osdu:reference-data--IndexPropertyPathConfiguration:work-product-component--WellboreMarkerSet:1.:1687552840965025",
"osdu:reference-data--IndexPropertyPathConfiguration:wks:master-data--Well:1.:1687552840965025",
"osdu:reference-data--IndexPropertyPathConfiguration:wks:master-data--Wellbore:1.:1687552840965025"
]
}
</details>
<details><summary>Retrieved Well configuration record for verification </summary>
{
"data": {
"Name": "Well-IndexPropertyPathConfiguration",
"Description": "The index property list for master-data--Well:1., valid for all master-data--Well kinds for major version 1.",
"Code": "osdu:wks:master-data--Well:1.",
"AttributionAuthority": "OSDU",
"Configurations": [
{
"Name": "CountryNamesKTJun23",
"Policy": "ExtractAllMatches",
"Paths": [
{
"RelatedObjectsSpec": {
"RelationshipDirection": "ChildToParent",
"RelatedObjectID": "data.GeoContexts[].GeoPoliticalEntityID",
"RelatedObjectKind": "osdu:wks:master-data--GeoPoliticalEntity:1.",
"RelatedConditionMatches": [
"osdu:reference-data--GeoPoliticalEntityType:Country:"
],
"RelatedConditionProperty": "data.GeoContexts[].GeoTypeID"
},
"ValueExtraction": {
"ValuePath": "data.GeoPoliticalEntityName"
}
}
],
"UseCase": "As a user I want to find objects by a country name, with the understanding that an object may extend over country boundaries."
},
{
"Name": "WellUWIKTJun23",
"Policy": "ExtractFirstMatch",
"Paths": [
{
"ValueExtraction": {
"RelatedConditionMatches": [
"osdu:reference-data--AliasNameType:UniqueIdentifier:",
"osdu:reference-data--AliasNameType:RegulatoryName:",
"osdu:reference-data--AliasNameType:PreferredName:",
"osdu:reference-data--AliasNameType:CommonName:",
"osdu:reference-data--AliasNameType:ShortName:"
],
"RelatedConditionProperty": "data.NameAliases[].AliasNameTypeID",
"ValuePath": "data.NameAliases[].AliasName"
}
}
],
"UseCase": "As a user I want to discover and match Wells by their UWI. I am aware that this is not globally reliable, however, I am able to specify a prioritized AliasNameType list to look up value in the NameAliases array."
}
]
},
"meta": [],
"modifyUser": "admin-main@testing.com",
"modifyTime": "2023-06-23T20:40:41.335Z",
"id": "osdu:reference-data--IndexPropertyPathConfiguration:wks:master-data--Well:1.",
"version": 1687552840965025,
"kind": "osdu:wks:reference-data--IndexPropertyPathConfiguration:1.0.0",
"acl": {
"viewers": [
"data.default.viewers@osdu.example.com"
],
"owners": [
"data.default.owners@osdu.example.com"
]
},
"legal": {
"legaltags": [
"osdu-AugmIdxExt-Legal-Tag-Test"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "admin-main@testing.com",
"createTime": "2023-06-15T17:35:37.889Z"
}
</details>
6. Re-index all the kinds that have extended properties using the reindex API
<details><summary>re-index --Well:1.0.0, --Well:1.1.0, --Well:1.2.0</summary>
curl --location 'https://osdu.r3m18.preshiptesting.osdu.aws/api/indexer/v2/reindex?force_clean=true' \
--header 'Content-Type: application/json' \
--header 'data-partition-id: osdu' \
--header 'Authorization: Bearer eyJraWQiOi...truncated...CG4HUDHg' \
--data '{
"kind": "osdu:wks:master-data--Well:1.0.0"
}'
Response 200 OK
</details>
7. Test search with the extended properties
<details><summary>Search and it's results</summary>
curl --location 'https://osdu.r3m18.preshiptesting.osdu.aws/api/search/v2/query' \
--header 'Authorization: Bearer eyJraWQiOi...Truncated...BAy-bDbtQ' \
--header 'data-partition-id: osdu' \
--header 'Content-Type: application/json' \
--data '{
"kind": "osdu:wks:master-data--Well:1.*",
"query": "_exists_:data.WellUWIKTJun23",
"returnedFields": ["id", "kind", "data.WellUWIKTJun23"]
}'
Response 200 OK
{
"results": [],
"aggregations": [],
"totalCount": 0
}
</details>
Also tried similar steps in GC where we have access to look at whether the feature is available or not. We found that feature is not enabled
and do not have permissions/access to enable the feature.
<details><summary>In GC R3 M18 Pre-ship environment</summary>
curl --location 'https://preship.gcp.gnrg-osdu.projects.epam.com/api/partition/v1/partitions/odesprod' \
--header 'Content-Type: application/json' \
--header 'data-partition-id: odesprod' \
--header 'Authorization: Bearer ya29.a0AWY7Ckmj...truncated...fpwHiQ0167' \
--data ''
Response 200 OK
{
"kubernetes-secret-name": {
"sensitive": false,
"value": "eds-odesprod"
},
"elasticsearch.password": {
"sensitive": true,
"value": "ELASTIC_PASS_ODESPROD"
},
"serviceAccount": {
"sensitive": false,
"value": "datafier@osdu-service-prod.iam.gserviceaccount.com"
},
"dataPartitionId": {
"sensitive": false,
"value": "odesprod"
},
"bucket": {
"sensitive": false,
"value": "osdu-data-prod-odesprod-records"
},
"index-augmenter-enabled": {
"sensitive": false,
"value": "false"
},
...Truncated...
{
"indexer.service.account": {
"sensitive": false,
"value": "workload-indexer-gcp@osdu-service-prod.iam.gserviceaccount.com"
},
"projectId": {
"sensitive": false,
"value": "osdu-data-prod"
}
}
When I try to enable the index-augmenter, I get the response of 403 Forbidden - RBAC: access denied.
</details>M18 - Release 0.21Dzmitry Malkevich (EPAM)Yong ZengDzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/529AWS M18 Preship - eds_ingest unable to fetch data records from Azure Preship2023-06-26T20:27:59ZPriyanka BhongadeAWS M18 Preship - eds_ingest unable to fetch data records from Azure Preshiphttps://5999f1b0-bc27-432c-b834-b3e0814512d0.c8.us-east-2.airflow.amazonaws.com/log?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-06-19T08%3A21%3A44%2B00%3A00&map_index=-1
eds_ingest - AWS M18 Preship is target and Azure Pe...https://5999f1b0-bc27-432c-b834-b3e0814512d0.c8.us-east-2.airflow.amazonaws.com/log?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-06-19T08%3A21%3A44%2B00%3A00&map_index=-1
eds_ingest - AWS M18 Preship is target and Azure Peship is Provider . AWS makes Search API call to Azure for Well data.
Search query is as below:
[2023-06-19, 08:21:55 UTC] {{src_dags_fetch_and_ingest.py:334}} INFO - QUERY BODY:{**'kind': 'osdu:wks:master-data--Well:1.0.0', 'query': '* AND ((createTime: [2020-01-01T00:00:00 TO 2023-06-19T08:21:54]) OR (modifyTime: [2020-01-01T00:00:00 TO 2023-06-19T08:21:54]))', 'sort': {'field': ['createTime'], 'order': ['ASC']}, 'limit': 100}**
Issue - it Doesnt fetch any results as shown below, where as same query fetch 1300 approx data records on postman
[2023-06-19, 08:21:55 UTC] {{src_dags_fetch_and_ingest.py:158}} ****ERROR - {'status': 'error', 'message': KeyError('results')}****https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/497Erroneous Inconsistent search results when adding or removing records2024-02-22T15:26:10ZMichaelErroneous Inconsistent search results when adding or removing recordsWe are adding and removing records from OSDU Storage, and we see erroneous results where a record that we add is not shown in the search result or a record that we delete is shown.
We tried waiting for the expected result from the sear...We are adding and removing records from OSDU Storage, and we see erroneous results where a record that we add is not shown in the search result or a record that we delete is shown.
We tried waiting for the expected result from the search and then trying again. We see erroneous results sometimes when we find it in the search results, but the next search does not find it. Similarly, for deleted records, we sometimes see the deletion only to see it not deleted in the next search.
Here is a video that demonstrates the inconsistent search results after a record has been created: [osdu_indexing_consistency.zip](/uploads/88858771b3a497a7336dd3bf47bb0886/osdu_indexing_consistency.zip)
Here is the postman collection that is used in the video: [Indexing_Test.postman_collection.json](/uploads/6958cd6c0b3da081a54a46488e1a2497/Indexing_Test.postman_collection.json)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/493Dataset API to get storageInstructions for kindSubType=dataset--FileCollectio...2024-02-08T12:24:12ZKamlesh TodaiDataset API to get storageInstructions for kindSubType=dataset--FileCollection.Generic is not consistent across all the CSPsThe implementation of GET storageInstruction for DATASET API is not consistent across all the CSPs
e.g. https://{{DATASET_HOST}}/storageInstructions?kindSubType=dataset--FileCollection.Generic when it is executed the output is various am...The implementation of GET storageInstruction for DATASET API is not consistent across all the CSPs
e.g. https://{{DATASET_HOST}}/storageInstructions?kindSubType=dataset--FileCollection.Generic when it is executed the output is various among all the CSPs. The usage example provided by AWS works in AWS environment but cannot get it to work in other CSP's environments, because other CSPs are not returning the same kind of information when request is made for storageInstructions as
the documentation is lacking and the working example is missing, on how to consume/use the information to create the collection of multiple files (FileCollection.Generic) using the registerDataset
**Details**
`**AWS**
**curl --location --request POST 'https://r3m17ps.preshiptesting.osdu.aws/api/dataset/v1/storageInstructions?kindSubType=dataset--FileCollection.Generic'** \
--header 'data-partition-id: osdu' \
--header 'Authorization: Bearer eyJraWQiOi...g0QA8ImMw'
Response 200 OK
{
"providerKey": "AWS_S3",
"storageLocation": {
**"unsignedUrl"**: "s3://r3m17ps-shared-561735291427-ca-central-1-file/osdu/FHugmJALXGTlPTC5VpxBxEmQfGy68jXp/",
"createdAt": "2023-05-06T11:32:37.167Z",
"connectionString": "AccessKeyId=ASIAYFSQIKIRU352WEF6;SecretAccessKey=n0QRlrK/YwDooN9+n2q8ky1whKhjFUCLK05U5aAZ;SessionToken=IQoJb3JpZ2luX2VjEBMaDGNhLWNlbnRyYWwtMSJHMEUCIQD2F8q01x6RAwLUrEhxH64m3FEVUtf058bPkrnOZfKjQwIgNuLtSmO7hIExCuSquCOq5+K1DyXgVxuiGNezuAkDzIsqpgUILRACGgw1NjE3MzUyOTE0MjciDL77xgCuhM4FLyVm3CqDBZw9vX/uBEhZ2RpwCmld1E6qSYgE2rKqdeNlv8JLGjENSq66gw3VA7Rw73UiQfq4ilOp32zf5tSvXN2P4BNZbtuv+t2bgjGHMhqH0eaQn+y5VJHMJOVJJuI7aajgFr6c0bGE2zxi9RrHX7d7luV79arzBlIO1XeeSLlDYlyPVHRB+pc4VmAXslpwNeFkHh/ih1P+6sy/VkHlmwYMTuz2q0rG0M0ME+SwK1udd0jAukLbAu0Ao2zAJ3yQ5QXq/L8DtHOx8nc4w40p/eocLcy4HOwKyY0omeM5QfQ2gCrPcBPS0jLwkUCnSjDVODffuakXhT9pALeqoX7stqDf2F2PrTxt8J+XMTRxo4tIMMtpzpiJzGpGxuqQIDlroCYf+AUmg4GxyCXm0oCNxH8uoSF8pnxsu3dQvSO8+dXf6QzOzWCmyyh9YjRrZLveITJ0Y08vcbBQGCw04HVfexhFxDOmaNf8X4PfSQr6Czwnbxl7p5S3NLVo56lxEe195+nPk9KHFOHBHHmq3du/gy2IrE0a4zMZdMG/p8vSAFvux8VUqR2dB9FJlhh3DiKq3MoBvFBo/D4MCI4JZ7v3hUJFo+fZP2ew981vz5OT1xGLH2MvY+53/jtMjYvt7HqN1HNv3SZuAzboqxfb79G7Y09ZiKjKiincxt6/FVM0jVolXXQeyaZgZMkh+asxiFlztZAL8KueGxXnHrPlsiUD+nFuWYv4Qz3vmBTNjNK1gqCczg4rgb7kM/AwNZ4U3MUanVVbXUX15T0gOeu826e2Adv+aJy5twR37Gi1+1Zp4tN4/Q+liI5ms3xQJAwg4BJpwVptNMAXaWSkojhnFvvNt8vgfQnBPy9aPdQw1fXYogY6nQFSWKU1AthSv4P/CbCc5mauZyAT8m5UYU3DoNu07Lb2+rPgiRji/H0cvCalDFPa4Typ4vzYWWjWNxOHkc3lWbmI3tI69IRBVjdTGBeA5P706YTb3h00M8CbwpTGI3WkR9FEKgSTtv0tgEZHt8dy89BqyG3R/5IMZrTS2HRMht/kDzwVoFEo6ruAA+Na89DF/V8d6T8WW45aPBZSR1lu;Expiration=2023-05-06T12:32:37Z",
"credentials": {
**"accessKeyId"**: "ASIAYFSQIKIRU352WEF6",
**"secretAccessKey"**: "n0QRlrK/YwDooN9+n2q8ky1whKhjFUCLK05U5aAZ",
**"sessionToken"**: "IQoJb3JpZ2luX2VjEBMaDGNhLWNlbnRyYWwtMSJHMEUCIQD2F8q01x6RAwLUrEhxH64m3FEVUtf058bPkrnOZfKjQwIgNuLtSmO7hIExCuSquCOq5+K1DyXgVxuiGNezuAkDzIsqpgUILRACGgw1NjE3MzUyOTE0MjciDL77xgCuhM4FLyVm3CqDBZw9vX/uBEhZ2RpwCmld1E6qSYgE2rKqdeNlv8JLGjENSq66gw3VA7Rw73UiQfq4ilOp32zf5tSvXN2P4BNZbtuv+t2bgjGHMhqH0eaQn+y5VJHMJOVJJuI7aajgFr6c0bGE2zxi9RrHX7d7luV79arzBlIO1XeeSLlDYlyPVHRB+pc4VmAXslpwNeFkHh/ih1P+6sy/VkHlmwYMTuz2q0rG0M0ME+SwK1udd0jAukLbAu0Ao2zAJ3yQ5QXq/L8DtHOx8nc4w40p/eocLcy4HOwKyY0omeM5QfQ2gCrPcBPS0jLwkUCnSjDVODffuakXhT9pALeqoX7stqDf2F2PrTxt8J+XMTRxo4tIMMtpzpiJzGpGxuqQIDlroCYf+AUmg4GxyCXm0oCNxH8uoSF8pnxsu3dQvSO8+dXf6QzOzWCmyyh9YjRrZLveITJ0Y08vcbBQGCw04HVfexhFxDOmaNf8X4PfSQr6Czwnbxl7p5S3NLVo56lxEe195+nPk9KHFOHBHHmq3du/gy2IrE0a4zMZdMG/p8vSAFvux8VUqR2dB9FJlhh3DiKq3MoBvFBo/D4MCI4JZ7v3hUJFo+fZP2ew981vz5OT1xGLH2MvY+53/jtMjYvt7HqN1HNv3SZuAzboqxfb79G7Y09ZiKjKiincxt6/FVM0jVolXXQeyaZgZMkh+asxiFlztZAL8KueGxXnHrPlsiUD+nFuWYv4Qz3vmBTNjNK1gqCczg4rgb7kM/AwNZ4U3MUanVVbXUX15T0gOeu826e2Adv+aJy5twR37Gi1+1Zp4tN4/Q+liI5ms3xQJAwg4BJpwVptNMAXaWSkojhnFvvNt8vgfQnBPy9aPdQw1fXYogY6nQFSWKU1AthSv4P/CbCc5mauZyAT8m5UYU3DoNu07Lb2+rPgiRji/H0cvCalDFPa4Typ4vzYWWjWNxOHkc3lWbmI3tI69IRBVjdTGBeA5P706YTb3h00M8CbwpTGI3WkR9FEKgSTtv0tgEZHt8dy89BqyG3R/5IMZrTS2HRMht/kDzwVoFEo6ruAA+Na89DF/V8d6T8WW45aPBZSR1lu",
"expiration": "2023-05-06T12:32:37.000+00:00"
},
"createdBy": "admin@testing.com",
**"region":** "ca-central-1"
}
}
**Using the information from the response of the previous request upload the file
Authorization Tab (Settings)
Type: AWS Signature
AccessKey
SecretKey
AWS Region
Service Name: s3
Session Token**
**curl --location --request PUT 'https://r3m17ps-shared-561735291427-ca-central-1-file.s3.amazonaws.com/osdu/FHugmJALXGTlPTC5VpxBxEmQfGy68jXp/file1' \**
--header 'Content-Type: text/csv' \
--header 'X-Amz-Security-Token: IQoJb3JpZ2luX2VjEBMaDGNhLWNlbnRyYWwtMSJHMEUCIQD2F8q01x6RAwLUrEhxH64m3FEVUtf058bPkrnOZfKjQwIgNuLtSmO7hIExCuSquCOq5+K1DyXgVxuiGNezuAkDzIsqpgUILRACGgw1NjE3MzUyOTE0MjciDL77xgCuhM4FLyVm3CqDBZw9vX/uBEhZ2RpwCmld1E6qSYgE2rKqdeNlv8JLGjENSq66gw3VA7Rw73UiQfq4ilOp32zf5tSvXN2P4BNZbtuv+t2bgjGHMhqH0eaQn+y5VJHMJOVJJuI7aajgFr6c0bGE2zxi9RrHX7d7luV79arzBlIO1XeeSLlDYlyPVHRB+pc4VmAXslpwNeFkHh/ih1P+6sy/VkHlmwYMTuz2q0rG0M0ME+SwK1udd0jAukLbAu0Ao2zAJ3yQ5QXq/L8DtHOx8nc4w40p/eocLcy4HOwKyY0omeM5QfQ2gCrPcBPS0jLwkUCnSjDVODffuakXhT9pALeqoX7stqDf2F2PrTxt8J+XMTRxo4tIMMtpzpiJzGpGxuqQIDlroCYf+AUmg4GxyCXm0oCNxH8uoSF8pnxsu3dQvSO8+dXf6QzOzWCmyyh9YjRrZLveITJ0Y08vcbBQGCw04HVfexhFxDOmaNf8X4PfSQr6Czwnbxl7p5S3NLVo56lxEe195+nPk9KHFOHBHHmq3du/gy2IrE0a4zMZdMG/p8vSAFvux8VUqR2dB9FJlhh3DiKq3MoBvFBo/D4MCI4JZ7v3hUJFo+fZP2ew981vz5OT1xGLH2MvY+53/jtMjYvt7HqN1HNv3SZuAzboqxfb79G7Y09ZiKjKiincxt6/FVM0jVolXXQeyaZgZMkh+asxiFlztZAL8KueGxXnHrPlsiUD+nFuWYv4Qz3vmBTNjNK1gqCczg4rgb7kM/AwNZ4U3MUanVVbXUX15T0gOeu826e2Adv+aJy5twR37Gi1+1Zp4tN4/Q+liI5ms3xQJAwg4BJpwVptNMAXaWSkojhnFvvNt8vgfQnBPy9aPdQw1fXYogY6nQFSWKU1AthSv4P/CbCc5mauZyAT8m5UYU3DoNu07Lb2+rPgiRji/H0cvCalDFPa4Typ4vzYWWjWNxOHkc3lWbmI3tI69IRBVjdTGBeA5P706YTb3h00M8CbwpTGI3WkR9FEKgSTtv0tgEZHt8dy89BqyG3R/5IMZrTS2HRMht/kDzwVoFEo6ruAA+Na89DF/V8d6T8WW45aPBZSR1lu' \
--header 'X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' \
--header 'X-Amz-Date: 20230506T113518Z' \
--header 'Authorization: AWS4-HMAC-SHA256 Credential=ASIAYFSQIKIRU352WEF6/20230506/ca-central-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=46fda8f99c22d1e881f37b12b94724079a1f2c6559a639fe86799a671f26cb91' \
--data '@Postman Collection/31_CICD_Setup_CSVIngestion/wellbore.csv'
**Azure**
curl --location --request POST 'https://osdu-ship.msft-osdu-test.org/api/dataset/v1/storageInstructions?kindSubType=dataset--FileCollection.Generic' \
--header 'data-partition-id: opendes' \
--header 'Authorization: Bearer eyJ0eXAiOiJ...e9tIuUDbg'
Response 200 OK
{
"providerKey": "AZURE",
"storageLocation": {
**"signedUrl"**: "https://osdumvpdp1shipbbnghiercl.dfs.core.windows.net/datalake-staging-area/osdu-user-1683373961973-2023-05-06-11-52-41-973-276de2f88be64f29861d7f23de482330?sv=2020-08-04&spr=https&st=2023-05-06T11%3A52%3A42Z&se=2023-05-13T11%3A52%3A42Z&sr=d&sp=racwl&sdd=1&sig=Si5JnpGmg15ToaXDlz3voXPPmcWdgEOYzGvjqR%2BTEdk%3D",
**"fileCollectionSource"**: "/osdu-user-1683373961973-2023-05-06-11-52-41-973-276de2f88be64f29861d7f23de482330",
"createdBy": "osdu-user"
}
}
**How to use the information to load the multiple files to create a collection**__
**GC**
curl --location --request POST 'https://preship.gcp.gnrg-osdu.projects.epam.com/api/dataset/v1/storageInstructions?kindSubType=dataset--FileCollection.Generic' \
--header 'data-partition-id: odesprod' \
--header 'Authorization: Bearer ya29.a0AWY7Ckmv_...thKfg0169'
Response 200 OK
{
"providerKey": "GCP",
"storageLocation": {
**"url"**: "https://storage.googleapis.com/osdu-data-prod-odesprod-staging-area/",
**"signingOptions"**: {
**"bucket"**: "osdu-data-prod-odesprod-staging-area",
**"connectionString"**: "ya29.dr.AfMbl1iSb-YCAWAR64myZ33i8nYVBFRLX7BT4G4G5x7CbCe6EeJBioAzBgbut_O0LVCynVd9ebQSVIwMYy_O6O22cM43CFt-AC5S_FHD_Up_OS1pDiBJ0SS-rAAK052taCvek_vAspNC1kFDEAZ-bgOMGSTe5uLYhzfjLX2v89JDPO2I_ytLQkQGGjG9sn8wR2xuSFfEfOhH3O9eGa5r9LwCAUsVus7j1ZKjvVnAoiFVBSWcg-lrOiG8yUycp-xAfHdYNErxELPq_yBeIqlgnOkF3Di5JZDNZSDQrp8n6CCnUTYIXiBgUl-E193cTHb5UwnYVatn1oMUmpN3MP76yMdCvMPbgmxLZCB1fMGQOCjNF49AwoibPZbyk1FU7jr9OlAiJ-lmGimx5V3V6Vp1NbGHLlA_UAbIcvmtFE4oOAIo9NXvIohL9p4gaktCbJbCde11xDw-87OUT7XnmBxsT7fG0HTXx6tpmdDqh9URPUhot-bj2n67fjUYirIO1EgCpQc-YNaX8pSB03DVFsdeNw33xWKJ7WiU4U0sP4gVr0iKzCjP5LFmxUpHDIcx2dHkNRKEYdDTyn3947rsXQEdYrgW-fSp1hFvwgAcRTqW_qIcWZmRt15OAoPZJompCEUnslqfH6SkSESXVlsWtME4a2mgaYvzRMCfuvtmsMQpAR8a0o6MbqxnQ6Lj5EUkMt_kqtv7CgW1XuKvkqhA4gkZPoJRKu55PtrEvjEpOgzxBbG0e4MZT68KngAtSsfO0IuG7bPXb1_KQrHBOpRYObe2RTAquj9tTGOCXnEWm-evKN3MBiHbdJsLXtbFSODcD9HqdJM8UsTBotigEAMX_7S6bkzE9JMpc9j3g2xrBbpHhDkR7iSB71TSxhLalFC_IeUTj9XBlHd3WmyqYa1JCmGyj8vrckQiR8yYfyUyHpdfFe2EtczeA10323UAnI2G8HZw2M2ViHKdBDfTk-AfruUR5iU8T-0nBko761nWJ1R7zPg1d50XCdiWAxu9g_9Q8qW-s9ss3DVAmhBb_QPUQblRIGWmawZQIUJPPpzSOGNinR9MH4R_un2vpA1j1M5zRZU4td1yzDUkyLbC2-X8GrKtZAnBwZC9GGu_6TOMwe0wIb0RQY2yGBR0-MdBIiDTVkwVyZQpGHZb6JTlSGB4sl3H_1qXp58F2koDKbHKLNdt7eRryWjPQ-9VQIFH2V1CELo05_0Sc0sCqBNAwHf9XD_r93DPgLlwpv-FOc8Zl4lJi48-WmFwNhTTON8juKT7MNmUS9UFI0nDKHTpWVBeGKbT-xjNt75BzunsVdM8ZuWW-i6CzO7KJaFGzL_7l8-1pyayBm0sXqz1T4j8KLlUPsCsV01RY5516OwR2YVI0araTVnDrqaUW4TiAFnxfxIIsNeTYXSvI436ityUFXjCEp4mAxnid5fEjZAI2ujQCDYjjSzwNqXBfmkswK6fh8NOMl8J8BGHp90FiP_j2HHPYngGzo6LP8Ry4-bkutUQqBGxu5v2lbzMKsNePO5vxzvd7W_7vavycmQEmDtsomnZtEBFt5xQpVQe1frMr0h6raXb-O_Oo_Z6ctPNgOQ30VFy3gMAfj0Bav2B9KWWVa5Gdw-p_einENiJzrVp_qax3A-ui7-B_KvqtMdDAnDjyr0jrpT2-wUubcmjFK3Ado_sn64aEuHWo8x9j0Fzdgv559ddUC-h6gQ7t3MqaJBHUcAVoqd0C38IrvtogE3lMTJAIVLEPhiTaC_6vlTw6l57ujDip4mNn1hdODN-N2XmbLtFq6aw886oZP1lheJr7uJfY5igKXKMlgyIbiykeOEYxZMERpkf5BDEdmpiEixAFuUC0JehKeZ8e4xTld2LcPGqZNyPm_Mt-gHlRUxdXbHhe-ID-yLVzYDmbTOp5-zvqA4A_vIMpV2dRWVUaXznmF_F7Y90Ou1SGKNZ2uLL6_1UXDvJeGF9dR_CI3OlK9ojY48D0FGMrua0kjOY5Pqe76dqIFL3Q2KPB2Q-tlgz-EHei4QQmZFzfwhMXYtCVPnvwHKT8PmLQopBrmYQVVDF9inew0iI2eZYjIsRqGSCQvpBBOt43Pk1OJJ4-D3xicu-45m4xgPn58wfoHyBjf1VYjFeTXrXQ-0qTNSN_IVYaSEtlthtB6YgfDjUWAoIs1Zpl-chhLjd3nEmXvuHT3vqph-PUx_raodOb1LQ9nUw6Ck0t1-x3XQTecaIGvBJUF-cBXSu7Ahz3QNwW4nshIvnt3u_jAijfbDPibJyRymFTVqNh5w-TNqXKWiL7PwrBlHAUlfF4FY-OTl8rA7DesF-t_2semd5BVLQQUnfUOBoi1rejFFXznEOsi4-ikRp5KmDC6OvZycKcZM8S0Jm2XH97yDp8ZzBXxcYX7BubcqSBc23nZs8ItQV_suikKE6x0Sshz5fEtQo4vltc4EaPZF82sT1YsZGGzLe6NdBLB4jmF5rnvepiNfOUtf-XgHG6BwDqzpkA9FIFGfRxjV3d4hVKHjJ0Qn827nCISpSQjjoEuj4OYW7npoC",
**"filepath"**: "70219d0ce05a4e14b828e719eaf6f552/"
},
**"fileCollectionSource"**: "70219d0ce05a4e14b828e719eaf6f552",
"createdBy": "preshipping_test_user@osdu-gcp.go3-nrg.projects.epam.com"
}
}
**How to use the information to load the multiple files to create a collection**__
`
Attached is the Postman collection that works on AWS
[DatatsetFileCollection_Test.postman_collection.json](/uploads/805790ab17f8fecae4c7bc557125da24/DatatsetFileCollection_Test.postman_collection.json)M20 - Release 0.23Denis Karpenok (EPAM)Dzmitry Malkevich (EPAM)Yauhen Shaliou [EPAM/GCP]Denis Karpenok (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/491EDS - M17 feature and fixes Testing on Azure Preship with detailed observatio...2023-06-13T10:59:51ZPriyanka BhongadeEDS - M17 feature and fixes Testing on Azure Preship with detailed observation and findings- [x] eds_scheduler run details after changes in constant file and including airflow utility
- eds_scheduler dag link : https://osdu-ship.msft-osdu-test.org/airflow2/graph?dag_id=eds_scheduler
- [x] Logger for eds_ingest - Display...- [x] eds_scheduler run details after changes in constant file and including airflow utility
- eds_scheduler dag link : https://osdu-ship.msft-osdu-test.org/airflow2/graph?dag_id=eds_scheduler
- [x] Logger for eds_ingest - Display message for sample record - https://github.com/ExxonMobil/osdu-platform/issues/318
- [x] Logger for eds_ingest - Display Osdu_ingest run ID explicitly in logs - https://github.com/ExxonMobil/osdu-platform/issues/318
- eds_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/log?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-05-02T07%3A59%3A56%2B00%3A00
- ![image](/uploads/bcf7d5da196ebef814667114a3bc337d/image.png)
- [x] Password Credential OAuth flow type - EDS - Include Password Credentials OAuth Flow Type #303
- eds_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/xcom?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-05-02T11%3A45%3A47%2B00%3A00
CSRE : opendes:master-data--ConnectedSourceRegistryEntry:KatalystTestingM17
![image](/uploads/7178379b3df27dfc0b32f2e411503bcb/image.png)
![image](/uploads/f56a814c55a2c69300f916f1f46eba64/image.png)
- [x] Validate the expiry of refresh token and generate a new refresh token and update secret vault - https://github.com/ExxonMobil/osdu-platform/issues/324
- eds_ingest dag link :
- [x] EDS - Adding Dynamic Schema Authority for Kind of CSRE ,CSDJ and ExternalReferenceValueMapping from Airflow Variable - https://github.com/ExxonMobil/osdu-platform/issues/360
- [x] EDS - Raise exception when Airflow Variable not found or None - https://github.com/ExxonMobil/osdu-platform/issues/359
- eds_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/log?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-05-02T08%3A10%3A55%2B00%3A00
-
![image](/uploads/0e48aa06635e320ea51b9b130bd28fa6/image.png)
- [ ] EDS: Conversion of the ConnectedSourceDataPartitionID to OnIngestionDataPartitionID for Array Datatype - https://github.com/ExxonMobil/osdu-platform/issues/322
- OSDU Ingest Link:
- [x] Test on Non-OSDU compliant Provider for master and WPC data
- Master Data -- SeismicAcquisitionSurvey
- eds_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/log?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-05-02T11%3A54%3A02%2B00%3A00
- Osdu_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/xcom?dag_id=Osdu_ingest&task_id=provide_manifest_integrity_task&execution_date=2023-05-02T11%3A54%3A15.263543%2B00%3A00
- Work Product Component -- issue with Wrapper - unable to provide data as "()" in the query
- eds_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/log?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-05-02T12%3A00%3A21%2B00%3A00
- Osdu_ingest dag link :
- [x] Test on OSDU Compliant Provider for master and WPC data - AWS M16
- Master Data -- Well
- eds ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/xcom?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-05-02T07%3A59%3A56%2B00%3A00
- Osdu_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/xcom?dag_id=Osdu_ingest&task_id=process_single_manifest_file_task&execution_date=2023-05-02T08%3A00%3A28.724429%2B00%3A00
- Work Product Component Data -- WellLog
- eds ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/xcom?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-05-02T08%3A06%3A35%2B00%3A00
- Osdu_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/xcom?dag_id=Osdu_ingest&task_id=process_single_manifest_file_task&execution_date=2023-05-02T08%3A06%3A48.542915%2B00%3A00
- [ ] Ingest any record and it will be added with "NameAliases" like below which has the source record id - https://github.com/ExxonMobil/osdu-platform/issues/289
"NameAliases": [
{
"AliasName": "odesprod:master-data--Basin:Kam2_02",
"AliasNameTypeID": ":reference-data--AliasNameType:EDSConnectedSourceIdentifier:",
"DefinitionOrganisationID": null
}
]
*Removed data-partition-id to avoid osdu_ingestion error
- [ ] Create provider specific External Reference Value Mapping - https://github.com/ExxonMobil/osdu-platform/issues/376https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/484M17 AWS - WITSML Parser - Trajectory data type - failure at schema validation...2023-05-02T21:04:01ZDebasis ChatterjeeM17 AWS - WITSML Parser - Trajectory data type - failure at schema validation stage[M17-AWS-WITSML-Trajectory-steps-and-data-Debasis.zip](/uploads/59370f20510a3deae8f2dd1398e39e3d/M17-AWS-WITSML-Trajectory-steps-and-data-Debasis.zip)
Please note that this is different problem than what we experienced earlier with work...[M17-AWS-WITSML-Trajectory-steps-and-data-Debasis.zip](/uploads/59370f20510a3deae8f2dd1398e39e3d/M17-AWS-WITSML-Trajectory-steps-and-data-Debasis.zip)
Please note that this is different problem than what we experienced earlier with work-product schema 1.1.0.
This time, it fails in schema validation stage. for Trajectory station type.https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/458For AWS platform query to get all kinds is not returning any records.2023-06-13T11:00:00ZKamlesh TodaiFor AWS platform query to get all kinds is not returning any records.The query to retrieve all the kinds is not returning any results (records)
curl --location 'https://r3m16-ue1.preshiptesting.osdu.aws/api/storage/v2/query/kinds' \
--header 'data-partition-id: osdu' \
--header 'Accept: application/json'...The query to retrieve all the kinds is not returning any results (records)
curl --location 'https://r3m16-ue1.preshiptesting.osdu.aws/api/storage/v2/query/kinds' \
--header 'data-partition-id: osdu' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer eyJraWQiOiJ...7kPscDabFJ3sEPeNA'
The response 200 OK (with results being empty)
{
"results": []
}
The collection used can be found at https://community.opengroup.org/osdu/platform/testing/-/blob/master/Postman%20Collection/12_CICD_Setup_StorageAPI/Storage%20API%20CI-CD%20v1.11.postman_collection.json
The request name is "01 Storage - Get all kinds success scenario"
@chad @debasischttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/448Pre-shipping: AWS CloudWatch does not show correct Container Mapping and No R...2023-04-06T12:34:51ZNaufal Mohamed NooriPre-shipping: AWS CloudWatch does not show correct Container Mapping and No Relevant Log From Any API runsI am currently testing non destructive operational procedure for AWS. The test involves monitor logs obtained from AWS console (CloudWatch) from any API runs from Postman (i.e. search or ingestion or storage API).
I encountered 2 peculi...I am currently testing non destructive operational procedure for AWS. The test involves monitor logs obtained from AWS console (CloudWatch) from any API runs from Postman (i.e. search or ingestion or storage API).
I encountered 2 peculiar issues:
a) I dont see any relevant logs retrieved from CloudWatch --> Log Groups --> /aws/containerinsights/r3-m16-eks-main-cluster/application
either from os-search (search API made from Postman) or storage API. In the previous milestone release, I am able to see all related logs when I ran search API from POSTMAN through cloudwatch.
![image](/uploads/7ebf9d0969056dee6819257a1f73e7c8/image.png)
b) The container insight map does not show r3m16 resources but only shows r3m12. It is weird as from the log group I can see the r3m16 resource built but the container map does not show any related map related to the current release.
![image](/uploads/799f012d870a1126f1e5e0f187b68cf2/image.png)M16 - Release 0.19https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/368EDS job erroneously changes the kind name when transferring record2022-10-11T04:52:00ZAndre MosleyEDS job erroneously changes the kind name when transferring recordWhen transferring a well from a remote OSDU system (i.e. a data provider's environment) into a local OSDU environment, the EDS (External Data Source) job changes the "authority" portion of the kind name from its original value to the loc...When transferring a well from a remote OSDU system (i.e. a data provider's environment) into a local OSDU environment, the EDS (External Data Source) job changes the "authority" portion of the kind name from its original value to the local environment's partition name. I've included an example below to highlight the issue:
_Here is the well on the remote system (AWS Pre-shipping environment in this case):_
![image](/uploads/d20835554aec30bc1169996a95444503/image.png)
_Here is the resulting well after being copied to the local system (Azure Pre-shipping in this case):_
![image](/uploads/c89857c5cb84183b2f2f16644885e24a/image.png)Rene von Borstel [EPAM]Ashish SaxenaNisha ThakranRene von Borstel [EPAM]https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/367Unable to fetch & ingest WPC from external data source2022-10-05T08:39:48ZAndre MosleyUnable to fetch & ingest WPC from external data sourceI am able to successfully setup an EDS ConnectedSourceDataJob to fetch a well or wellbore master data record, but am unable to fetch a WPC such as a well log. For a scheduled job, the DAG is able to find the WPC on the external source, a...I am able to successfully setup an EDS ConnectedSourceDataJob to fetch a well or wellbore master data record, but am unable to fetch a WPC such as a well log. For a scheduled job, the DAG is able to find the WPC on the external source, and begin to fetch it. However, during the process of prepping the data for ingestion, the Airflow logs always show failure with the errors below:
![image](/uploads/799ae3a23d9f197a0fa67fba8df8e3d2/image.png)
The full log can be found here: https://osdu-ship.msft-osdu-test.org/airflow2/log?dag_id=eds_ingest&task_id=fetch_client&execution_date=2022-09-15T23%3A58%3A00%2B00%3A00
The welllog WPC that was preloaded onto the AWS pre-ship environment (as the remote source) is provided in the attached [welllogWPC.json](/uploads/bed61e01c4e02a4e8bab8e3e26a9fad0/welllogWPC.json) file.
ConnectedSourceDataJob (on Azure "opendes" partition): opendes:master-data--ConnectedSourceDataJob:a9d70013-e645-4d7a-a721-89f88cvxtest_wellLog
ConnectedSourceRegistryEntryID (on Azure "opendes" partition): opendes:master-data--ConnectedSourceRegistryEntry:0ae7be1c-5df7-5656-98c3-f7c550awspreshipRene von Borstel [EPAM]Ashish SaxenaRene von Borstel [EPAM]https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/357IBM M13 - Manifest Ingestion - Performance Load testing 50k records - 504 res...2022-09-09T15:15:41ZChad LeongIBM M13 - Manifest Ingestion - Performance Load testing 50k records - 504 responseWhile submitting 50k records for performance load testing in manifest ingestion:
`https://cpd-osdu.odi-og-osdu-ba8e38d4c011d627379af1a4280c4e35-0000.us-south.containers.appdomain.cloud//osdu-workflow/api/workflow/v1/workflow/Osdu_ingest...While submitting 50k records for performance load testing in manifest ingestion:
`https://cpd-osdu.odi-og-osdu-ba8e38d4c011d627379af1a4280c4e35-0000.us-south.containers.appdomain.cloud//osdu-workflow/api/workflow/v1/workflow/Osdu_ingest/workflowRun`
### Got the following response:
```
<Response [504]>
b'20220527_1916 x86_64\n'
```
## 50k Records:
[Records_IBM_50000.zipx](/uploads/1838e5d6b46af934af4fcd35c2dc474c/Records_IBM_50000.zipx)
## Airflow response:
N/AShrikant GargShrikant Garghttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/332Azure - Spotfire Reporting Dashboard - Error retrieving data2022-08-23T11:28:04ZThiago MoreiraAzure - Spotfire Reporting Dashboard - Error retrieving dataI am not being able to connect the Reporting Portal to Azure. It is not retrieving data and I get this error:
Error message: Error retrieving Provider Data: Provider is Null in Credentials
Data connection:
Open Subsurface Data Universe ...I am not being able to connect the Reporting Portal to Azure. It is not retrieving data and I get this error:
Error message: Error retrieving Provider Data: Provider is Null in Credentials
Data connection:
Open Subsurface Data Universe (OSDU)
Provider Name:
Microsoft
ImportException at Spotfire.Dxp.Data:
Error retrieving Provider Data: Provider is Null in Credentials
Data connection:
Open Subsurface Data Universe (OSDU)
Provider Name:
Microsoft (HRESULT: 80131500)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/325GCP- Spotfire Reporting Dashboard - Cannot retrieve well data properly2022-08-23T11:28:05ZThiago MoreiraGCP- Spotfire Reporting Dashboard - Cannot retrieve well data properlyThe connector works and I can see the wells in the map, but when I try to get well data to populate the graphs, WellLog_ID columns has no data in many wells. Even one well that has the WellLog_ID information is not showing any data in th...The connector works and I can see the wells in the map, but when I try to get well data to populate the graphs, WellLog_ID columns has no data in many wells. Even one well that has the WellLog_ID information is not showing any data in the graphs.
I tried to run the Volume Dashboard and it did not get any data.https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/294Azure R3M11 - Manifest Ingestion FoR CRS: skipped_ids "Entity doesn't pass th...2022-08-23T13:29:58ZEsmira RafigayevaAzure R3M11 - Manifest Ingestion FoR CRS: skipped_ids "Entity doesn't pass the schema validation"Manifest Ingestion FoR CRS test execution: indicating in json payload
"id": "osdu:master-data--Well:1115ER",
"kind": "osdu:wks:master-data--Well:1.0.0",
getting skipped ids.
"runId": "fd4e79f9-acc...Manifest Ingestion FoR CRS test execution: indicating in json payload
"id": "osdu:master-data--Well:1115ER",
"kind": "osdu:wks:master-data--Well:1.0.0",
getting skipped ids.
"runId": "fd4e79f9-accd-42d7-8992-11d648eb29eb"
Airflow response:
XCom
Key Value
saved_record_ids {'process_single_manifest_file_task': ['opendes:reference-data--ResourceSecurityClassification:RESTRICTED']}
skipped_ids {'validate_manifest_schema_task': [{'id': 'osdu:master-data--Well:1115ER', 'kind': 'osdu:wks:master-data--Well:1.0.0', 'reason': "Entity doesn't pass the schema validation."}]}
Appreciate your quick response. Regards, Esmira cc: @sehuboy @ankurrawat @krganesan @krveduru @NikhilSingh @debasiscM11 - Release 0.14Nikhil Singh[MicroSoft]Esmira RafigayevaNikhil Singh[MicroSoft]https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/292Azure - WITSML Parser collection. Suggestions and feedback2023-09-28T13:07:16ZDebasis ChatterjeeAzure - WITSML Parser collection. Suggestions and feedbackI have a few improvement suggestions for this collection.
![Azure-WITSML-Parser-collection-steps](/uploads/298062b85e86bb7dab3de418b0810336/Azure-WITSML-Parser-collection-steps.PNG)
Why is this called “1. Record id”?
Combine steps 1.0...I have a few improvement suggestions for this collection.
![Azure-WITSML-Parser-collection-steps](/uploads/298062b85e86bb7dab3de418b0810336/Azure-WITSML-Parser-collection-steps.PNG)
Why is this called “1. Record id”?
Combine steps 1.0 and 1.4 – in sub-folder “Pre-requisite steps”.
Then have another sub-folder (name = “Repeat for each data type”) for steps 1.1-1.3, 1.5-1.9.
Keep a companion guide which will instruct the user/tester to perform the repeatable steps by using test data of “Well” record.
Ensure that “Well” mentioned in Wellbore test data is actually the one created above.
Then steps for Log, Marker, Trajectory, Tubular can come in any order.
Each should ensure that mentioned Wellbore ID actually exists (from previous steps).
Ideally, please create dedicated folder for WITSML in Azure/Preship site with guide (word document), Postman Collection and 6 sample data files.
https://community.opengroup.org/osdu/platform/pre-shipping/-/tree/main/R3-M11/Azure-M11/Services/Ingestion
Thanks
cc @krveduruKishore BattulaKishore Battula