Pre Shipping issueshttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues2023-11-22T04:15:10Zhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/641EDS bug in fetch_client.py2023-11-22T04:15:10ZShane HutchinsEDS bug in fetch_client.pyFrom airflow eds_ingest fetch_client on Azure. When attempting to pull from GCP M21 Preship.
Reading remote log from wasb-airflowlog/opendes:master-data--ConnectedSourceDataJob:AWSPreshipSH202311__2023-11-14T16:05:00+00:00/eds_ingest/fe...From airflow eds_ingest fetch_client on Azure. When attempting to pull from GCP M21 Preship.
Reading remote log from wasb-airflowlog/opendes:master-data--ConnectedSourceDataJob:AWSPreshipSH202311__2023-11-14T16:05:00+00:00/eds_ingest/fetch_client/2023-11-14T16:05:00+00:00/None/1.log.
While it's possible the credentials are no longer good or other issues. fetch_client.py is not checking return status and assuming a 200 return and "results" in the response (hence the `KeyError`).
```
[2023-11-14, 16:05:08 UTC] {check_oauth_flow_type.py:28} INFO - Auth Code Generated for RefreshTokenKeyName
[2023-11-14, 16:05:08 UTC] {check_oauth_flow_type.py:29} INFO - _build_auth_code_request---end
[2023-11-14, 16:05:08 UTC] {fetch_client.py:61} INFO - QUERY BODY:{'kind': 'osdu:wks:master-data--Basin:1.0.0', 'query': '* AND ((createTime: [2023-02-01T00:00:00 TO 2023-11-14T16:05:06]) OR (modifyTime: [2023-02-01T00:00:00 TO 2023-11-14T16:05:06]))', 'sort': {'field': ['createTime'], 'order': ['ASC']}, 'limit': 5}
[2023-11-14, 16:05:08 UTC] {src_dags_fetch_and_ingest.py:152} ERROR - {'status': 'error', 'message': KeyError('results')}
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/osdu_airflow/eds/eds_ingest/src_dags_fetch_and_ingest.py", line 92, in fetch_and_ingest
records = self.fetch_client.fetch_details(criteria_dict,self.auth)
File "/home/airflow/.local/lib/python3.8/site-packages/osdu_airflow/eds/eds_ingest/implementation/fetch/fetch_client.py", line 67, in fetch_details
records = search_body["results"]
KeyError: 'results'
```M21 - Release 0.24Ashish SaxenaAshish Saxenahttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/368EDS job erroneously changes the kind name when transferring record2022-10-11T04:52:00ZAndre MosleyEDS job erroneously changes the kind name when transferring recordWhen transferring a well from a remote OSDU system (i.e. a data provider's environment) into a local OSDU environment, the EDS (External Data Source) job changes the "authority" portion of the kind name from its original value to the loc...When transferring a well from a remote OSDU system (i.e. a data provider's environment) into a local OSDU environment, the EDS (External Data Source) job changes the "authority" portion of the kind name from its original value to the local environment's partition name. I've included an example below to highlight the issue:
_Here is the well on the remote system (AWS Pre-shipping environment in this case):_
![image](/uploads/d20835554aec30bc1169996a95444503/image.png)
_Here is the resulting well after being copied to the local system (Azure Pre-shipping in this case):_
![image](/uploads/c89857c5cb84183b2f2f16644885e24a/image.png)Rene von Borstel [EPAM]Ashish SaxenaNisha ThakranRene von Borstel [EPAM]https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/491EDS - M17 feature and fixes Testing on Azure Preship with detailed observatio...2023-06-13T10:59:51ZPriyanka BhongadeEDS - M17 feature and fixes Testing on Azure Preship with detailed observation and findings- [x] eds_scheduler run details after changes in constant file and including airflow utility
- eds_scheduler dag link : https://osdu-ship.msft-osdu-test.org/airflow2/graph?dag_id=eds_scheduler
- [x] Logger for eds_ingest - Display...- [x] eds_scheduler run details after changes in constant file and including airflow utility
- eds_scheduler dag link : https://osdu-ship.msft-osdu-test.org/airflow2/graph?dag_id=eds_scheduler
- [x] Logger for eds_ingest - Display message for sample record - https://github.com/ExxonMobil/osdu-platform/issues/318
- [x] Logger for eds_ingest - Display Osdu_ingest run ID explicitly in logs - https://github.com/ExxonMobil/osdu-platform/issues/318
- eds_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/log?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-05-02T07%3A59%3A56%2B00%3A00
- ![image](/uploads/bcf7d5da196ebef814667114a3bc337d/image.png)
- [x] Password Credential OAuth flow type - EDS - Include Password Credentials OAuth Flow Type #303
- eds_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/xcom?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-05-02T11%3A45%3A47%2B00%3A00
CSRE : opendes:master-data--ConnectedSourceRegistryEntry:KatalystTestingM17
![image](/uploads/7178379b3df27dfc0b32f2e411503bcb/image.png)
![image](/uploads/f56a814c55a2c69300f916f1f46eba64/image.png)
- [x] Validate the expiry of refresh token and generate a new refresh token and update secret vault - https://github.com/ExxonMobil/osdu-platform/issues/324
- eds_ingest dag link :
- [x] EDS - Adding Dynamic Schema Authority for Kind of CSRE ,CSDJ and ExternalReferenceValueMapping from Airflow Variable - https://github.com/ExxonMobil/osdu-platform/issues/360
- [x] EDS - Raise exception when Airflow Variable not found or None - https://github.com/ExxonMobil/osdu-platform/issues/359
- eds_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/log?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-05-02T08%3A10%3A55%2B00%3A00
-
![image](/uploads/0e48aa06635e320ea51b9b130bd28fa6/image.png)
- [ ] EDS: Conversion of the ConnectedSourceDataPartitionID to OnIngestionDataPartitionID for Array Datatype - https://github.com/ExxonMobil/osdu-platform/issues/322
- OSDU Ingest Link:
- [x] Test on Non-OSDU compliant Provider for master and WPC data
- Master Data -- SeismicAcquisitionSurvey
- eds_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/log?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-05-02T11%3A54%3A02%2B00%3A00
- Osdu_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/xcom?dag_id=Osdu_ingest&task_id=provide_manifest_integrity_task&execution_date=2023-05-02T11%3A54%3A15.263543%2B00%3A00
- Work Product Component -- issue with Wrapper - unable to provide data as "()" in the query
- eds_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/log?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-05-02T12%3A00%3A21%2B00%3A00
- Osdu_ingest dag link :
- [x] Test on OSDU Compliant Provider for master and WPC data - AWS M16
- Master Data -- Well
- eds ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/xcom?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-05-02T07%3A59%3A56%2B00%3A00
- Osdu_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/xcom?dag_id=Osdu_ingest&task_id=process_single_manifest_file_task&execution_date=2023-05-02T08%3A00%3A28.724429%2B00%3A00
- Work Product Component Data -- WellLog
- eds ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/xcom?dag_id=eds_ingest&task_id=fetch_client&execution_date=2023-05-02T08%3A06%3A35%2B00%3A00
- Osdu_ingest dag link : https://osdu-ship.msft-osdu-test.org/airflow2/xcom?dag_id=Osdu_ingest&task_id=process_single_manifest_file_task&execution_date=2023-05-02T08%3A06%3A48.542915%2B00%3A00
- [ ] Ingest any record and it will be added with "NameAliases" like below which has the source record id - https://github.com/ExxonMobil/osdu-platform/issues/289
"NameAliases": [
{
"AliasName": "odesprod:master-data--Basin:Kam2_02",
"AliasNameTypeID": ":reference-data--AliasNameType:EDSConnectedSourceIdentifier:",
"DefinitionOrganisationID": null
}
]
*Removed data-partition-id to avoid osdu_ingestion error
- [ ] Create provider specific External Reference Value Mapping - https://github.com/ExxonMobil/osdu-platform/issues/376https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/563Energistics XML parser DAG not able to execute successfully. Fails while cr...2023-08-16T10:04:39ZKamlesh TodaiEnergistics XML parser DAG not able to execute successfully. Fails while creating new pod for the oepratorWhen one looks at the DAG runs there has not been any successful run. It is failing while creating new pod.
```plaintext
[2023-08-04 20:00:50,255] {pod_launcher.py:128} WARNING - Pod not yet started: witsml-parser-task.10c656bb27934c3c9...When one looks at the DAG runs there has not been any successful run. It is failing while creating new pod.
```plaintext
[2023-08-04 20:00:50,255] {pod_launcher.py:128} WARNING - Pod not yet started: witsml-parser-task.10c656bb27934c3c90951ea67d10e8a0
[2023-08-04 20:00:50,272] {taskinstance.py:1463} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 367, in execute
final_state, remote_pod, result = self.create_new_pod_for_operator(labels, launcher)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 520, in create_new_pod_for_operator
launcher.start_pod(self.pod, startup_timeout=self.startup_timeout_seconds)
File "/home/airflow/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 329, in wrapped_f
return self.call(f, *args, **kw)
File "/home/airflow/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 409, in call
do = self.iter(retry_state=retry_state)
File "/home/airflow/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 356, in iter
return fut.result()
File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 437, in result
return self.__get_result()
File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/home/airflow/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 412, in call
result = fn(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/utils/pod_launcher.py", line 131, in start_pod
raise AirflowException("Pod took too long to start")
airflow.exceptions.AirflowException: Pod took too long to start
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1165, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1283, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1313, in _execute_task
result = task_copy.execute(context=context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 374, in execute
raise AirflowException(f'Pod Launching failed: {ex}')
airflow.exceptions.AirflowException: Pod Launching failed: Pod took too long to start
[2023-08-04 20:00:50,273] {taskinstance.py:1506} INFO - Marking task as FAILED. dag_id=Energistics_xml_ingest, task_id=witsml_parser_task, execution_date=20230804T195547, start_date=20230804T195549, end_date=20230804T200050
[2023-08-04 20:00:50,318] {local_task_job.py:151} INFO - Task exited with return code 1
[2023-08-04 20:00:50,339] {local_task_job.py:261} INFO - 0 downstream tasks scheduled from follow-on schedule check
```Anuj Guptavikas ranaAnuj Guptahttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/497Erroneous Inconsistent search results when adding or removing records2024-02-22T15:26:10ZMichaelErroneous Inconsistent search results when adding or removing recordsWe are adding and removing records from OSDU Storage, and we see erroneous results where a record that we add is not shown in the search result or a record that we delete is shown.
We tried waiting for the expected result from the sear...We are adding and removing records from OSDU Storage, and we see erroneous results where a record that we add is not shown in the search result or a record that we delete is shown.
We tried waiting for the expected result from the search and then trying again. We see erroneous results sometimes when we find it in the search results, but the next search does not find it. Similarly, for deleted records, we sometimes see the deletion only to see it not deleted in the next search.
Here is a video that demonstrates the inconsistent search results after a record has been created: [osdu_indexing_consistency.zip](/uploads/88858771b3a497a7336dd3bf47bb0886/osdu_indexing_consistency.zip)
Here is the postman collection that is used in the video: [Indexing_Test.postman_collection.json](/uploads/6958cd6c0b3da081a54a46488e1a2497/Indexing_Test.postman_collection.json)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/458For AWS platform query to get all kinds is not returning any records.2023-06-13T11:00:00ZKamlesh TodaiFor AWS platform query to get all kinds is not returning any records.The query to retrieve all the kinds is not returning any results (records)
curl --location 'https://r3m16-ue1.preshiptesting.osdu.aws/api/storage/v2/query/kinds' \
--header 'data-partition-id: osdu' \
--header 'Accept: application/json'...The query to retrieve all the kinds is not returning any results (records)
curl --location 'https://r3m16-ue1.preshiptesting.osdu.aws/api/storage/v2/query/kinds' \
--header 'data-partition-id: osdu' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer eyJraWQiOiJ...7kPscDabFJ3sEPeNA'
The response 200 OK (with results being empty)
{
"results": []
}
The collection used can be found at https://community.opengroup.org/osdu/platform/testing/-/blob/master/Postman%20Collection/12_CICD_Setup_StorageAPI/Storage%20API%20CI-CD%20v1.11.postman_collection.json
The request name is "01 Storage - Get all kinds success scenario"
@chad @debasischttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/674GC M22 - Core Services - Search "AsIngestedCoordinates" not returning2024-01-29T15:53:54ZJuliana Fernandesjuliana.fernandes@iesbrazil.com.brGC M22 - Core Services - Search "AsIngestedCoordinates" not returningHello,
I'm trying to complete the the task **Search shows AsIngested** from pre-shipping and I'm not getting the expected result.
First I need to Get the record and this one shows AsIngestedCoordinates, but when I search it does not ap...Hello,
I'm trying to complete the the task **Search shows AsIngested** from pre-shipping and I'm not getting the expected result.
First I need to Get the record and this one shows AsIngestedCoordinates, but when I search it does not appear.
So I made a search for the AsIngestedCoordinates without success and finally made a search to look for the AsIngestedCoordinates in a record without success to...
What it's happening?
Debasis posted an example for AWS, here: https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/Test_plan_Results_M22/Core%20Services/M22-AWS-Core-Services-search-returns-AsIngested-coordinates-Debasis.docx?ref_type=heads
Here is what I'm doing for Search shows AsIngested:
- Collection: [GC_OSDU_Smoke_Tests.postman_collection.json](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/GC-M22/GC_OSDU_Smoke_Tests.postman_collection.json)
- Environment: [OSDU_R3_GC_Pre-ship.postman_environment.json](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/GC-M22/OSDU_R3_GC_Pre-ship.postman_environment.json)
## 1. GET the Record
GET https://{{STORAGE_HOST}}/records/m19:master-data--Wellbore:9254
**BODY CURL VERSION**
```
curl --location 'https://preship.gcp.gnrg-osdu.projects.epam.com/api/storage/v2/records/m19:master-data--Wellbore:9254' \
--header 'data-partition-id: m19' \
--header 'Authorization: Bearer <Token>'
```
**RESPONSE**
```json
{
"data": {
"CurrentOperatorID": "m19:master-data--Organisation:Total%20E%26P%20Nederland%20B.V.:",
"DefaultVerticalMeasurementID": "Measured_From",
"DrillingReasons": [
{
"DrillingReasonTypeID": "m19:reference-data--DrillingReasonType:EXP-HC:"
}
],
"FacilityEvents": [
{
"EffectiveDateTime": "1995-12-09T00:00:00",
"FacilityEventTypeID": "m19:reference-data--FacilityEventType:Sidetrack:"
},
{
"EffectiveDateTime": "1996-02-17T00:00:00",
"FacilityEventTypeID": "m19:reference-data--FacilityEventType:TDReached:"
}
],
"FacilityName": "9254",
"FacilityStates": [
{
"FacilityStateTypeID": "m19:reference-data--FacilityStateType:ShutIn:"
}
],
"FacilityTypeID": "m19:reference-data--FacilityType:Wellbore:",
"GeoContexts": [
{
"GeoPoliticalEntityID": "m19:master-data--GeoPoliticalEntity:K04_BlockID:",
"GeoTypeID": "m19:reference-data--GeoPoliticalEntityType:BlockID:"
},
{
"FieldID": "m19:master-data--Field:K04-A:",
"GeoTypeID": "Field"
}
],
"InitialOperatorID": "m19:master-data--Organisation:Elf%20Petroland:",
"NameAliases": [
{
"AliasName": "9254",
"AliasNameTypeID": "m19:reference-data--AliasNameType:UWI:"
},
{
"AliasName": "K04-09B-S2",
"AliasNameTypeID": "m19:reference-data--AliasNameType:BoreholeCode:"
},
{
"AliasName": "K04-09B-SIDETRACK2",
"AliasNameTypeID": "m19:reference-data--AliasNameType:Borehole:"
}
],
"OperatingEnvironmentID": "m19:reference-data--OperatingEnvironment:Off:",
"PrimaryMaterialID": "m19:reference-data--MaterialType:Gas:",
"ProjectedBottomHoleLocation": {
"AsIngestedCoordinates": {
"CoordinateReferenceSystemID": "m19:reference-data--CoordinateReferenceSystem:23031:",
"features": [
{
"geometry": {
"coordinates": [
517701.0,
5957457.0
],
"type": "AnyCrsPoint"
},
"properties": {},
"type": "AnyCrsFeature"
}
],
"persistableReferenceCrs": "{\"authCode\":{\"auth\":\"EPSG\",\"code\":\"23031\"},\"name\":\"ED_1950_UTM_Zone_31N\",\"type\":\"LBC\",\"ver\":\"PE_10_3_1\",\"wkt\":\"PROJCS[\\\"ED_1950_UTM_Zone_31N\\\",GEOGCS[\\\"GCS_European_1950\\\",DATUM[\\\"D_European_1950\\\",SPHEROID[\\\"International_1924\\\",6378388.0,297.0]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],PROJECTION[\\\"Transverse_Mercator\\\"],PARAMETER[\\\"False_Easting\\\",500000.0],PARAMETER[\\\"False_Northing\\\",0.0],PARAMETER[\\\"Central_Meridian\\\",3.0],PARAMETER[\\\"Scale_Factor\\\",0.9996],PARAMETER[\\\"Latitude_Of_Origin\\\",0.0],UNIT[\\\"Meter\\\",1.0],AUTHORITY[\\\"EPSG\\\",23031]]\"}",
"type": "AnyCrsFeatureCollection"
}
},
"SequenceNumber": 3,
"SpatialLocation": {
"AsIngestedCoordinates": {
"CoordinateReferenceSystemID": "m19:reference-data--CoordinateReferenceSystem:23031:",
"features": [
{
"geometry": {
"coordinates": [
520507.0,
5955994.0
],
"type": "AnyCrsPoint"
},
"properties": {},
"type": "AnyCrsFeature"
}
],
"persistableReferenceCrs": "{\"authCode\":{\"auth\":\"EPSG\",\"code\":\"23031\"},\"name\":\"ED_1950_UTM_Zone_31N\",\"type\":\"LBC\",\"ver\":\"PE_10_3_1\",\"wkt\":\"PROJCS[\\\"ED_1950_UTM_Zone_31N\\\",GEOGCS[\\\"GCS_European_1950\\\",DATUM[\\\"D_European_1950\\\",SPHEROID[\\\"International_1924\\\",6378388.0,297.0]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433]],PROJECTION[\\\"Transverse_Mercator\\\"],PARAMETER[\\\"False_Easting\\\",500000.0],PARAMETER[\\\"False_Northing\\\",0.0],PARAMETER[\\\"Central_Meridian\\\",3.0],PARAMETER[\\\"Scale_Factor\\\",0.9996],PARAMETER[\\\"Latitude_Of_Origin\\\",0.0],UNIT[\\\"Meter\\\",1.0],AUTHORITY[\\\"EPSG\\\",23031]]\"}",
"type": "AnyCrsFeatureCollection"
},
"Wgs84Coordinates": {
"features": [
{
"geometry": {
"coordinates": [
3.30959248,
53.75027988
],
"type": "Point"
},
"properties": {},
"type": "Feature"
}
],
"type": "FeatureCollection"
}
},
"TrajectoryTypeID": "m19:reference-data--WellboreTrajectoryType:Deviated:",
"VerticalMeasurements": [
{
"VerticalMeasurement": 5193.0,
"VerticalMeasurementID": "TD-Original",
"VerticalMeasurementPathID": "m19:reference-data--VerticalMeasurementPath:MD:",
"VerticalMeasurementTypeID": "m19:reference-data--VerticalMeasurementType:TD:",
"VerticalReferenceID": "Measured_From"
},
{
"VerticalMeasurement": 3887.61,
"VerticalMeasurementID": "TVD",
"VerticalMeasurementPathID": "m19:reference-data--VerticalMeasurementPath:TVD:",
"VerticalMeasurementTypeID": "m19:reference-data--VerticalMeasurementType:TD:",
"VerticalReferenceID": "Measured_From"
},
{
"VerticalCRSID": "m19:reference-data--CoordinateReferenceSystem:MSL:",
"VerticalMeasurement": 40.0,
"VerticalMeasurementID": "Measured_From",
"VerticalMeasurementPathID": "m19:reference-data--VerticalMeasurementPath:Elevation:",
"VerticalMeasurementTypeID": "m19:reference-data--VerticalMeasurementType:RotaryTable:"
}
],
"WellID": "m19:master-data--Well:8988:"
},
"meta": [
{
"kind": "Unit",
"name": "m",
"persistableReference": "{\"abcd\":{\"a\":0.0,\"b\":1.0,\"c\":1.0,\"d\":0.0},\"symbol\":\"m\",\"baseMeasurement\":{\"ancestry\":\"L\",\"type\":\"UM\"},\"type\":\"UAD\"}",
"propertyNames": [
"VerticalMeasurements[].VerticalMeasurement"
],
"unitOfMeasureID": "m19:reference-data--UnitOfMeasure:m:"
}
],
"id": "m19:master-data--Wellbore:9254",
"version": 1689778069245048,
"kind": "osdu:wks:master-data--Wellbore:1.0.0",
"acl": {
"viewers": [
"data.default.viewers@m19.group"
],
"owners": [
"data.default.owners@m19.group"
]
},
"legal": {
"legaltags": [
"m19-default-data-tag"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "datafier@osdu-service-prod.iam.gserviceaccount.com",
"createTime": "2023-07-19T14:47:50.784Z"
}
```
## 2. Search the Record
### You should see AsIngested coordinates in the response.
POST {{SEARCH_HOST}}/query
**BODY CURL VERSION**
```
curl --location 'https://preship.gcp.gnrg-osdu.projects.epam.com/api/search/v2/query' \
--header 'Authorization: Bearer <Token>' \
--header 'data-partition-id: m19' \
--header 'Content-Type: application/json' \
--data '{
"kind": "osdu:wks:master-data--Wellbore:*",
"query": "id: \"m19:master-data--Wellbore:9254\""
}'
```
**BODY**
```json
{
"kind": "{{authority}}:{{schemaSource}}:master-data--Wellbore:*",
"query": "id: \"m19:master-data--Wellbore:9254\""
}
```
**RESPONSE**
```json
{
"results": [
{
"data": {
"VirtualProperties.DefaultLocation.QuantitativeAccuracyBandID": null,
"DefaultVerticalMeasurementID": "Measured_From",
"ResourceCurationStatus": null,
"FacilityName": "9254",
"DrillingReasons": [
{
"DrillingReasonTypeID": "m19:reference-data--DrillingReasonType:EXP-HC:"
}
],
"VirtualProperties.DefaultName": "9254",
"VirtualProperties.DefaultLocation.CoordinateQualityCheckPerformedBy": null,
"ResourceSecurityClassification": null,
"SequenceNumber": 3,
"DataSourceOrganisationID": null,
"SpatialLocation.SpatialParameterTypeID": null,
"ExistenceKind": null,
"FacilityTypeID": "m19:reference-data--FacilityType:Wellbore:",
"FacilityEvents": [
{
"EffectiveDateTime": "1995-12-09T00:00:00+0000",
"FacilityEventTypeID": "m19:reference-data--FacilityEventType:Sidetrack:"
},
{
"EffectiveDateTime": "1996-02-17T00:00:00+0000",
"FacilityEventTypeID": "m19:reference-data--FacilityEventType:TDReached:"
}
],
"VirtualProperties.DefaultLocation.IsDecimated": false,
"ProjectedBottomHoleLocation.QuantitativeAccuracyBandID": null,
"CurrentOperatorID": "m19:master-data--Organisation:Total%20E%26P%20Nederland%20B.V.:",
"ProjectedBottomHoleLocation.SpatialGeometryTypeID": null,
"SpatialLocation.SpatialGeometryTypeID": null,
"OperatingEnvironmentID": "m19:reference-data--OperatingEnvironment:Off:",
"ProjectedBottomHoleLocation.QualitativeSpatialAccuracyTypeID": null,
"GeographicBottomHoleLocation.QuantitativeAccuracyBandID": null,
"PrimaryMaterialID": "m19:reference-data--MaterialType:Gas:",
"VirtualProperties.DefaultLocation.QualitativeSpatialAccuracyTypeID": null,
"InitialOperatorID": "m19:master-data--Organisation:Elf%20Petroland:",
"GeographicBottomHoleLocation.SpatialGeometryTypeID": null,
"WellID": "m19:master-data--Well:8988:",
"FacilityStates": [
{
"FacilityStateTypeID": "m19:reference-data--FacilityStateType:ShutIn:"
}
],
"GeographicBottomHoleLocation.SpatialParameterTypeID": null,
"GeographicBottomHoleLocation.CoordinateQualityCheckPerformedBy": null,
"ResourceLifecycleStatus": null,
"TargetFormation": null,
"SpatialLocation.Wgs84Coordinates": {
"geometries": [
{
"coordinates": [
3.30959248,
53.75027988
],
"type": "point"
}
],
"type": "geometrycollection"
},
"TechnicalAssuranceID": null,
"VirtualProperties.DefaultLocation.SpatialGeometryTypeID": null,
"Source": null,
"DefinitiveTrajectoryID": null,
"FacilityID": null,
"GeographicBottomHoleLocation.QualitativeSpatialAccuracyTypeID": null,
"VerticalMeasurements": [
{
"WellboreTVDTrajectoryID": null,
"VerticalCRSID": null,
"VerticalMeasurementSourceID": null,
"VerticalReferenceID": "Measured_From",
"VerticalMeasurementID": "TD-Original",
"VerticalMeasurementPathID": "m19:reference-data--VerticalMeasurementPath:MD:",
"VerticalMeasurement": 5193.0,
"VerticalMeasurementTypeID": "m19:reference-data--VerticalMeasurementType:TD:",
"VerticalMeasurementDescription": null,
"VerticalMeasurementUnitOfMeasureID": null
},
{
"WellboreTVDTrajectoryID": null,
"VerticalCRSID": null,
"VerticalMeasurementSourceID": null,
"VerticalReferenceID": "Measured_From",
"VerticalMeasurementID": "TVD",
"VerticalMeasurementPathID": "m19:reference-data--VerticalMeasurementPath:TVD:",
"VerticalMeasurement": 3887.61,
"VerticalMeasurementTypeID": "m19:reference-data--VerticalMeasurementType:TD:",
"VerticalMeasurementDescription": null,
"VerticalMeasurementUnitOfMeasureID": null
},
{
"WellboreTVDTrajectoryID": null,
"VerticalCRSID": "m19:reference-data--CoordinateReferenceSystem:MSL:",
"VerticalMeasurementSourceID": null,
"VerticalReferenceID": null,
"VerticalMeasurementID": "Measured_From",
"VerticalMeasurementPathID": "m19:reference-data--VerticalMeasurementPath:Elevation:",
"VerticalMeasurement": 40.0,
"VerticalMeasurementTypeID": "m19:reference-data--VerticalMeasurementType:RotaryTable:",
"VerticalMeasurementDescription": null,
"VerticalMeasurementUnitOfMeasureID": null
}
],
"VersionCreationReason": null,
"KickOffWellbore": null,
"SpatialLocation.CoordinateQualityCheckPerformedBy": null,
"NameAliases": [
{
"AliasName": "9254",
"AliasNameTypeID": "m19:reference-data--AliasNameType:UWI:",
"DefinitionOrganisationID": null
},
{
"AliasName": "K04-09B-S2",
"AliasNameTypeID": "m19:reference-data--AliasNameType:BoreholeCode:",
"DefinitionOrganisationID": null
},
{
"AliasName": "K04-09B-SIDETRACK2",
"AliasNameTypeID": "m19:reference-data--AliasNameType:Borehole:",
"DefinitionOrganisationID": null
}
],
"ProjectedBottomHoleLocation.CoordinateQualityCheckPerformedBy": null,
"GeoContexts": [
{
"BasinID": null,
"FieldID": null,
"PlayID": null,
"GeoPoliticalEntityID": "m19:master-data--GeoPoliticalEntity:K04_BlockID:",
"GeoTypeID": "m19:reference-data--GeoPoliticalEntityType:BlockID:",
"ProspectID": null
},
{
"BasinID": null,
"FieldID": "m19:master-data--Field:K04-A:",
"PlayID": null,
"GeoPoliticalEntityID": null,
"GeoTypeID": "Field",
"ProspectID": null
}
],
"ProjectedBottomHoleLocation.SpatialParameterTypeID": null,
"SpatialLocation.QualitativeSpatialAccuracyTypeID": null,
"VirtualProperties.DefaultLocation.SpatialParameterTypeID": null,
"ResourceHomeRegionID": null,
"VirtualProperties.DefaultLocation.Wgs84Coordinates": {
"geometries": [
{
"coordinates": [
3.30959248,
53.75027988
],
"type": "point"
}
],
"type": "geometrycollection"
},
"SpatialLocation.QuantitativeAccuracyBandID": null,
"TrajectoryTypeID": "m19:reference-data--WellboreTrajectoryType:Deviated:"
},
"kind": "osdu:wks:master-data--Wellbore:1.0.0",
"source": "wks",
"acl": {
"viewers": [
"data.default.viewers@m19.group"
],
"owners": [
"data.default.owners@m19.group"
]
},
"type": "master-data--Wellbore",
"version": 1689778069245048,
"tags": {
"normalizedKind": "osdu:wks:master-data--Wellbore:1"
},
"createTime": "2023-07-19T14:47:50.784Z",
"authority": "osdu",
"namespace": "osdu:wks",
"legal": {
"legaltags": [
"m19-dataimport-legaltag"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "datafier@osdu-service-prod.iam.gserviceaccount.com",
"id": "m19:master-data--Wellbore:9254"
}
],
"aggregations": null,
"phraseSuggestions": [],
"totalCount": 1
}
```
When I Search for AsIngestedCoordinates I get an empty result:
## Serch AsIngestedCoordinates
POST https://{{SEARCH_HOST}}/query
**BODY CURL VERSION**
```
curl --location 'https://preship.gcp.gnrg-osdu.projects.epam.com/api/search/v2/query' \
--header 'Authorization: Bearer <Token>' \
--header 'data-partition-id: m19' \
--header 'Content-Type: application/json' \
--data '{
"kind": ["osdu:wks:master-data--Wellbore:*"],
"returnedFields": ["id", "data.SpatialLocation", "data.FacilityName"],
"query":
"data.SpatialLocation.AsIngestedCoordinates.CoordinateReferenceSystemID:\"m19:reference-data--CoordinateReferenceSystem:23031:\""
}'
```
**BODY**
```json
{
"kind": ["{{authority}}:{{schemaSource}}:master-data--Wellbore:*"],
"returnedFields": ["id", "data.SpatialLocation", "data.FacilityName"],
"query":
"data.SpatialLocation.AsIngestedCoordinates.CoordinateReferenceSystemID:\"m19:reference-data--CoordinateReferenceSystem:23031:\""
}
```
**RESPONSE**
```json
{
"results": [],
"aggregations": [],
"phraseSuggestions": [],
"totalCount": 0
}
```
To make sure the AsIngestedCoordinates exist I made the following search and didn't found it:
## Search for AsIngestedCoordinates in the Record
POST https://{{SEARCH_HOST}}/query
**BODY CURL VERSION**
```
curl --location 'https://preship.gcp.gnrg-osdu.projects.epam.com/api/search/v2/query' \
--header 'Authorization: Bearer <Token>' \
--header 'data-partition-id: m19' \
--header 'Content-Type: application/json' \
--data '{
"kind": "osdu:*:master-data--Wellbore:*",
"offset": 0,
"limit": 1,
"trackTotalCount": true,
"highlightedFields": ["id", "data.ProjectedBottomHoleLocation.AsIngestedCoordinates", "data.FacilityName", "kind"]
}'
```
**BODY**
```json
{
"kind": "osdu:*:master-data--Wellbore:*",
"offset": 0,
"limit": 1,
"trackTotalCount": true,
"highlightedFields": ["id", "data.ProjectedBottomHoleLocation.AsIngestedCoordinates", "data.FacilityName", "kind"]
}
```
**RESPONSE**
```json
{
"results": [
{
"data": {
"VirtualProperties.DefaultLocation.QuantitativeAccuracyBandID": null,
"DefaultVerticalMeasurementID": "Measured_From",
"ResourceCurationStatus": null,
"FacilityName": "9254",
"DrillingReasons": [
{
"DrillingReasonTypeID": "m19:reference-data--DrillingReasonType:EXP-HC:"
}
],
"VirtualProperties.DefaultName": "9254",
"VirtualProperties.DefaultLocation.CoordinateQualityCheckPerformedBy": null,
"ResourceSecurityClassification": null,
"SequenceNumber": 3,
"DataSourceOrganisationID": null,
"SpatialLocation.SpatialParameterTypeID": null,
"ExistenceKind": null,
"FacilityTypeID": "m19:reference-data--FacilityType:Wellbore:",
"FacilityEvents": [
{
"EffectiveDateTime": "1995-12-09T00:00:00+0000",
"FacilityEventTypeID": "m19:reference-data--FacilityEventType:Sidetrack:"
},
{
"EffectiveDateTime": "1996-02-17T00:00:00+0000",
"FacilityEventTypeID": "m19:reference-data--FacilityEventType:TDReached:"
}
],
"VirtualProperties.DefaultLocation.IsDecimated": false,
"ProjectedBottomHoleLocation.QuantitativeAccuracyBandID": null,
"CurrentOperatorID": "m19:master-data--Organisation:Total%20E%26P%20Nederland%20B.V.:",
"ProjectedBottomHoleLocation.SpatialGeometryTypeID": null,
"SpatialLocation.SpatialGeometryTypeID": null,
"OperatingEnvironmentID": "m19:reference-data--OperatingEnvironment:Off:",
"ProjectedBottomHoleLocation.QualitativeSpatialAccuracyTypeID": null,
"GeographicBottomHoleLocation.QuantitativeAccuracyBandID": null,
"PrimaryMaterialID": "m19:reference-data--MaterialType:Gas:",
"VirtualProperties.DefaultLocation.QualitativeSpatialAccuracyTypeID": null,
"InitialOperatorID": "m19:master-data--Organisation:Elf%20Petroland:",
"GeographicBottomHoleLocation.SpatialGeometryTypeID": null,
"WellID": "m19:master-data--Well:8988:",
"FacilityStates": [
{
"FacilityStateTypeID": "m19:reference-data--FacilityStateType:ShutIn:"
}
],
"GeographicBottomHoleLocation.SpatialParameterTypeID": null,
"GeographicBottomHoleLocation.CoordinateQualityCheckPerformedBy": null,
"ResourceLifecycleStatus": null,
"TargetFormation": null,
"SpatialLocation.Wgs84Coordinates": {
"geometries": [
{
"coordinates": [
3.30959248,
53.75027988
],
"type": "point"
}
],
"type": "geometrycollection"
},
"TechnicalAssuranceID": null,
"VirtualProperties.DefaultLocation.SpatialGeometryTypeID": null,
"Source": null,
"DefinitiveTrajectoryID": null,
"FacilityID": null,
"GeographicBottomHoleLocation.QualitativeSpatialAccuracyTypeID": null,
"VerticalMeasurements": [
{
"WellboreTVDTrajectoryID": null,
"VerticalCRSID": null,
"VerticalMeasurementSourceID": null,
"VerticalReferenceID": "Measured_From",
"VerticalMeasurementID": "TD-Original",
"VerticalMeasurementPathID": "m19:reference-data--VerticalMeasurementPath:MD:",
"VerticalMeasurement": 5193.0,
"VerticalMeasurementTypeID": "m19:reference-data--VerticalMeasurementType:TD:",
"VerticalMeasurementDescription": null,
"VerticalMeasurementUnitOfMeasureID": null
},
{
"WellboreTVDTrajectoryID": null,
"VerticalCRSID": null,
"VerticalMeasurementSourceID": null,
"VerticalReferenceID": "Measured_From",
"VerticalMeasurementID": "TVD",
"VerticalMeasurementPathID": "m19:reference-data--VerticalMeasurementPath:TVD:",
"VerticalMeasurement": 3887.61,
"VerticalMeasurementTypeID": "m19:reference-data--VerticalMeasurementType:TD:",
"VerticalMeasurementDescription": null,
"VerticalMeasurementUnitOfMeasureID": null
},
{
"WellboreTVDTrajectoryID": null,
"VerticalCRSID": "m19:reference-data--CoordinateReferenceSystem:MSL:",
"VerticalMeasurementSourceID": null,
"VerticalReferenceID": null,
"VerticalMeasurementID": "Measured_From",
"VerticalMeasurementPathID": "m19:reference-data--VerticalMeasurementPath:Elevation:",
"VerticalMeasurement": 40.0,
"VerticalMeasurementTypeID": "m19:reference-data--VerticalMeasurementType:RotaryTable:",
"VerticalMeasurementDescription": null,
"VerticalMeasurementUnitOfMeasureID": null
}
],
"VersionCreationReason": null,
"KickOffWellbore": null,
"SpatialLocation.CoordinateQualityCheckPerformedBy": null,
"NameAliases": [
{
"AliasName": "9254",
"AliasNameTypeID": "m19:reference-data--AliasNameType:UWI:",
"DefinitionOrganisationID": null
},
{
"AliasName": "K04-09B-S2",
"AliasNameTypeID": "m19:reference-data--AliasNameType:BoreholeCode:",
"DefinitionOrganisationID": null
},
{
"AliasName": "K04-09B-SIDETRACK2",
"AliasNameTypeID": "m19:reference-data--AliasNameType:Borehole:",
"DefinitionOrganisationID": null
}
],
"ProjectedBottomHoleLocation.CoordinateQualityCheckPerformedBy": null,
"GeoContexts": [
{
"BasinID": null,
"FieldID": null,
"PlayID": null,
"GeoPoliticalEntityID": "m19:master-data--GeoPoliticalEntity:K04_BlockID:",
"GeoTypeID": "m19:reference-data--GeoPoliticalEntityType:BlockID:",
"ProspectID": null
},
{
"BasinID": null,
"FieldID": "m19:master-data--Field:K04-A:",
"PlayID": null,
"GeoPoliticalEntityID": null,
"GeoTypeID": "Field",
"ProspectID": null
}
],
"ProjectedBottomHoleLocation.SpatialParameterTypeID": null,
"SpatialLocation.QualitativeSpatialAccuracyTypeID": null,
"VirtualProperties.DefaultLocation.SpatialParameterTypeID": null,
"ResourceHomeRegionID": null,
"VirtualProperties.DefaultLocation.Wgs84Coordinates": {
"geometries": [
{
"coordinates": [
3.30959248,
53.75027988
],
"type": "point"
}
],
"type": "geometrycollection"
},
"SpatialLocation.QuantitativeAccuracyBandID": null,
"TrajectoryTypeID": "m19:reference-data--WellboreTrajectoryType:Deviated:"
},
"kind": "osdu:wks:master-data--Wellbore:1.0.0",
"source": "wks",
"acl": {
"viewers": [
"data.default.viewers@m19.group"
],
"owners": [
"data.default.owners@m19.group"
]
},
"type": "master-data--Wellbore",
"version": 1689778069245048,
"tags": {
"normalizedKind": "osdu:wks:master-data--Wellbore:1"
},
"createTime": "2023-07-19T14:47:50.784Z",
"authority": "osdu",
"namespace": "osdu:wks",
"legal": {
"legaltags": [
"m19-dataimport-legaltag"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "datafier@osdu-service-prod.iam.gserviceaccount.com",
"id": "m19:master-data--Wellbore:9254"
}
],
"aggregations": null,
"phraseSuggestions": [],
"totalCount": 6668
}
```M22 - Release 0.25Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/325GCP- Spotfire Reporting Dashboard - Cannot retrieve well data properly2022-08-23T11:28:05ZThiago MoreiraGCP- Spotfire Reporting Dashboard - Cannot retrieve well data properlyThe connector works and I can see the wells in the map, but when I try to get well data to populate the graphs, WellLog_ID columns has no data in many wells. Even one well that has the WellLog_ID information is not showing any data in th...The connector works and I can see the wells in the map, but when I try to get well data to populate the graphs, WellLog_ID columns has no data in many wells. Even one well that has the WellLog_ID information is not showing any data in the graphs.
I tried to run the Volume Dashboard and it did not get any data.https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/659GCZ Azure M22 - Feature layers do not display on map service2024-01-16T09:51:36ZStefania Gerbaudo LarongaGCZ Azure M22 - Feature layers do not display on map serviceHello,
Seismic 3D Live traces do not display on the ArcGIS map service (url: https://osdu-gcz.msft-osdu-test.org/ignite-provider/rest/services/test/FeatureServer/7).
Regards,
Stefania Laronga
For info: @debasiscHello,
Seismic 3D Live traces do not display on the ArcGIS map service (url: https://osdu-gcz.msft-osdu-test.org/ignite-provider/rest/services/test/FeatureServer/7).
Regards,
Stefania Laronga
For info: @debasiscSrinivasan Narayananshivani karipesaketh somarajuSrinivasan Narayananhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/658GCZ M22 Azure returns empty features2024-01-16T09:57:14ZStefania Gerbaudo LarongaGCZ M22 Azure returns empty featuresHello,
Well kind, wellbore kind, 3D polygons, seismic 2D lines do not return any feature [].
Only 3D live traces returns results from ingestion.
Regards,
Stefania
For info @debasiscHello,
Well kind, wellbore kind, 3D polygons, seismic 2D lines do not return any feature [].
Only 3D live traces returns results from ingestion.
Regards,
Stefania
For info @debasiscSrinivasan Narayananshivani karipesaketh somarajuSrinivasan Narayananhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/291IBM M11 - Well Delivery Invalid responses2022-08-23T11:25:07ZMichaelIBM M11 - Well Delivery Invalid responsesI am getting responses where the valid=false for some of the requests in the collection: https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M11/IBM-M11/IBM_ODI_R3_WellDelivery.postman_collection.json
The attached ...I am getting responses where the valid=false for some of the requests in the collection: https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M11/IBM-M11/IBM_ODI_R3_WellDelivery.postman_collection.json
The attached document contains details on the failed requests:
[IBM_M11_Well_Delivery_Failed_Requests.docx](/uploads/26d5a619b72b434cdcc14894fab6eb0c/IBM_M11_Well_Delivery_Failed_Requests.docx)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/357IBM M13 - Manifest Ingestion - Performance Load testing 50k records - 504 res...2022-09-09T15:15:41ZChad LeongIBM M13 - Manifest Ingestion - Performance Load testing 50k records - 504 responseWhile submitting 50k records for performance load testing in manifest ingestion:
`https://cpd-osdu.odi-og-osdu-ba8e38d4c011d627379af1a4280c4e35-0000.us-south.containers.appdomain.cloud//osdu-workflow/api/workflow/v1/workflow/Osdu_ingest...While submitting 50k records for performance load testing in manifest ingestion:
`https://cpd-osdu.odi-og-osdu-ba8e38d4c011d627379af1a4280c4e35-0000.us-south.containers.appdomain.cloud//osdu-workflow/api/workflow/v1/workflow/Osdu_ingest/workflowRun`
### Got the following response:
```
<Response [504]>
b'20220527_1916 x86_64\n'
```
## 50k Records:
[Records_IBM_50000.zipx](/uploads/1838e5d6b46af934af4fcd35c2dc474c/Records_IBM_50000.zipx)
## Airflow response:
N/AShrikant GargShrikant Garghttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/558IBM M19 Policy engine incorrect settings2023-08-29T10:55:12ZDadong ZhouIBM M19 Policy engine incorrect settingsKamlesh reported the errors when calling the Policy Evaluate api:
```
{
"result": {
"records": [
{
"errors": [
{
"code": 404,
"id...Kamlesh reported the errors when calling the Policy Evaluate api:
```
{
"result": {
"records": [
{
"errors": [
{
"code": 404,
"id": "opendes:master-data--Well:test1111111111",
"message": "Entitlements response 404 Error 404 - Not Found ",
"reason": "Unauthorized"
},
{
"code": 404,
"id": "opendes:master-data--Well:test1111111111",
"message": "Legal response 404 Error 404 - Not Found ",
"reason": "Error from compliance service"
},
{
"code": "403",
"id": "opendes:master-data--Well:test1111111111",
"message": "The user is not authorized to perform this action",
"reason": "Access denied"
}
],
"id": "opendes:master-data--Well:test1111111111"
}
]
}
}
```
I confirmed the error in IBM M19 environment.
From the error messages, it seems the two environmental variables ENTITLEMENTS_BASE_URL and LEGAL_BASE_URL in Policy engine are not set correctly. Please check and correct.
Thanks.
cc @todaiksAnuj Guptavikas ranaAnuj Guptahttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/682IBM M22 - Core Services - CRS Conversion V32024-02-07T07:01:55ZJuliana Fernandesjuliana.fernandes@iesbrazil.com.brIBM M22 - Core Services - CRS Conversion V3Hello,
I'm trying to run 2 requests for CRS Conversion V3 without Success.
here is what I have:
- Collection: [CRS Conversion Service V3 CI-CD v1.0.postman_collection.json](https://community.opengroup.org/osdu/platform/pre-shipping/-...Hello,
I'm trying to run 2 requests for CRS Conversion V3 without Success.
here is what I have:
- Collection: [CRS Conversion Service V3 CI-CD v1.0.postman_collection.json](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/QA_Artifacts_M22/envFilesAndCollections/CRS%20Conversion%20Service%20V3%20CI-CD%20v1.0.postman_collection.json?ref_type=heads)
- Environment: [OSDU_R3_IBM_ODI_M22_ENV.postman_environment.json](https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/IBM-M22/Environment/OSDU_R3_IBM_ODI_M22_ENV.postman_environment.json?ref_type=heads)
## 2. Convert a list of points
POST https://{{CRS_CONVERSION_HOST_V3}}/convert
**BODY CURL VERSION**
```
curl --location 'https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-crs-conversion/api/crs/converter/v3/convert' \
--header 'data-partition-id: opendes' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer <Token>' \
--header 'Cookie: 5e7c4a992d2558462d1c9fcde6f1cee7=d7c00795bd9f4e77085f06413828a62c' \
--data '{
"fromCRS": "opendes:reference-data--CoordinateReferenceSystem:BoundProjected:EPSG::28992_EPSG::1672:",
"toCRS": "opendes:reference-data--CoordinateReferenceSystem:Projected:EPSG::32632:",
"points": [
{
"x": 400000,
"y": 190000,
"z": 0
},
{
"x": 399787.00835051807,
"y": 6500567.67783067,
"z": -173.79810638019308
}
]
}'
```
**BODY**
```json
{
"fromCRS": "{{data-partition-id}}:reference-data--CoordinateReferenceSystem:BoundProjected:EPSG::28992_EPSG::1672:",
"toCRS": "{{data-partition-id}}:reference-data--CoordinateReferenceSystem:Projected:EPSG::32632:",
"points": [
{
"x": 400000,
"y": 190000,
"z": 0
},
{
"x": 399787.00835051807,
"y": 6500567.67783067,
"z": -173.79810638019308
}
]
}
```
**RESPONSE**
```json
{
"code": 400,
"message": "record not found: opendes:reference-data--CoordinateReferenceSystem:BoundProjected:EPSG::28992_EPSG::1672/"
}
```
## 3. Convert Bin Grid
POST https://{{CRS_CONVERSION_HOST_V3}}/convertBinGrid
**BODY CURL VERSION**
```
curl --location 'https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-crs-conversion/api/crs/converter/v3/convertBinGrid' \
--header 'data-partition-id: opendes' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer <Token>' \
--header 'Cookie: 5e7c4a992d2558462d1c9fcde6f1cee7=d7c00795bd9f4e77085f06413828a62c' \
--data '{
"toCRS": "opendes:reference-data--CoordinateReferenceSystem:BoundProjected:EPSG::32064_EPSG::15851:",
"inBinGrid": {
"CoveragePercent": 1.0,
"ABCDBinGridSpatialLocation": {
"AsIngestedCoordinates": {
"type": "AnyCrsFeatureCollection",
"CoordinateReferenceSystemID": "opendes:reference-data--CoordinateReferenceSystem:Projected:EPSG::32615:",
"features": [
{
"type": "AnyCrsFeature",
"properties": {
"Kind": "osdu:osdu:AbstractGeoJson.PropertiesBinGridCorners:1.0.0",
"PointProperties": [
{
"Inline": 1,
"Crossline": 1000
}
]
},
"geometry": {
"type": "AnyCrsPoint",
"coordinates": [
500000.00,
3000000.00
]
}
},
{
"type": "AnyCrsFeature",
"properties": {
"Kind": "osdu:osdu:AbstractGeoJson.PropertiesBinGridCorners:1.0.0",
"PointProperties": [
{
"Inline": 1,
"Crossline": 2000
}
]
},
"geometry": {
"type": "AnyCrsPoint",
"coordinates": [
500000.00,
3100000.00
]
}
},
{
"type": "AnyCrsFeature",
"properties": {
"Kind": "osdu:osdu:AbstractGeoJson.PropertiesBinGridCorners:1.0.0",
"PointProperties": [
{
"Inline": 101,
"Crossline": 1000
}
]
},
"geometry": {
"type": "AnyCrsPoint",
"coordinates": [
600000.00,
3000000.00
]
}
},
{
"type": "AnyCrsFeature",
"properties": {
"Kind": "osdu:osdu:AbstractGeoJson.PropertiesBinGridCorners:1.0.0",
"PointProperties": [
{
"Inline": 101,
"Crossline": 2000
}
]
},
"geometry": {
"type": "AnyCrsPoint",
"coordinates": [
600000.00,
3100000.00
]
}
}
]
}
},
"P6ScaleFactorOfBinGrid": 1,
"P6BinNodeIncrementOnIaxis": 1,
"P6BinNodeIncrementOnJaxis": 1
}
}'
```
**BODY**
```json
{
"toCRS": "{{data-partition-id}}:reference-data--CoordinateReferenceSystem:BoundProjected:EPSG::32064_EPSG::15851:",
"inBinGrid": {
"CoveragePercent": 1.0,
"ABCDBinGridSpatialLocation": {
"AsIngestedCoordinates": {
"type": "AnyCrsFeatureCollection",
"CoordinateReferenceSystemID": "{{data-partition-id}}:reference-data--CoordinateReferenceSystem:Projected:EPSG::32615:",
"features": [
{
"type": "AnyCrsFeature",
"properties": {
"Kind": "{{authority}}:{{schemaSource}}:AbstractGeoJson.PropertiesBinGridCorners:1.0.0",
"PointProperties": [
{
"Inline": 1,
"Crossline": 1000
}
]
},
"geometry": {
"type": "AnyCrsPoint",
"coordinates": [
500000.00,
3000000.00
]
}
},
{
"type": "AnyCrsFeature",
"properties": {
"Kind": "{{authority}}:{{schemaSource}}:AbstractGeoJson.PropertiesBinGridCorners:1.0.0",
"PointProperties": [
{
"Inline": 1,
"Crossline": 2000
}
]
},
"geometry": {
"type": "AnyCrsPoint",
"coordinates": [
500000.00,
3100000.00
]
}
},
{
"type": "AnyCrsFeature",
"properties": {
"Kind": "{{authority}}:{{schemaSource}}:AbstractGeoJson.PropertiesBinGridCorners:1.0.0",
"PointProperties": [
{
"Inline": 101,
"Crossline": 1000
}
]
},
"geometry": {
"type": "AnyCrsPoint",
"coordinates": [
600000.00,
3000000.00
]
}
},
{
"type": "AnyCrsFeature",
"properties": {
"Kind": "{{authority}}:{{schemaSource}}:AbstractGeoJson.PropertiesBinGridCorners:1.0.0",
"PointProperties": [
{
"Inline": 101,
"Crossline": 2000
}
]
},
"geometry": {
"type": "AnyCrsPoint",
"coordinates": [
600000.00,
3100000.00
]
}
}
]
}
},
"P6ScaleFactorOfBinGrid": 1,
"P6BinNodeIncrementOnIaxis": 1,
"P6BinNodeIncrementOnJaxis": 1
}
}
```
**RESPONSE**
```json
{
"code": 400,
"message": "record not found: opendes:reference-data--CoordinateReferenceSystem:BoundProjected:EPSG::32064_EPSG::15851/"
}
```M22 - Release 0.25Ashwani PandeyAshwani Pandeyhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/677IBM M22 - Core Services - Register and Notification2024-01-24T19:40:47ZJuliana Fernandesjuliana.fernandes@iesbrazil.com.brIBM M22 - Core Services - Register and NotificationHello,
I'm trying to do the Register and notification but I was not able to find any collection for Register.
This feature is enabled for IBM?
Also IBM is able to provide some documentation/clarification for Notification?
Regards,
Jul...Hello,
I'm trying to do the Register and notification but I was not able to find any collection for Register.
This feature is enabled for IBM?
Also IBM is able to provide some documentation/clarification for Notification?
Regards,
JulianaM22 - Release 0.25https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/670IBM M22 - CRS Conversion - Coordinate Reference System not Loaded2024-01-27T06:33:10ZJuliana Fernandesjuliana.fernandes@iesbrazil.com.brIBM M22 - CRS Conversion - Coordinate Reference System not LoadedHello,
I was trying to complete the task FoR CRS Conversion for Manifest ingestion but I was not able to complete.
In one of the first steps were I need to check if the Coordinate Reference System is loaded into OSDU I got an empty ret...Hello,
I was trying to complete the task FoR CRS Conversion for Manifest ingestion but I was not able to complete.
In one of the first steps were I need to check if the Coordinate Reference System is loaded into OSDU I got an empty return.
- Collection used:https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/QA_Artifacts_M22/envFilesAndCollections/Manifest_Based_Ingestion_Osdu_ingest_CI-CD_v2.0.postman_collection.json?ref_type=heads
- Environment used: https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M22/IBM-M22/Environment/OSDU_R3_IBM_ODI_M22_ENV.postman_environment.json?ref_type=heads
### 02 - Search Coordinate Reference System (CRS) Record
**Body Curl Version**
```
curl --location 'https://cpd-osdu.apps.osdu-preship.ibmodi.com/osdu-search/api/search/v2/query' \
--header 'Content-Type: application/json' \
--header 'data-partition-id: opendes' \
--header 'Authorization: Bearer <Token>' \
--header 'Cookie: 5e7c4a992d2558462d1c9fcde6f1cee7=c87320314b80622dec9caac6b93398c9' \
--data '{
"kind": "osdu:wks:reference-data--CoordinateReferenceSystem:1.0.0",
"query" : "id:\"opendes:reference-data--CoordinateReferenceSystem:23031024\"",
"limit": 1000
}'
```
**Response**
```
{
"results": [],
"aggregations": [],
"phraseSuggestions": [],
"totalCount": 0
}
```
Is it the Coordinate Reference System not loaded or some wrong code?
Regards,
Juliana FernandesM22 - Release 0.25https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/669IBM M22 Preship - WITSML Parser V1 fails2024-01-19T05:37:54ZDebasis ChatterjeeIBM M22 Preship - WITSML Parser V1 fails@vikasrana , @nikhil_patil , @ishakumari
As reported by email, I tried the first step (Create "Well" record) and it is failing for me.
"runId": "7a554510-5b1d-4495-ad7a-7859113e8d45",
See steps.[M22-IBM-WITSML-V1-Parser-Well-steps-Deb...@vikasrana , @nikhil_patil , @ishakumari
As reported by email, I tried the first step (Create "Well" record) and it is failing for me.
"runId": "7a554510-5b1d-4495-ad7a-7859113e8d45",
See steps.[M22-IBM-WITSML-V1-Parser-Well-steps-Debasis.txt](/uploads/6472c311bc1777d346caaca939757b3a/M22-IBM-WITSML-V1-Parser-Well-steps-Debasis.txt)
Enclosed is the log file from Airflow.
[2024-01-16-WITSML_V1-Parser-Airflow-log-failure.txt](/uploads/ef69ab1085ad65e611be0be9455e5e83/2024-01-16-WITSML_V1-Parser-Airflow-log-failure.txt)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/679IBM M22 - wbdutil SSL certificate2024-01-26T07:47:26ZChad LeongIBM M22 - wbdutil SSL certificatewbdutil relies on httpx client lbi which requires SSL to be turned on.
```
(env) D:\osdu\wbdutil>wbdutil ingest wellbore -p "D:\osdu\wbdutil\1068_asn01_1982_comp.las" -c %CONFIGPATH% -t %OSDUTOKEN% --verbose
Beginning ingest of LAS file...wbdutil relies on httpx client lbi which requires SSL to be turned on.
```
(env) D:\osdu\wbdutil>wbdutil ingest wellbore -p "D:\osdu\wbdutil\1068_asn01_1982_comp.las" -c %CONFIGPATH% -t %OSDUTOKEN% --verbose
Beginning ingest of LAS file: D:\osdu\wbdutil\1068_asn01_1982_comp.las
Ingest failed: 1068_asn01_1982_comp.las (see summary for details)
SUMMARY - files not ingested:
1068_asn01_1982_comp.las ([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129))
```vikas ranavikas ranahttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/575IBM R3 M19 environment having issue with Search and or indexer service.2023-08-11T09:53:55ZKamlesh TodaiIBM R3 M19 environment having issue with Search and or indexer service.After ingesting the data using manifest ingestion (Osdu_ingest), one can retrieve the data using storage API, but cannot retrieve it using the Search API.
Even after waiting for more then 15 minutes.
Using Storage API
```
curl --locat...After ingesting the data using manifest ingestion (Osdu_ingest), one can retrieve the data using storage API, but cannot retrieve it using the Search API.
Even after waiting for more then 15 minutes.
Using Storage API
```
curl --location 'https://cpd-osdu.apps.ibmosdu-preship.lndu.p1.openshiftapps.com/osdu-storage/api/storage/v2/records/opendes:master-data--GeoPoliticalEntity:Limburg' \
--header 'data-partition-id: opendes' \
--header 'Authorization: Bearer eyJhbGciOi...Truncated...3k3p_JLqg'
Response 200 OK
{
"data": {
"GeoPoliticalEntityName": "Limburg",
"Source": "TNO",
"VirtualProperties.DefaultName": "Limburg"
},
"meta": null,
"modifyUser": "osdu-bvt@osdu.opengroup.org",
"modifyTime": "2023-08-08T13:40:37.481Z",
"id": "opendes:master-data--GeoPoliticalEntity:Limburg",
"version": 1691502037052901,
"kind": "osdu:wks:master-data--GeoPoliticalEntity:1.0.0",
"acl": {
"viewers": [
"data.default.viewers@opendes.ibm.com"
],
"owners": [
"data.default.owners@opendes.ibm.com"
]
},
"legal": {
"legaltags": [
"opendes-R3FullManifest-Legal-Tag-Test7791699"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "osdu-bvt@osdu.opengroup.org",
"createTime": "2023-08-04T20:51:59.686Z"
}
```
But when one performs the search using Search API, the results returned are zero for the count.
```
curl --location 'https://cpd-osdu.apps.ibmosdu-preship.lndu.p1.openshiftapps.com/osdu-search/api/search/v2/query' \
--header 'Content-Type: application/json' \
--header 'data-partition-id: opendes' \
--header 'Authorization: Bearer eyJhbGciOi...Truncated...3p_JLqg' \
--data '{
"kind": "osdu:wks:master-data--GeoPoliticalEntity:1.0.0",
"query" : "*",
"limit" : 500,
"returnedFields" : ["id"]
}'
Response 200 OK
{
"results": [],
"aggregations": [],
"totalCount": 0
}
```
Does not matter what search is being performed, the total count returned is zero.
@chad @debasisc @davidglassAnuj Guptavikas ranaAnuj Guptahttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/189[IBM R3M9](Segy to Ovds conversion) Air flow DAG failure after run Segy to Vd...2022-08-23T12:46:22Zkenneth liew[IBM R3M9](Segy to Ovds conversion) Air flow DAG failure after run Segy to Vds” conversationI had run a [R3M9 IBM ] “Segy to Vds” conversation task, but I found that the DAG(openvds_import) is failed to process my workflow.
```
Request :
Post Method: {{WORKFLOW_HOST}}/workflow/{{workflow_name_openvds}}/workflowRun
Body:
{
...I had run a [R3M9 IBM ] “Segy to Vds” conversation task, but I found that the DAG(openvds_import) is failed to process my workflow.
```
Request :
Post Method: {{WORKFLOW_HOST}}/workflow/{{workflow_name_openvds}}/workflowRun
Body:
{
"executionContext": {
"url_connection": "sdauthorityurl=https://{{osdu-cpd}}/osdu-seismic/api/v3;sdapikey=xx;sdtoken={{access_token}};EndpointOverride={{minio-url}}",
"input_connection": "sdauthorityurl=https://{{osdu-cpd}}/osdu-seismic/api/v3;sdapikey=xx;sdtoken={{access_token}};EndpointOverride={{minio-url}}",
"segy_file": "sd://opendes/kenneth-test/ST10010ZC11_PZ_PSDM_KIRCH_NEAR_D.MIG_FIN.POST_STACK.3D.JS-017536_Version4.segy",
"url": "sd://opendes/kenneth-test/"
}
}
```
```
Response:
{
"workflowId": "b133573249894456a32813caac1bf692",
"runId": "e831ba7b-6e80-4c0a-ba83-05cfa55cf309",
"startTimeStamp": 1639650166798,
"status": "submitted",
"submittedBy": "preshipteama@osdu.opengroup.org"
}
```
AirFlow DAG log-openvds_import
```
2021-12-16 10:24:51,867] {taskinstance.py:1501} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 366, in execute
final_state, remote_pod, result = self.create_new_pod_for_operator(labels, launcher)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 519, in create_new_pod_for_operator
launcher.start_pod(self.pod, startup_timeout=self.startup_timeout_seconds)
File "/home/airflow/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 329, in wrapped_f
return self.call(f, *args, **kw)
File "/home/airflow/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 409, in call
do = self.iter(retry_state=retry_state)
File "/home/airflow/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 356, in iter
return fut.result()
File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 437, in result
return self.__get_result()
File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/home/airflow/.local/lib/python3.8/site-packages/tenacity/__init__.py", line 412, in call
result = fn(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/utils/pod_launcher.py", line 131, in start_pod
raise AirflowException("Pod took too long to start")
airflow.exceptions.AirflowException: Pod took too long to start
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1157, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1331, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1361, in _execute_task
result = task_copy.execute(context=context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 373, in execute
raise AirflowException(f'Pod Launching failed: {ex}')
airflow.exceptions.AirflowException: Pod Launching failed: Pod took too long to start
[2021-12-16 10:24:51,873] {taskinstance.py:1544} INFO - Marking task as FAILED. dag_id=openvds_import, task_id=OPENVDS, execution_date=20211216T102247, start_date=20211216T102249, end_date=20211216T102451
[2021-12-16 10:24:51,959] {local_task_job.py:151} INFO - Task exited with return code 1
```
[AirFlow_IBM_r3m9_Error_.txt](/uploads/af9d88da349f1b210a9e529f5c623252/AirFlow_IBM_r3m9_Error_.txt)
[VDS_Convert_Request_IBM_R3m9_.txt](/uploads/466a043ce3070bbedb4a7f5071f13796/VDS_Convert_Request_IBM_R3m9_.txt)
cc @anujgupta ,@dsouzawalter ,@shamazum