Pre Shipping issueshttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues2021-11-18T21:15:13Zhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/2WITSML Parser does not perform integrity check - AWS, R3M72021-11-18T21:15:13ZDebasis ChatterjeeWITSML Parser does not perform integrity check - AWS, R3M7Provide-manifest-integrity-task component is missing.
I am showing "graph view" from AWS and GCP for comparison.
AWS
![WITSML-Integrity-check-AWS-missed](/uploads/13a30d68dabcbba38b73a06c4685de21/WITSML-Integrity-check-AWS-missed.PNG)
G...Provide-manifest-integrity-task component is missing.
I am showing "graph view" from AWS and GCP for comparison.
AWS
![WITSML-Integrity-check-AWS-missed](/uploads/13a30d68dabcbba38b73a06c4685de21/WITSML-Integrity-check-AWS-missed.PNG)
GCP
![WITSML-Integrity-check-GCP](/uploads/f6d412361861627dbc12131a2e8ae7e3/WITSML-Integrity-check-GCP.PNG)
Please see reference from IBM, as provided by @chad
![WITSML-Integrity-check-IBM](/uploads/0493e2c753994e3f32bbbd4073cae80b/WITSML-Integrity-check-IBM.png)
cc - @Wibben and @spencerhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/3Storage Service. Version endpoint. AWS. R3M72021-09-30T13:27:09ZDebasis ChatterjeeStorage Service. Version endpoint. AWS. R3M7GET {{osduonaws_base_url}}/api/storage/v2/info
Check enclosed response. Shows 0.11 instead of 0.10.
```
{
"groupId": "org.opengroup.osdu",
"artifactId": "storage-aws",
"version": "0.11.0-SNAPSHOT",
"buildTime": "2021-07...GET {{osduonaws_base_url}}/api/storage/v2/info
Check enclosed response. Shows 0.11 instead of 0.10.
```
{
"groupId": "org.opengroup.osdu",
"artifactId": "storage-aws",
"version": "0.11.0-SNAPSHOT",
"buildTime": "2021-07-28T18:27:50.468Z",
"branch": "refs/heads/release/r3-m7-0",
"commitId": "b98a7d5d8e1bba56921ddf381836f6d61a751e1c",
"commitMessage": "Merge",
"connectedOuterServices": []
}
```
When I try this in GCP, I see "0.10"
```
{
"groupId": "org.opengroup.osdu",
"artifactId": "storage-gcp",
"version": "0.10.1",
"buildTime": "2021-07-29T06:35:33.192Z",
"branch": "v0.10.1",
"commitId": "39a6ca57e1e62b8a5b6573943dc8f305f6cb106d",
"commitMessage": "Creating Release Commit",
"connectedOuterServices": []
}
```https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/4Manifest-based Ingestion, XCom summary. Does not show details of failed IDs -...2021-10-14T01:04:11ZDebasis ChatterjeeManifest-based Ingestion, XCom summary. Does not show details of failed IDs - AWS R3M7In my test case, I am trying to create 2 Master data records.
One of them refers to existing Reference value.
The second record refers to non existing Reference value.
[AWS-Manifest-check-XCom-summary.txt](/uploads/8de8debb0a6635802523...In my test case, I am trying to create 2 Master data records.
One of them refers to existing Reference value.
The second record refers to non existing Reference value.
[AWS-Manifest-check-XCom-summary.txt](/uploads/8de8debb0a66358025234b68370119b8/AWS-Manifest-check-XCom-summary.txt)
As expected, one record is created and the other record fails integrity test.
The issue is that XCom summary from the very last step (update status finished task) does not show Xcom summary of two rows - processed record, failed ID.
There seems to be difference in behaviour from GCP.
![GCP-Update_Status_Finished_task-XCom-summary](/uploads/3d119b25e921b0d8a049f046db769aa9/GCP-Update_Status_Finished_task-XCom-summary.PNG)
Please let me know if you have any question about the test case, results.
Thank you
cc: @Wibben , @spencerhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/5Question about integrity check - Manifest-based Ingestion2022-01-15T01:00:11ZDebasis ChatterjeeQuestion about integrity check - Manifest-based IngestionCase in point is reference to CoordinateReferenceSystem.
I verified that this reference value exists for the Reference entity CoordinateReferenceSystem .
"id": "osdu:reference-data--CoordinateReferenceSystem:ProjectedCRS::EPSG::23031"
...Case in point is reference to CoordinateReferenceSystem.
I verified that this reference value exists for the Reference entity CoordinateReferenceSystem .
"id": "osdu:reference-data--CoordinateReferenceSystem:ProjectedCRS::EPSG::23031"
But the problem occurs when I refer to this value while loading Seismic Horizon work-product component.
"CoordinateReferenceSystemID": "osdu:reference-data--CoordinateReferenceSystem:ProjectedCRS::EPSG::23031:",
Airflow log shows - Missing ids '{'osdu:reference-data--CoordinateReferenceSystem:ProjectedCRS::EPSG::23031'}'
Wondering if there is a known problem with handling of special characters (":") within the value.
Elsewhere I have seen evidence of work-around such as in "EncodingFormatTypeID": "osdu:reference-data--EncodingFormatType:text%2Fcsv:",
Do you think it will be necessary to load up Reference value of CoordinateSystemReference suitably? Such as by using "%3A" instead of colon?
==========
Please check this site for suggested values from Data Definition team.
https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/ReferenceValues/Manifests/reference-data
https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/ReferenceValues/Manifests/reference-data/LOCAL/CoordinateReferenceSystem.1.0.0.json
These show use of ":" inside the value.M9 - Release 0.12Debasis ChatterjeeDebasis Chatterjeehttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/185GCP R3M9 - Versioning Endpoints Issues2022-08-23T12:47:19ZMaritsa EconomoGCP R3M9 - Versioning Endpoints IssuesI’ve tested all the GCP Versioning Endpoints collection and had issues with the following: -
GET Unit Version:
{
"code": 401,
"reason": "Unauthorized",
"message": "The user is not authorized to perform this action"
}
GET C...I’ve tested all the GCP Versioning Endpoints collection and had issues with the following: -
GET Unit Version:
{
"code": 401,
"reason": "Unauthorized",
"message": "The user is not authorized to perform this action"
}
GET CRS Catalog Version:
{
"code": 401,
"reason": "Unauthorized",
"message": "The user is not authorized to perform this action"
}
GET CRS Conversion Version: -
{
"code": 401,
"reason": "Entitlement Error",
"message": "Error making request to Entitlements service. Check the inner HttpResponse for more info."
}
All other tests worked fine – can you please check the above and advise.Dzmitry Malkevich (EPAM)Maritsa EconomoDzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/182GCP: 50K loading test failed2022-08-23T12:47:09ZWira YarmanGCP: 50K loading test failedDAG failed for 50K records - run id: d8130b55-66b2-43f0-bdbc-23471724278eDAG failed for 50K records - run id: d8130b55-66b2-43f0-bdbc-23471724278eM9 - Release 0.12Aliaksandr Ramanovich (EPAM)Siarhei Khaletski (EPAM)Dzmitry Malkevich (EPAM)Aliaksandr Ramanovich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/174GCP R3M9 - Unable to retrieve file for surrogate key using Dataset service2022-08-23T12:47:20ZSteven EvansGCP R3M9 - Unable to retrieve file for surrogate key using Dataset serviceUnable to retrieve file for surrogate key using Dataset service replicating @debasisc successful run last week - worked example attached
[GCP_R3M9_worked_example_manifest_surrogate_key_and_file_retrieval_error.txt](/uploads/007453a0f706f...Unable to retrieve file for surrogate key using Dataset service replicating @debasisc successful run last week - worked example attached
[GCP_R3M9_worked_example_manifest_surrogate_key_and_file_retrieval_error.txt](/uploads/007453a0f706fc049ed60220305ab3ae/GCP_R3M9_worked_example_manifest_surrogate_key_and_file_retrieval_error.txt)Aliaksandr Ramanovich (EPAM)Dzmitry Malkevich (EPAM)Aliaksandr Ramanovich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/171GCP R3M9 - Manifest data WP WPC - fails to create a record id for work product2022-08-23T12:47:22ZSteven EvansGCP R3M9 - Manifest data WP WPC - fails to create a record id for work productFails to process the WP record, but successfully processes WPC and dataset record ids. Complains about a bad URL for kind wks:work-product
skipped_ids [{'id': 'odesprod:work-product--WorkProduct:0d409294-effd-4103-bb89-b7fb7471c444:SJE-...Fails to process the WP record, but successfully processes WPC and dataset record ids. Complains about a bad URL for kind wks:work-product
skipped_ids [{'id': 'odesprod:work-product--WorkProduct:0d409294-effd-4103-bb89-b7fb7471c444:SJE-WP1}', 'kind': 'odesprod:wks:work-product--WorkProduct:1.0.0', 'reason': '400 Client Error: Bad Request for url: https://preship.osdu.club/api/storage/v2/records'}]
For GCP the data-partition-id is set to odesprod and this appears to work fine for “wks” for both reference and master data and for the WPC and dataset records. I did also change the value to osdu for work product kind just to test the scenario and the same error occurred. Appears to be an issue with the url : https://preship.osdu.club/api/storage/v2/records.
worked example attached:
[WPWPC_issue_skipped_ids_WP.txt](/uploads/129e904532d867d763dad77d9f81e869/WPWPC_issue_skipped_ids_WP.txt)Aliaksandr Ramanovich (EPAM)Steven EvansDzmitry Malkevich (EPAM)Aliaksandr Ramanovich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/165GCP R3M9 - SegY to oVDS - Step 3.1 (create catalog records) - request clarifi...2022-08-23T12:47:11ZDebasis ChatterjeeGCP R3M9 - SegY to oVDS - Step 3.1 (create catalog records) - request clarification before we run thisPlease check step 6. I am using OpenVDS folder requests from GCP Smoke test collection
PUT https://{{DATASET_HOST}}/registerDataset
Body
```
{
"datasetRegistries": [
{
"id": "{{data-partition-id}}:dataset--File....Please check step 6. I am using OpenVDS folder requests from GCP Smoke test collection
PUT https://{{DATASET_HOST}}/registerDataset
Body
```
{
"datasetRegistries": [
{
"id": "{{data-partition-id}}:dataset--File.Generic:{{$guid}}",
"kind": "{{data-partition-id}}:{{schemaSource}}:dataset--File.Generic:1.0.0",
"acl": {
"viewers": [
"{{New_ViewerDataGroup}}@{{data-partition-id}}{{domain}}"
],
"owners": [
"{{New_OwnerDataGroup}}@{{data-partition-id}}{{domain}}"
]
},
"legal": {
"legaltags": [
"{{tagName}}"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"data": {
"DatasetProperties": {
"FileSourceInfo": {
"FileSource": "{{UnsignedUrl}}{{SignedUploadFileName}}",
"PreLoadFilePath": ""
}
},
"ResourceSecurityClassification": "{{data-partition-id}}:reference-data--ResourceSecurityClassification:RESTRICTED:"
},
"meta": [],
"tags": {}
}
]
}
```
Response
```
{
"code": 400,
"reason": "Bad Request",
"message": "Invalid dataset metadata"
}
```
cc - @aliaksandr_ramanovich , @kennethliew , @ramansingh , @gmarblestone and @sehuboy for informationDenis Karpenok (EPAM)Dzmitry Malkevich (EPAM)Denis Karpenok (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/164GCP R3M9 - SegY-to-oVDS - issue with Dataset service from Smoke test collecti...2022-08-23T12:47:23ZDebasis ChatterjeeGCP R3M9 - SegY-to-oVDS - issue with Dataset service from Smoke test collection (minor fix needed)Please check step-4. I get failure. Works after I add https://
**SegY-to-VDS**
Step-4
GET {{DATASET_HOST}}/getStorageInstructions?kindSubType=dataset--File.Generic - **this fails**
GET https://{{DATASET_HOST}}/getStorageInstructions?ki...Please check step-4. I get failure. Works after I add https://
**SegY-to-VDS**
Step-4
GET {{DATASET_HOST}}/getStorageInstructions?kindSubType=dataset--File.Generic - **this fails**
GET https://{{DATASET_HOST}}/getStorageInstructions?kindSubType=dataset--File.Generic - **this works**
cc - @kennethliewDzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/163GCP - R3M9 - SegY to zgy conversion fails2022-08-23T12:47:22ZDebasis ChatterjeeGCP - R3M9 - SegY to zgy conversion failsAs I have problem with sdutil, I used a file that is already loaded by @Yan_Sushchynski into SDSTORE.
sd://gcp-test/gcp-test/volve/ST10010ZC11_PZ_PSDM_KIRCH_FULL_T.MIG_FIN.POST_STACK.3D.JS-017536.segy
Using that I went ahead to create ...As I have problem with sdutil, I used a file that is already loaded by @Yan_Sushchynski into SDSTORE.
sd://gcp-test/gcp-test/volve/ST10010ZC11_PZ_PSDM_KIRCH_FULL_T.MIG_FIN.POST_STACK.3D.JS-017536.segy
Using that I went ahead to create 4 catalog records (wp, wpc1, wpc2, dataset).
And then tried the conversion.
I get failure during conversion.
Postman collection used -
[Segy--Zgy._Demo.postman_collection.json](/uploads/48a2290d54c153daad0ad90a96b74573/Segy--Zgy._Demo.postman_collection.json)
Step for creating catalog records
[Step2-Storage-PUT-create-4-records.txt](/uploads/9ae97bba598aac285adfb946b14f6ee0/Step2-Storage-PUT-create-4-records.txt)
Step for SegY to zgy conversion
[Step3-Trigger-workflow-convert-Zgy-issue.txt](/uploads/e3f0c0b623a16d2d51952844846d94e3/Step3-Trigger-workflow-convert-Zgy-issue.txt)
Airflow log showing errors
[Step3-Trigger-workflow-convert-Zgy-Airflow-log-issue.txt](/uploads/bc5a07ccafba756c8ae8ea62e4f35363/Step3-Trigger-workflow-convert-Zgy-Airflow-log-issue.txt)
cc - @aliaksandr_ramanovich , @dzmitry_malkevich for informationAliaksandr Ramanovich (EPAM)Yan Sushchynski (EPAM)Aliaksandr Ramanovich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/158GCP R3M9 - Seismic DDMS sdutil - cp or ls command fails "not authorized"2022-08-23T12:47:10ZDebasis ChatterjeeGCP R3M9 - Seismic DDMS sdutil - cp or ls command fails "not authorized"```
(sdutilenv) C:\seismic-store-sdutil-master>python sdutil stat sd://DCtest01Dec --idtoken=%ID_TOKEN%
Wrong Command: sd://DCtest01Dec is not a valid seismic store dataset path.
The seismic store dataset path should matc...```
(sdutilenv) C:\seismic-store-sdutil-master>python sdutil stat sd://DCtest01Dec --idtoken=%ID_TOKEN%
Wrong Command: sd://DCtest01Dec is not a valid seismic store dataset path.
The seismic store dataset path should match one of the standard forms:
Tenant: sd://<tenant_nane>.
Subproject: sd://<tenant_nane>/<subproject_name>.
Dataset: sd://<tenant_nane>/<subproject_name>/<path>/<dataset_name>.
For more information type "python sdutil stat" to open the command help menu.
```
Details are here
[Step1-sdutil-copy-file-R3M9.txt](/uploads/673181ccba20ff223bd1e7f3eeb16a13/Step1-sdutil-copy-file-R3M9.txt)
cc - @dzmitry_malkevich for informationAleksandr Spivakov (EPAM)Dzmitry Malkevich (EPAM)Aleksandr Spivakov (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/159GCP R3M9 - Smoke test, CSV Ingestion - comments2022-08-23T12:47:05ZDebasis ChatterjeeGCP R3M9 - Smoke test, CSV Ingestion - commentsPlease check. The steps are using deprecated feature from Storage service.
Recommended approach is to use Schema service to create custom schema.
cc - @todaiks , @sehuboy , @esmira.rafigayeva and @TaylorGraber for informationPlease check. The steps are using deprecated feature from Storage service.
Recommended approach is to use Schema service to create custom schema.
cc - @todaiks , @sehuboy , @esmira.rafigayeva and @TaylorGraber for informationDenis Karpenok (EPAM)Dzmitry Malkevich (EPAM)Denis Karpenok (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/200In GCP pre-ship env R3M9 segy_to_zgy conversion is failing2022-08-23T11:28:57ZKamlesh TodaiIn GCP pre-ship env R3M9 segy_to_zgy conversion is failing**In pre-ship environment ( preship-asm.osdu-gcp.go3-nrg.projects.epam.com) When I am trying to perform segy_to_zgy_converison, the DAG is failing. Upon looking at the airflow logs I see the following:**
[2022-01-24 17:08:04,117] {pod_l...**In pre-ship environment ( preship-asm.osdu-gcp.go3-nrg.projects.epam.com) When I am trying to perform segy_to_zgy_converison, the DAG is failing. Upon looking at the airflow logs I see the following:**
[2022-01-24 17:08:04,117] {pod_launcher.py:149} INFO - [0.000384] Data partition ID : [odesprod]
[2022-01-24 17:08:04,117] {pod_launcher.py:149} INFO - [0.000397] About to get record [odesprod:work-product--WorkProduct:713012-681176:] version 18446744073709551615
[2022-01-24 17:08:04,117] {pod_launcher.py:149} INFO - [0.000428] GET [https://preship.osdu.club/api/storage/v2/records/odesprod%3Awork-product--WorkProduct%3A713012-681176]
[2022-01-24 17:08:04,118] {pod_launcher.py:149} INFO - Error resolving address
[2022-01-24 17:08:05,124] {rest.py:228} DEBUG - response body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"segy-to-zgy.06931d094bdf4ad1aecee87278ecbdec","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/segy-to-zgy.06931d094bdf4ad1aecee87278ecbdec","uid":"5ebf1ba1-24df-41f2-865d-97b6a1e33d22","resourceVersion":"53964314","creationTimestamp":"2022-01-24T17:06:29Z","labels":{"airflow_version":"2.1.2-composer","dag_id":"Segy_to_zgy_conversion","execution_date":"2022-01-24T1706140000-c08f6646f","kubernetes_pod_operator":"True","task_id":"segy-to-zgy","try_number":"1"},"managedFields":[{"manager":"OpenAPI-Generator","operation":"Update","apiVersion":"v1","time":"2022-01-24T17:06:29Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:airflow_version":{},"f:dag_id":{},"f:execution_date":{},"f:kubernetes_pod_operator":{},"f:task_id":{},"f:try_number":{}}},"f:spec":{"f:affinity":{".":{},"f:nodeAffinity":{".":{},"f:requiredDuringSchedulingIgnoredDuringExecution":{".":{},"f:nodeSelectorTerms":{}}}},"f:containers":{"k:{\"name\":\"base\"}":{".":{},"f:args":{},"f:env":{".":{},"k:{\"name\":\"OSDU_DATAPARTITIONID\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"SD_READ_CACHE_MAX_PAGES\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"SD_READ_CACHE_PAGE_SIZE\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"SD_SVC_API_KEY\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"SD_SVC_TOKEN\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"SD_SVC_URL\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"SEGYTOZGY_GENERATE_INDEX\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"SEGYTOZGY_VERBOSITY\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"STORAGE_SVC_API_KEY\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"STORAGE_SVC_TOKEN\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"STORAGE_SVC_URL\"}":{".":{},"f:name":{},"f:value":{}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{".":{},"f:limits":{".":{},"f:cpu":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kube-scheduler","operation":"Update","apiVersion":"v1","time":"2022-01-24T17:06:30Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{".":{},"k:{\"type\":\"PodScheduled\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2022-01-24T17:08:04Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.0.13.3\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"default-token-dd5hc","secret":{"secretName":"default-token-dd5hc","defaultMode":420}}],"containers":[{"name":"base","image":"gcr.io/nice-etching-277309/segy-to-zgy-conversion/segy-to-zgy-conversion-gcp:v0.12.2","args":["--osdu","odesprod:dataset--FileCollection.SEGY:713012-681176:","odesprod:work-product--WorkProduct:713012-681176:"],"env":[{"name":"STORAGE_SVC_URL","value":"https://preship.osdu.club/api/storage/v2"},{"name":"SD_SVC_URL","value":"https://preship.osdu.club/api/seismic-store/v3"},{"name":"SD_SVC_TOKEN","value":"eyJhbGciOiJSUzI1NiIsImtpZCI6ImNhMDA2MjBjNWFhN2JlOGNkMDNhNmYzYzY4NDA2ZTQ1ZTkzYjNjYWIiLCJ0eXAiOiJKV1QifQ.eyJpc3MiOiJodHRwczovL2FjY291bnRzLmdvb2dsZS5jb20iLCJhenAiOiI2MDU0NTczNTcxNDMtNmg2dXF1bnE2N2Y1M205amVpYm4zOGd1cGQyN2JzZmIuYXBwcy5nb29nbGV1c2VyY29udGVudC5jb20iLCJhdWQiOiI2MDU0NTczNTcxNDMtNmg2dXF1bnE2N2Y1M205amVpYm4zOGd1cGQyN2JzZmIuYXBwcy5nb29nbGV1c2VyY29udGVudC5jb20iLCJzdWIiOiIxMTA3MDM5ODQzMzM5MDg0ODc0NDIiLCJoZCI6Im9zZHUtZ2NwLmdvMy1ucmcucHJvamVjdHMuZXBhbS5jb20iLCJlbWFpbCI6InByZXNoaXBwaW5nX3Rlc3RfdXNlckBvc2R1LWdjcC5nbzMtbnJnLnByb2plY3RzLmVwYW0uY29tIiwiZW1haWxfdmVyaWZpZWQiOnRydWUsImF0X2hhc2giOiJQWlR1Mk9heUs3bEc3SmhMeTVBZWhnIiwibmFtZSI6InByZXNoaXBwaW5nIHRlc3RfdXNlciIsInBpY3R1cmUiOiJodHRwczovL2xoMy5nb29nbGV1c2VyY29udGVudC5jb20vYS9BQVRYQUp3UHZMRFIyWW50OGprWXlXUHJTRl8zaW95aWoxVlV0N2RTQ0xLZT1zOTYtYyIsImdpdmVuX25hbWUiOiJwcmVzaGlwcGluZyIsImZhbWlseV9uYW1lIjoidGVzdF91c2VyIiwibG9jYWxlIjoiZW4iLCJpYXQiOjE2NDMwNDM4MjUsImV4cCI6MTY0MzA0NzQyNX0.mWt3ap7x14djeuBgpK1-5FtLNWDUzKQdnwyqeL5V0XDTfV6ZHOQHzv0USNUTbzcDGuAa7FaCuXYMBOEzHVkOJ7bh-5-giAXEqBu8MW1jq9CtBei9mKMeJr2sW8yVHBvpr0aPU_vXe_voU-74MB2oit_HfaYURDR0-rDSEIgBE794SzvVMIcdukkODvGNoiAFTdzUtqFkWT_Y3_JZokZTG7ag6VsZI01b9_NIO17CqQNTLsSwy5GSKT_Ap13eJCXGkeIn7RRbN01S3PHl4HPhEBchtlwp7HLhF5RiA6GoX37dlmedWxy1w0sW8mfR-CCeBClhfVZXawFOooAM7LuX3Q"},{"name":"STORAGE_SVC_TOKEN","value":"Bearer ya29.A0ARrdaM8PBVwtyNGOCzE_Boja1qtNaHyxdu6YgD61LKsZGks-sCWdK7DvQPTxHnQMNoOb0PtjLeY9bbLUzK2EdN-ZoWNuhw-c8k2QgxBcprwv7Ufmoox0CQXf2oQhiFQ42N8LjlKzdMkyh-T-ITgNl4NnTqHtFw"},{"name":"STORAGE_SVC_API_KEY","value":"test"},{"name":"SD_SVC_API_KEY","value":"test"},{"name":"OSDU_DATAPARTITIONID","value":"odesprod"},{"name":"SD_READ_CACHE_PAGE_SIZE","value":"4195024"},{"name":"SD_READ_CACHE_MAX_PAGES","value":"256"},{"name":"SEGYTOZGY_VERBOSITY","value":"3"},{"name":"SEGYTOZGY_GENERATE_INDEX","value":"1"}],"resources":{"limits":{"cpu":"1","memory":"4Gi"},"requests":{"cpu":"200m","memory":"4Gi"}},"volumeMounts":[{"name":"default-token-dd5hc","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Never","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"gke-us-central1-airflow--default-pool-8179635e-w7nx","securityContext":{},"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"cloud.google.com/gke-nodepool","operator":"In","values":["default-pool","pool-1","pool-2"]}]}]}}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Failed","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2022-01-24T17:07:48Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2022-01-24T17:08:05Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2022-01-24T17:08:05Z","reason":"ContainersNotReady","message":"containers with unready status: [base]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2022-01-24T17:07:47Z"}],"hostIP":"10.128.0.154","podIP":"10.0.13.3","podIPs":[{"ip":"10.0.13.3"}],"startTime":"2022-01-24T17:07:48Z","containerStatuses":[{"name":"base","state":{"terminated":{"exitCode":1,"reason":"Error","startedAt":"2022-01-24T17:08:04Z","finishedAt":"2022-01-24T17:08:04Z","containerID":"containerd://05139c2a63dadba959322e2ec1888c5da483c3b38907ff5047ecce8cb1dc9268"}},"lastState":{},"ready":false,"restartCount":0,"image":"gcr.io/nice-etching-277309/segy-to-zgy-conversion/segy-to-zgy-conversion-gcp:v0.12.2","imageID":"gcr.io/nice-etching-277309/segy-to-zgy-conversion/segy-to-zgy-conversion-gcp@sha256:45eaafd41220c587a076b0ff637f81171e94c131220e3d30b8d190b5e2ebcf67","containerID":"containerd://05139c2a63dadba959322e2ec1888c5da483c3b38907ff5047ecce8cb1dc9268","started":false}],"qosClass":"Burstable"}}
**[{"name":"STORAGE_SVC_URL","value":"https://preship.osdu.club/api/storage/v2"},{"name":"SD_SVC_URL","value":"https://preship.osdu.club/api/seismic-store/v3"},**
**It is trying to access the temp domain that was set for GCP earlier when the pre-ship env had some issue. Is that environment still active and should this DAG be set to use that temp env or should it be set to use the actual preship env?**M9 - Release 0.12Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/281GCP - M11 - Well Delivery - Query returning no results in well delivery postm...2022-08-23T12:46:56ZMichaelGCP - M11 - Well Delivery - Query returning no results in well delivery postman collectionI am testing the well delivery postman collection and one of the postman queries is returning no results when atleast one should be returned.
Postman collection used: https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/mai...I am testing the well delivery postman collection and one of the postman queries is returning no results when atleast one should be returned.
Postman collection used: https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M11/QA_Artifacts_M11/Collections/Well%20Delivery%20DDMS%20CI-CD%20v3.0.postman_collection.json
Postman environment used: https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M11/GCP-M11/Postman_env_file.json
The following variable needs to be added to the environment for the collection to work:
welldeliveryURL: https://preship.gcp.gnrg-osdu.projects.epam.com/api/well-delivery
The following 2 postman queries are returning a 404 response:
1. UC3- Data1- query_BHArun_by_well
curl --location --request POST 'https://preship.gcp.gnrg-osdu.projects.epam.com/api/well-delivery/query/v1/by_well/bhaRun:batch' \
--header 'data-partition-id: odesprod' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer ...' \
--data-raw '[
"well01"
]'
Response:
{
"message": "Not found planned BHA runs with well ids: well01"
}
2. UC3 - Data1 - query_wellboreTrajectory_by_well
curl --location --request POST 'https://preship.gcp.gnrg-osdu.projects.epam.com/api/well-delivery/query/v1/by_well/wellboreTrajectory:batch' \
--header 'data-partition-id: odesprod' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer ...' \
--data-raw '[
"well01"
]'
Response:
{
"message": "Not found trajectories with well ids: well01"
}
![image](/uploads/b372ae8b07815a040e51e44ccedcea85/image.png)
The following file contains all the steps that were executed for this test:
[OSDU_PTP_M11_GCP_WELL_DELIVERY_RESULTS.txt](/uploads/2028085f5c925c91b66b4aa9d72e993b/OSDU_PTP_M11_GCP_WELL_DELIVERY_RESULTS.txt)Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/290GCP SegY to oVDS conversion2022-08-23T12:47:00ZDebasis ChatterjeeGCP SegY to oVDS conversion@dzmitry_malkevich - As mentioned during this morning call, we are curious to know if the Postman Collection is consistent with latest changes whereby WPC record will be updated by the Converter to leave a link with the File Collection (...@dzmitry_malkevich - As mentioned during this morning call, we are curious to know if the Postman Collection is consistent with latest changes whereby WPC record will be updated by the Converter to leave a link with the File Collection (oVDS files).
cc - @Yan_Sushchynski , @gmarblestone , @ramansingh , @marneson , @todaiks for informationDzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/301GCP Search service throwing "elasticsearch.host not configured correctly for ...2022-06-06T09:36:25ZRakesh SharmaGCP Search service throwing "elasticsearch.host not configured correctly for partition odesprod"Hi,
we are using https://preship.gcp.gnrg-osdu.projects.epam.com/api/search/v2/query search endpoint but getting following error
```
Internal Server Error {"code":500,"reason":"Vital property not present in the environment","message":"...Hi,
we are using https://preship.gcp.gnrg-osdu.projects.epam.com/api/search/v2/query search endpoint but getting following error
```
Internal Server Error {"code":500,"reason":"Vital property not present in the environment","message":"elasticsearch.host not configured correctly for partition odesprod"}
```
Is there change in some configuration which needs to be done on our side?Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/289GCP 50K Batch Performance Loading Fails2023-01-04T15:19:01ZSehubo AkinyanmiGCP 50K Batch Performance Loading FailsHello,
I have run the BATCH performance loading for 500 (runId: 6e861338-c55d-42c8-8c74-1a6707a8658a) & 1000 (runId: 96a5ffd6-9b12-4e03-9715-3b1b4b3a5391) which runs fine and completed okay with data evidenced in the xcomm summary.
Whe...Hello,
I have run the BATCH performance loading for 500 (runId: 6e861338-c55d-42c8-8c74-1a6707a8658a) & 1000 (runId: 96a5ffd6-9b12-4e03-9715-3b1b4b3a5391) which runs fine and completed okay with data evidenced in the xcomm summary.
When I run the 50,000 batch loading (twice); it doesn't generate any runid and same error is observed as noted below;
DEBUG:root:Response: 413
DEBUG:root:json = {"code": 413, "reason": "Failed to send request.", "message": **"Unable to send request to Airflow.** 413 Request Entity Too Large_<html>__<head><title>413 Request Entity Too Large</title></head>__<body>__<center><h1>413 Request Entity Too Large</h1></center>__<hr><center>nginx</center>__</body>__</html>__"}
>>>https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/303Failed challenge response check while registering subscriber for notification...2023-05-03T07:21:22ZRakesh SharmaFailed challenge response check while registering subscriber for notifications on GCP preshippingHi,
I am trying to register subscriber for receiving notification but getting bad request error.Also it seems that register service is not sending any request to challenge endpoint https://dataalloys.com/aw/notification/consumer
Here ...Hi,
I am trying to register subscriber for receiving notification but getting bad request error.Also it seems that register service is not sending any request to challenge endpoint https://dataalloys.com/aw/notification/consumer
Here is the response of request
```{
"code": 400,
"reason": "Bad Request",
"message": "Failed challenge response check to GET https://dataalloys.com/aw/notification/consumer"
}
```
Endpoint
```
https://preship.gcp.gnrg-osdu.projects.epam.com/api/register/v1/subscription
```
Payload
```
{
"name": "alloysubscription",
"description": "Alloy Subscription",
"topic": "records-changed",
"pushEndpoint": "https://dataalloys.com/aw/notification/consumer",
"secret": {
"secretType": "HMAC",
"value": "testsecret"
}
}
```
Please let me know if something is missing from our side.Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/304GCP - Seismic SDUTIL VDS File Download Issue2022-07-05T13:58:52ZSamiullah GhousudeenGCP - Seismic SDUTIL VDS File Download Issue
Able to execute the `SEGY --> OpenVDS` conversion workflow successfully using postman collection -
<pre><code> https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M11/GCP-M11/OpenVDS_SSDMS_to_SSDMS_conversion_C...
Able to execute the `SEGY --> OpenVDS` conversion workflow successfully using postman collection -
<pre><code> https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M11/GCP-M11/OpenVDS_SSDMS_to_SSDMS_conversion_CI-CD.postman_collection.json </code></pre>
After successful run, OpenVDS file created in seismic store sd path and file collection record created in data storage - <pre><code> odesprod:dataset--FileCollection.Bluware.OpenVDS:0fce29c55fec406a967b67e61314d823 </code></pre>
<pre><code>
>> python sdutil stat sd://odesprod/sami-subprojectid21/ST0202R08_PS_PSDM_RAW_PP_TIME.MIG_RAW.POST_STACK.3D.JS-017534.vds --idtoken=$env:ID_TOKEN
- Name: sd://odesprod/sami-subprojectid21/ST0202R08_PS_PSDM_RAW_PP_TIME.MIG_RAW.POST_STACK.3D.JS-017534.vds
- Created By: 110703984333908487442
- Created Date: Wed Jun 22 2022 15:04:44 GMT+0000 (Coordinated Universal Time)
- Size: 213.3 MB
- ReadOnly: False
</code></pre>
However when trying to download converted vds file using sdutil copy command, getting following error :
> Python sdutil cp sd://odesprod/sami-subprojectid21/ST0202R08_PS_PSDM_RAW_PP_TIME.MIG_RAW.POST_STACK.3D.JS-017534.vds .\ST0202R08_PS_PSDM_RAW_PP_TIME.MIG_RAW.POST_STACK.3D.JS-017534.vds --idtoken=$env:ID_TOKEN
<p>
<details>
<summary>ERROR </summary>
<pre><code>[404] {
"error": {
"code": 404,
"message": "No such object: ss-odesprod-2ikfmxucplrey4r4/5ce881a5-ef3a-4045-8530-c7f3bae94452/0",
"errors": [
{
"message": "No such object: ss-odesprod-2ikfmxucplrey4r4/5ce881a5-ef3a-4045-8530-c7f3bae94452/0",
"domain": "global",
"reason": "notFound"
}
]
}
}
</code></pre>
</details>
</p>M13 - Release 0.16Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/306GCP M12 - Well Delivery - Query returning 404 in well delivery postman collec...2022-10-17T08:26:21ZMichaelGCP M12 - Well Delivery - Query returning 404 in well delivery postman collection (GONRG-5326)I am testing the well delivery postman collection and on eof the postman queries is returning a 404 error.
Postman collection used: https://community.opengroup.org/osdu/platform/testing/-/blob/master/Postman%20Collection/44_CICD_Well_De...I am testing the well delivery postman collection and on eof the postman queries is returning a 404 error.
Postman collection used: https://community.opengroup.org/osdu/platform/testing/-/blob/master/Postman%20Collection/44_CICD_Well_Delivery_DMS/WellDelivery%20DDMS%20CI-CD%20v3.1.postman_collection.json
Postman environment used: Used existing M11 environment from https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M11/GCP-M11/Postman_env_file.json The following variable needs to be added to the environment for the collection to work:
welldeliveryURL: https://preship.gcp.gnrg-osdu.projects.epam.com/api/well-delivery
The following postman query returns a 404 response:
UC1-well_activity_programs_latest_get_by_well:
curl --location --request GET 'https://preship.gcp.gnrg-osdu.projects.epam.com/api/well-delivery/wellActivityPrograms/v1/by_well/wellWD_autotest7849598' \
--header 'data-partition-id: odesprod' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer ...' \
--data-raw ''
Response:
404 error
{
"message": "Could not find wellbore with well id: wellWD_autotest7849598"
}
The following file contains all the steps that were executed for this test: [GCP_M12_Well_Delivery_Tests.docx](/uploads/ed74590ebd5e4c87cd91935085afbc31/GCP_M12_Well_Delivery_Tests.docx)Andrei IonescuDzmitry Malkevich (EPAM)Andrei Ionescuhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/331GCP - CRS Catalog V3 - gives response, but incomplete result2022-09-06T18:50:20ZDebasis ChatterjeeGCP - CRS Catalog V3 - gives response, but incomplete resultIt is, as if actual search is not executed.
[M12-GCP-CRS-Catalog-V3-Debasis-problem.txt](/uploads/3e3eed4dc392479f9aa961ef4889ab62/M12-GCP-CRS-Catalog-V3-Debasis-problem.txt)
Can you check and advise?
Thanks
cc : @Yauhen_Shaliou , @D...It is, as if actual search is not executed.
[M12-GCP-CRS-Catalog-V3-Debasis-problem.txt](/uploads/3e3eed4dc392479f9aa961ef4889ab62/M12-GCP-CRS-Catalog-V3-Debasis-problem.txt)
Can you check and advise?
Thanks
cc : @Yauhen_Shaliou , @Dzmitry_Poloyko for informationDzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/354GCP - Audit and Metric - The KPI for "data size transferred" fails even after...2023-03-14T02:04:54ZDebasis ChatterjeeGCP - Audit and Metric - The KPI for "data size transferred" fails even after we provide right tenant and sub-project namesSee enclosed file.
[M13-GCP-audit-metric-KPI-data-size-transferred.txt](/uploads/a57c3a5bd5f9271605f49ee8c497bd3d/M13-GCP-audit-metric-KPI-data-size-transferred.txt)See enclosed file.
[M13-GCP-audit-metric-KPI-data-size-transferred.txt](/uploads/a57c3a5bd5f9271605f49ee8c497bd3d/M13-GCP-audit-metric-KPI-data-size-transferred.txt)Mohd Asad ShaikhDzmitry Malkevich (EPAM)Yauhen Shaliou [EPAM/GCP]Mohd Asad Shaikhhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/443IBM-CSV Ingestion-Platform Validation-Error File. The problem is also seen in...2023-03-12T16:54:37ZAnkit GoyalIBM-CSV Ingestion-Platform Validation-Error File. The problem is also seen in Pre-ship environment[2023-02-28 20:55:07,709] {taskinstance.py:1107} INFO - Executing <Task(KubernetesPodOperator): csv-parser> on 2023-02-28T20:50:03.419560+00:00
[2023-02-28 20:55:07,714] {standard_task_runner.py:52} INFO - Started process 2868 to run tas...[2023-02-28 20:55:07,709] {taskinstance.py:1107} INFO - Executing <Task(KubernetesPodOperator): csv-parser> on 2023-02-28T20:50:03.419560+00:00
[2023-02-28 20:55:07,714] {standard_task_runner.py:52} INFO - Started process 2868 to run task
[2023-02-28 20:55:07,720] {standard_task_runner.py:76} INFO - Running: ['***', 'tasks', 'run', 'csv-parser-dag', 'csv-parser', '2023-02-28T20:50:03.419560+00:00', '--job-id', '58359', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/csv_ingestion_all_steps.py', '--cfg-path', '/tmp/tmpuolct0ay', '--error-file', '/tmp/tmpeb5k26k9']
[2023-02-28 20:55:07,721] {standard_task_runner.py:77} INFO - Job 58359: Subtask csv-parser
[2023-02-28 20:55:07,780] {logging_mixin.py:104} INFO - Running <TaskInstance: csv-parser-dag.csv-parser 2023-02-28T20:50:03.419560+00:00 [running]> on host ***-worker-0.***-worker.osdu-***.svc.cluster.local
[2023-02-28 20:55:07,846] {taskinstance.py:1501} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1157, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1294, in _prepare_and_execute_task_with_callbacks
self.render_templates(context=context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1795, in render_templates
self.task.render_template_fields(context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1003, in render_template_fields
self._do_render_template_fields(self, self.template_fields, context, jinja_env, set())
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1016, in _do_render_template_fields
rendered_content = self.render_template(content, context, jinja_env, seen_oids)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1067, in render_template
return [self.render_template(element, context, jinja_env) for element in content]
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1067, in <listcomp>
return [self.render_template(element, context, jinja_env) for element in content]
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 1053, in render_template
return jinja_env.from_string(content).render(**context)
File "/home/airflow/.local/lib/python3.8/site-packages/jinja2/environment.py", line 1090, in render
self.environment.handle_exception()
File "/home/airflow/.local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/home/airflow/.local/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
raise value.with_traceback(tb)
File "<template>", line 1, in top-level template code
jinja2.exceptions.UndefinedError: 'dict object' has no attribute 'correlationId'
[2023-02-28 20:55:07,851] {taskinstance.py:1544} INFO - Marking task as FAILED. dag_id=csv-parser-dag, task_id=csv-parser, execuAnuj GuptaPintu GuptaAnuj Guptahttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/397GCP M15 CSV Ingestion message Application run failed2023-01-06T18:20:30ZAnkit GoyalGCP M15 CSV Ingestion message Application run failed[2022-12-22, 18:00:35 UTC] {pod_launcher.py:149} INFO - {"timestamp":"2022-12-22 18:00:35.153","thread":"main","logger":"org.opengroup.osdu.csvparser.provider.gcp.Application","message":"Starting Application v0.18.2 on csv-parser.bafbec4...[2022-12-22, 18:00:35 UTC] {pod_launcher.py:149} INFO - {"timestamp":"2022-12-22 18:00:35.153","thread":"main","logger":"org.opengroup.osdu.csvparser.provider.gcp.Application","message":"Starting Application v0.18.2 on csv-parser.bafbec4b888a42848a29d4516ed6f150 with PID 7 (/app/csv-parser-gcp.jar started by root in /app)","context":"default","severity":"INFO"}
[2022-12-22, 18:00:35 UTC] {pod_launcher.py:149} INFO - {"timestamp":"2022-12-22 18:00:35.242","thread":"main","logger":"org.opengroup.osdu.csvparser.provider.gcp.Application","message":"Running with Spring Boot v2.2.11.RELEASE, Spring v5.2.10.RELEASE","context":"default","severity":"DEBUG"}
[2022-12-22, 18:00:35 UTC] {pod_launcher.py:149} INFO - {"timestamp":"2022-12-22 18:00:35.243","thread":"main","logger":"org.opengroup.osdu.csvparser.provider.gcp.Application","message":"The following profiles are active: gcp","context":"default","severity":"INFO"}
[2022-12-22, 18:00:41 UTC] {pod_launcher.py:149} INFO - {"timestamp":"2022-12-22 18:00:41.398","thread":"main","logger":"org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker","message":"Bean 'tenantInfoFactory' of type [org.springframework.aop.scope.ScopedProxyFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)","context":"default","severity":"INFO"}
[2022-12-22, 18:00:41 UTC] {pod_launcher.py:149} INFO - {"timestamp":"2022-12-22 18:00:41.448","thread":"main","logger":"org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker","message":"Bean 'appConfiguration' of type [org.opengroup.osdu.csvparser.config.AppConfiguration$$EnhancerBySpringCGLIB$$cc7e415b] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)","context":"default","severity":"INFO"}
[2022-12-22, 18:00:41 UTC] {pod_launcher.py:149} INFO - {"timestamp":"2022-12-22 18:00:41.551","thread":"main","logger":"org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker","message":"Bean 'objectMapper' of type [com.fasterxml.jackson.databind.ObjectMapper] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)","context":"default","severity":"INFO"}
[2022-12-22, 18:00:41 UTC] {pod_launcher.py:149} INFO - {"timestamp":"2022-12-22 18:00:41.554","thread":"main","logger":"org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker","message":"Bean 'httpResponseBodyMapper' of type [org.opengroup.osdu.core.common.http.json.HttpResponseBodyMapper] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)","context":"default","severity":"INFO"}
[2022-12-22, 18:00:45 UTC] {pod_launcher.py:149} INFO - {"timestamp":"2022-12-22 18:00:45.145","thread":"main","logger":"org.springframework.http.converter.json.Jackson2ObjectMapperBuilder","message":"For Jackson Kotlin classes support please add \"com.fasterxml.jackson.module:jackson-module-kotlin\" to the classpath","context":"default","severity":"WARN"}
[2022-12-22, 18:00:45 UTC] {pod_launcher.py:149} INFO - {"timestamp":"2022-12-22 18:00:45.623","thread":"main","logger":"org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor","message":"Autowired annotation is not supported on static fields: private static java.lang.String org.opengroup.osdu.core.common.search.Config.QUERY_DEFAULT_LIMIT","context":"default","severity":"INFO"}
[2022-12-22, 18:00:45 UTC] {pod_launcher.py:149} INFO - {"timestamp":"2022-12-22 18:00:45.626","thread":"main","logger":"org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor","message":"Autowired annotation is not supported on static fields: private static java.lang.String org.opengroup.osdu.core.common.search.Config.QUERY_LIMIT_MAXIMUM","context":"default","severity":"INFO"}
[2022-12-22, 18:00:46 UTC] {pod_launcher.py:149} INFO - {"timestamp":"2022-12-22 18:00:46.511","thread":"main","logger":"org.springframework.aop.framework.CglibAopProxy","message":"Unable to proxy interface-implementing method [public final java.net.URL org.opengroup.osdu.core.gcp.obm.driver.gcs.GcsDriver.getSignedUrlForDownload(java.lang.String,org.opengroup.osdu.core.gcp.obm.persistence.ObmDestination,java.lang.String,long,java.util.concurrent.TimeUnit,org.apache.commons.lang3.tuple.Pair[])] because it is marked as final: Consider using interface-based JDK proxies instead!","context":"default","severity":"INFO"}
[2022-12-22, 18:00:46 UTC] {pod_launcher.py:149} INFO - {"timestamp":"2022-12-22 18:00:46.512","thread":"main","logger":"org.springframework.aop.framework.CglibAopProxy","message":"Unable to proxy interface-implementing method [public final java.net.URL org.opengroup.osdu.core.gcp.obm.driver.gcs.GcsDriver.getSignedUrlForUpload(java.lang.String,org.opengroup.osdu.core.gcp.obm.persistence.ObmDestination,java.lang.String,long,java.util.concurrent.TimeUnit,org.apache.commons.lang3.tuple.Pair[])] because it is marked as final: Consider using interface-based JDK proxies instead!","context":"default","severity":"INFO"}
[2022-12-22, 18:00:46 UTC] {pod_launcher.py:149} INFO - {"timestamp":"2022-12-22 18:00:46.513","thread":"main","logger":"org.springframework.aop.framework.CglibAopProxy","message":"Unable to proxy interface-implementing method [public final java.net.URL org.opengroup.osdu.core.gcp.obm.driver.gcs.GcsDriver.getSignedUrlForDelete(java.lang.String,org.opengroup.osdu.core.gcp.obm.persistence.ObmDestination,java.lang.String,long,java.util.concurrent.TimeUnit,org.apache.commons.lang3.tuple.Pair[])] because it is marked as final: Consider using interface-based JDK proxies instead!","context":"default","severity":"INFO"}
[2022-12-22, 18:00:48 UTC] {pod_launcher.py:149} INFO - {"timestamp":"2022-12-22 18:00:48.845","thread":"main","logger":"org.opengroup.osdu.csvparser.provider.gcp.Application","message":"Started Application in 17.023 seconds (JVM running for 20.301)","context":"default","severity":"INFO"}
[2022-12-22, 18:00:49 UTC] {pod_launcher.py:149} INFO - {"timestamp":"2022-12-22 18:00:49.347","thread":"main","logger":"org.springframework.boot.autoconfigure.logging.ConditionEvaluationReportLoggingListener","message":"\n\nError starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.","context":"default","severity":"INFO"}
[2022-12-22, 18:00:49 UTC] {pod_launcher.py:149} INFO - {"timestamp":"2022-12-22 18:00:49.360","thread":"main","logger":"org.springframework.boot.SpringApplication","message":"Application run failed","context":"default","exception":"java.lang.IllegalStateException: Failed to execute CommandLineRunner\n\tat org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:787)\n\tat org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:768)\n\tat org.springframework.boot.SpringApplication.run(SpringApplication.java:322)\n\tat org.opengroup.osdu.csvparser.provider.gcp.Application.main(Application.java:38)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:87)\n\tat org.springframework.boot.loader.Launcher.launch(Launcher.java:51)\n\tat org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52)\nCaused by: org.opengroup.osdu.core.common.exception.BadRequestException: Invalid Ingestion request: \n correlationId: must not be empty\n\n\tat org.opengroup.osdu.csvparser.CsvIngestionRunner.validate(CsvIngestionRunner.java:85)\n\tat org.opengroup.osdu.csvparser.CsvIngestionRunner.run(CsvIngestionRunner.java:45)\n\tat org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:784)\n\t... 11 common frames omitted\n","severity":"ERROR"}
[2022-12-22, 18:00:49 UTC] {pod_launcher.py:149} INFO - {"timestamp":"2022-12-22 18:00:49.410","thread":"main","logger":"org.opengroup.osdu.core.destination.obm.GcsTenantObmDestinationResolver","message":"On pre-destroy","context":"default","severity":"INFO"}
[2022-12-22, 18:00:49 UTC] {pod_launcher.py:149} INFO - {"timestamp":"2022-12-22 18:00:49.413","thread":"main","logger":"org.opengroup.osdu.core.destination.oqm.PsTenantOqmDestinationResolver","message":"On pre-destroy. 0 topic client(s) & 0 subscription clients to shutdown","context":"default","severity":"INFO"}
[2022-12-22, 18:00:51 UTC] {pod_launcher.py:198} INFO - Event: csv-parser.bafbec4b888a42848a29d4516ed6f150 had an event of type Running
[2022-12-22, 18:00:51 UTC] {pod_launcher.py:171} INFO - Pod csv-parser.bafbec4b888a42848a29d4516ed6f150 has state running
[2022-12-22, 18:00:53 UTC] {pod_launcher.py:198} INFO - Event: csv-parser.bafbec4b888a42848a29d4516ed6f150 had an event of type Failed
[2022-12-22, 18:00:53 UTC] {pod_launcher.py:308} ERROR - Event with job id csv-parser.bafbec4b888a42848a29d4516ed6f150 Failed
[2022-12-22, 18:00:53 UTC] {pod_launcher.py:198} INFO - Event: csv-parser.bafbec4b888a42848a29d4516ed6f150 had an event of type Failed
[2022-12-22, 18:00:53 UTC] {pod_launcher.py:308} ERROR - Event with job id csv-parser.bafbec4b888a42848a29d4516ed6f150 Failed
[2022-12-22, 18:00:53 UTC] {taskinstance.py:1776} ERROR - Task failed with exceptionDzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/401GCP R3M15-CSV Ingestion : In execut raise PipelineFailedError("Dag failed")2023-01-05T12:50:23ZREETU RAGINIGCP R3M15-CSV Ingestion : In execut raise PipelineFailedError("Dag failed")*** Reading remote log from gs://us-central1-airflow-v2-36b0ff41-bucket/logs/csv_ingestion/update_status_finished/2022-12-27T15:27:44.295473+00:00/1.log.
[2022-12-27, 15:34:14 UTC] {taskinstance.py:1044} INFO - Dependencies all met for <...*** Reading remote log from gs://us-central1-airflow-v2-36b0ff41-bucket/logs/csv_ingestion/update_status_finished/2022-12-27T15:27:44.295473+00:00/1.log.
[2022-12-27, 15:34:14 UTC] {taskinstance.py:1044} INFO - Dependencies all met for <TaskInstance: csv_ingestion.update_status_finished test_workflow707265 [queued]>
[2022-12-27, 15:34:14 UTC] {taskinstance.py:1044} INFO - Dependencies all met for <TaskInstance: csv_ingestion.update_status_finished test_workflow707265 [queued]>
[2022-12-27, 15:34:14 UTC] {taskinstance.py:1250} INFO -
--------------------------------------------------------------------------------
[2022-12-27, 15:34:14 UTC] {taskinstance.py:1251} INFO - Starting attempt 1 of 2
[2022-12-27, 15:34:14 UTC] {taskinstance.py:1252} INFO -
--------------------------------------------------------------------------------
[2022-12-27, 15:34:14 UTC] {taskinstance.py:1271} INFO - Executing <Task(UpdateStatusOperator): update_status_finished> on 2022-12-27 15:27:44.295473+00:00
[2022-12-27, 15:34:14 UTC] {standard_task_runner.py:52} INFO - Started process 185313 to run task
[2022-12-27, 15:34:14 UTC] {standard_task_runner.py:79} INFO - Running: ['airflow', 'tasks', 'run', 'csv_ingestion', 'update_status_finished', 'test_workflow707265', '--job-id', '217613', '--raw', '--subdir', 'DAGS_FOLDER/csv-parser/csv_ingestion_all_steps.py', '--cfg-path', '/tmp/tmpltcu6ban', '--error-file', '/tmp/tmpfmf51_8b']
[2022-12-27, 15:34:14 UTC] {standard_task_runner.py:80} INFO - Job 217613: Subtask update_status_finished
[2022-12-27, 15:34:15 UTC] {task_command.py:298} INFO - Running <TaskInstance: csv_ingestion.update_status_finished test_workflow707265 [running]> on host airflow-worker-6467cc5845-fqzgm
[2022-12-27, 15:34:15 UTC] {taskinstance.py:1448} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_EMAIL=airflow@example.com
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=csv_ingestion
AIRFLOW_CTX_TASK_ID=update_status_finished
AIRFLOW_CTX_EXECUTION_DATE=2022-12-27T15:27:44.295473+00:00
AIRFLOW_CTX_DAG_RUN_ID=test_workflow707265
[2022-12-27, 15:34:15 UTC] {update_status.py:67} INFO - There are failed tasks before this one. So it has status FAILED
[2022-12-27, 15:34:16 UTC] {warnings.py:109} WARNING - /opt/python3.8/lib/python3.8/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'preship.gcp.gnrg-osdu.projects.epam.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
[2022-12-27, 15:34:17 UTC] {taskinstance.py:1776} ERROR - Task failed with exception
Traceback (most recent call last):
File "/opt/python3.8/lib/python3.8/site-packages/osdu_airflow/operators/update_status.py", line 153, in execute
raise PipelineFailedError("Dag failed")
osdu_ingestion.libs.exceptions.PipelineFailedError: Dag failed
[2022-12-27, 15:34:17 UTC] {taskinstance.py:1279} INFO - Marking task as UP_FOR_RETRY. dag_id=csv_ingestion, task_id=update_status_finished, execution_date=20221227T152744, start_date=20221227T153414, end_date=20221227T153417
[2022-12-27, 15:34:17 UTC] {standard_task_runner.py:93} ERROR - Failed to execute job 217613 for task update_status_finished (Dag failed; 185313)
[2022-12-27, 15:34:17 UTC] {local_task_job.py:154} INFO - Task exited with return code 1
[2022-12-27, 15:34:17 UTC] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule checkDzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/414GCP M15 Indexing not working2023-05-03T13:23:44ZDadong ZhouGCP M15 Indexing not workingFor E&O policy testing: I created a new data record with storage api and I can see the data record with storage api but search api cannot find the new data record. Please check the indexing service. Thanks
Here is the data record:
```
{...For E&O policy testing: I created a new data record with storage api and I can see the data record with storage api but search api cannot find the new data record. Please check the indexing service. Thanks
Here is the data record:
```
{
"data": {
"msg": "hello world from Data Lake",
"FacilityID": "BAS-03"
},
"meta": [
{}
],
"id": "odesprod:master-data--Well:TestData-851007",
"version": 1672847261687073,
"kind": "odesprod:wks:master-data--Well:1.0.0",
"acl": {
"viewers": [
"data.policytest38161.viewers@odesprod.osdu-gcp.go3-nrg.projects.epam.com"
],
"owners": [
"data.policytest38161.viewers@odesprod.osdu-gcp.go3-nrg.projects.epam.com"
]
},
"legal": {
"legaltags": [
"odesprod-demo-legaltag"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "preshipping_test_user@osdu-gcp.go3-nrg.projects.epam.com",
"createTime": "2023-01-04T15:47:42.650Z"
}
```Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/417GCP preshipping. Step Get 08 Workflow - Get work status: Status is Failed2023-01-12T15:44:50ZRakesh BorgeGCP preshipping. Step Get 08 Workflow - Get work status: Status is FailedGET: https://{{WORKFLOW_URL}}/workflow/Osdu_ingest_by_reference/workflowRun/{{run_id}}
Body: 1
Response:
{
"workflowId": "a995c63b-f1cf-40bc-b3c0-c76f99ca6457",
"runId": "1e6d3e6c-00a2-4a1e-b939-573cc8a06ac3",
"startTimeStamp...GET: https://{{WORKFLOW_URL}}/workflow/Osdu_ingest_by_reference/workflowRun/{{run_id}}
Body: 1
Response:
{
"workflowId": "a995c63b-f1cf-40bc-b3c0-c76f99ca6457",
"runId": "1e6d3e6c-00a2-4a1e-b939-573cc8a06ac3",
"startTimeStamp": 1672932068263,
"status": "running",
"submittedBy": "preshipping_test_user@osdu-gcp.go3-nrg.projects.epam.com"
}
{
"workflowId": "a995c63b-f1cf-40bc-b3c0-c76f99ca6457",
"runId": "1e6d3e6c-00a2-4a1e-b939-573cc8a06ac3",
"startTimeStamp": 1672932068263,
"endTimeStamp": 1672932102048,
"status": "failed",
"submittedBy": "preshipping_test_user@osdu-gcp.go3-nrg.projects.epam.com"
}
Collection:
[R15-Master_Data_manifestByReference_ingestion.postman_collection.json](/uploads/10dd47570a4e16cbc72dddfb28e1f678/R15-Master_Data_manifestByReference_ingestion.postman_collection.json)
[R15-Reference_Data_manifestByReference_ingestion.postman_collection.json](/uploads/3e3d78ed93284d18ced24ed6a4aa9067/R15-Reference_Data_manifestByReference_ingestion.postman_collection.json)
Environment:
[OSDU_R3_M15_GCP_pre-ship.postman_environment.json](/uploads/2a825885383e05baa24f84fa85ae438c/OSDU_R3_M15_GCP_pre-ship.postman_environment.json)
JSON File:
[ManifestTest_recordsNew.json](/uploads/84f58196a7496b04373777978125bf84/ManifestTest_recordsNew.json)
[ManifestTest_ReferenceDataRecord.json](/uploads/9d55aaecf51a7d94f2ec7d0c580f89ed/ManifestTest_ReferenceDataRecord.json)Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)2023-01-12https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/426GCP M15 SEGY to oVDS Conversion Issue: "reason":"Access denied","message":"Th...2023-01-14T17:34:45ZAnkit GoyalGCP M15 SEGY to oVDS Conversion Issue: "reason":"Access denied","message":"The user is not authorized to perform this action"Used: GCP Provided Collection and Environment
Error Log at Stage 3, Ingest Seismic: Seismic Manifest ingestion:
[2023-01-10, 01:35:41 UTC] {taskinstance.py:1271} INFO - Executing <Task(UpdateStatusOperator): update_status_finished_task> ...Used: GCP Provided Collection and Environment
Error Log at Stage 3, Ingest Seismic: Seismic Manifest ingestion:
[2023-01-10, 01:35:41 UTC] {taskinstance.py:1271} INFO - Executing <Task(UpdateStatusOperator): update_status_finished_task> on 2023-01-10 01:35:24.624858+00:00
[2023-01-10, 01:35:41 UTC] {standard_task_runner.py:52} INFO - Started process 42875 to run task
[2023-01-10, 01:35:41 UTC] {standard_task_runner.py:79} INFO - Running: ['airflow', 'tasks', 'run', 'Osdu_ingest', 'update_status_finished_task', 'cacb5e35-712b-47bb-b499-d39a02b45c22', '--job-id', '232377', '--raw', '--subdir', 'DAGS_FOLDER/ingestion-dags/osdu-ingest-r3.py', '--cfg-path', '/tmp/tmpqv8c0tee', '--error-file', '/tmp/tmpot2jc8fg']
[2023-01-10, 01:35:41 UTC] {standard_task_runner.py:80} INFO - Job 232377: Subtask update_status_finished_task
[2023-01-10, 01:35:43 UTC] {task_command.py:298} INFO - Running <TaskInstance: Osdu_ingest.update_status_finished_task cacb5e35-712b-47bb-b499-d39a02b45c22 [running]> on host airflow-worker-6467cc5845-s6gcw
[2023-01-10, 01:35:43 UTC] {taskinstance.py:1448} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=Osdu_ingest
AIRFLOW_CTX_TASK_ID=update_status_finished_task
AIRFLOW_CTX_EXECUTION_DATE=2023-01-10T01:35:24.624858+00:00
AIRFLOW_CTX_DAG_RUN_ID=cacb5e35-712b-47bb-b499-d39a02b45c22
[2023-01-10, 01:35:43 UTC] {update_status.py:67} INFO - There are failed tasks before this one. So it has status FAILED
[2023-01-10, 01:35:43 UTC] {warnings.py:109} WARNING - /opt/python3.8/lib/python3.8/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'preship.gcp.gnrg-osdu.projects.epam.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
[2023-01-10, 01:35:44 UTC] {authorization.py:137} ERROR - {"code":503,"reason":"Access denied","message":"The user is not authorized to perform this action"}
[2023-01-10, 01:35:44 UTC] {taskinstance.py:1776} ERROR - Task failed with exception
Traceback (most recent call last):
File "/opt/python3.8/lib/python3.8/site-packages/osdu_airflow/operators/update_status.py", line 145, in execute
status_updater.update_workflow_status()
File "/opt/python3.8/lib/python3.8/site-packages/osdu_ingestion/libs/update_status.py", line 108, in update_workflow_status
self.update_status_request()
File "/opt/python3.8/lib/python3.8/site-packages/osdu_ingestion/libs/update_status.py", line 83, in update_status_request
response = self.workflow_client.update_workflow_run(update_status_request, self.workflow_name, self.run_id)
File "/opt/python3.8/lib/python3.8/site-packages/osdu_api/clients/ingestion_workflow/ingestion_workflow_client.py", line 74, in update_workflow_run
return self.make_request(method=HttpMethod.PUT, url='{}{}{}{}{}'.format(self.ingestion_workflow_url, '/workflow/', workflow_name, '/workflowRun/', run_id),
File "/opt/python3.8/lib/python3.8/site-packages/osdu_api/clients/base_client.py", line 217, in make_request
response = self._send_request_with_token_refresher(headers, method, url, data, params)
File "/opt/python3.8/lib/python3.8/site-packages/osdu_api/auth/authorization.py", line 159, in _wrapper
return send_request_with_auth_header(_token_refresher,Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/421GCP M15 SEGY to ZGY conversion Issue: ERROR - Event with job id segy-to-zgy.7...2023-06-15T12:48:49ZAnkit GoyalGCP M15 SEGY to ZGY conversion Issue: ERROR - Event with job id segy-to-zgy.729db93e72234dc5a33c259bdc0d5f53 FailedEnclosed the detailed job log with this issue ticket, please let me know if you need more information.[SEGY_TO_oZGY_error.txt](/uploads/99c2a2aec7d5dac31014467e9854a5d8/SEGY_TO_oZGY_error.txt)Enclosed the detailed job log with this issue ticket, please let me know if you need more information.[SEGY_TO_oZGY_error.txt](/uploads/99c2a2aec7d5dac31014467e9854a5d8/SEGY_TO_oZGY_error.txt)Ivan Medeiros MonteiroDzmitry Malkevich (EPAM)Ivan Medeiros Monteirohttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/416GCP R3M15- Pre-shipping- CSV Ingestion -Server Error- Bad Gateway2023-01-09T09:07:37ZREETU RAGINIGCP R3M15- Pre-shipping- CSV Ingestion -Server Error- Bad Gateway<html>
<head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>502 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered a temporary error ...<html>
<head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>502 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.
</h2>
<h2></h2>
</body>
</html>Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/409GCP M15 Missing Search Partition Policy2023-01-04T15:28:49ZDadong ZhouGCP M15 Missing Search Partition PolicyRunning Policy postman collection: https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M15/GCP-M15/Policy/GCP_OSDUR3M15_Policy_Tests.postman_collection.json
Running Step 3 (Get Search Policy) in Policy folder: http...Running Policy postman collection: https://community.opengroup.org/osdu/platform/pre-shipping/-/blob/main/R3-M15/GCP-M15/Policy/GCP_OSDUR3M15_Policy_Tests.postman_collection.json
Running Step 3 (Get Search Policy) in Policy folder: https://{{POLICY_HOST}}/policies/osdu/partition/{{data-partition-id}}/search.rego
Error is returned: { "detail": "Unexpected result from OPA storage_not_found_error: policy id "osdu/partition/odesprod/search.rego"" }
Please check if the default search partition policy is deployed. Thanks.Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/360In GCP environment, Search queries created by the CRS Conversion Services V3,...2022-12-14T16:02:36ZKamlesh TodaiIn GCP environment, Search queries created by the CRS Conversion Services V3, have some issue, because of which failure are seen while using CRS Conversion V3 servicePOST https://preship.gcp.gnrg-osdu.projects.epam.com/api/crs/converter/v3/convert
Body
{
"fromCRS": "odesprod:reference-data--CoordinateReferenceSystem:BoundProjected:EPSG::29193_EPSG::1867:",
"toCRS": "odesprod:reference-data--Coor...POST https://preship.gcp.gnrg-osdu.projects.epam.com/api/crs/converter/v3/convert
Body
{
"fromCRS": "odesprod:reference-data--CoordinateReferenceSystem:BoundProjected:EPSG::29193_EPSG::1867:",
"toCRS": "odesprod:reference-data--CoordinateReferenceSystem:Geographic2D:EPSG::4326:",
"points": [
{
"x": 697339.525,
"y": 7239989.403,
"z": 0
}
]
}
Response
{
"code": 400,
"message": "record not found: odesprod:reference-data--CoordinateReferenceSystem:BoundProjected:EPSG::29193_EPSG::1867/"
}
We do find the two following records in GCP/Preship environment, using the Search API
odesprod:reference-data--CoordinateReferenceSystem:BoundProjected:EPSG::29193_EPSG::1867 (SAD69 UTM Zone 23S)
odesprod:reference-data--CoordinateReferenceSystem:Geographic2D:EPSG::4326 for WGS84
The collection to test can be found here
https://community.opengroup.org/osdu/platform/testing/-/blob/master/Postman%20Collection/18_CICD_Setup_CRSConversionAPI/CRS%20Conversion%20Service%20V3.skip_postman_collection.json
Some details about this issue can also be found here
https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/353
@debasisc @chadYan Sushchynski (EPAM)Dzmitry Malkevich (EPAM)Yan Sushchynski (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/365GCP M13 - Manifest Ingestion - Performance Load testing 50k records - 502 res...2023-01-04T15:19:01ZChad LeongGCP M13 - Manifest Ingestion - Performance Load testing 50k records - 502 responseWhile submitting 50k records for performance load testing in manifest ingestion:
`https://preship.gcp.gnrg-osdu.projects.epam.com/api/workflow/v1/workflow/Osdu_ingest/workflowRun `
### Got the following response:
```
<Response [502]>...While submitting 50k records for performance load testing in manifest ingestion:
`https://preship.gcp.gnrg-osdu.projects.epam.com/api/workflow/v1/workflow/Osdu_ingest/workflowRun `
### Got the following response:
```
<Response [502]>
'{"code":502,"reason":"Failed to send request.","message":"Unable to send request to Airflow. 502 Bad Gateway__ Bad Gateway__<html><head>_<meta http-equiv=\\"content-type\\" content=\\"text/html;charset=utf-8\ Server Error<\">_<title>502 Server Error</title>_</head>_<body text=#000000 bgcolor=#ffffff>_<h1>Error: Server countered a te
Error</h1>_<h2>The server encountered a temporary error and could not complete your request.<p>Ple>_</body></htm<html><head>_<meta http-equiv=\\"content-type\\" content=\\"text/html;charset=utf-8\\">_<title>502 Server Error</titlease try again in 30 seconds.</h2>_<h2></h2>_</body></html>_"}'
```
## 50k Records:
[Records_GCP_50000.zipx](/uploads/95c7f0b3993ceb7b21d78668b3964658/Records_GCP_50000.zipx)
## Airflow response:
N/ADzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/384GCP R3M14: Seismic DDMS: SDUTILS, Create Subfolder and existing file stats er...2022-12-21T14:47:20ZAnkit GoyalGCP R3M14: Seismic DDMS: SDUTILS, Create Subfolder and existing file stats error: [400] [seismic-store-service] Bad Request@Yauhen_Shaliou @Yan_Sushchynski
Create Folder error:
(sdutilenv) C:\OSDU\seismic-store-sdutil-master>python sdutil mk "sd://odesprod/ankittest" “preshipping_test_user@osdu-gcp.go3-nrg.projects.epam.com” "odesprod-Segy-Legal-Tag-138257...@Yauhen_Shaliou @Yan_Sushchynski
Create Folder error:
(sdutilenv) C:\OSDU\seismic-store-sdutil-master>python sdutil mk "sd://odesprod/ankittest" “preshipping_test_user@osdu-gcp.go3-nrg.projects.epam.com” "odesprod-Segy-Legal-Tag-138257" --idtoken=%ID_TOKEN%
[1] REGIONAL
[2] MULTI_REGIONAL
[3] NEARLINE
[4] COLDLINE
Select the bucket storage class: 2
[1] ASIA
[2] EU
[3] US
Select the bucket storage location: 3
> Registering the subproject ankittest(tenant=odesprod) with admin@osdu-gcp.go3-nrg.projects.epam.com as admin ...
[400] [seismic-store-service] Bad Request
Error while generating the Stat of an existing file:
(sdutilenv) C:\OSDU\seismic-store-sdutil-master>python sdutil stat "sd://odesprod/autotesting/ST10010ZC11_PZ_PSDM_KIRCH_FULL_T.MIG_FIN.POST_STACK.3D.JS-017536.segy" --idtoken=%ID_TOKEN%
[400] [seismic-store-service] Bad RequestSacha BrantsDzmitry Malkevich (EPAM)Yauhen Shaliou [EPAM/GCP]Sacha Brantshttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/394GCP M14 - Manifest Ingestion - Performance Load testing - Maximum Payload Siz...2023-01-04T15:19:01ZSamiullah GhousudeenGCP M14 - Manifest Ingestion - Performance Load testing - Maximum Payload Size LimitGCP M14 - Manifest Ingestion - Performance Load testing - Maximum Payload Size Limit
Please confirm maximum payload limit in pre-shipping for manifest-based ingestion testing so we could plan test-case for performance loading in M15 Pre...GCP M14 - Manifest Ingestion - Performance Load testing - Maximum Payload Size Limit
Please confirm maximum payload limit in pre-shipping for manifest-based ingestion testing so we could plan test-case for performance loading in M15 Pre-shipping scope.
cc @todaiks @debasisc @davidglass @chadDenis Karpenok (EPAM)Dzmitry Malkevich (EPAM)Denis Karpenok (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/370GCP - M14 - Ingest by Reference - fails in first step2022-10-28T03:02:19ZDebasis ChatterjeeGCP - M14 - Ingest by Reference - fails in first stepSee airflow log
[M_4-GCP-ingest-by-Reference-error-validate-manifest-schema-task.txt](/uploads/54d100cfa992c3c49008758baf745fb8/M_4-GCP-ingest-by-Reference-error-validate-manifest-schema-task.txt)
Airflow RunID
```
{
"workflowId": ...See airflow log
[M_4-GCP-ingest-by-Reference-error-validate-manifest-schema-task.txt](/uploads/54d100cfa992c3c49008758baf745fb8/M_4-GCP-ingest-by-Reference-error-validate-manifest-schema-task.txt)
Airflow RunID
```
{
"workflowId": "a995c63b-f1cf-40bc-b3c0-c76f99ca6457",
"runId": "bf4ca995-9407-4f41-8167-ea121e90b36e",
"startTimeStamp": 1666294752152,
"status": "submitted",
"submittedBy": "preshipping_test_user@osdu-gcp.go3-nrg.projects.epam.com"
}
```Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/369Not able to trigger the DAG In R3 M13 GCP pre-ship environment the DAG "Segy...2022-09-28T16:19:04ZKamlesh TodaiNot able to trigger the DAG In R3 M13 GCP pre-ship environment the DAG "Segy_to_vds_conversion_sdms"The airflow DAG indicates that the DAG is running. Upon registering the workflow one gets the 409 conflict message as expected, But when one tries to trigger the DAG for execution, the response code returned is 500 with the following mes...The airflow DAG indicates that the DAG is running. Upon registering the workflow one gets the 409 conflict message as expected, But when one tries to trigger the DAG for execution, the response code returned is 500 with the following message:
{
"code": 500,
"reason": "Failed to send the request.",
"message": "Unable to send request to Airflow. 500 INTERNAL SERVER ERROR___<!DOCTYPE html>_<html lang=\"en\">_ <head>_ <link rel=\"stylesheet\" href=\"https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css\">_ </head>_ <body>_ <div class=\"container\">_ <h1> Ooops! </h1>_ <div>_ <pre>_Something bad has happened.__Airflow is used by many users, and it is very likely that others had similar problems and you can easily find_a solution to your problem.__Consider following these steps:__ * gather the relevant information (detailed logs with errors, reproduction steps, details of your deployment)__ * find similar issues using:_ .... }Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/353CRS reference values missing from GCP pre-ship R3 M13 environment when testin...2022-10-13T03:47:48ZKamlesh TodaiCRS reference values missing from GCP pre-ship R3 M13 environment when testing with V3 collectionsUsing
https://community.opengroup.org/osdu/platform/testing/-/blob/master/Postman%20Collection/16_CICD_Setup_%20CRSCatalogServiceAPI/CRS%20Catalog%20Service%20API%20V3.skip_postman_collection.json
https://community.opengroup.org/osdu/pl...Using
https://community.opengroup.org/osdu/platform/testing/-/blob/master/Postman%20Collection/16_CICD_Setup_%20CRSCatalogServiceAPI/CRS%20Catalog%20Service%20API%20V3.skip_postman_collection.json
https://community.opengroup.org/osdu/platform/testing/-/blob/master/Postman%20Collection/18_CICD_Setup_CRSConversionAPI/CRS%20Conversion%20Service%20V3.skip_postman_collection.json
These are passing on AWS and Azure, but are failing on GCP because the reference values are not present.
Notes from Debasis email -
(Drill down to FIXED, LOCAL, OPEN for exact JSON files)
https://community.opengroup.org/osdu/data/data-definitions/-/tree/master/ReferenceValues/Manifests/reference-data
If the reference data is sourced from here, then we can get good coverage.
Note to Dzmitry – It is not any collection. We request population of standard set of Reference Values from Data Definition team.
In turn, that will help validation of CRS Catalog V3 and CRS Conversion V3 services.Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/342GCP - M13 - bulk log data load error2022-08-31T13:04:50ZGrant MarblestoneGCP - M13 - bulk log data load errorHowdy,
Trying to post log data via Postman to an existing log record but getting error when posting bulk data.
Creating log using
curl --location --request POST 'https://preship.gcp.gnrg-osdu.projects.epam.com/api/os-wellbore-ddms/ddms...Howdy,
Trying to post log data via Postman to an existing log record but getting error when posting bulk data.
Creating log using
curl --location --request POST 'https://preship.gcp.gnrg-osdu.projects.epam.com/api/os-wellbore-ddms/ddms/v3/welllogs' \
--header 'data-partition-id: odesprod' \
--header 'Authorization: Bearer xxxxx' \
--header 'Content-Type: application/json' \
--data-raw '[{
"acl": {
"viewers": [
"data.default.viewers@odesprod.osdu-gcp.go3-nrg.projects.epam.com"
],
"owners": [
"data.default.owners@odesprod.osdu-gcp.go3-nrg.projects.epam.com"
]
},
"data": {
"Curves": [
{
"CurveID": "GR_ID",
"Mnemonic": "GR",
"CurveUnit": "odesprod:reference-data--UnitOfMeasure:m:",
"LogCurveFamilyID": "odesprod:reference-data--LogCurveFamily:GammaRay:"
},
{
"CurveID": "POR_ID",
"Mnemonic": "NPOR",
"CurveUnit": "odesprod:reference-data--UnitOfMeasure:m:",
"LogCurveFamilyID": "odesprod:reference-data--LogCurveFamily:NeutronPorosity:"
},
{
"CurveID": "Bulk Density",
"Mnemonic": "RHOB",
"CurveUnit": "odesprod:reference-data--UnitOfMeasure:m:",
"LogCurveFamilyID": "odesprod:reference-data--LogCurveFamily:BulkDensity:"
}
],
"WellboreID": "odesprod:master-data--Wellbore:AutoTest_999951556951:",
"CreationDateTime": "2013-03-22T11:16:03Z",
"VerticalMeasurement": {
"VerticalMeasurement": 2680.5,
"VerticalMeasurementPathID": "odesprod:reference-data--VerticalMeasurementPath:MD:",
"VerticalMeasurementUnitOfMeasureID": "odesprod:reference-data--UnitOfMeasure:ft:"
},
"TopMeasuredDepth": 12345.6,
"BottomMeasuredDepth": 13856.25,
"Name": "AutoTest_999951556951",
"ExtensionProperties": {
"step": {
"unitKey": "ft",
"value": 0.1
},
"dateModified": "2013-03-22T11:16:03Z"
}
},
"id": "odesprod:work-product-component--WellLog:AutoTest_999951556951",
"kind": "osdu:wks:work-product-component--WellLog:1.1.0",
"legal": {
"legaltags": [
"odesprod-Wellbore-Legal-Tag-Test999951556951"
],
"otherRelevantDataCountries": [
"US",
"FR"
]
},
"meta": [{
"kind": "Unit",
"name": "ft",
"persistableReference": "{\"scaleOffset\":{\"scale\":0.3048,\"offset\":0.0},\"symbol\":\"ft\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}",
"propertyNames": [
"stop.value",
"elevationReference.elevationFromMsl.value",
"start.value",
"step.value",
"reference.unitKey"
],
"propertyValues": [
"ft"
]
}, {
"kind": "DateTime",
"name": "datetime",
"persistableReference": "{\"format\":\"yyyy-MM-ddTHH:mm:ssZ\",\"timeZone\":\"UTC\",\"type\":\"DTM\"}",
"propertyNames": [
"dateModified",
"dateCreated"
]
}
]
}
]'
trying to post using
https://preship.gcp.gnrg-osdu.projects.epam.com/api/os-wellbore-ddms/ddms/v3/welllogs/odesprod:work-product-component--WellLog:AutoTest_999951556951/data
{
"detail": "The number of columns for curve(s): {'GR_ID': 1, 'POR_ID': 1, 'Bulk Density': 1} in the bulk data do(es) not match the 'NumberOfColumns' property value in the WellLog record for CurveID: {'GR_ID': None, 'POR_ID': None, 'Bulk Density': None} ."
}
thanks,
GrantLuc YriarteDzmitry Malkevich (EPAM)Luc Yriartehttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/345GCP - M13 - Ingestion by Reference failure2022-09-02T11:40:27ZDebasis ChatterjeeGCP - M13 - Ingestion by Reference failurePlease see enclosed files.
Steps
[M13-GCP-Ingest-by-Reference-3-master-records-steps-Debasis.txt](/uploads/d692af3f9bf35e6150d4c3d81e1773a0/M13-GCP-Ingest-by-Reference-3-master-records-steps-Debasis.txt)
Airflow log showing failure
[M1...Please see enclosed files.
Steps
[M13-GCP-Ingest-by-Reference-3-master-records-steps-Debasis.txt](/uploads/d692af3f9bf35e6150d4c3d81e1773a0/M13-GCP-Ingest-by-Reference-3-master-records-steps-Debasis.txt)
Airflow log showing failure
[M13-GCP-Ingest-by-Ref-test-25Aug-failure-Airflow-log.txt](/uploads/aed224f5deb22736ca7349ce98bd27ca/M13-GCP-Ingest-by-Ref-test-25Aug-failure-Airflow-log.txt)
Not obvious where from this is finding PreLoadFilePath.
I checked the Dataset record which I created and this property is not mentioned.
Please check.Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/349R3M13 GCP - CSV Ingestion - Airflow logs update issue2022-08-31T12:39:47ZSamiullah GhousudeenR3M13 GCP - CSV Ingestion - Airflow logs update issue**R3M13 GCP - CSV Ingestion - Airflow logs update issue**
There's no log updates in Airflow for successful CSV ingestion workflow.
As trying to fetch the ingested record id but I can see only below details from Airflow log for "runId"...**R3M13 GCP - CSV Ingestion - Airflow logs update issue**
There's no log updates in Airflow for successful CSV ingestion workflow.
As trying to fetch the ingested record id but I can see only below details from Airflow log for "runId": "test_workflow375096" -
`*** Reading remote log from gs://us-central1-airflow-v2-36b0ff41-bucket/logs/csv_ingestion/csv-parser/2022-08-30T16:07:44.008522+00:00/1.log.`Yan Sushchynski (EPAM)Dzmitry Malkevich (EPAM)Yan Sushchynski (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/280GCP - M11 - SEGY to ZGY conversion DAG fails2022-08-23T11:25:06ZRaman SinghGCP - M11 - SEGY to ZGY conversion DAG failsSEGY to ZGY conversion DAG fails on GCP. See attached Airflow Log.
[gcp_airflow_log_runid_30f6aa12-aafe-40ae-afe0-2419446b6296.txt](/uploads/8bb7c878d3be0bb2ff7749a0709853d7/gcp_airflow_log_runid_30f6aa12-aafe-40ae-afe0-2419446b6296.txt)SEGY to ZGY conversion DAG fails on GCP. See attached Airflow Log.
[gcp_airflow_log_runid_30f6aa12-aafe-40ae-afe0-2419446b6296.txt](/uploads/8bb7c878d3be0bb2ff7749a0709853d7/gcp_airflow_log_runid_30f6aa12-aafe-40ae-afe0-2419446b6296.txt)M12 - Release 0.15Yan Sushchynski (EPAM)Dzmitry Malkevich (EPAM)Yan Sushchynski (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/276GCP - M11 - Core Smoke Collection/Unit - Not getting a desired answer in the ...2022-08-23T11:25:08ZJuliana Fernandesjuliana.fernandes@iesbrazil.com.brGCP - M11 - Core Smoke Collection/Unit - Not getting a desired answer in the requestHello,
I'm trying to do the basic check on the Core Smoke Collection and I'm not getting the right response in one request from Unit.
The request is Unit -> POST Retrieve the same record by using Search Service
```
curl --location --re...Hello,
I'm trying to do the basic check on the Core Smoke Collection and I'm not getting the right response in one request from Unit.
The request is Unit -> POST Retrieve the same record by using Search Service
```
curl --location --request POST 'https://preship.gcp.gnrg-osdu.projects.epam.com/api/search/v2/query'
```
The body synthax is:
```
{
"kind": "{{authority}}:{{schemaSource}}:master-data--SeismicAcquisitionSurvey:1.0.0",
"query": "id: \"{{recordID}}\"",
"returnedFields": ["kind", "id", "data"]
}
```
And the response is:
```
{
"results": [],
"aggregations": [],
"totalCount": 0
}
```
The test results are:
![GCP_result](/uploads/8e9ee7a004813f7e56cbdb4daca7c70d/GCP_result.PNG)
Important to note that this request worked fine in AWS and IBM.
I'm using the collection and the environment provided here (QA Artifacts):
https://community.opengroup.org/osdu/platform/pre-shipping/-/tree/main/R3-M11/QA_Artifacts_M11
Regards,
Juliana FernandesDzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/274GCP - M11 - Setting Spotfire to connect to OSDU2022-08-23T11:25:10ZJuliana Fernandesjuliana.fernandes@iesbrazil.com.brGCP - M11 - Setting Spotfire to connect to OSDUHello,
I'm trying to set the variables to connect Spotfire into OSDU and I think I'm having problem with the Callback Template. Please, could you provide to me the correct variables for:
- Discovery URL
- Callback Template
- Ports
Att...Hello,
I'm trying to set the variables to connect Spotfire into OSDU and I think I'm having problem with the Callback Template. Please, could you provide to me the correct variables for:
- Discovery URL
- Callback Template
- Ports
Attached you will find a screen shot from my settings in Spotfire.
Regards,
Juliana Fernandes!
[GCP_Spotfire](/uploads/538d39a02b82007586e95437fc138a7f/GCP_Spotfire.PNG)Denis Karpenok (EPAM)Dzmitry Malkevich (EPAM)Denis Karpenok (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/253Spotfire connector GCP2022-08-23T11:26:58ZEsmira RafigayevaSpotfire connector GCPCould you advise if the authentication details can be shared with Andrew (Tibco) in order to proceed with the testing? Thank you, Esmira @dzmitry_malkevich @Kateryna_Kurach @Denis_Karpenok cc: @debasisc @aberridg @sehuboyCould you advise if the authentication details can be shared with Andrew (Tibco) in order to proceed with the testing? Thank you, Esmira @dzmitry_malkevich @Kateryna_Kurach @Denis_Karpenok cc: @debasisc @aberridg @sehuboyM10 - Release 0.13Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/209In GCP R3 M10 environment having trouble with Seismic DDMS and performing seg...2022-03-04T18:34:53ZKamlesh TodaiIn GCP R3 M10 environment having trouble with Seismic DDMS and performing segy to zgy conversionR3M10 GCP pre-ship environment
Problem with Seismic DDMS and segy to zgy conversion
1) Cannot create new subproject under **odesprod** tenant using **sdutil**
2) Cannot access the file in the existing subproject (**autotesting**) that wa...R3M10 GCP pre-ship environment
Problem with Seismic DDMS and segy to zgy conversion
1) Cannot create new subproject under **odesprod** tenant using **sdutil**
2) Cannot access the file in the existing subproject (**autotesting**) that was created in M9
3) When I create the new tenant using Seismic DDMS, I can retrieve the tenant using Seismic DDMS API, but when I try to list the subprojects under **odesprod **using **sdutil**, I get the message of permission denied (500)
**Created a tenant (**autotest**) using Seismic DDMS API**
I can retrieve the created tenant using Seismic DDMS API
curl --location --request GET https://preship.gcp.gnrg-osdu.projects.epam.com/api/seismic-store/v3/tenant/autotest' \
--header 'Content-Type: application/json' \
--header 'data-partition-id: odesprod' \
--header 'Authorization: Bearer eyJhbGciOiJSU…t8bNiR5lakLVEstpsSZA'
Response:
{
"esd": "odesprod.osdu-gcp.go3-nrg.projects.epam.com",
"gcpid": "AutoTest_project_id",
"default_acls": "users.datalake.admins@odesprod.osdu-gcp.go3-nrg.projects.epam.com",
"name": "autotest"
}
**(sdutilenv) D:\OSDU\dev\seismic-store-sdutil-master>**python sdutil ls sd://autotest --idtoken=%ID_TOKEN%**
[500] 7 PERMISSION_DENIED: Permission denied on resource project AutoTest_project_id.**
**Then when I try to create the new subproject e.g. (**autotestkt**) I run into access problem (403)**
(sdutilenv) D:\OSDU\dev\seismic-store-sdutil-master>**python sdutil mk "sd://odesprod/autotestkt" "admin@odesprod.osdu-gcp.go3-nrg.projects.epam.com" "odesprod-openZgy-Legal-Tag-7851742" --idtoken=%ID_TOKEN%**
[1] REGIONAL
[2] MULTI_REGIONAL
[3] NEARLINE
[4] COLDLINE
Select the bucket storage class: 2
[1] ASIA
[2] EU
[3] US
Select the bucket storage location: 3
> **Registering the subproject autotestkt(tenant=odesprod) with admin@odesprod.osdu-gcp.go3-nrg.projects.epam.com as admin ...
[403] workload-seismic-store-gcp@osdu-service-prod.iam.gserviceaccount.com does not have storage.buckets.create access to the Google Cloud project.**
Also, when I try to access **segy** file listed in the existing **autotesting** subproject created in **R3M9**, I can list the files but I cannot access the files, I get the message invalid legal tag.
(sdutilenv) D:\OSDU\dev\seismic-store-sdutil-master>python **sdutil ls sd://odesprod/autotesting --idtoken=%ID_TOKEN%**
raman-test/
ST10010ZC11_PZ_PSDM_KIRCH_FULL_T.MIG_FIN.POST_STACK.3D.JS-017536.3a81a4cf-82ff-4dfc-a7b8-203db5bd4940.zgy
ST10010ZC11_PZ_PSDM_KIRCH_FULL_T.MIG_FIN.POST_STACK.3D.JS-017536.74cbacfc-9de0-4379-85c8-ea7a51a32f56.zgy
**ST10010ZC11_PZ_PSDM_KIRCH_FULL_T.MIG_FIN.POST_STACK.3D.JS-017536.segy**
ST10010ZC11_PZ_PSDM_KIRCH_FULL_T.MIG_FIN.POST_STACK.3D.JS-017536.segy.idx
ST10010ZC11_PZ_PSDM_KIRCH_FULL_T.MIG_FIN.POST_STACK.3D.JS-017536.segy.idx.bin
(sdutilenv) D:\OSDU\dev\seismic-store-sdutil-master>**python sdutil stat sd://odesprod/autotesting/ST10010ZC11_PZ_PSDM_KIRCH_FULL_T.MIG_FIN.POST_STACK.3D.JS-017536.segy --idtoken=%ID_TOKEN%**
**[404] [seismic-store-service] The legal tag 'odesprod-Wellbore-Legal-Tag-Test3816976' is not valid.**
**Also, when I try to perform segy to zgy conversion using the DAG, it fails with log message indicating that the legal is not valid.**
[2022-02-14 22:53:25,652] {pod_launcher.py:149} INFO - SDFileReader::SDFileReader: Error opening sd file:
[2022-02-14 22:53:25,653] {pod_launcher.py:149} INFO - SeismicStore::DatasetLock: Error executing an HTTP request [ HTTP 404, [seismic-store-service] The legal tag 'odesprod-Wellbore-Legal-Tag-Test3816976' is not valid. ]
[2022-02-14 22:53:25,653] {pod_launcher.py:149} INFO - SDFileReader::SDFileReader: Error opening sd file: SeismicStore::DatasetLock: **Error executing an HTTP request [ HTTP 404, [seismic-store-service] The legal tag 'odesprod-Wellbore-Legal-Tag-Test3816976' is not valid. ]**
[2022-02-14 22:53:26,897] {taskinstance.py:615} DEBUG - Refreshing TaskInstance <TaskInstance: Segy_to_zgy_conversion.segy-to-zgy 2022-02-14T22:51:40.800163+00:00 [running]> from DB
[2022-02-14 22:53:26,943] {taskinstance.py:650} DEBUG - Refreshed TaskInstance <TaskInstance: Segy_to_zgy_conversion.segy-to-zgy 2022-02-14T22:51:40.800163+00:00 [running]>
[2022-02-14 22:53:26,947] {base_job.py:227} DEBUG - [heartbeat]
@Kateryna_Kurach @Denis_Karpenok @sehuboy @debasiscM10 - Release 0.13Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/212GCP R3M10 Manifest testing - Surrogate key, entity fails schema validation2022-08-23T12:47:22ZSteven EvansGCP R3M10 Manifest testing - Surrogate key, entity fails schema validationSurrogate keys does not create any processed ids for WP or WPC, it only creates a record id for dataset, stating that the entity doesn’t pass the schema validation.
skipped_ids [{'id': 'surrogate-key:sje-wpc-1', 'kind': 'osdu:wks:work-pr...Surrogate keys does not create any processed ids for WP or WPC, it only creates a record id for dataset, stating that the entity doesn’t pass the schema validation.
skipped_ids [{'id': 'surrogate-key:sje-wpc-1', 'kind': 'osdu:wks:work-product-component--WellboreMarkerSet:1.0.0', 'reason': "Entity doesn't pass the schema validation."}]
Storage service returns the dataset and file retrieval service returns the loaded csv as expected.[GCP_R3M10_Manifest_Ingestion_Suurogate_Key_file_retrieval_WP_and_WPC_ids_not_created.txt](/uploads/47c575687b21364a7f7ed29992a54c36/GCP_R3M10_Manifest_Ingestion_Suurogate_Key_file_retrieval_WP_and_WPC_ids_not_created.txt)Kateryna Kurach (EPAM)Dzmitry Malkevich (EPAM)Kateryna Kurach (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/213GCP R3M10 Manifest ingestion - Frame of Reference, fails to create record ids2022-08-23T12:47:15ZSteven EvansGCP R3M10 Manifest ingestion - Frame of Reference, fails to create record idsFrame of Reference (unit conversion) fails to create any record ids due to “missing parents”
skipped_ids {'provide_manifest_integrity_task': [{'id': 'odesprod:master-data--SeismicAcquisitionSurvey:ST0202R08-Steve-17Feb', 'kind': 'osdu:wk...Frame of Reference (unit conversion) fails to create any record ids due to “missing parents”
skipped_ids {'provide_manifest_integrity_task': [{'id': 'odesprod:master-data--SeismicAcquisitionSurvey:ST0202R08-Steve-17Feb', 'kind': 'osdu:wks:master-data--SeismicAcquisitionSurvey:1.0.0', 'reason': 'Missing parents: {SRN: odesprod:reference-data--UnitOfMeasure:ms, SRN: odesprod:reference-data--UnitOfMeasure:ft}'}]}
This workflow replicates that of Esmira’s successful run performed for GCP R3M9 and uploaded in the test plan folder for R3M9.[GCP_R3M10_Manifest_Ingestion_Frame_of_Reference_Unit_conversion_Missing_Parents_Failed.txt](/uploads/b64ce35e50b98952791d79d3602c8cdf/GCP_R3M10_Manifest_Ingestion_Frame_of_Reference_Unit_conversion_Missing_Parents_Failed.txt)Kateryna Kurach (EPAM)Dzmitry Malkevich (EPAM)Kateryna Kurach (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/236GCP 50K Performance Loading Fails2022-08-23T11:26:57ZSehubo AkinyanmiGCP 50K Performance Loading Fails**Please can you help investigate why 50K performance loading is failing for GCP?**
**DAG Run ID for further investigation is fdcdf45b-76be-4060-80f5-3d6e0023acd6**
I have attached a sample of the 1K record which passed OK. Same code w...**Please can you help investigate why 50K performance loading is failing for GCP?**
**DAG Run ID for further investigation is fdcdf45b-76be-4060-80f5-3d6e0023acd6**
I have attached a sample of the 1K record which passed OK. Same code was used to generate the 50K manifest records.[odesprod_master_data_Organisation_recs1000_2022-02-28_14-46-22.json](/uploads/ca4d049542a4040d6348fb13043f436d/odesprod_master_data_Organisation_recs1000_2022-02-28_14-46-22.json)
Below is the interface error observed towards the end.
CURRENT DAG RUN STATUS: running
INFO:root:Wait 10 sec ... check DAG Osdu_ingest for runId: fdcdf45b-76be-4060-80f5-3d6e0023acd6 ...
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): preship.gcp.gnrg-osdu.projects.epam.com:443
DEBUG:urllib3.connectionpool:https://preship.gcp.gnrg-osdu.projects.epam.com:443 "GET /api/workflow/v1/workflow/Osdu_ingest/workflowRun/fdcdf45b-76be-4060-80f5-3d6e0023acd6 HTTP/1.1" 200 None
DEBUG:root:HTTP GET https://preship.gcp.gnrg-osdu.projects.epam.com/api/workflow/v1/workflow/Osdu_ingest/workflowRun/fdcdf45b-76be-4060-80f5-3d6e0023acd6
DEBUG:root:args = {"headers": {"data-partition-id": "odesprod", "Accept": "application/json", "Authorization": "Bearer ya29.A0ARrdaM-4rHRs0iYoZQ852a9pS69xDmdao1JF3L-xCO5W_m7FSGLGnB4yKcIiUXRDeVfGJSLsDkjZwBd-gEQMjo_oH8WapBkECE2rgPjHq7bMjA27sjU5Qe4Wzy0gzzbXWU29KguWhQhl0eEVMSqN_bCamLBfxQ"}, "verify": false}
DEBUG:root:Response: 200
DEBUG:root:json = {"workflowId": "ef82cba0-0e45-4df3-91bf-4df1553102d3", "runId": "fdcdf45b-76be-4060-80f5-3d6e0023acd6", "startTimeStamp": 1646253900184, "status": "running", "submittedBy": "preshipping_test_user@osdu-gcp.go3-nrg.projects.epam.com"}
CURRENT DAG RUN STATUS: running
INFO:root:Wait 10 sec ... check DAG Osdu_ingest for runId: fdcdf45b-76be-4060-80f5-3d6e0023acd6 ...
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): preship.gcp.gnrg-osdu.projects.epam.com:443
DEBUG:urllib3.connectionpool:https://preship.gcp.gnrg-osdu.projects.epam.com:443 "GET /api/workflow/v1/workflow/Osdu_ingest/workflowRun/fdcdf45b-76be-4060-80f5-3d6e0023acd6 HTTP/1.1" 200 None
DEBUG:root:HTTP GET https://preship.gcp.gnrg-osdu.projects.epam.com/api/workflow/v1/workflow/Osdu_ingest/workflowRun/fdcdf45b-76be-4060-80f5-3d6e0023acd6
DEBUG:root:args = {"headers": {"data-partition-id": "odesprod", "Accept": "application/json", "Authorization": "Bearer ya29.A0ARrdaM-4rHRs0iYoZQ852a9pS69xDmdao1JF3L-xCO5W_m7FSGLGnB4yKcIiUXRDeVfGJSLsDkjZwBd-gEQMjo_oH8WapBkECE2rgPjHq7bMjA27sjU5Qe4Wzy0gzzbXWU29KguWhQhl0eEVMSqN_bCamLBfxQ"}, "verify": false}
DEBUG:root:Response: 200
DEBUG:root:json = {"workflowId": "ef82cba0-0e45-4df3-91bf-4df1553102d3", "runId": "fdcdf45b-76be-4060-80f5-3d6e0023acd6", "startTimeStamp": 1646253900184, "status": "running", "submittedBy": "preshipping_test_user@osdu-gcp.go3-nrg.projects.epam.com"}
CURRENT DAG RUN STATUS: running
INFO:root:Wait 10 sec ... check DAG Osdu_ingest for runId: fdcdf45b-76be-4060-80f5-3d6e0023acd6 ...
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): preship.gcp.gnrg-osdu.projects.epam.com:443
DEBUG:urllib3.connectionpool:https://preship.gcp.gnrg-osdu.projects.epam.com:443 "GET /api/workflow/v1/workflow/Osdu_ingest/workflowRun/fdcdf45b-76be-4060-80f5-3d6e0023acd6 HTTP/1.1" 200 None
DEBUG:root:HTTP GET https://preship.gcp.gnrg-osdu.projects.epam.com/api/workflow/v1/workflow/Osdu_ingest/workflowRun/fdcdf45b-76be-4060-80f5-3d6e0023acd6
DEBUG:root:args = {"headers": {"data-partition-id": "odesprod", "Accept": "application/json", "Authorization": "Bearer ya29.A0ARrdaM-4rHRs0iYoZQ852a9pS69xDmdao1JF3L-xCO5W_m7FSGLGnB4yKcIiUXRDeVfGJSLsDkjZwBd-gEQMjo_oH8WapBkECE2rgPjHq7bMjA27sjU5Qe4Wzy0gzzbXWU29KguWhQhl0eEVMSqN_bCamLBfxQ"}, "verify": false}
DEBUG:root:Response: 200
DEBUG:root:json = {"workflowId": "ef82cba0-0e45-4df3-91bf-4df1553102d3", "runId": "fdcdf45b-76be-4060-80f5-3d6e0023acd6", "startTimeStamp": 1646253900184, "endTimeStamp": 1646254465687, "status": "failed", "submittedBy": "preshipping_test_user@osdu-gcp.go3-nrg.projects.epam.com"}
CURRENT DAG RUN STATUS: failed
<osdu_client.OsduClient object at 0x0000022932DCA470>
>>>[odesprod_master_data_Organisation_recs1000_2022-02-28_14-46-22.json](/uploads/d2d334fe143642504ce3171036ff1288/odesprod_master_data_Organisation_recs1000_2022-02-28_14-46-22.json)M10 - Release 0.13Aliaksandr Ramanovich (EPAM)Yan Sushchynski (EPAM)Kateryna Kurach (EPAM)Dzmitry Malkevich (EPAM)Aliaksandr Ramanovich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/240GCP - Well Delivery DMS | Problems to run the use cases (UC)2022-08-23T12:47:11ZJuliana Fernandesjuliana.fernandes@iesbrazil.com.brGCP - Well Delivery DMS | Problems to run the use cases (UC)I`m trying to run the UC1 from the Well Delivery and I`m getting the following error (Status: 400 Bad Request):
{
"message": "Entity type in API(well ) and body(well) are not same."
}
I used the collection provided by AWS, once the...I`m trying to run the UC1 from the Well Delivery and I`m getting the following error (Status: 400 Bad Request):
{
"message": "Entity type in API(well ) and body(well) are not same."
}
I used the collection provided by AWS, once there is not a collection in the Platform Validation. The Schemas ran well!Kamlesh TodaiDzmitry Malkevich (EPAM)Kamlesh Todaihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/229GCP - Bad Gateway issue - Witsml parser - dataset storage instructions2022-08-23T12:47:10Zetienne peyssonGCP - Bad Gateway issue - Witsml parser - dataset storage instructions![image](/uploads/38c581e1ee5d94c11a860d1f110cb4ac/image.png)
![image](/uploads/0b0fa7e848d6f307109113c7af42188a/image.png)![image](/uploads/38c581e1ee5d94c11a860d1f110cb4ac/image.png)
![image](/uploads/0b0fa7e848d6f307109113c7af42188a/image.png)M10 - Release 0.13Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/214GCP Version Endpoint Validation - Issue2022-08-23T11:26:55ZSamiullah GhousudeenGCP Version Endpoint Validation - Issue
GCP Version Endpoint Validation - Kindly check below findings and update environment file and postman collections if valid.
1. Indexer-queue Version - INDEXER_QUEUE_HOST variable incorrect in environment file
Getting response with IND...
GCP Version Endpoint Validation - Kindly check below findings and update environment file and postman collections if valid.
1. Indexer-queue Version - INDEXER_QUEUE_HOST variable incorrect in environment file
Getting response with INDEXER_QUEUE_HOST - preship.gcp.gnrg-osdu.projects.epam.com/api/indexer/v2
whereas in environment file, value is INDEXER_QUEUE_HOST - preship.gcp.gnrg-osdu.projects.epam.com , which returns "NULL" response
2. Unit Version - UNIT_HOST requires update in environment file
Works fine with UNIT_HOST - preship.gcp.gnrg-osdu.projects.epam.com/api/unit
whereas in environment file, value is UNIT_HOST - preship.gcp.gnrg-osdu.projects.epam.com/api/unit/v3 , which returns "401-Unauthorized" response.
3. Workflow Version - WORKFLOW_HOST requires update in environment file
Works fine with WORKFLOW_HOST - preship.gcp.gnrg-osdu.projects.epam.com/api/workflow
but in environment file, value is WORKFLOW_HOST - preship.gcp.gnrg-osdu.projects.epam.com/api/workflow/v1 , which returns "404-Not Found" response.
4. CRS Catalog Version - Update postman collection as this request requires "Data-Partition-ID" as header .
without data-partition-id header returns "401-Unauthorized" response
curl --location --request GET 'https://preship.gcp.gnrg-osdu.projects.epam.com/api/crs/catalog/v2/info' \
--header 'data-partition-id: odesprod'M10 - Release 0.13Denis Karpenok (EPAM)Dzmitry Malkevich (EPAM)Denis Karpenok (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/211M10 Airflow Login Details2022-02-16T13:27:23ZSehubo AkinyanmiM10 Airflow Login DetailsHello, please can you send me an email with the authorization details to access the Airflow console?
Please email **sehubo.akinyanmi2@uk.bp.com** the Airflow URL and Login Info.
Thanks
SehuboHello, please can you send me an email with the authorization details to access the Airflow console?
Please email **sehubo.akinyanmi2@uk.bp.com** the Airflow URL and Login Info.
Thanks
SehuboM10 - Release 0.13Aliaksandr Ramanovich (EPAM)Kateryna Kurach (EPAM)Dzmitry Malkevich (EPAM)Aliaksandr Ramanovich (EPAM)2022-02-16https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/202Request to add redirect URL to the GCP instance for testing TIBCO Spotfire co...2022-01-26T07:24:23ZVinoth ManamalaRequest to add redirect URL to the GCP instance for testing TIBCO Spotfire connector for OSDUWe are trying to connect TIBCO Spotfire to your R3M9 OSDU environment on your GCP instance using custom OSDU connector. We are running to the following error. "Error 400: redirect_uri_mismatch".
Our ask is , is there some way you can ad...We are trying to connect TIBCO Spotfire to your R3M9 OSDU environment on your GCP instance using custom OSDU connector. We are running to the following error. "Error 400: redirect_uri_mismatch".
Our ask is , is there some way you can add the following Redirect URLs to your GCP instance http://127.0.0.1:55932/authorize/code and http://127.0.0.1:55931/authorize/code ?
If the above doesn't work for some reason, what is the best way to deal with this?Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/199GCP - Version Endpoint for Indexer2022-01-17T12:03:21ZSehubo AkinyanmiGCP - Version Endpoint for IndexerRelated to #198.
Please can you check on why Indexer is not returning version info?
GET https://preship.gcp.gnrg-osdu.projects.epam.com/api/indexer/v2/info
{
"code": 500,
"reason": "Internal server error",
"message": "An erro...Related to #198.
Please can you check on why Indexer is not returning version info?
GET https://preship.gcp.gnrg-osdu.projects.epam.com/api/indexer/v2/info
{
"code": 500,
"reason": "Internal server error",
"message": "An error occurred when accessing third-party APIs"
}
Response Body
{"code":500,"reason":"Internal server error","message":"An error occurred when accessing third-party APIs"}M9 - Release 0.12Aliaksandr Ramanovich (EPAM)Kateryna Kurach (EPAM)Dzmitry Malkevich (EPAM)Aliaksandr Ramanovich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/197GCP - R3M9 Patch2 - Manifest-Ingestion - Master data with F0R (CRS conversion...2022-08-23T12:47:07ZDebasis ChatterjeeGCP - R3M9 Patch2 - Manifest-Ingestion - Master data with F0R (CRS conversion) problemPlease see enclosed test case.
I had a clean run earlier in R3M8.
Now it does not seem to do proper conversion.
[OSDU_PTP_M9-Patch2-Manifest-Ingestion-Master-w-CRS-conversion.txt](/uploads/8d61d69933f41ddad1582649652da72c/OSDU_PTP_M9-...Please see enclosed test case.
I had a clean run earlier in R3M8.
Now it does not seem to do proper conversion.
[OSDU_PTP_M9-Patch2-Manifest-Ingestion-Master-w-CRS-conversion.txt](/uploads/8d61d69933f41ddad1582649652da72c/OSDU_PTP_M9-Patch2-Manifest-Ingestion-Master-w-CRS-conversion.txt)Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/196GCP R3M9 - File service - created record cannot be found from Search2022-08-23T12:47:23ZDebasis ChatterjeeGCP R3M9 - File service - created record cannot be found from SearchPlease check this record.
"id": "odesprod:dataset--File.Generic:b6e3f4a4-f3ce-4a95-bda4-0a49f0f680ad"
Created using
POST {{FILE_HOST}}/files/metadata
I can retrieve using FILE Service
GET {{FILE_HOST}}/files/{{FILE_ID}}/metadata
I c...Please check this record.
"id": "odesprod:dataset--File.Generic:b6e3f4a4-f3ce-4a95-bda4-0a49f0f680ad"
Created using
POST {{FILE_HOST}}/files/metadata
I can retrieve using FILE Service
GET {{FILE_HOST}}/files/{{FILE_ID}}/metadata
I can also retrieve using Storage service (GET).
GET https://{{STORAGE_HOST}}/records/odesprod:dataset--File.Generic:b6e3f4a4-f3ce-4a95-bda4-0a49f0f680ad
But cannot find from Search.
Tried to reindex.
Even after that count of record for this entity stays stuck at 84.
Also check this record 'odesprod:dataset--File.Generic:DC-FILE12Jan'
I created this from Manifest-based Ingestion.Riabokon Stanislav(EPAM)[GCP]Dzmitry Malkevich (EPAM)Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/195GCP R3M9-patch - probem running sdutil2022-08-23T12:46:59ZDebasis ChatterjeeGCP R3M9-patch - probem running sdutilFrom command line (Windows), I get this problem now. This setup was working earlier on 10-Dec-2021.
C:\seismic-store-sdutil-master>python sdutil ls sd://dctest10dec/dcproj --idtoken=%ID_TOKEN%
No module named 'alive_progress'
C:\seismi...From command line (Windows), I get this problem now. This setup was working earlier on 10-Dec-2021.
C:\seismic-store-sdutil-master>python sdutil ls sd://dctest10dec/dcproj --idtoken=%ID_TOKEN%
No module named 'alive_progress'
C:\seismic-store-sdutil-master>python sdutil stat sd://dctest10dec/dcproj/osdu-volve.segy --idtoken=%ID_TOKEN%
No module named 'alive_progress'
C:\seismic-store-sdutil-master>
=======
Later I requested my colleague @kennethliew to check from his end. He gets a different problem.
(sdutilenv) C: \seismic-store-sdutil-master>python sdutil ls sd://tenant-ken-test1 --idtoken=%ID_TOKEN%
HTTPSConnectionPool(host='preship.osdu.club', port=443): Max retries exceeded with url: /api/seismic-store/v3/utility/ls?sdpath=sd%3A%2F%2Ftenant-ken-test1 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x000002960DBCFF10>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))
Please check and advise.
cc - @Denis_Karpenok , @aliaksandr_ramanovich and @Yan_Sushchynski for informationDzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/133GCP R3M9 - CSV Ingestion (custom schema) - unable to retrieve created record ...2022-08-23T12:47:20ZDebasis ChatterjeeGCP R3M9 - CSV Ingestion (custom schema) - unable to retrieve created record by using Search serviceUsed collection-31 from Platform Validation.
https://community.opengroup.org/osdu/platform/testing/-/tree/master/Postman%20Collection
Records get created and I can retrieve by using Storage service.
But I cannot retrieve by using Search...Used collection-31 from Platform Validation.
https://community.opengroup.org/osdu/platform/testing/-/tree/master/Postman%20Collection
Records get created and I can retrieve by using Storage service.
But I cannot retrieve by using Search Service.
cc - @todaiks , @sehuboy , @aliaksandr_ramanovich
See enclosed.
Steps -
[GCP-R3M9-CSV-custom-schema-steps.txt](/uploads/5cde156a9ae9f0b30779ebc993e72bf9/GCP-R3M9-CSV-custom-schema-steps.txt)
Airflow console log
[GCP-R3M9-CSV-custom-Airflow-log.txt](/uploads/4eaaa71655569a82286c0948dc37dd4b/GCP-R3M9-CSV-custom-Airflow-log.txt)
Source CSV file
[wellbore.csv](/uploads/ef74695aad0cd00b65b45446e72896c3/wellbore.csv)
For troubleshooting, I removed CRS and DateTime from meta block, left only "Unit".
Conversion status (Storage) shows success, still record cannot be retrieved by using Search.
```
"conversionStatuses": [
{
"id": "odesprod:testEntity_284349:wellbore-U1A0MzU4NDQ5MjEyODgxMDAxUzBCMQ",
"status": "SUCCESS",
"errors": []
}
]
```Debasis ChatterjeeDebasis Chatterjeehttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/142GCP R3M9 - Seismic DDMS (from Smoke Test Collection) - issue to retrieve sub-...2022-08-23T12:47:18ZDebasis ChatterjeeGCP R3M9 - Seismic DDMS (from Smoke Test Collection) - issue to retrieve sub-project although creation was successfulStep-1 Create tenant
POST https://{{SEISMICDMS_HOST}}/tenant/{{autotestTenantid}}
Body (also check the content here)
```
{
"gcpid": "AutoTest_project_id",
"esd": "{{data-partition-id}}{{domain}}",
"default_acl": "users.datalake....Step-1 Create tenant
POST https://{{SEISMICDMS_HOST}}/tenant/{{autotestTenantid}}
Body (also check the content here)
```
{
"gcpid": "AutoTest_project_id",
"esd": "{{data-partition-id}}{{domain}}",
"default_acl": "users.datalake.admins@{{data-partition-id}}{{domain}}"
}
```
Variable SEISMICDDMSHOST seems not set properly.
The value is "preship.osdu.club/api/v3"
I get 404 (Not found) error when I execute the request.
Please check.
cc - @ramansingh , @gmarblestone , @aliaksandr_ramanovich and @Kateryna_Kurach for informationSacha BrantsDzmitry Malkevich (EPAM)Sacha Brantshttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/150Create Admin/Owner Role User for Tester2022-08-23T12:47:06ZNaufal Mohamed NooriCreate Admin/Owner Role User for TesterHi,
I am currently testing GCP console for multiple operational procedures i.e. logging, monitoring, provisioning and user management.
Currently I am using this user: preshipping_test_user@osdu-gcp.go3-nrg.projects.epam.com to conduct...Hi,
I am currently testing GCP console for multiple operational procedures i.e. logging, monitoring, provisioning and user management.
Currently I am using this user: preshipping_test_user@osdu-gcp.go3-nrg.projects.epam.com to conduct multiple tests based on guideline here: https://community.opengroup.org/osdu/documentation/-/wikis/Releases/R2.0/GCP/GCP-Operation
However, I have limited roles for this user and few tests are not allowed to be completed (i.e. monitoring/filter activity etc). Hopefully you can create a user or add appropriate roles (admin/owner) to the existing user so I can conduct the relevant test accordingly.
Let me know if you have any concerns.Naufal Mohamed NooriNaufal Mohamed Noorihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/148GCP R3M9 - Storage service - Custom schema - populated data cannot be retriev...2022-08-23T12:47:21ZDebasis ChatterjeeGCP R3M9 - Storage service - Custom schema - populated data cannot be retrieved from SearchPlease check the workflow.
[Core-services-Storage-custom-schema.txt](/uploads/581b31d340c5b4802c4e7937c8978495/Core-services-Storage-custom-schema.txt)
Watch steps - create schema, add data, retrieve using Storage service, retrieve usin...Please check the workflow.
[Core-services-Storage-custom-schema.txt](/uploads/581b31d340c5b4802c4e7937c8978495/Core-services-Storage-custom-schema.txt)
Watch steps - create schema, add data, retrieve using Storage service, retrieve using Search service.
cc @aliaksandr_ramanovichDzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/151Backup Service Host Returns 5002022-08-23T12:47:08ZNaufal Mohamed NooriBackup Service Host Returns 500Running few Backup host API:
- ScheduleBackup (POST https://{{BACKUP_HOST}}/collection)
- CheckBackupStatus (GET https://{{BACKUP_HOST}}/collection/states/platform-configuration)
- RestoreBackup (PATCH https://{{BACKUP_HOST}}/backup/v1/...Running few Backup host API:
- ScheduleBackup (POST https://{{BACKUP_HOST}}/collection)
- CheckBackupStatus (GET https://{{BACKUP_HOST}}/collection/states/platform-configuration)
- RestoreBackup (PATCH https://{{BACKUP_HOST}}/backup/v1/collection/restore/0a1f276b878c495ab047e12bf49812a4)
BACKUP_HOST: preship-asm.osdu-gcp.go3-nrg.projects.epam.com/api/backup/v1
Return the following:
`
{
"timestamp": "2021-11-29T08:58:36.655+00:00",
"status": 500,
"error": "Internal Server Error",
"message": "",
"path": "/api/backup/v1/collection"
}
`Dzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/143GCP R3M9 - Wellbore DDMS in GCP environment is failing when trying to post w...2022-08-23T11:28:55ZKamlesh TodaiGCP R3M9 - Wellbore DDMS in GCP environment is failing when trying to post welllog data and while creating wellog sessionWellbore DDMS collection https://community.opengroup.org/osdu/platform/testing/-/blob/master/Postman%20Collection/28_CICD_Setup_WellboreDDMSAPI/Wellbore%20DDMS%20CI-CD%20v3.0.postman_collection.json
is failing when trying to test v3/Well...Wellbore DDMS collection https://community.opengroup.org/osdu/platform/testing/-/blob/master/Postman%20Collection/28_CICD_Setup_WellboreDDMSAPI/Wellbore%20DDMS%20CI-CD%20v3.0.postman_collection.json
is failing when trying to test v3/Wellog API
https://preship.osdu.club/api/os-wellbore-ddms/ddms/v3/welllogs/odesprod:work-product-component--WellLog:AutoTest_999532639041/data
Response code: 500 Internal Server Error
https://preship.osdu.club/api/os-wellbore-ddms/ddms/v3/welllogs/odesprod:work-product-component--WellLog:AutoTest_999532639041/sessions
Response code: 500 Internal Server Error
Details can be found in the attached file. The same collection works on Azure and IBM platforms
[gcp.Wellbore_DDMS_CI-CD_v3_0_report_extra.html](/uploads/dc2fadae98de5bd5f8fa77be01b6720f/gcp.Wellbore_DDMS_CI-CD_v3_0_report_extra.html)
@aliaksandr_ramanovich @gmarblestone @Kateryna_Kurach @debasiscKamlesh TodaiDzmitry Malkevich (EPAM)Kamlesh Todaihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/134GCP R3M9 - Core Service (Storage) - Frame of Reference - Unit conversion issu...2022-08-23T12:47:15ZDebasis ChatterjeeGCP R3M9 - Core Service (Storage) - Frame of Reference - Unit conversion issues (created record cannot be found by using Search)Please check test case here.
[GCP-Storage-service-Unit-conversion-issue.txt](/uploads/06dc14dfe3599efe290450fdc6d1c30c/GCP-Storage-service-Unit-conversion-issue.txt)
cc - @aliaksandr_ramanovich , @todaiks , @esmira.rafigayeva , @sehubo...Please check test case here.
[GCP-Storage-service-Unit-conversion-issue.txt](/uploads/06dc14dfe3599efe290450fdc6d1c30c/GCP-Storage-service-Unit-conversion-issue.txt)
cc - @aliaksandr_ramanovich , @todaiks , @esmira.rafigayeva , @sehuboy , @gmarblestone for informationDzmitry Malkevich (EPAM)Dzmitry Malkevich (EPAM)https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/6CSV Parser -R3M7/AWS - issue loading Basin Type data2021-09-02T01:29:50Zkenneth liewCSV Parser -R3M7/AWS - issue loading Basin Type dataThe CSV Parser(DAG Portal) contains some errors when import Basin Type Master Data by CSV Ingestion.
[_R3M7_AWS_CSV_Ingestion.txt](/uploads/ddcb5b29680b8383177ebdc7c48f50e4/_R3M7_AWS_CSV_Ingestion.txt)
[BasinType.csv](/uploads/6311a0e84...The CSV Parser(DAG Portal) contains some errors when import Basin Type Master Data by CSV Ingestion.
[_R3M7_AWS_CSV_Ingestion.txt](/uploads/ddcb5b29680b8383177ebdc7c48f50e4/_R3M7_AWS_CSV_Ingestion.txt)
[BasinType.csv](/uploads/6311a0e844e869bd80e3918832e9bb24/BasinType.csv)
[CSVParserErrorLog.txt](/uploads/c71ff4835981391648a0f88544542243/CSVParserErrorLog.txt)M8 - Release 0.11https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/262Request to add redirect URL to support TIBCO OSDU connector authentication2023-10-04T11:54:49ZVinoth ManamalaRequest to add redirect URL to support TIBCO OSDU connector authenticationhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/7CSV Ingestion -R3M7/AWS - issue loading Basin Type data2022-08-28T14:53:51Zkenneth liewCSV Ingestion -R3M7/AWS - issue loading Basin Type dataThe CSV Parser(DAG Portal) contains some errors when import Basin Type Master Data by CSV Ingestion.
[_R3M7_AWS_CSV_Ingestion.txt](/uploads/ddcb5b29680b8383177ebdc7c48f50e4/_R3M7_AWS_CSV_Ingestion.txt)
[BasinType.csv](/uploads/6311a0e84...The CSV Parser(DAG Portal) contains some errors when import Basin Type Master Data by CSV Ingestion.
[_R3M7_AWS_CSV_Ingestion.txt](/uploads/ddcb5b29680b8383177ebdc7c48f50e4/_R3M7_AWS_CSV_Ingestion.txt)
[BasinType.csv](/uploads/6311a0e844e869bd80e3918832e9bb24/BasinType.csv)
[CSVParserErrorLog.txt](/uploads/c71ff4835981391648a0f88544542243/CSVParserErrorLog.txt)M12 - Release 0.15https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/9Test email issue2021-09-03T19:04:30ZSergey Krupenin (EPAM)Test email issueTest ticketTest tickethttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/11Manifest Ingestion2022-09-03T02:16:02ZSehubo AkinyanmiManifest IngestionPre-Shipping R3-M8ivar Soerheimivar Soerheimhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/12Retrieving entitlements/groups2021-09-03T14:26:39Zetienne peyssonRetrieving entitlements/groupsFor testing purposes, I'm trying to get the proper ACL groups from the entitlement endpoint as follow :
```
curl --location --request GET 'https://osdu-cpd-osdu.odi-osdu-og-fa7661852f2ab29a6be32f560b2f5573-0000.us-south.containers.appdo...For testing purposes, I'm trying to get the proper ACL groups from the entitlement endpoint as follow :
```
curl --location --request GET 'https://osdu-cpd-osdu.odi-osdu-og-fa7661852f2ab29a6be32f560b2f5573-0000.us-south.containers.appdomain.cloud/osdu-entitlements/api/entitlements/v1/groups' \
--header 'data-partition-id: opendes' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer HIDDEN'
```
But I'm reaching a 502 BAD gateway response.
I'ven't found a reference to this API in the postman collections provided.https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/31Seismic DDMS2022-08-24T14:21:41ZEsmira RafigayevaSeismic DDMSPre-Shipping R3-M8MichaelMichaelhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/14No provider found in config - Airflow - Witsml ingestion2021-09-30T05:47:17Zetienne peyssonNo provider found in config - Airflow - Witsml ingestion
Could you check if the provider is present within the osdu_api.ini ?
Here is the error I'm having once the workflow is triggered for a trajectory witsml ingestion :
```
[2021-09-06 16:15:28,601] {logging_mixin.py:112} INFO - [2021-09...
Could you check if the provider is present within the osdu_api.ini ?
Here is the error I'm having once the workflow is triggered for a trajectory witsml ingestion :
```
[2021-09-06 16:15:28,601] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,601] {pod_launcher.py:125} INFO - b'IBMBlobStorageFactory().s3_client: <botocore.client.S3 object at 0x7fcb516cb828>\n'
[2021-09-06 16:15:28,601] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,601] {pod_launcher.py:125} INFO - b'Traceback (most recent call last):\n'
[2021-09-06 16:15:28,602] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,602] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.6/configparser.py", line 1138, in _unify_values\n'
[2021-09-06 16:15:28,602] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,602] {pod_launcher.py:125} INFO - b' sectiondict = self._sections[section]\n'
[2021-09-06 16:15:28,602] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,602] {pod_launcher.py:125} INFO - b"KeyError: 'provider'\n"
[2021-09-06 16:15:28,603] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,602] {pod_launcher.py:125} INFO - b'\n'
[2021-09-06 16:15:28,603] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,603] {pod_launcher.py:125} INFO - b'During handling of the above exception, another exception occurred:\n'
[2021-09-06 16:15:28,603] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,603] {pod_launcher.py:125} INFO - b'\n'
[2021-09-06 16:15:28,603] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,603] {pod_launcher.py:125} INFO - b'Traceback (most recent call last):\n'
[2021-09-06 16:15:28,604] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,603] {pod_launcher.py:125} INFO - b' File "main.py", line 45, in <module>\n'
[2021-09-06 16:15:28,604] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,604] {pod_launcher.py:125} INFO - b' manifest = json.dumps(main(json.loads(args.context), args.file_service))\n'
[2021-09-06 16:15:28,604] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,604] {pod_launcher.py:125} INFO - b' File "main.py", line 35, in main\n'
[2021-09-06 16:15:28,723] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,722] {pod_launcher.py:125} INFO - b' manifest = manifest_creator.create_manifest(execution_context)\n'
[2021-09-06 16:15:28,723] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,723] {pod_launcher.py:125} INFO - b' File "/home/witsml_parser/create_energistics_manifest.py", line 231, in create_manifest\n'
[2021-09-06 16:15:28,723] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,723] {pod_launcher.py:125} INFO - b' dataset_dms_client = DatasetDmsClient()\n'
[2021-09-06 16:15:28,724] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,723] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.6/site-packages/osdu_api/clients/base_client.py", line 33, in __init__\n'
[2021-09-06 16:15:28,724] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,724] {pod_launcher.py:125} INFO - b' self._parse_config()\n'
[2021-09-06 16:15:28,724] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,724] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.6/site-packages/osdu_api/clients/base_client.py", line 69, in _parse_config\n'
[2021-09-06 16:15:28,724] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,724] {pod_launcher.py:125} INFO - b" self.provider = config_parser.get('provider', 'name')\n"
[2021-09-06 16:15:28,725] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,724] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.6/configparser.py", line 781, in get\n'
[2021-09-06 16:15:28,725] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,725] {pod_launcher.py:125} INFO - b' d = self._unify_values(section, vars)\n'
[2021-09-06 16:15:28,725] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,725] {pod_launcher.py:125} INFO - b' File "/usr/local/lib/python3.6/configparser.py", line 1141, in _unify_values\n'
[2021-09-06 16:15:28,725] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,725] {pod_launcher.py:125} INFO - b' raise NoSectionError(section)\n'
[2021-09-06 16:15:28,726] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,725] {pod_launcher.py:125} INFO - b"configparser.NoSectionError: No section: 'provider'\n"
[2021-09-06 16:15:28,883] {logging_mixin.py:112} INFO - [2021-09-06 16:15:28,883] {pod_launcher.py:129} INFO - Container witsml-parser-task-4fe66f0c has state running
[2021-09-06 16:15:33,305] {logging_mixin.py:112} INFO - [2021-09-06 16:15:33,305] {pod_launcher.py:217} INFO - Running command... cat /airflow/xcom/return.json
[2021-09-06 16:15:33,628] {logging_mixin.py:112} INFO - [2021-09-06 16:15:33,628] {pod_launcher.py:224} INFO - cat: can't open '/airflow/xcom/return.json': No such file or directory
```
![Screenshot_from_2021-09-06_18-58-42](/uploads/ce42ce6cc08d7092e765fcf8421d02f0/Screenshot_from_2021-09-06_18-58-42.png)Anuj GuptaAnuj Guptahttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/42Load/performance/benchmark Testing2022-08-23T12:49:23ZKamlesh TodaiLoad/performance/benchmark TestingPre-Shipping R3-M8Kamlesh TodaiKamlesh Todaihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/32Seismic DDMS2022-08-23T13:29:59ZEsmira RafigayevaSeismic DDMSPre-Shipping R3-M8Debasis ChatterjeeDebasis Chatterjeehttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/33Seismic DDMS2022-08-23T13:29:53ZEsmira RafigayevaSeismic DDMSPre-Shipping R3-M8Kamlesh TodaiKamlesh Todaihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/41Seismic DDMS Integration (oZGY)- IBM2022-08-23T13:29:52ZKamlesh TodaiSeismic DDMS Integration (oZGY)- IBMPre-Shipping R3-M8Kamlesh TodaiKamlesh Todaihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/40Seismic DDMS Intergation (oVDS) - IBM2022-08-23T13:29:51ZKamlesh TodaiSeismic DDMS Intergation (oVDS) - IBMPre-Shipping R3-M8Kamlesh TodaiKamlesh Todaihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/15Version Endpoint2022-08-23T13:29:56ZEsmira RafigayevaVersion EndpointPre-Shipping R3-M8Sehubo AkinyanmiSehubo Akinyanmihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/18Version Endpoint2022-08-23T13:29:57ZEsmira RafigayevaVersion EndpointPre-Shipping R3-M8Sehubo AkinyanmiSehubo Akinyanmihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/17Version Endpoint2022-08-23T13:29:57ZEsmira RafigayevaVersion EndpointPre-Shipping R3-M8Sehubo AkinyanmiSehubo Akinyanmihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/20Manifest Ingestion2022-09-03T02:23:32ZEsmira RafigayevaManifest IngestionPre-Shipping R3-M8Yanbin ZhangYanbin Zhanghttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/21Airflow Console/Log2022-09-03T02:23:16ZEsmira RafigayevaAirflow Console/LogPre-Shipping R3-M8ivar Soerheimivar Soerheimhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/22Airflow Console/Log2022-09-03T02:21:59ZEsmira RafigayevaAirflow Console/LogPre-Shipping R3-M8Naufal Mohamed NooriNaufal Mohamed Noorihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/23Airflow Console/Log2022-09-03T02:21:05ZEsmira RafigayevaAirflow Console/LogPre-Shipping R3-M8Yanbin ZhangYanbin Zhanghttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/25CSV Ingestion2022-09-03T02:24:04ZEsmira RafigayevaCSV IngestionPre-Shipping R3-M8Valery GinakValery Ginakhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/30Seismic DDMS - Can copy files using sdutil2022-08-23T13:29:53ZEsmira RafigayevaSeismic DDMS - Can copy files using sdutil**Seismic DDMS oZgy:**
When trying to copy file using `sdutil` user is getting
`('Connection aborted.', timeout('The write operation timed out'))` issue**Seismic DDMS oZgy:**
When trying to copy file using `sdutil` user is getting
`('Connection aborted.', timeout('The write operation timed out'))` issuePre-Shipping R3-M8Raman SinghGrant MarblestoneRaman Singhhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/44Load/performance/benchmark Testing2022-09-03T02:17:18ZKamlesh TodaiLoad/performance/benchmark TestingPre-Shipping R3-M8Kamlesh TodaiKamlesh Todaihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/43Load/performance/benchmark Testing2022-09-03T02:17:51ZKamlesh TodaiLoad/performance/benchmark TestingPre-Shipping R3-M8Kamlesh TodaiKamlesh Todaihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/45Load/performance/benchmark Testing2022-09-03T02:16:58ZKamlesh TodaiLoad/performance/benchmark TestingPre-Shipping R3-M8Kamlesh TodaiKamlesh Todaihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/29WITSML Parser2022-09-03T02:21:29ZEsmira RafigayevaWITSML ParserPre-Shipping R3-M8etienne peyssonetienne peyssonhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/28WITSML Parser2022-09-03T02:21:28ZEsmira RafigayevaWITSML ParserPre-Shipping R3-M8etienne peyssonetienne peyssonhttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/34Wellbore DDMS2022-09-03T02:18:36ZEsmira RafigayevaWellbore DDMSPre-Shipping R3-M8Debasis ChatterjeeDebasis Chatterjeehttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/35Well Delivery DDMS2022-06-02T18:45:03ZEsmira RafigayevaWell Delivery DDMSPre-Shipping R3-M8Akkaiah PrathipatiAkkaiah Prathipatihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/36Operating Procedures - Non Destructive Testing2022-08-24T14:22:03ZAkkaiah PrathipatiOperating Procedures - Non Destructive TestingPre-Shipping R3-M8Akkaiah PrathipatiAkkaiah Prathipatihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/37Operating Procedures - Non Destructive Testing2022-09-03T02:22:26ZAkkaiah PrathipatiOperating Procedures - Non Destructive TestingPre-Shipping R3-M8Akkaiah PrathipatiAkkaiah Prathipatihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/38Operating Procedures - Non Destructive Testing2022-09-03T02:18:19ZAkkaiah PrathipatiOperating Procedures - Non Destructive TestingPre-Shipping R3-M8Akkaiah PrathipatiAkkaiah Prathipatihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/39Operating Procedures - Non Destructive Testing2022-09-03T02:20:49ZAkkaiah PrathipatiOperating Procedures - Non Destructive TestingPre-Shipping R3-M8Akkaiah PrathipatiAkkaiah Prathipatihttps://community.opengroup.org/osdu/platform/pre-shipping/-/issues/26CSV Ingestion2022-09-03T02:21:43ZEsmira RafigayevaCSV IngestionPre-Shipping R3-M8Steven EvansSteven Evans