Not able to download the uploaded file using DATASET APIs
The workflow is
- Get upload instructions to upload the file (/getStorageInstructions?kindSubType=dataset--File.Genric)
- Upload the text file using the signedUrl
- Register the dataset (/registerDataset)
- Get Retrieval instructions to downdload the file (/getRetrievalInstructions) The postman collection that we have was provided by GCP and in this request for retrieval in the test tab, it is expecting to receive **unsignedURL **and connection string. It uses/parses the connection string to get the bucket_name, folder_name and file_name. It then tries to use these values in the next request to download the text file. What is noticed is that retrieval request is only returning signedUrl
- Download the text file. https://storage.googleapis.com/storage/v1/b/{{bucket_name}}/o/{{folder_name}}%2f{{file_name_1}}?alt=media and the URL (https://storage.googlsapis.com is hardcoded) Obviously request 5 fails as the variables are not parsed with any values.
Dataset_-_GCP.postman_collection.json
Request to get retrieval instructions
curl --location --request POST 'https://dev2.gcp.gnrg-osdu.projects.epam.com/api/dataset/v1/getRetrievalInstructions'
--header 'Data-Partition-Id: devtwo'
--header 'Authorization: Bearer ya29.A0ARrdaM80iLCu94gO0v5CLr1DsnP55XGUrZh8ESSdjcupn8RxzTyKyJ_PJa__D1_cqxpdjkSG7FYUyFXGLxwVu5zcxIm9LpyrZpYUNPnU8moMWxMf0aSLZ9V5rClGvJA7BRwynxJTauGE2cDj8s8mAnLTNQjWaWg'
--header 'Content-Type: application/json'
--data-raw '{
"datasetRegistryIds": [
"devtwo:dataset--File.Generic:autotest5729559"
]
}'
Response: 200 OK { "delivery": [ { "datasetRegistryId": "devtwo:dataset--File.Generic:autotest5729559", "retrievalProperties": { "signedUrl": "https://storage.googleapis.com/osdu-dev2-persistent-area/5b57190a-337a-42e4-be1b-7997de7af1d9/168c6cc9c0104b9f8959d9d01c3d6add?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=workload-file-gcp%40osdu-dev2.iam.gserviceaccount.com%2F20220331%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20220331T163449Z&X-Goog-Expires=604800&X-Goog-SignedHeaders=host&X-Goog-Signature=a174491103e0d72e49512...512f1b615", "createdBy": "kamlesh_todai@osdu-gcp.go3-nrg.projects.epam.com" }, "providerKey": "GCP" } ] }
We have started with GCP but would like this type of collection to be running in all CSPs environment. If any CSP has the collection or this workflow working in their environment then please share.
@dzmitry_malkevich @debasisc @anujgupta @krveduru @fhoueto.amz
Also please refer to the following issue also #35 (closed)