Dataset API to get storageInstructions for kindSubType=dataset--FileCollection.Generic is not consistent across all the CSPs
The implementation of GET storageInstruction for DATASET API is not consistent across all the CSPs e.g. https://{{DATASET_HOST}}/storageInstructions?kindSubType=dataset--FileCollection.Generic when it is executed the output is various among all the CSPs. The usage example provided by AWS works in AWS environment but cannot get it to work in other CSP's environments, because other CSPs are not returning the same kind of information when request is made for storageInstructions as the documentation is lacking and the working example is missing, on how to consume/use the information to create the collection of multiple files (FileCollection.Generic) using the registerDataset
Details
`AWS
curl --location --request POST 'https://r3m17ps.preshiptesting.osdu.aws/api/dataset/v1/storageInstructions?kindSubType=dataset--FileCollection.Generic'
--header 'data-partition-id: osdu'
--header 'Authorization: Bearer eyJraWQiOi...g0QA8ImMw'
Response 200 OK
{
"providerKey": "AWS_S3",
"storageLocation": {
**"unsignedUrl"**: "s3://r3m17ps-shared-561735291427-ca-central-1-file/osdu/FHugmJALXGTlPTC5VpxBxEmQfGy68jXp/",
"createdAt": "2023-05-06T11:32:37.167Z",
"connectionString": "AccessKeyId=ASIAYFSQIKIRU352WEF6;SecretAccessKey=n0QRlrK/YwDooN9+n2q8ky1whKhjFUCLK05U5aAZ;SessionToken=IQoJb3JpZ2luX2VjEBMaDGNhLWNlbnRyYWwtMSJHMEUCIQD2F8q01x6RAwLUrEhxH64m3FEVUtf058bPkrnOZfKjQwIgNuLtSmO7hIExCuSquCOq5+K1DyXgVxuiGNezuAkDzIsqpgUILRACGgw1NjE3MzUyOTE0MjciDL77xgCuhM4FLyVm3CqDBZw9vX/uBEhZ2RpwCmld1E6qSYgE2rKqdeNlv8JLGjENSq66gw3VA7Rw73UiQfq4ilOp32zf5tSvXN2P4BNZbtuv+t2bgjGHMhqH0eaQn+y5VJHMJOVJJuI7aajgFr6c0bGE2zxi9RrHX7d7luV79arzBlIO1XeeSLlDYlyPVHRB+pc4VmAXslpwNeFkHh/ih1P+6sy/VkHlmwYMTuz2q0rG0M0ME+SwK1udd0jAukLbAu0Ao2zAJ3yQ5QXq/L8DtHOx8nc4w40p/eocLcy4HOwKyY0omeM5QfQ2gCrPcBPS0jLwkUCnSjDVODffuakXhT9pALeqoX7stqDf2F2PrTxt8J+XMTRxo4tIMMtpzpiJzGpGxuqQIDlroCYf+AUmg4GxyCXm0oCNxH8uoSF8pnxsu3dQvSO8+dXf6QzOzWCmyyh9YjRrZLveITJ0Y08vcbBQGCw04HVfexhFxDOmaNf8X4PfSQr6Czwnbxl7p5S3NLVo56lxEe195+nPk9KHFOHBHHmq3du/gy2IrE0a4zMZdMG/p8vSAFvux8VUqR2dB9FJlhh3DiKq3MoBvFBo/D4MCI4JZ7v3hUJFo+fZP2ew981vz5OT1xGLH2MvY+53/jtMjYvt7HqN1HNv3SZuAzboqxfb79G7Y09ZiKjKiincxt6/FVM0jVolXXQeyaZgZMkh+asxiFlztZAL8KueGxXnHrPlsiUD+nFuWYv4Qz3vmBTNjNK1gqCczg4rgb7kM/AwNZ4U3MUanVVbXUX15T0gOeu826e2Adv+aJy5twR37Gi1+1Zp4tN4/Q+liI5ms3xQJAwg4BJpwVptNMAXaWSkojhnFvvNt8vgfQnBPy9aPdQw1fXYogY6nQFSWKU1AthSv4P/CbCc5mauZyAT8m5UYU3DoNu07Lb2+rPgiRji/H0cvCalDFPa4Typ4vzYWWjWNxOHkc3lWbmI3tI69IRBVjdTGBeA5P706YTb3h00M8CbwpTGI3WkR9FEKgSTtv0tgEZHt8dy89BqyG3R/5IMZrTS2HRMht/kDzwVoFEo6ruAA+Na89DF/V8d6T8WW45aPBZSR1lu;Expiration=2023-05-06T12:32:37Z",
"credentials": {
**"accessKeyId"**: "ASIAYFSQIKIRU352WEF6",
**"secretAccessKey"**: "n0QRlrK/YwDooN9+n2q8ky1whKhjFUCLK05U5aAZ",
**"sessionToken"**: "IQoJb3JpZ2luX2VjEBMaDGNhLWNlbnRyYWwtMSJHMEUCIQD2F8q01x6RAwLUrEhxH64m3FEVUtf058bPkrnOZfKjQwIgNuLtSmO7hIExCuSquCOq5+K1DyXgVxuiGNezuAkDzIsqpgUILRACGgw1NjE3MzUyOTE0MjciDL77xgCuhM4FLyVm3CqDBZw9vX/uBEhZ2RpwCmld1E6qSYgE2rKqdeNlv8JLGjENSq66gw3VA7Rw73UiQfq4ilOp32zf5tSvXN2P4BNZbtuv+t2bgjGHMhqH0eaQn+y5VJHMJOVJJuI7aajgFr6c0bGE2zxi9RrHX7d7luV79arzBlIO1XeeSLlDYlyPVHRB+pc4VmAXslpwNeFkHh/ih1P+6sy/VkHlmwYMTuz2q0rG0M0ME+SwK1udd0jAukLbAu0Ao2zAJ3yQ5QXq/L8DtHOx8nc4w40p/eocLcy4HOwKyY0omeM5QfQ2gCrPcBPS0jLwkUCnSjDVODffuakXhT9pALeqoX7stqDf2F2PrTxt8J+XMTRxo4tIMMtpzpiJzGpGxuqQIDlroCYf+AUmg4GxyCXm0oCNxH8uoSF8pnxsu3dQvSO8+dXf6QzOzWCmyyh9YjRrZLveITJ0Y08vcbBQGCw04HVfexhFxDOmaNf8X4PfSQr6Czwnbxl7p5S3NLVo56lxEe195+nPk9KHFOHBHHmq3du/gy2IrE0a4zMZdMG/p8vSAFvux8VUqR2dB9FJlhh3DiKq3MoBvFBo/D4MCI4JZ7v3hUJFo+fZP2ew981vz5OT1xGLH2MvY+53/jtMjYvt7HqN1HNv3SZuAzboqxfb79G7Y09ZiKjKiincxt6/FVM0jVolXXQeyaZgZMkh+asxiFlztZAL8KueGxXnHrPlsiUD+nFuWYv4Qz3vmBTNjNK1gqCczg4rgb7kM/AwNZ4U3MUanVVbXUX15T0gOeu826e2Adv+aJy5twR37Gi1+1Zp4tN4/Q+liI5ms3xQJAwg4BJpwVptNMAXaWSkojhnFvvNt8vgfQnBPy9aPdQw1fXYogY6nQFSWKU1AthSv4P/CbCc5mauZyAT8m5UYU3DoNu07Lb2+rPgiRji/H0cvCalDFPa4Typ4vzYWWjWNxOHkc3lWbmI3tI69IRBVjdTGBeA5P706YTb3h00M8CbwpTGI3WkR9FEKgSTtv0tgEZHt8dy89BqyG3R/5IMZrTS2HRMht/kDzwVoFEo6ruAA+Na89DF/V8d6T8WW45aPBZSR1lu",
"expiration": "2023-05-06T12:32:37.000+00:00"
},
"createdBy": "admin@testing.com",
**"region":** "ca-central-1"
}
}
**Using the information from the response of the previous request upload the file
Authorization Tab (Settings)
Type: AWS Signature
AccessKey
SecretKey
AWS Region
Service Name: s3
Session Token**
**curl --location --request PUT 'https://r3m17ps-shared-561735291427-ca-central-1-file.s3.amazonaws.com/osdu/FHugmJALXGTlPTC5VpxBxEmQfGy68jXp/file1' \**
--header 'Content-Type: text/csv' \
--header 'X-Amz-Security-Token: IQoJb3JpZ2luX2VjEBMaDGNhLWNlbnRyYWwtMSJHMEUCIQD2F8q01x6RAwLUrEhxH64m3FEVUtf058bPkrnOZfKjQwIgNuLtSmO7hIExCuSquCOq5+K1DyXgVxuiGNezuAkDzIsqpgUILRACGgw1NjE3MzUyOTE0MjciDL77xgCuhM4FLyVm3CqDBZw9vX/uBEhZ2RpwCmld1E6qSYgE2rKqdeNlv8JLGjENSq66gw3VA7Rw73UiQfq4ilOp32zf5tSvXN2P4BNZbtuv+t2bgjGHMhqH0eaQn+y5VJHMJOVJJuI7aajgFr6c0bGE2zxi9RrHX7d7luV79arzBlIO1XeeSLlDYlyPVHRB+pc4VmAXslpwNeFkHh/ih1P+6sy/VkHlmwYMTuz2q0rG0M0ME+SwK1udd0jAukLbAu0Ao2zAJ3yQ5QXq/L8DtHOx8nc4w40p/eocLcy4HOwKyY0omeM5QfQ2gCrPcBPS0jLwkUCnSjDVODffuakXhT9pALeqoX7stqDf2F2PrTxt8J+XMTRxo4tIMMtpzpiJzGpGxuqQIDlroCYf+AUmg4GxyCXm0oCNxH8uoSF8pnxsu3dQvSO8+dXf6QzOzWCmyyh9YjRrZLveITJ0Y08vcbBQGCw04HVfexhFxDOmaNf8X4PfSQr6Czwnbxl7p5S3NLVo56lxEe195+nPk9KHFOHBHHmq3du/gy2IrE0a4zMZdMG/p8vSAFvux8VUqR2dB9FJlhh3DiKq3MoBvFBo/D4MCI4JZ7v3hUJFo+fZP2ew981vz5OT1xGLH2MvY+53/jtMjYvt7HqN1HNv3SZuAzboqxfb79G7Y09ZiKjKiincxt6/FVM0jVolXXQeyaZgZMkh+asxiFlztZAL8KueGxXnHrPlsiUD+nFuWYv4Qz3vmBTNjNK1gqCczg4rgb7kM/AwNZ4U3MUanVVbXUX15T0gOeu826e2Adv+aJy5twR37Gi1+1Zp4tN4/Q+liI5ms3xQJAwg4BJpwVptNMAXaWSkojhnFvvNt8vgfQnBPy9aPdQw1fXYogY6nQFSWKU1AthSv4P/CbCc5mauZyAT8m5UYU3DoNu07Lb2+rPgiRji/H0cvCalDFPa4Typ4vzYWWjWNxOHkc3lWbmI3tI69IRBVjdTGBeA5P706YTb3h00M8CbwpTGI3WkR9FEKgSTtv0tgEZHt8dy89BqyG3R/5IMZrTS2HRMht/kDzwVoFEo6ruAA+Na89DF/V8d6T8WW45aPBZSR1lu' \
--header 'X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' \
--header 'X-Amz-Date: 20230506T113518Z' \
--header 'Authorization: AWS4-HMAC-SHA256 Credential=ASIAYFSQIKIRU352WEF6/20230506/ca-central-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=46fda8f99c22d1e881f37b12b94724079a1f2c6559a639fe86799a671f26cb91' \
--data '@Postman Collection/31_CICD_Setup_CSVIngestion/wellbore.csv'
Azure
curl --location --request POST 'https://osdu-ship.msft-osdu-test.org/api/dataset/v1/storageInstructions?kindSubType=dataset--FileCollection.Generic'
--header 'data-partition-id: opendes'
--header 'Authorization: Bearer eyJ0eXAiOiJ...e9tIuUDbg'
Response 200 OK
{
"providerKey": "AZURE",
"storageLocation": {
**"signedUrl"**: "https://osdumvpdp1shipbbnghiercl.dfs.core.windows.net/datalake-staging-area/osdu-user-1683373961973-2023-05-06-11-52-41-973-276de2f88be64f29861d7f23de482330?sv=2020-08-04&spr=https&st=2023-05-06T11%3A52%3A42Z&se=2023-05-13T11%3A52%3A42Z&sr=d&sp=racwl&sdd=1&sig=Si5JnpGmg15ToaXDlz3voXPPmcWdgEOYzGvjqR%2BTEdk%3D",
**"fileCollectionSource"**: "/osdu-user-1683373961973-2023-05-06-11-52-41-973-276de2f88be64f29861d7f23de482330",
"createdBy": "osdu-user"
}
}
**How to use the information to load the multiple files to create a collection**__
GC
curl --location --request POST 'https://preship.gcp.gnrg-osdu.projects.epam.com/api/dataset/v1/storageInstructions?kindSubType=dataset--FileCollection.Generic'
--header 'data-partition-id: odesprod'
--header 'Authorization: Bearer ya29.a0AWY7Ckmv_...thKfg0169'
Response 200 OK
{
"providerKey": "GCP",
"storageLocation": {
**"url"**: "https://storage.googleapis.com/osdu-data-prod-odesprod-staging-area/",
**"signingOptions"**: {
**"bucket"**: "osdu-data-prod-odesprod-staging-area",
**"connectionString"**: "ya29.dr.AfMbl1iSb-YCAWAR64myZ33i8nYVBFRLX7BT4G4G5x7CbCe6EeJBioAzBgbut_O0LVCynVd9ebQSVIwMYy_O6O22cM43CFt-AC5S_FHD_Up_OS1pDiBJ0SS-rAAK052taCvek_vAspNC1kFDEAZ-bgOMGSTe5uLYhzfjLX2v89JDPO2I_ytLQkQGGjG9sn8wR2xuSFfEfOhH3O9eGa5r9LwCAUsVus7j1ZKjvVnAoiFVBSWcg-lrOiG8yUycp-xAfHdYNErxELPq_yBeIqlgnOkF3Di5JZDNZSDQrp8n6CCnUTYIXiBgUl-E193cTHb5UwnYVatn1oMUmpN3MP76yMdCvMPbgmxLZCB1fMGQOCjNF49AwoibPZbyk1FU7jr9OlAiJ-lmGimx5V3V6Vp1NbGHLlA_UAbIcvmtFE4oOAIo9NXvIohL9p4gaktCbJbCde11xDw-87OUT7XnmBxsT7fG0HTXx6tpmdDqh9URPUhot-bj2n67fjUYirIO1EgCpQc-YNaX8pSB03DVFsdeNw33xWKJ7WiU4U0sP4gVr0iKzCjP5LFmxUpHDIcx2dHkNRKEYdDTyn3947rsXQEdYrgW-fSp1hFvwgAcRTqW_qIcWZmRt15OAoPZJompCEUnslqfH6SkSESXVlsWtME4a2mgaYvzRMCfuvtmsMQpAR8a0o6MbqxnQ6Lj5EUkMt_kqtv7CgW1XuKvkqhA4gkZPoJRKu55PtrEvjEpOgzxBbG0e4MZT68KngAtSsfO0IuG7bPXb1_KQrHBOpRYObe2RTAquj9tTGOCXnEWm-evKN3MBiHbdJsLXtbFSODcD9HqdJM8UsTBotigEAMX_7S6bkzE9JMpc9j3g2xrBbpHhDkR7iSB71TSxhLalFC_IeUTj9XBlHd3WmyqYa1JCmGyj8vrckQiR8yYfyUyHpdfFe2EtczeA10323UAnI2G8HZw2M2ViHKdBDfTk-AfruUR5iU8T-0nBko761nWJ1R7zPg1d50XCdiWAxu9g_9Q8qW-s9ss3DVAmhBb_QPUQblRIGWmawZQIUJPPpzSOGNinR9MH4R_un2vpA1j1M5zRZU4td1yzDUkyLbC2-X8GrKtZAnBwZC9GGu_6TOMwe0wIb0RQY2yGBR0-MdBIiDTVkwVyZQpGHZb6JTlSGB4sl3H_1qXp58F2koDKbHKLNdt7eRryWjPQ-9VQIFH2V1CELo05_0Sc0sCqBNAwHf9XD_r93DPgLlwpv-FOc8Zl4lJi48-WmFwNhTTON8juKT7MNmUS9UFI0nDKHTpWVBeGKbT-xjNt75BzunsVdM8ZuWW-i6CzO7KJaFGzL_7l8-1pyayBm0sXqz1T4j8KLlUPsCsV01RY5516OwR2YVI0araTVnDrqaUW4TiAFnxfxIIsNeTYXSvI436ityUFXjCEp4mAxnid5fEjZAI2ujQCDYjjSzwNqXBfmkswK6fh8NOMl8J8BGHp90FiP_j2HHPYngGzo6LP8Ry4-bkutUQqBGxu5v2lbzMKsNePO5vxzvd7W_7vavycmQEmDtsomnZtEBFt5xQpVQe1frMr0h6raXb-O_Oo_Z6ctPNgOQ30VFy3gMAfj0Bav2B9KWWVa5Gdw-p_einENiJzrVp_qax3A-ui7-B_KvqtMdDAnDjyr0jrpT2-wUubcmjFK3Ado_sn64aEuHWo8x9j0Fzdgv559ddUC-h6gQ7t3MqaJBHUcAVoqd0C38IrvtogE3lMTJAIVLEPhiTaC_6vlTw6l57ujDip4mNn1hdODN-N2XmbLtFq6aw886oZP1lheJr7uJfY5igKXKMlgyIbiykeOEYxZMERpkf5BDEdmpiEixAFuUC0JehKeZ8e4xTld2LcPGqZNyPm_Mt-gHlRUxdXbHhe-ID-yLVzYDmbTOp5-zvqA4A_vIMpV2dRWVUaXznmF_F7Y90Ou1SGKNZ2uLL6_1UXDvJeGF9dR_CI3OlK9ojY48D0FGMrua0kjOY5Pqe76dqIFL3Q2KPB2Q-tlgz-EHei4QQmZFzfwhMXYtCVPnvwHKT8PmLQopBrmYQVVDF9inew0iI2eZYjIsRqGSCQvpBBOt43Pk1OJJ4-D3xicu-45m4xgPn58wfoHyBjf1VYjFeTXrXQ-0qTNSN_IVYaSEtlthtB6YgfDjUWAoIs1Zpl-chhLjd3nEmXvuHT3vqph-PUx_raodOb1LQ9nUw6Ck0t1-x3XQTecaIGvBJUF-cBXSu7Ahz3QNwW4nshIvnt3u_jAijfbDPibJyRymFTVqNh5w-TNqXKWiL7PwrBlHAUlfF4FY-OTl8rA7DesF-t_2semd5BVLQQUnfUOBoi1rejFFXznEOsi4-ikRp5KmDC6OvZycKcZM8S0Jm2XH97yDp8ZzBXxcYX7BubcqSBc23nZs8ItQV_suikKE6x0Sshz5fEtQo4vltc4EaPZF82sT1YsZGGzLe6NdBLB4jmF5rnvepiNfOUtf-XgHG6BwDqzpkA9FIFGfRxjV3d4hVKHjJ0Qn827nCISpSQjjoEuj4OYW7npoC",
**"filepath"**: "70219d0ce05a4e14b828e719eaf6f552/"
},
**"fileCollectionSource"**: "70219d0ce05a4e14b828e719eaf6f552",
"createdBy": "preshipping_test_user@osdu-gcp.go3-nrg.projects.epam.com"
}
}
**How to use the information to load the multiple files to create a collection**__
` Attached is the Postman collection that works on AWS DatatsetFileCollection_Test.postman_collection.json