RI: File service end-to-end workflow scenario
Summary
This is a post to summarize the existing behavior of the File service end-to-end workflow scenario across the platform providers today and aims to provide a standardized behavior.
A user should be able to securely upload and retrieve the file from the OSDU data platform.
Steps to reproduce
- Generate signed URL
| RI | AWS | Azure | GC | IBM |
|---|---|---|---|---|
GET https://osdu.bm-preship.gcp.gnrg-osdu.projects.epam.com/api/file/v2/files/uploadURL |
GET https://osdu.r3m18.preshiptesting.osdu.aws/api/file/v2/files/uploadURL |
GET https://osdu-ship.msft-osdu-test.org/api/file/v2/files/uploadURL |
GET https://preship.gcp.gnrg-osdu.projects.epam.com/api/file/v2/files/uploadURL |
GET https://cpd-osdu.apps.ibmosdu-preship.lndu.p1.openshiftapps.com/osdu-file/api/file/v2/files/uploadURL |
{"FileID": "a547911d165342efb2b1a0a2b0d62be0","Location": {"SignedURL": "https://s3.bm-preship.gcp.gnrg-osdu.projects.epam.com/refi-osdu-staging-area/b694b155-ab58-4219-90e1-62ad05340c1c/a547911d165342efb2b1a0a2b0d62be0?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=fileUser%2F20230807%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230807T203924Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=a985ad6e9c96797e3200b780e557c04f5147dc6b35de1f00fc6701926543df79","FileSource": "/b694b155-ab58-4219-90e1-62ad05340c1c/a547911d165342efb2b1a0a2b0d62be0"}} |
{ "FileID": "624d080b600641abaf3af970f6312676", "Location": { "SignedURL": "https://psosdu-shared-file-20230607030747431500000005.s3.us-east-2.amazonaws.com/osdu/mngFQIw0uAnFGTfASsb6A33xFDOfzmo6/624d080b600641abaf3af970f6312676?X-Amz-Security-Token=IQoJb3JpZ2luX2VjEM%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMiJHMEUCIQD6O%2F3HiYMc22n0NjW33UfkP6LynnyOKftkcSFSJNGFUAIgNtWeF0ziMXd1vlqqENBI6%2BcXayHxYD%2Fl%2FG6GVg%2B%2BFKEqpgUIeBACGgw1NjE3MzUyOTE0MjciDNFmocR6p9x5wb8WIiqDBYyB7XWL7yAZNo27Fa7mCF9L4TJfDky%2Fd5QvUYE9QKp7x60DP7GtC%2Bh1%2Fo0wOykyp1PKKcsjmiFN97%2BZkLhtwvCAHRnIrlt%2FObnOU6RnPORGm73yZTMxe94iGjCfmgrM0ac6HZx2GI7SCnGs9%2FOpfSNi75z4jXPDH6RIjLMx3iHexLU3sM6n0Dn5aA8sAy0xYcUmOfsyGBnFgQpN3Szw6P8NjnrsqnwIlua%2BK3AhMgVAHucJud7ORPXhdnkOph8U%2FH1ePfCphtg7H6ub2seVJA%2B%2B9brAvg87uR4Txxg%2F%2F9LGit%2FM6AEJqlVmgMeUDj234WJkB4zp6i96etxH7UOjSYbAZ7%2Bs35f4oSZNwtsbtyiamCo75XzMam%2BLGCiFRQ0KDJ7oS9lk7A3gh6dWx6dxObxb1UrXSjrMTCrQkMMn5p9tm5MYxgF9288Eb%2BlRNM%2FiUOP1SVhvLgzDlbWGWtdHIH7c2PuzRNJxc4pK0pWxGA2z3QbBdFKTmlLGuEnzZbONXKXJ4JdUTp%2FKqpEipclYqqcAFBbF2xr6sBQ6PIAZJ%2FQ5JT1ZkFDA8nuHVVOEJvuuifAXskHn3FDGG3NtEGO%2Fn4uKi%2FisCXIi1u7yJ%2B9WDoDzJ0VB5s0vTkXslppR23T%2BXtLI%2FXOkW6QOgiCeZviS40bPM6sSJFdV6w0TdpkMcUVxmLuOytpXnNNhig%2FlQJQA6aaXIbHJe8NgAce8nVFxlxlI4xI9PUwtgXSw7Xat0cHu19tiAoFAVap%2Bvf6f7HN5oaFuzVjxvgB1D%2BGZczt84RhR2OIQR%2FR13zOT0H05qTctk40PoXYv%2FrVTDIu1xlfJ0joboH01zMfUXHglb3g0t5ahIX8w6JTEpgY6nQGV8ZN9rthZJHdpIpwAKTNM%2Fxd7vut0OsO0%2BFNm6KmP749WiLSSt%2BHIyzV2ggbd7YTc2kwrxLeYzn8q9538aVhdWb8uxPC6DdaEbcZBZcE0%2BZ%2BdlXROyLz7YiGFkGrzxFBZ4ss1xJIKanirDui9P9tZ%2Fiz3OIyN%2FsrVFQMPcqNhHslor7I2ZgKEGGDz285gLCqxDExSfaORuxLmtS%2FJ&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230807T151448Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86399&X-Amz-Credential=ASIAYFSQIKIRV7YB25NL%2F20230807%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Signature=2c4bcbc19b45e9fdf7faba7314885189ea013e81d61ad91805dfa83bfa49248c", "FileSource": "s3://psosdu-shared-file-20230607030747431500000005/osdu/mngFQIw0uAnFGTfASsb6A33xFDOfzmo6/624d080b600641abaf3af970f6312676"}} |
{ "FileID": "91018f5735bf46ff907a3f9657584daa", "Location": { "SignedURL": "https://osdumvpdp1shipbbngdata.blob.core.windows.net/file-staging-area/osdu-user%2F1691481200990-2023-08-08-07-53-20-990%2F91018f5735bf46ff907a3f9657584daa?sv=2020-08-04&st=2023-08-08T07%3A53%3A21Z&se=2023-08-08T08%3A53%3A21Z&skoid=04e80323-039e-48fe-b76b-3a65c3cf7bcf&sktid=58975fd3-4977-44d0-bea8-37af0baac100&skt=2023-08-08T07%3A53%3A21Z&ske=2023-08-08T08%3A53%3A21Z&sks=b&skv=2020-08-04&sr=b&sp=cw&sig=j0cJNQ21TOcCBltbmuymaTaJiMssi7bXMqcgjwNzWuI%3D", "FileSource": "/osdu-user/1691481200990-2023-08-08-07-53-20-990/91018f5735bf46ff907a3f9657584daa" }} |
{ "FileID": "1ba9be427638426aba6e3ebaa6109690", "Location": { "SignedURL": "https://storage.googleapis.com/osdu-data-prod-odesprod-staging-area/b33afab6-2865-4347-b76e-1071ca738bb2/1ba9be427638426aba6e3ebaa6109690?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=workload-file-gcp%40osdu-service-prod.iam.gserviceaccount.com%2F20230807%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20230807T111802Z&X-Goog-Expires=3600&X-Goog-SignedHeaders=host&X-Goog-Signature=c5d6854dc74ccc8a0d4fdcfb97a09b7e22a93e866d1478262534a8fe31546ed4cd26cca7a8202c0c2f3b2a5d6d4d1191180b02089e1a3c641fc56cd58e80e33694158919e68953a797f5743da6f7628efa4142662d8dd93adbb2c1bff6eddb1332a96ddf63e19e3296532bea31f334212bec8ab49b5eb15c1cc97ce790288e6c393771314ee831586cd32af8e7a7071f0d2c11b26e6193a67b57553268a2ad753090275e7e4ec201223c7821b3a7c7c241564324817abed2c4bb0b8307fb471a0d34d400bf285356194e2c88fc4abffe9c277f3cdd34f160a02849439017dc83b34de214585a1dcc78b1b15129041301d4047ba69c1df7cfc1fa094af79658cb", "FileSource": "/b33afab6-2865-4347-b76e-1071ca738bb2/1ba9be427638426aba6e3ebaa6109690" }} |
{ "FileID": "899f355b50204081acac0b4fcde57fa5", "Location": { "SignedURL": "https://minio-osdu.apps.ibmosdu-preship.lndu.p1.openshiftapps.com/oc-cpd-opendes-staging-bucket/899f355b50204081acac0b4fcde57fa5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230808T075530Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=minio%2F20230808%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=35cd4e12e801fd7873145ff245db43f91ec648801c0e17c6abd927a0c15bd354", "FileSource": "899f355b50204081acac0b4fcde57fa5" }} |
- Uploading to signed URL
- Create file metadata
| RI | AWS | Azure | GC | IBM |
|---|---|---|---|---|
POST https://osdu.bm-preship.gcp.gnrg-osdu.projects.epam.com/api/file/v2/files/metadata |
POST https://osdu.r3m18.preshiptesting.osdu.aws/api/file/v2/files/metadata |
POST https://osdu-ship.msft-osdu-test.org/api/file/v2/files/metadata |
POST https://preship.gcp.gnrg-osdu.projects.epam.com/api/file/v2/files/metadata |
POST https://cpd-osdu.apps.ibmosdu-preship.lndu.p1.openshiftapps.com/osdu-file/api/file/v2/files/metadata |
Request Body Example:
Click to expand
{
"kind": "osdu:wks:dataset--File.Generic:1.0.0",
"meta": [
{
"name": {
"id": [
{
"kind": "12587932"
}
]
}
}
],
"tags": {
"dataflowId": "dataflowId",
"dataflowId-2": "dataflowId"
},
"acl": {
"owners": [
"data.default.owners@osdu.group"
],
"viewers": [
"data.default.viewers@osdu.group"
]
},
"legal": {
"legaltags": [
"osdu-FileAPI-Legal-Tag-Test"
],
"otherRelevantDataCountries": [
"US"
]
},
"data": {
"ResourceHomeRegionID": "osdu:reference-data--OSDURegion:AWSEastUSA:",
"ResourceHostRegionIDs": [
"namespace:reference-data--OSDURegion:AWSEastUSA:"
],
"ResourceCurationStatus": "osdu:reference-data--ResourceCurationStatus:CREATED:",
"ResourceLifecycleStatus": "osdu:reference-data--ResourceLifecycleStatus:LOADING:",
"ResourceSecurityClassification": "osdu:reference-data--ResourceSecurityClassification:RESTRICTED:",
"Source": "Example Data Source",
"ExistenceKind": "osdu:reference-data--ExistenceKind:Prototype:",
"Name": "Well_Headers_and_Completion_Reports 2017.csv",
"Description": "As originally delivered by ACME.com.",
"TotalSize": "1324521",
"EncodingFormatTypeID": "osdu:reference-data--EncodingFormatType:text%2Fcsv:",
"SchemaFormatTypeID": "osdu:reference-data--SchemaFormatType:CWLS%20LAS3:",
"Endian": "BIG",
"DatasetProperties": {
"FileSourceInfo": {
"FileSource": "/922679a4-48d3-4c47-9d23-ad12e7628825/e6bb3aff28d2429eb6f2928fc40e1a67",
"PreloadFilePath": "s3://staging-area/r7/raw-data/provided/documents/1000.witsml",
"PreloadFileCreateUser": "somebody@acme.org",
"PreloadFileCreateDate": "2019-12-16T11:46:20.163Z",
"PreloadFileModifyUser": "somebody.else@acme.org",
"PreloadFileModifyDate": "2019-12-20T17:20:05.356Z",
"Name": "tesing,copy.txt",
"FileSize": "95463",
"EncodingFormatTypeID": "osdu:reference-data--EncodingFormatType:application%2Fgeo%2Bjson:",
"Checksum": "checksum",
"ChecksumAlgorithm": "SHA-256"
}
},
"ExtensionProperties": {
"Classification": "Raw File",
"Description": "An text further describing this file example.",
"ExternalIds": [],
"FileDateCreated": {},
"FileDateModified": {},
"FileContentsDetails": {
"TargetKind": "slb:NewZealandCSV:wellbore:1.0.0",
"FileType": "csv",
"FrameOfReference": [
{
"kind": "CRS",
"name": "GCS_WGS_1984",
"persistableReference": "{\"wkt\":\"GEOGCS[\\\"GCS_WGS_1984\\\",DATUM[\\\"D_WGS_1984\\\",SPHEROID[\\\"WGS_1984\\\",6378137.0,298.257223563]],PRIMEM[\\\"Greenwich\\\",0.0],UNIT[\\\"Degree\\\",0.0174532925199433],AUTHORITY[\\\"EPSG\\\",4326]]\",\"ver\":\"PE_10_3_1\",\"name\":\"GCS_WGS_1984\",\"authCode\":{\"auth\":\"EPSG\",\"code\":\"4326\"},\"type\":\"LBC\"}",
"propertyNames": [
"LATITUDE",
"LONGITUDE"
],
"propertyValues": [
"deg"
],
"uncertainty": 0
},
{
"kind": "DateTime",
"persistableReference": "{\"type\": \"DAT\", \"format\": \"MM-dd-yyyy\"}",
"propertyNames": [
"PERMIT_DATE",
"INITIAL_COMPLETION_DATE",
"STATUS_DATE",
"SPUD_date"
],
"propertyValues": [],
"uncertainty": 0
},
{
"kind": "Unit",
"name": "ft",
"persistableReference": "{\"scaleOffset\":{\"scale\":0.3048,\"offset\":0.0},\"symbol\":\"ft\",\"baseMeasurement\":{\"ancestry\":\"Length\",\"type\":\"UM\"},\"type\":\"USO\"}",
"propertyNames": [
"MD",
"TVD",
"ELEVATION"
],
"propertyValues": [
"ft"
],
"uncertainty": 0
}
]
}
}
}
}Response body: 201
{
"id": "osdu:dataset--File.Generic:c79b2ec7-83c0-486b-b413-b3bf5536f316"
}- Generating a signed url to download file
| RI | AWS | Azure | GC | IBM |
|---|---|---|---|---|
| GET https://osdu.bm-preship.gcp.gnrg-osdu.projects.epam.com/api/file/v2/files/osdu:dataset--File.Generic:d505785d-479e-4bfa-a5f4-28c37a8e8ec2/downloadURL?expiryTime=15M | GET https://osdu.r3m18.preshiptesting.osdu.aws/api/file/v2/files/osdu:dataset--File.Generic:c79b2ec7-83c0-486b-b413-b3bf5536f316/downloadURL?expiryTime=15M | GET https://osdu-ship.msft-osdu-test.org/api/file/v2/files/opendes:dataset--File.Generic:459be362-facb-4c44-b805-7da4e165b0a6/downloadURL?expiryTime=15M | GET https://preship.gcp.gnrg-osdu.projects.epam.com/api/file/v2/files/odesprod:dataset--File.Generic:39d4cc35-5135-4b10-b742-95d838b43638/downloadURL?expiryTime=15M | GET https://cpd-osdu.apps.ibmosdu-preship.lndu.p1.openshiftapps.com/osdu-file/api/file/v2/files/opendes:dataset--File.Generic:00602949-656b-4f99-a523-c29c100dac6c/downloadURL?expiryTime=15M |
{ "SignedUrl": "https://s3.bm-preship.gcp.gnrg-osdu.projects.epam.com/refi-osdu-persistent-area/922679a4-48d3-4c47-9d23-ad12e7628825/e6bb3aff28d2429eb6f2928fc40e1a67?response-content-type=text%2Fplain&response-content-disposition=attachment%3B%20filename%3Dtesing%2Ccopy.txt&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=fileUser%2F20230808%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230808T075134Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=ddfbd5830f80ad8e9f2aa53c095bf0dee73c1de63653e72280cc87943c18cfef"} |
{ "SignedUrl": "https://psosdu-shared-file-20230607030747431500000005.s3.us-east-2.amazonaws.com/osdu/BbdSbYXtJnxF8F5BfW8fngl0ObIHsTzO/81723dde071042e5ba9799511c1b972e?X-Amz-Security-Token=IQoJb3JpZ2luX2VjEOD%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMiJHMEUCICHSjHHYquPNhPauPr%2Bj0mLbTnzrbbO0LROgfIUCybUyAiEA0Y7urQSEoGkj%2F6tMRijNIe%2FrIvy5kwzdUvuCwLdw8rEqkAUIif%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARACGgw1NjE3MzUyOTE0MjciDFn%2FI6Ebin8xMtVwKSrkBLNAzhK0JceuhqmBoQ%2Ba3zCpuyXDR6NTkIv8RfEu0Qe%2BtommPQgBbU%2Fp6NqrHFVD28EUvlUNmTRS7O49qHnKFeRv5o536oWQK0dTUw2LVcl4%2BLMgO5rHYasaywLeJw4AesrRYf3sEvDAxAdjrvlC9yzoi27bTvvJeswaHo2D7TLFq2lM%2BY8CWWfW5zl%2FY7%2FNWnnZsZ%2B2G37LEvq0c9vxdvxJBX58uADlqS8Msrl%2F1HN514NQN8rGTAzvCPEXujnnHsQse7Pj%2FkC6lGBs5xzYF6quRiAkB7kyMMM%2F2nMkkX5v%2BUef7JWS2NKVdr4jwGaSvBsVhyquINXJFtdUMHsAbiB8SfHpap5pWKTw%2BNE75ZRHzFXpyVcXbWaT1V95UhViHCaOmwGfvrB4dJizriUTlBaqZbe1Uva9uvfuCc1zf1B%2Fi4ojCpEF2kzcvRd1J3HbDoeTBn1GHy92OudDnP7xbdlCkOBb3oHBqfN%2Bnut2V0VLF1BFnPDehog3w6Uu5zhaabAsvy6fBtNhALElZXBHdSBEByCh4aL47kNT67dIsmJ4B7BuxQ5htuHIzyLT99tkCQqoEJajhkZCtkcugkV8sGID7WpLcn8KngEZb0x0TJ6vknUScS72I3UZHt53JNXkqKY4tlAZwpe24LNkJtlnuqJVLimjRawRm6%2FhYSUiewwHEJuOp%2FcuOqtBo8GrqcxoSx15iuzGAPW35ERv22n64SptkPAUOnsrn5yH%2Banui3U5SfT6SWVfQlwYZZWy5UthZdAMctxEx0FxdpdJzU8JryqKidl8caVLDhWgFcW%2FhUIbgyk3UDDu6MemBjqdAdraWXyjQJ2n5wNnzGlOxR0rMcwThmV5xdKakoXcyrSbQArcioyRTFf%2FHmrw%2B3y1XYdcAe5roe1kIPg%2FLEogp7MSErYCt52LbBoQDByh8xUuj5lIqYUEsiKL%2BtDfJH%2BMmo5z9hcSs7uneEfWAkR3SCpJqsQMB87Kqr2cFisZeNY6%2F2Ce77Mn%2F7zxnfnudUCF4TrynwVeW4Ysydym1Fg%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230808T075318Z&X-Amz-SignedHeaders=host&X-Amz-Expires=899&X-Amz-Credential=ASIAYFSQIKIR6PYRBPWH%2F20230808%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Signature=8e516594ad4be350c7b1bc9ad7f86e58407c3003c7aeec813ac25ea0218e72de"} |
{ "SignedUrl": "https://osdumvpdp1shipbbngdata.blob.core.windows.net/file-persistent-area/osdu-user%2F1691481200990-2023-08-08-07-53-20-990%2F91018f5735bf46ff907a3f9657584daa?sv=2020-08-04&se=2023-08-08T08%3A08%3A29Z&skoid=04e80323-039e-48fe-b76b-3a65c3cf7bcf&sktid=58975fd3-4977-44d0-bea8-37af0baac100&skt=2023-08-08T07%3A53%3A29Z&ske=2023-08-08T08%3A08%3A29Z&sks=b&skv=2020-08-04&sr=b&sp=r&sig=lKcTIxIW1iyFrJxK0bwPdzbt7%2FlbSsOBD8Q%2FIP79HyE%3D&rscd=attachment%3B%20filename%3D%20tesing%2Ccopy.txt&rsct=text%2Fplain"} |
{ "SignedUrl": "https://storage.googleapis.com/osdu-data-prod-odesprod-persistent-area/68bb9081-443a-45ac-bb9d-eb21aac553fd/0bd8018e8d674cb9a6c958aeb6b4f114?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=workload-file-gcp%40osdu-service-prod.iam.gserviceaccount.com%2F20230808%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20230808T075840Z&X-Goog-Expires=900&X-Goog-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dtesing%2Ccopy.txt&response-content-type=text%2Fplain&X-Goog-Signature=b0803ed77d01ba22a356a8c44e211f5d99024b07c6d70fd3738315f3e94470f841506b36802bba44cd95ec96ea0f9f9b4141fc834f30eee12040ec859dbbfe16c30264845120deb0564162b7252b750fa98dd7f46f6309efff5b9b99f57f099cbda1f9c3c4d5cd05fe422e76b1878e927e3a0ea3165dc17e4e169f52676ada2bd8ab826c915087d08c1184f3f8be61bdad8d1e1647334cd4d0f30b1f4ff2b14eaca9f5c5ef8051b8089d92f6ce28996f84eca4d30713272fdbdae5fa611cb0b3b4514246a1ce5c303d788dc0f732320132b65f133721e37196236a17ce41a6e4c22bc4d53f273064c752c499bded8be44bfaa2edca20c1ab120f9fa90bb2613c"} |
{ "SignedUrl": "https://minio-osdu.apps.ibmosdu-preship.lndu.p1.openshiftapps.com/oc-cpd-opendes-persistent-bucket/899f355b50204081acac0b4fcde57fa5?response-content-disposition=attachment%3B%20filename%20%3D%22tesing%2Ccopy.txt%22&response-content-type=text%2Fplain&response-expires=15M&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230808T075542Z&X-Amz-SignedHeaders=host&X-Amz-Expires=899&X-Amz-Credential=minio%2F20230808%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=009d8ac0e613b42d740cc2e7562b38e4d7ff54bf5445561d57025d1c3efe59a0"} |
Intended Behavior
The downloaded file should honor and preserve the Name attribute in the metadata as per the schema and in our test scenario, return the following file as tesing,copy.txt (A non-standard file name with a comma has been chosen for validation purposes)
{
...
"DatasetProperties": {
"FileSourceInfo": {
"Name": "tesing,copy.txt"
}
}
...
}Edited by Chad Leong