Checksum computed by FILE API does not match the expected checksum as seen in the local computer from where the file is uploaded.
When the request is made to ingest/create the metadata record for the file, checksum is not provided and is expected to be computed by the FILE API.
The checksum is getting computed, but the value for the checksum does not match the expected value seen on the computer from where the file loaded from. For the same file and same postman request in different environments (AWS, Azure and GC), the checksum computed is different.
e.g.
the request to create metadata
curl --location 'https://osdu-ship.msft-osdu-test.org/api/file/v2/files/metadata' \
--header 'Data-Partition-Id: opendes' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer eyJ0eXAiOi...truncated...YCIMwsA' \
--header 'Cookie: JSESSIONID=08F04CDD138A5392623F07D5BDEC2416' \
--data-raw '{
"data": {
"Endian": "BIG",
"Description": "As originally delivered by ACME.com.",
"DatasetProperties": {
"FileSourceInfo": {
"FileSource": "/osdu-user/1687458073930-2023-06-22-18-21-13-930/bcade1152ef4488c97a85253eb83dab4",
"Name": "7004_a1501_1978_comp.las",
"PreLoadFilePath": "",
"PreloadFileCreateUser": "user1",
"PreloadFileModifyDate": "mar 11",
"PreloadFileModifyUser": "mar 11"
}
},
"TotalSize": "13245217273",
"Source": "Example Data Source",
"Name": "Dataset X221/15"
},
"kind": "osdu:wks:dataset--File.Generic:1.0.0",
"acl": {
"viewers": [
"data.default.viewers@opendes.contoso.com"
],
"owners": [
"data.default.owners@opendes.contoso.com"
]
},
"legal": {
"legaltags": [
"opendes-FileAPI-Legal-Tag-Test"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "",
"createTime": "",
"modifyUser": "",
"modifyTime": ""
}'
Checksum seen on local computer
D:\OSDU\git\qa\Postman Collection\21_CICD_Setup_FileAPI>dir 7004_a1501_1978_comp.las
Volume in drive D is Kamlesh SecondDrive
Volume Serial Number is 80AA-DDE3
Directory of D:\OSDU\git\qa\Postman Collection\21_CICD_Setup_FileAPI
05/22/2023 02:05 PM 1,615,618 7004_a1501_1978_comp.las
1 File(s) 1,615,618 bytes <================================
0 Dir(s) 391,477,112,832 bytes free
D:\OSDU\git\qa\Postman Collection\21_CICD_Setup_FileAPI>certutil -hashfile 7004_a1501_1978_comp.las MD5
MD5 hash of 7004_a1501_1978_comp.las:
6e62bc8f6c906bba620970bdb3bc9dd7 <=========================
CertUtil: -hashfile command completed successfully.
Checksum computed on AWS
{
"id": "osdu:dataset--File.Generic:69f1ed86-31de-428a-8e1e-50bed005f0d2",
"kind": "osdu:wks:dataset--File.Generic:1.0.0",
"legal": {
"legaltags": [
"osdu-FileAPI-Legal-Tag-Test"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"data": {
"Name": "Dataset X221/15",
"Description": "As originally delivered by ACME.com.",
"TotalSize": "13245217273", <=========================================
"Source": "Example Data Source",
"DatasetProperties": {
"FileSourceInfo": {
"FileSource": "s3://psosdu-shared-file-20230607030747431500000005/osdu/cqiBOcwCCeqIbVcz9O3l02kSpux06Vww/5adbc1ce3b5442c0b74922654125cac6",
"PreloadFileCreateUser": "user1",
"PreloadFileModifyUser": "mar 11",
"PreloadFileModifyDate": "mar 11",
"Name": "7004_a1501_1978_comp.las",
"Checksum": "747b6c19fb42ec76b8fe926b4244f530", <=================
"ChecksumAlgorithm": "MD5" <===========================
}
},
"Endian": "BIG"
},
"version": 1687386740078690,
"meta": [],
"acl": {
"viewers": [
"data.default.viewers@osdu.example.com"
],
"owners": [
"data.default.owners@osdu.example.com"
]
}
}
Checksum computed in Azure
{
"id": "opendes:dataset--File.Generic:9d445cbd-1e59-4a63-a0ab-26dbe0e7d32c",
"kind": "osdu:wks:dataset--File.Generic:1.0.0",
"legal": {
"legaltags": [
"opendes-FileAPI-Legal-Tag-Test"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"data": {
"Name": "Dataset X221/15",
"Description": "As originally delivered by ACME.com.",
"TotalSize": "13245217273",
"Source": "Example Data Source",
"DatasetProperties": {
"FileSourceInfo": {
"FileSource": "/osdu-user/1687458073930-2023-06-22-18-21-13-930/bcade1152ef4488c97a85253eb83dab4",
"PreloadFileCreateUser": "user1",
"PreloadFileModifyUser": "mar 11",
"PreloadFileModifyDate": "mar 11",
"Name": "7004_a1501_1978_comp.las",
"Checksum": "76630be1d6c8c0eaaf94dd85c9f900c1", <==================
"ChecksumAlgorithm": "MD5" <=======================
}
},
"Endian": "BIG"
},
"version": 1687458098789171,
"acl": {
"viewers": [
"data.default.viewers@opendes.contoso.com"
],
"owners": [
"data.default.owners@opendes.contoso.com"
]
}
}
Checksum seen on GC
{
"id": "odesprod:dataset--File.Generic:1ee6e3c1-f8d4-4f14-8ab4-a6930fe2aa63",
"kind": "osdu:wks:dataset--File.Generic:1.0.0",
"legal": {
"legaltags": [
"odesprod-FileAPI-Legal-Tag-Test"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"data": {
"Name": "Dataset X221/15",
"Description": "As originally delivered by ACME.com.",
"TotalSize": "13245217273", <===================================
"Source": "Example Data Source",
"DatasetProperties": {
"FileSourceInfo": {
"FileSource": "/5cef6ed4-46e3-4076-9340-8654f0b80a7d/f0fe9ecb5ba849599c8869d7fe44f073",
"PreloadFileCreateUser": "user1",
"PreloadFileModifyUser": "mar 11",
"PreloadFileModifyDate": "mar 11",
"Name": "7004_a1501_1978_comp.las",
"Checksum": "e91d53ba7affe69ffc62fa2306f62a1e", <=========================
"ChecksumAlgorithm": "MD5" <=========================
}
},
"Endian": "BIG"
},
"version": 1687387101698635,
"acl": {
"viewers": [
"data.default.viewers@odesprod.osdu-gcp.go3-nrg.projects.epam.com"
],
"owners": [
"data.default.owners@odesprod.osdu-gcp.go3-nrg.projects.epam.com"
]
}
}