@gehrmann - Thank you for sharing your inputs.
Below is my understanding on OSDU Platform –
Based on your inputs above, if additional properties allowed inside data block of WKS schema and OSDU platform doesn’t restrict invalid attributes , then I’m wondering the use of extension properties and custom schemas as most of the data can be ingested into WKS schema with their own source data attributes.
Please correct me on my above understandings.
@Yan_Sushchynski - Thanks for checking on this, as you mentioned schema missing the "additionalProperties": false"
in data block, so in that case do you think we need an update in all WKS schemas to include "additionalProperties": false"
inside data block to work as expected.
As it's failing in WKS Well schema as well, as per @todaiks attached doc here.
@debasisc - yes, same expectations for me as well so far, however when I tested to confirm it's not validating the custom fields inside 'data' block in WKS schema.
Manifest-based ingestion workflow - Custom attributes check fails
Below reference data manifest with custom attributes successfully ingested into WKS schema-osdu:wks:reference-data--FacilityType:1.0.0
using Manifest-based ingestion workflow.
Looks like manifest ingestion workflow fails to validate the custom attributes based on defined WKS schema in manifest.
"data": {
"Code": "Well",
"ID": "Well",
"Name-test": "Well",
"Source": "Workbook Published/FacilityTypeType.1.0.0.xlsx; commit SHA 0b4db59a.",
"test-ingestion": [
{
"test-id": "12345"
}
]
},
"meta": null,
"modifyUser": "preshipping@azureglobal1.onmicrosoft.com",
"modifyTime": "2023-11-08T10:34:42.243Z",
"id": "opendes:reference-data--FacilityType:Well-06112023",
"version": 1699439681616809,
"kind": "osdu:wks:reference-data--FacilityType:1.0.0",
"acl": {
"viewers": [
"data.default.viewers@opendes.contoso.com"
],
"owners": [
"data.default.owners@opendes.contoso.com"
]
},
"legal": {
"legaltags": [
"opendes-Test-Legal-Tag-1007568"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"createUser": "preshipping@azureglobal1.onmicrosoft.com",
"createTime": "2023-11-06T18:14:31.762Z"
}
Samiullah Ghousudeen (54063365) at 13 Jun 09:36
Replace Planned_Scope_for_M18.xlsx
@thulasi_dass - fyi, I can have user input record id using the Dataset API, so this issue with File API only.
Samiullah Ghousudeen (8053951e) at 08 Jun 14:57
Replace Planned_Scope_for_M18.xlsx
File Service rejects Record ID from dataset--File.Generic manifest and generate always GUID as record id
Please check below CURL request to register metadata record for file dataset, I have included ID in the manifest, however all time it rejects this id and create system generated GUID for this registered record.
Create File metadata - Request
curl --location 'https://osdu-ship.msft-osdu-test.org/api/file/v2/files/metadata'
--header 'Data-Partition-Id: opendes'
--header 'Authorization: Bearer ...'
--header 'Content-Type: application/json'
--header 'Cookie: JSESSIONID=F4452A7D9F8752E8A82DC6E354D29B26'
--data-raw '{
"kind": "osdu:wks:dataset--File.Generic:1.0.0",
"id":"opendes:dataset--File.Generic:sami-test1",
"acl": {
"viewers": [
"data.default.viewers@opendes.contoso.com"
],
"owners": [
"data.default.viewers@opendes.contoso.com"
]
},
"legal": {
"legaltags": [
"opendes-Test-Legal-Tag-4766549"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"data": {
"Endian": "BIG",
"DatasetProperties": {
"FileSourceInfo": {
"FileSource": "/osdu-user/1686225883215-2023-06-08-12-04-43-215/4a62ec123d43427e93af2a4a1c515a6b"
}
}
}
}'
Create File metadata - Response
{ "id": "opendes:dataset--File.Generic:d5c226d6-3eb2-4825-8b9b-e0834d0464cb" }
Samiullah Ghousudeen (ef809c9a) at 06 Jun 12:45
Replace Planned_Scope_for_M18.xlsx
Samiullah Ghousudeen (0af7fa19) at 05 Jun 17:43
Replace Planned_Scope_for_M18.xlsx
Samiullah Ghousudeen (b68d4549) at 01 Jun 14:35
Upload New File
Samiullah Ghousudeen (5dd36e8f) at 31 May 09:44
Delete .gitkeep
Samiullah Ghousudeen (055a8eec) at 31 May 09:44
Delete .gitkeep
Samiullah Ghousudeen (35dd3ed8) at 31 May 09:44
Add new directory
Samiullah Ghousudeen (a2a426c4) at 31 May 09:44
Add new directory
Samiullah Ghousudeen (eebe63c9) at 31 May 09:43
Add new directory
Samiullah Ghousudeen (93c53396) at 31 May 09:43
Add new directory
Samiullah Ghousudeen (38a3179d) at 31 May 09:20
Update 2 files
Samiullah Ghousudeen (b89b4f3b) at 31 May 09:17
Add new directory
Samiullah Ghousudeen (835da34a) at 31 May 09:17
Add new directory