Manifest ingestion not picking dataset id generated by file service
In Azure to refer a file in manifest following steps are performed
- Generate signed url
- Upload content to signed url
- Create file metadata
- Refer the id received from create file metadata in the manifest.
The above steps are needed because manifest ingestion doesn't talk to file service which will reduce step 3 (which will be part of manifest ingestion).
Because of this limitation manifest file will have actual id from platform when referring a file. Master code of manifest ingestion is throwing exception when validating dataset ids. Please refer below for logs
Failed validating 'pattern' in schema['properties']['data']['allOf'][1]['properties']['Datasets']['items']:
{'description': 'The SRN which identifies this OSDU File resource.',
'pattern': '^(surrogate-key:.+|[\\w\\-\\.]+:dataset\\-\\-[\\w\\-\\.]+:[\\w\\-\\.\\:\\%]+:[0-9]*)$',
'type': 'string',
'x-osdu-relationship': [{'GroupType': 'dataset'}]}
On instance['data']['Datasets'][0]:
'opendes:dataset--File.Generic:253ea910-a13d-4759-a168-73310a8b2b2e'