Skip to content

GitLab

Closed
Created by Kishore Battula@kibattulOwner

Manifest ingestion not picking dataset id generated by file service

In Azure to refer a file in manifest following steps are performed

  1. Generate signed url
  2. Upload content to signed url
  3. Create file metadata
  4. Refer the id received from create file metadata in the manifest.

The above steps are needed because manifest ingestion doesn't talk to file service which will reduce step 3 (which will be part of manifest ingestion).

Because of this limitation manifest file will have actual id from platform when referring a file. Master code of manifest ingestion is throwing exception when validating dataset ids. Please refer below for logs

Failed validating 'pattern' in schema['properties']['data']['allOf'][1]['properties']['Datasets']['items']:
    {'description': 'The SRN which identifies this OSDU File resource.',
     'pattern': '^(surrogate-key:.+|[\\w\\-\\.]+:dataset\\-\\-[\\w\\-\\.]+:[\\w\\-\\.\\:\\%]+:[0-9]*)$',
     'type': 'string',
     'x-osdu-relationship': [{'GroupType': 'dataset'}]}

On instance['data']['Datasets'][0]:
    'opendes:dataset--File.Generic:253ea910-a13d-4759-a168-73310a8b2b2e'