Reimplement work with File Service
There are changes required in this file: https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics-osdu-integration/-/blob/feature/Port_to_R3_schemas/energistics/src/witsml_parser/create_energistics_manifest.py
At the current moment, the DAG expects preloadFilePath
(an original XML file path in some bucket or storage).
What the main script (create_energistics_manifest.py) does:
- gets a file's content from
preloadFilePath
(a file path in a bucket); - uploads the file using File Service;
- creates the file's metarecord using File Service and gets the file's id;
- gets its SignedUrl by the file's Id;
- downloads the file's contents and parses it;
- returns the Manifest.
The main concern is that the first three steps (uploading the file to File Service, creating its metarecord etc.) must be delegated to a client.
The DAG should expect only a file id and the script(create_energistics_manifest.py) should implement the last three steps.
What the main script (create_energistics_manifest.py) should do:
- get the file's SignedUrl by the file's Id;
- download the file's contents and parse it;
- return the Manifest.
Expected Workflow Payload after changes:
{
"executionContext": {
"Payload": {
"AppKey": "test-app",
"data-partition-id": <string>
},
"Context": {
"fileId": <string>
}
}
}
Current Workflow Payload:
{
"executionContext": {
"Payload": {
"AppKey": "test-app",
"data-partition-id": <string>
},
"Context": {
"acl": {
"viewers": [
<string>
],
"owners": [
<string>
]
},
"legal": {
"legaltags": [
<string>
],
"otherRelevantDataCountries": [
<string>
],
"status": "compliant"
},
"kind": <string>,
"file_name": <string>,
"preload_file_path": <string>,
"version": 5
}
}
}