Ingestion - replace FileID with JSON FileRecord
- File record will point to a manifest file, or to an opaque file, in which case no further processing required. This is a replacement for the FileID in the “Submit Pre-Loaded”
- It’s then up to the application to create the file record and hand it to the ingestion service; will then either create a manifest or create an opaque file record
- The DAG type (DataType) would be passed along with this so knows which type of DAG to run.
- Also add another method called createFileRecord(filename, unsignedURL, datatype, ???) which would return a FileRecord as created in the Storage service.
Context & Scope
This pattern more closely matches the ISV bulk loader pattern implemented by EPAM.
This would require re-orienting everything else in the Ingestion service which relies on FileID.
- Because of the way the OSDU DAG is implemented, the Ingestion service needs to call the OSDU (aka Manifest) DAG.
When to revisit
Tradeoff Analysis - Input to decision
Alternatives and implications
Replace FileID with JSON FileRecord
- Breaks referential integrity: creates records even if File doesn't exist because wasn't successfully loaded
- Aligns better with with ISV bulk loader test cases implemented by EPAM, simplifying testing
- Assumes ingestions is used only for the file type data
Implement Ingestion service as designed
- Preserves referential integrity
- Doesn't require re-working existing Ingestion services
- Requires greater alignment of test cases with ISV bulk loader test cases implemented by EPAM