GitLab

• File record will point to a manifest file, or to an opaque file, in which case no further processing required. This is a replacement for the FileID in the “Submit Pre-Loaded”
• It’s then up to the application to create the file record and hand it to the ingestion service; will then either create a manifest or create an opaque file record
• The DAG type (DataType) would be passed along with this so knows which type of DAG to run.
• Also add another method called createFileRecord(filename, unsignedURL, datatype, ???) which would return a FileRecord as created in the Storage service.

Status

• Proposed
• Trialing
• Under review
• Approved
• Retired

Context & Scope

This pattern more closely matches the ISV bulk loader pattern implemented by EPAM.

Decision

Rationale

Consequences

This would require re-orienting everything else in the Ingestion service which relies on FileID.

• Because of the way the OSDU DAG is implemented, the Ingestion service needs to call the OSDU (aka Manifest) DAG.

When to revisit

---

Tradeoff Analysis - Input to decision

Alternatives and implications

Replace FileID with JSON FileRecord

• Breaks referential integrity: creates records even if File doesn't exist because wasn't successfully loaded
• Aligns better with with ISV bulk loader test cases implemented by EPAM, simplifying testing
• Assumes ingestions is used only for the file type data

Implement Ingestion service as designed

• Preserves referential integrity
• Doesn't require re-working existing Ingestion services
• Requires greater alignment of test cases with ISV bulk loader test cases implemented by EPAM

Decision criteria and tradeoffs

Decision timeline