Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in / Register
  • D Documentation
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 27
    • Issues 27
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 1
    • Merge requests 1
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Open Subsurface Data Universe SoftwareOpen Subsurface Data Universe Software
  • Documentation
  • Issues
  • #66
Closed
Open
Issue created Mar 23, 2020 by Ferris Argyle@fargyleMaintainer1 of 5 checklist items completed1/5 checklist items

Ingestion - replace FileID with JSON FileRecord

  • File record will point to a manifest file, or to an opaque file, in which case no further processing required. This is a replacement for the FileID in the “Submit Pre-Loaded”
  • It’s then up to the application to create the file record and hand it to the ingestion service; will then either create a manifest or create an opaque file record
  • The DAG type (DataType) would be passed along with this so knows which type of DAG to run.
  • Also add another method called createFileRecord(filename, unsignedURL, datatype, ???) which would return a FileRecord as created in the Storage service.

Status

  • Proposed
  • Trialing
  • Under review
  • Approved
  • Retired

Context & Scope

This pattern more closely matches the ISV bulk loader pattern implemented by EPAM.

Decision

Rationale

Consequences

This would require re-orienting everything else in the Ingestion service which relies on FileID.

  • Because of the way the OSDU DAG is implemented, the Ingestion service needs to call the OSDU (aka Manifest) DAG.

When to revisit


Tradeoff Analysis - Input to decision

Alternatives and implications

Replace FileID with JSON FileRecord

  • Breaks referential integrity: creates records even if File doesn't exist because wasn't successfully loaded
  • Aligns better with with ISV bulk loader test cases implemented by EPAM, simplifying testing
  • Assumes ingestions is used only for the file type data

Implement Ingestion service as designed

  • Preserves referential integrity
  • Doesn't require re-working existing Ingestion services
  • Requires greater alignment of test cases with ISV bulk loader test cases implemented by EPAM

Decision criteria and tradeoffs

Decision timeline

Edited Mar 27, 2020 by Vlad Korniev
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Assignee
Assign to
Time tracking