Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
  • H Home
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 15
    • Issues 15
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Deployments
    • Deployments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • Insights
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
Collapse sidebar
  • Open Subsurface Data Universe SoftwareOpen Subsurface Data Universe Software
  • Platform
  • Data Flow
  • Data IngestionData Ingestion
  • Home
  • Issues
  • #41
Closed
Open
Issue created Sep 28, 2020 by Stephen Whitley (Invited Expert)@stephenwhitley1 of 6 checklist items completed1/6 checklist items

Ingestion enables data platform compliance through the core services

Ingestion enables data platform compliance through the core services

Status

  • Initiated
  • Proposed
  • Trialing
  • Under review
  • Approved
  • Retired

Context & Scope

The ingestion process acts as a bridge enabling external data to enter the data platform in a compliant manner.

As a bridge, it has to live partly in the external world and partly in the data platform world. Data platform obligations such as security, entitlements, immutability, and discoverability are introduced during the ingestion process and imposed by the data platform itself. However, these obligations cannot be fully imposed on every action within the data ingestion process itself or it will be unable to deal with external data and systems.

Decision

The Core Services represent the contract boundary of the data platform. Once data is introduced through these services it must comply with the obligations of the data platform and will do so by using, and not bypassing, the core services.

This

  • Allows the ingestion process to perform operations that are outside the data platform.
  • Requires the ingestion process to conform to the data platform requirements when interacting through the core services (it is not a privileged service of the data platform)
  • Ensures compliance obligations are imposed by the core services themselves rather than held as concerns outside the platform.

Rationale

By using the core services to enforce data platform obligation; we avoid having carrying obligations in two places:

  • The data platform which is permanent
  • The ingestion process which is transient

Consequences

When to revisit

Edited Sep 28, 2020 by Stephen Whitley (Invited Expert)
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Assignee
Assign to
Time tracking