Ingestion enables data platform compliance through the core services
Ingestion enables data platform compliance through the core services
Status
- Initiated
- Proposed
- Trialing
- Under review
- Approved
- Retired
Context & Scope
The ingestion process acts as a bridge enabling external data to enter the data platform in a compliant manner.
As a bridge, it has to live partly in the external world and partly in the data platform world. Data platform obligations such as security, entitlements, immutability, and discoverability are introduced during the ingestion process and imposed by the data platform itself. However, these obligations cannot be fully imposed on every action within the data ingestion process itself or it will be unable to deal with external data and systems.
Decision
The Core Services represent the contract boundary of the data platform. Once data is introduced through these services it must comply with the obligations of the data platform and will do so by using, and not bypassing, the core services.
This
- Allows the ingestion process to perform operations that are outside the data platform.
- Requires the ingestion process to conform to the data platform requirements when interacting through the core services (it is not a privileged service of the data platform)
- Ensures compliance obligations are imposed by the core services themselves rather than held as concerns outside the platform.
Rationale
By using the core services to enforce data platform obligation; we avoid having carrying obligations in two places:
- The data platform which is permanent
- The ingestion process which is transient