Skip to content
GitLab
  • Menu
Projects Groups Snippets
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in / Register
  • I Ingestion DAGs
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 34
    • Issues 34
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages & Registries
    • Packages & Registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Open Subsurface Data Universe Software
  • Platform
  • Data Flow
  • Data Ingestion
  • Ingestion DAGs
  • Merge requests
  • !24

Ingestion updates

  • Review changes

  • Download
  • Email patches
  • Plain diff
Merged Siarhei Khaletski (EPAM) requested to merge trusted-epam-gcp-ingestion-updates into master Feb 25, 2021
  • Overview 46
  • Commits 20
  • Pipelines 11
  • Changes 75

Type of change

  • Bug Fix
  • Feature

Does this introduce a change in the core logic?

  • [Yes]

Does this introduce a change in the cloud provider implementation, if so which cloud?

  • AWS
  • Azure
  • GCP
  • IBM

Updates description?

This MR comes with a batch of updates:

Features:

  • Validate entire manifest entity (GONRG-1783)
  • Surrogate keys replacement (GONRG-1652)
  • Auth logic uses Python SDK implementation (GONRG-1689)
  • New operator for manifest integrity (GONRG-1700)
  • New operator for schema validation (GONRG-1770)
  • Logic for ensuring Datasets, WPCs and WP referencial integrity (GONRG-1653)
  • Implementation of the batch uploading (GONRG-1650)
  • Added FileSource validation for Datasets (GONRG-1651)

Structure updates:

  • Removed obsolete dags (GONRG-1567)
  • README.md has been updated (GONRG-1591)
  • Fix id composing (GONRG-1700)
  • Cleaned-up and renamed airflow variables (GONRG-1719)

Bugfixes:

  • Handle file variable fix
  • Fix download_as_bytes not supported in storage==1.13.2
  • Removed : (colon) symbol from the end of reference ids (GONRG-1911)
Assignee
Assign to
Reviewer
Request review from
Time tracking
Source branch: trusted-epam-gcp-ingestion-updates