Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in / Register
  • I Ingestion DAGs
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 32
    • Issues 32
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 1
    • Merge requests 1
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Open Subsurface Data Universe SoftwareOpen Subsurface Data Universe Software
  • Platform
  • Data Flow
  • Data IngestionData Ingestion
  • Ingestion DAGs
  • Merge requests
  • !67

Enable Support for Packaged DAGs

  • Review changes

  • Download
  • Email patches
  • Plain diff
Merged harshit aggarwal requested to merge haaggarw/packaged-dags into master Aug 12, 2021
  • Overview 5
  • Commits 21
  • Pipelines 4
  • Changes 27

Type of change

  • Bug Fix
  • Feature

Does this introduce a change in the core logic?

  • [Yes]

Does this introduce a change in the cloud provider implementation, if so which cloud?

  • AWS
  • Azure
  • GCP
  • IBM

Updates description?

The MR here is making changes to support packaged DAGs for manifest, the ADR for this change has been approved

New folder structure

├── osdu_manifest
│   ├── __init__.py
│   ├── libs
│   │   ├── __init__.py
│   │   └── utils.py
│   └── operators
│   |    ├── __init__.py
│   |    └── customOperator1.py
|   |___ hooks
|   |    |__  __init__.py
|   | 
|   |___ configs
|        |__ __init__.py
|
|___ osdu-ingest-r3.py
         

The changes include in the MR include

  • Restructuring the folders
  • Fixing any import statements
  • Minor changes to run existing tests

Related Issue - #86 (closed)

ADR - https://community.opengroup.org/osdu/platform/data-flow/home/-/issues/47

Edited Aug 12, 2021 by harshit aggarwal
Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: haaggarw/packaged-dags