Commits · 7bda9686e9a9325426627a1db1ad478bdb56c3f0 · OSDU / OSDU Data Platform / Data Flow / Data Ingestion / Manifest Ingestion DAG · GitLab

Snippets Groups Projects

Apr 09, 2021
- GONRG-2185: Single manifest validation hiden under the flag · 7bda9686
  Siarhei Khaletski (EPAM) authored 3 years ago
  
  7bda9686
Apr 07, 2021
- Merge branch 'trusted-r3_documentation_updated' into 'master' · f2efbffd
  Siarhei Khaletski (EPAM) authored 3 years ago
  
  GONRG-2008: Documentation has been updated See merge request !36
  View commits for tag v0.8.0 v0.8.0
  
  f2efbffd
Apr 06, 2021
- GONRG-2008: Documentation has been updated · e547ec5e
  Siarhei Khaletski (EPAM) authored 3 years ago
  
  e547ec5e
Mar 23, 2021
- Merge branch 'trusted-feature/base-token-refresher' into 'master' · 9a251e54
  Siarhei Khaletski (EPAM) authored 4 years ago
  
  Added BaseTokenRefresher See merge request !34
  9a251e54
- Add BaseTokenRefresher · 48e9789a
  Yan Sushchynski (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  48e9789a
Mar 19, 2021
- Merge branch 'trusted-manifest-ingestion-fixes' into 'master' · 2d1ee64d
  Siarhei Khaletski (EPAM) authored 4 years ago
  
  Fix for FileSource; fix for pipelines with invalid manifest See merge request !32
  2d1ee64d
Mar 18, 2021
- GONRG-1961: Fix pipline fails · 64df438a
  Yan Sushchynski (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  64df438a
- GONRG-1941: Fix space char in FileSource · d38d6f2e
  Yan Sushchynski (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  d38d6f2e
Mar 15, 2021
- Merge branch 'trusted-manifest-integrity-refactoring' into 'master' · e52f30bf
  Siarhei Khaletski (EPAM) authored 4 years ago
  
  Refactor referential integrity (GONRG-1932) See merge request !30
  e52f30bf
Mar 11, 2021
- Merge branch 'azure-documentation' into 'master' · a62739d8
  Kishore Battula authored 4 years ago
  
  Added readme for azure See merge request !29
  a62739d8
- GONRG-1932: Refactor referential integrity · 4f02c1db
  Yan Sushchynski (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  4f02c1db
- Added readme for azure · f06ae1ae
  Kishore Battula authored 4 years ago
  
  f06ae1ae
Mar 05, 2021
- Merge branch 'trusted-updated-vendors-loading' into 'master' · 61ca4e2a
  Siarhei Khaletski (EPAM) authored 4 years ago
  
  GONRG-1866: add condition statement to import vendors dependencies See merge request !27
  View commits for tag v0.7.0 v0.7.0
  
  61ca4e2a
- Merge branch 'trusted-ingestion-updates-m4' into 'master' · 50dd96fa
  Siarhei Khaletski (EPAM) authored 4 years ago
  
  Ingestion updates m4 (tested) See merge request !28
  50dd96fa
- Fix adding to valid wpc · b75370d5
  Yan Sushchynski (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  b75370d5
- Add constant to single manifest processor · f7c5e607
  Yan Sushchynski (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  f7c5e607
- GONRG-1912: Add surrogate keys to patterns · 92bbd40c
  Yan Sushchynski (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  92bbd40c
- GONRG-1891: Artifacts processing in WPC · c448c54e
  Yan Sushchynski (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  c448c54e
- Merge branch 'trusted-fixes-ids-refs' into 'master' · 43b80d2f
  Siarhei Khaletski (EPAM) authored 4 years ago
  
  Fix for IDs refs. Updated Docker image link See merge request !26
  43b80d2f
Mar 04, 2021
- GONRG-1866: add condition statement to import vendors dependencies · 8938fc43
  Dmitrii Valuiskii (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  8938fc43
- GONRG-1910: fix bug with ingesting master data with pre-defined id · b1778896
  Dmitrii Valuiskii (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  b1778896
- GONRG-1936: fix error with colon symbol in referential ids · b223c6c8
  Dmitrii Valuiskii (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  b223c6c8
Mar 03, 2021
- GONRG-1910: add validation reference ids with versions · fb11caae
  Dmitrii Valuiskii (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  fb11caae
- GONRG-1788: Fixed image path for Ingestion-DAGs (Cloudbuild and Dockerfile... · bfa8eb73
  Aliaksandr Lubouski (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  GONRG-1788: Fixed image path for Ingestion-DAGs (Cloudbuild and Dockerfile added for Python-SDK pipeline to build public image for airflow)
  bfa8eb73
Mar 02, 2021
- Merge branch 'trusted-epam-gcp-ingestion-updates' into 'master' · b93ff05f
  Kishore Battula authored 4 years ago
  
  Ingestion updates See merge request !24
  b93ff05f
- GONRG-1881: add check for WP manifest completeness · c1cf95ec
  Dmitrii Valuiskii (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  c1cf95ec
Mar 01, 2021
- GONRG-1719: updated variables to support ENV approach · 3d0e0f7b
  Siarhei Khaletski (EPAM) authored 4 years ago
  
  3d0e0f7b
Feb 25, 2021
- Add trigger rule to skip batch tasks · 7f1c24f6
  Yan Sushchynski (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  (cherry picked from commit 8af8301d)
  7f1c24f6
- GONRG-1911: remove : (colon) symbol from the end of reference ids · a4643d8a
  Dmitrii Valuiskii (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  a4643d8a
- GONRG-1651: Add FileSource validation. · 6fbb5750
  Ernesto Gutierrez authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  6fbb5750
- GONRG-1770: hotfix variable name · 0b1dfba1
  Dmitrii Valuiskii (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  0b1dfba1
- GONRG-1770: add composable operator for schema validation · dbdd530b
  Dmitrii Valuiskii (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  dbdd530b
- GONRG-1719: clean-up and rename airflow variables · 4f2f58a2
  Dmitrii Valuiskii (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  4f2f58a2
- Handle file variable fix · d2cebc9c
  Yan Sushchynski (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  d2cebc9c
- GONRG-1567: Removed obsolete dags: · 008c172b
  Siarhei Khaletski (EPAM) authored 4 years ago
  
  - default-ingest - other-log-ingest - well-log-ingest
  008c172b
- GONRG-1653: Validate Datasets, WPCs and WP referencial integrity. · e2fc2d0a
  Ernesto Gutierrez authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  e2fc2d0a
- Fix dag. Remove subdags · b97d68d2
  Yan Sushchynski (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  b97d68d2
- GONRG-1700: fix id composing · 5a58168e
  Dmitrii Valuiskii (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  5a58168e
- GONRG-1650: Batch upload · 58d9e72c
  Yan Sushchynski (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  58d9e72c
- GONRG-1700: add operator for manifest integrity · aed37799
  Dmitrii Valuiskii (EPAM) authored 4 years ago and Siarhei Khaletski (EPAM) committed 4 years ago
  
  aed37799