Data Ingestion issueshttps://community.opengroup.org/groups/osdu/platform/data-flow/ingestion/-/issues2022-08-23T10:47:24Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/issues/20Master Data - Load manifest2022-08-23T10:47:24ZMeena RathinavelMaster Data - Load manifestM1 - Release 0.1Clifford PattersonJames O'BoyleRohit KurhekarClifford Pattersonhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/issues/13Perform ACL Check2022-08-23T10:47:24ZMeena RathinavelPerform ACL Check- Does my role permit me to add data?- Does my role permit me to add data?M1 - Release 0.1Clifford PattersonJames O'BoyleClifford Pattersonhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/issues/27Manifests by reference2022-04-06T18:16:16ZAlan HensonManifests by referenceRather than pass the full manifest payload to the Ingestion Workflow, pass a pointer (via dataset id, or some other construct) to the manifest to be processed.
This approach better handles larger manifests for manifest ingestion.
**De...Rather than pass the full manifest payload to the Ingestion Workflow, pass a pointer (via dataset id, or some other construct) to the manifest to be processed.
This approach better handles larger manifests for manifest ingestion.
**Details**
1. In this approach, a user or process would create one or more manifests (could be done using tooling)
2. The user or process then uploads the manifests to OSDU using the Dataset service (e.g., get storage instructions, upload file, store metadata record, get the record id)
3. Invoke the workflow service passing in on or more record ids that point to the uploaded manifest(s)
4. Create a DAG operator capable of fetching the manifest(s) from storage using the Dataset service (get retrieval instructions)
**Definition of Done**
- A new DAG operator exists that can be added to ingestion workflows that are capable of reading in manifests into a DAG operator from the dataset service
**Question for consideration**
What happens once the manifest is read and other DAG operators downstream in the workflow need to access the contents? Is it passed via XCom? Does it need to be passed?
@Kateryna_Kurach may be able to offer more details from a design and architecture perspective.Devendra RawatDevendra Rawathttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/22WITSML parser updates to reflect 1.0.0 schema data definition changes2022-08-23T10:47:22ZJana ScheyWITSML parser updates to reflect 1.0.0 schema data definition changesLaurent DenyLaurent Deny2021-04-09https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/18R3 parser documentation2022-09-15T23:49:37ZJana ScheyR3 parser documentationSample issueSample issueJana ScheyJana Schey2020-12-11https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/15Google Implementation of Energistics Parsers2021-04-08T18:27:29ZJana ScheyGoogle Implementation of Energistics ParsersSample issueSample issueJay HollingsworthJay Hollingsworth2021-04-09https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/14R3 Airflow operators and DAG for parsers2022-08-23T10:47:22ZJana ScheyR3 Airflow operators and DAG for parsersSample issueSample issueJay HollingsworthJay Hollingsworthhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/13R3 parser dev toolkit2020-11-03T16:29:30ZJana ScheyR3 parser dev toolkitSample issueSample issueJay HollingsworthJay Hollingsworthhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/issues/8Register new Ingestion DAG2021-01-14T20:22:51ZStephen Whitley (Invited Expert)Register new Ingestion DAGStephen Whitley (Invited Expert)Todd DixonSwapnilStephen Whitley (Invited Expert)https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/issues/7[Validation] Add manifest validation in Manifest Ingestion Workflow - Schema ...2021-01-25T16:55:14ZElizaveta Zeldina (EPAM)[Validation] Add manifest validation in Manifest Ingestion Workflow - Schema validationDetails can be found here:
https://community.opengroup.org/osdu/platform/data-flow/home/-/issues/15
This issue was also mentioned in Lessons Learned after Energistics demo. We need to add the following types of validation:
- Manifest st...Details can be found here:
https://community.opengroup.org/osdu/platform/data-flow/home/-/issues/15
This issue was also mentioned in Lessons Learned after Energistics demo. We need to add the following types of validation:
- Manifest structure conforms the Schema that is being used
- Mandatory fields are filledDmitriy RudkoDmitriy Rudko