... | ... | @@ -4,6 +4,14 @@ This page describes a walkthrough of an implementation of the Composable Ingesti |
|
|
|
|
|
This was presented to the R3 Ingestion framework project team on June 12th by Wei Sun and Pingjiang Wang from Microsoft. There will be more expansion of the alternatives over the next few weeks and a detailing of the business and technical criteria for framework selection based on this exercise.
|
|
|
|
|
|
#Ingestion Goals
|
|
|
Ingestion in OSDU is a fundamental concept and it is important to get this right. This is the process by which data will move into the platform, enabling integration of various data types, including files, opaque data, structure relational data, and a wide variety of data types.
|
|
|
The ingestion pipeline intends to also address the needs of consumption zones, mainly data consuming applications, where the format or structure of the data maybe different from how it is represented in the system.
|
|
|
|
|
|
Here's a diagram that illustrate the logical architecture of ingestion in the context of Seismic data:
|
|
|
|
|
|
![image](uploads/f26cf3953e6b1125d0703ad476bf418a/image.png)
|
|
|
|
|
|
# Background
|
|
|
|
|
|
* Airflow was used as the workflow engine in the "R2 Ingestion workflow orchestration non\-spike"
|
... | ... | |