|
|
**30 June 2021**
|
|
|
|
|
|
* Updated GitLab Issues Board
|
|
|
* Performance & Scalability Testing
|
|
|
|
|
|
Next steps:
|
|
|
|
|
|
* Publish status as per Publish Status ADR
|
|
|
* Multi-thread optimization
|
|
|
|
|
|
**9 June 2021**
|
|
|
|
|
|
* Nested Arrays
|
|
|
|
|
|
**2 June 2021**
|
|
|
|
|
|
* Finalize support OSDU relationships
|
|
|
|
|
|
Next steps:
|
|
|
|
|
|
* Nested arrays
|
|
|
|
|
|
Next steps:
|
|
|
|
|
|
* Nested arrays
|
|
|
|
|
|
**10 November 2020**
|
|
|
- CSV Ingestor DAG
|
|
|
- integrated with schema service
|
|
|
- integrated with partition service
|
|
|
- Custom operator for creating meta{} block for units in progress
|
|
|
- Sharing states between operators in progress
|
|
|
|
|
|
* CSV Ingestor DAG
|
|
|
* integrated with schema service
|
|
|
* integrated with partition service
|
|
|
* Custom operator for creating meta{} block for units in progress
|
|
|
* Sharing states between operators in progress
|
|
|
|
|
|
**27 October 2020**
|
|
|
- Composable breakdown of DAG design in progress (multi-step DAG)
|
|
|
- Two DAG's created
|
|
|
- Standard HTTP Operator
|
|
|
- Kubernetes Pod Operator for better scalability
|
|
|
|
|
|
* Composable breakdown of DAG design in progress (multi-step DAG)
|
|
|
* Two DAG's created
|
|
|
* Standard HTTP Operator
|
|
|
* Kubernetes Pod Operator for better scalability
|
|
|
|
|
|
Next steps:
|
|
|
- Workflow service to support component registration
|
|
|
- Register Custom Airflow Python Operator (self-encapsulated)
|
|
|
- Register Custom Kubernetes Pod Operator (Docker image)
|
|
|
|
|
|
* Workflow service to support component registration
|
|
|
* Register Custom Airflow Python Operator (self-encapsulated)
|
|
|
* Register Custom Kubernetes Pod Operator (Docker image)
|
|
|
|
|
|
**6 October 2020**
|
|
|
- Horizon 1 Development complete
|
|
|
- CSV Ingestor Merge Request to master submitted on 6 October
|
|
|
- Composability spike concluded
|
|
|
|
|
|
* Horizon 1 Development complete
|
|
|
* CSV Ingestor Merge Request to master submitted on 6 October
|
|
|
* Composability spike concluded
|
|
|
|
|
|
Next steps:
|
|
|
- Break 1-step DAG into multi-step DAG
|
|
|
- Register re-usable component
|
|
|
- Register a DAG using that component
|
|
|
|
|
|
* Break 1-step DAG into multi-step DAG
|
|
|
* Register re-usable component
|
|
|
* Register a DAG using that component
|
|
|
|
|
|
**22 September 2020**
|
|
|
- Another commit done to Gitlab
|
|
|
- Single step CSV Ingestor DAG is now working in Airflow
|
|
|
- Manual registration (placed in dag directory)
|
|
|
- Manual execution (via Airflow UI)
|
|
|
- Authentication is supported (same as in Core Services)
|
|
|
|
|
|
* Another commit done to Gitlab
|
|
|
* Single step CSV Ingestor DAG is now working in Airflow
|
|
|
* Manual registration (placed in dag directory)
|
|
|
* Manual execution (via Airflow UI)
|
|
|
* Authentication is supported (same as in Core Services)
|
|
|
|
|
|
Next steps:
|
|
|
- Finish Horizon 1 development (eta: 28 September)
|
|
|
- Integration tests
|
|
|
- Check for code vulnerabilities
|
|
|
- Once all above is done, Pull Request to merge to master
|
|
|
|
|
|
* Finish Horizon 1 development (eta: 28 September)
|
|
|
* Integration tests
|
|
|
* Check for code vulnerabilities
|
|
|
* Once all above is done, Pull Request to merge to master
|
|
|
|
|
|
**15 September 2020**
|
|
|
- Initial commit to [separate branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/tree/osdu-r3-code/os-csv-parser/provider/csv-parser-azure) for CSV Ingestor DAG
|
|
|
- Plan to commit every Monday going forward
|
|
|
- [ADR on R2 Workflow Service Enhancement](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/69) initial review on 14 September 2020 (new DAG registration endpoint).
|
|
|
|
|
|
* Initial commit to [separate branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/tree/osdu-r3-code/os-csv-parser/provider/csv-parser-azure) for CSV Ingestor DAG
|
|
|
* Plan to commit every Monday going forward
|
|
|
* [ADR on R2 Workflow Service Enhancement](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/issues/69) initial review on 14 September 2020 (new DAG registration endpoint).
|
|
|
|
|
|
**8 September 2020**
|
|
|
- Initial (simplified/1-step) DAG implemented for delimited ASCII files
|
|
|
- Plan to commit to OSDU Gitlab under review
|
|
|
- Synthetic data/schema created and published to slack [here](https://opensdu.slack.com/files/U015U9FJHMJ/F01AXKNF2E4/csv_ingestor_wellbore_data.zip).
|
|
|
|
|
|
**2 June 2021**
|
|
|
- Finalize support OSDU relationships
|
|
|
|
|
|
Next steps:
|
|
|
- Nested arrays
|
|
|
|
|
|
**9 June 2021**
|
|
|
- Nested Arrays
|
|
|
|
|
|
Next steps:
|
|
|
- Nested arrays |
|
|
\ No newline at end of file |
|
|
* Initial (simplified/1-step) DAG implemented for delimited ASCII files
|
|
|
* Plan to commit to OSDU Gitlab under review
|
|
|
* Synthetic data/schema created and published to slack [here](https://opensdu.slack.com/files/U015U9FJHMJ/F01AXKNF2E4/csv_ingestor_wellbore_data.zip). |
|
|
\ No newline at end of file |