WITSML Parser merge requestshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests2023-08-18T11:04:04Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/6Ing par to energistics migration2023-08-18T11:04:04ZMohammed Alimohammed.ali@energistics.orgIng par to energistics migrationmoving files and dirs from IngestionsParsers to energistics dirmoving files and dirs from IngestionsParsers to energistics dirM1 - Release 0.1Mohammed Alimohammed.ali@energistics.orgMohammed Alimohammed.ali@energistics.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/3rename the energistics-osdu-integration/witsml_parser folder to energistics-o...2023-08-18T11:04:06ZMohammed Alimohammed.ali@energistics.orgrename the energistics-osdu-integration/witsml_parser folder to energistics-osdu-integration/energistics folderUpdate energistics/src/dags/energistics/witsml_parser_dag.py, energistics/src/plugins/energistics/hooks/__init__.py, energistics/src/plugins/energistics/libs/energistics_parsers/OSDU_types/Document_1_0_0.py, energistics/src/plugins/energ...Update energistics/src/dags/energistics/witsml_parser_dag.py, energistics/src/plugins/energistics/hooks/__init__.py, energistics/src/plugins/energistics/libs/energistics_parsers/OSDU_types/Document_1_0_0.py, energistics/src/plugins/energistics/libs/energistics_parsers/OSDU_types/File_1_0_0.py, energistics/src/plugins/energistics/libs/energistics_parsers/OSDU_types/Seismic2DInterpretationProject_1_0_0.py, energistics/src/plugins/energistics/libs/energistics_parsers/OSDU_types/Seismic3DInterpretationProject_1_0_0.py, energistics/src/plugins/energistics/libs/energistics_parsers/OSDU_types/SeismicBinGrid_1_0_0.py, energistics/src/plugins/energistics/libs/energistics_parsers/OSDU_types/SeismicHorizon_1_0_0.py, energistics/src/plugins/energistics/libs/energistics_parsers/OSDU_types/WellLog_1_0_0.py, energistics/src/plugins/energistics/libs/energistics_parsers/OSDU_types/Well_1_0_0.py, energistics/src/plugins/energistics/libs/energistics_parsers/OSDU_types/WellboreMarker_1_0_0.py, energistics/src/plugins/energistics/libs/energistics_parsers/OSDU_types/WellboreTrajectory_1_0_0.py, energistics/src/plugins/energistics/libs/energistics_parsers/OSDU_types/Wellbore_1_0_0.py, energistics/src/plugins/energistics/libs/energistics_parsers/OSDU_types/WorkProduct_1_0_0.py, energistics/src/plugins/energistics/libs/energistics_parsers/OSDU_types/__init__.py, energistics/src/plugins/energistics/libs/energistics_parsers/resqml_2_0/__init__.py, energistics/src/plugins/energistics/libs/energistics_parsers/resqml_2_0/default_parser.py, energistics/src/plugins/energistics/libs/energistics_parsers/resqml_2_0/grid2drepresentation_parser.py, energistics/src/plugins/energistics/libs/energistics_parsers/resqml_2_0/resqml_2_0_1_xsd.py, energistics/src/plugins/energistics/libs/energistics_parsers/resqml_2_0/wellboreframerepresentation_parser.py, energistics/src/plugins/energistics/libs/energistics_parsers/resqml_2_0/wellboreinterpretation_parser.py, energistics/src/plugins/energistics/libs/energistics_parsers/resqml_2_0/wellboremarkerframerepresentation_parser.py, energistics/src/plugins/energistics/libs/energistics_parsers/resqml_2_0/wellboretrajectoryrepresentation_parser.py, energistics/src/plugins/energistics/libs/energistics_parsers/witsml_2_0/.gitkeep, energistics/src/plugins/energistics/libs/energistics_parsers/witsml_2_0/__init__.py, energistics/src/plugins/energistics/libs/energistics_parsers/witsml_2_0/log_parser.py, energistics/src/plugins/energistics/libs/energistics_parsers/witsml_2_0/trajectory_parser.py, energistics/src/plugins/energistics/libs/energistics_parsers/witsml_2_0/witsml_2_0_xsd_log.py, energistics/src/plugins/energistics/libs/energistics_parsers/witsml_2_0/witsml_2_0_xsd_trajectory.py, energistics/src/plugins/energistics/libs/energistics_parsers/witsml_2_0/witsml_2_0_xsd_well.py, energistics/src/plugins/energistics/libs/energistics_parsers/witsml_2_0/witsml_2_0_xsd_wellbore.py, energistics/src/plugins/energistics/libs/energistics_parsers/witsml_2_0/witsml_2_0_xsd_wellboremarkers.py, energistics/src/plugins/energistics/libs/energistics_parsers/__init__.py, energistics/src/plugins/energistics/libs/energistics_parsers/no_wpc_parser.py, energistics/src/plugins/energistics/libs/energistics_parsers/parser.py, energistics/src/plugins/energistics/libs/energistics_parsers/parser_factory.py, energistics/src/plugins/energistics/libs/__init__.py, energistics/src/plugins/energistics/operators/__init__.py, energistics/src/plugins/energistics/operators/process_energistics_op.py, energistics/.gitignore, energistics/README.md filesM1 - Release 0.1Mohammed Alimohammed.ali@energistics.orgMohammed Alimohammed.ali@energistics.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/2Pluggable structure for WITSML parser (GONRG-798)2023-08-18T11:04:07ZDmitriy RudkoPluggable structure for WITSML parser (GONRG-798)Whats in this MR:
```
root
|_witsml_parser/
| |_src/
| |_dags/
| | |_energistics/ # Dags code here
| |_plugins/
| |_energistics/
| |_hooks/
| |_libs/ # Energistics libs here (WITSML, RESQML)
| |_operators/
|_.gitlab-ci....Whats in this MR:
```
root
|_witsml_parser/
| |_src/
| |_dags/
| | |_energistics/ # Dags code here
| |_plugins/
| |_energistics/
| |_hooks/
| |_libs/ # Energistics libs here (WITSML, RESQML)
| |_operators/
|_.gitlab-ci.yml # CICD config with deployment to GCP
```M1 - Release 0.1Mohammed Alimohammed.ali@energistics.orgDmitriy RudkoMohammed Alimohammed.ali@energistics.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/1Witsml integration12023-08-18T11:04:10ZMohammed Alimohammed.ali@energistics.orgWitsml integration1Merge the WITSML_Integration1 branch to masterMerge the WITSML_Integration1 branch to masterM1 - Release 0.1Mohammed Alimohammed.ali@energistics.orgMohammed Alimohammed.ali@energistics.orghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/7GONRG: Energistics integration2023-08-18T11:04:03ZYan Sushchynski (EPAM)GONRG: Energistics integrationExample of Ingestion service payload
{
"FileID": "1121212121",
"DataType": "Energistics_xml_ingest",
"Context": {
"acl": {
"viewers": [
"data.default.viewers@osdu.osd...Example of Ingestion service payload
{
"FileID": "1121212121",
"DataType": "Energistics_xml_ingest",
"Context": {
"acl": {
"viewers": [
"data.default.viewers@osdu.osdu-gcp.go3-nrg.projects.epam.com"
],
"owners": [
"data.default.owners@osdu.osdu-gcp.go3-nrg.projects.epam.com"
]
},
"legal": {
"legaltags": [
"osdu-demo-legaltag"
],
"otherRelevantDataCountries": [
"US"
],
"status": "compliant"
},
"kind": "osdu:wks:dataset--File.Generic:1.0.0",
"file_name": "trace_trajectory",
"preload_file_path": "gs://test_kate/energistics_witsml_data_Well.xml",
"version": 5
}
}
}M4 - Release 0.7Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/9Changed deploy step to manual2023-08-18T11:03:59ZSiarhei Khaletski (EPAM)Changed deploy step to manualM5 - Release 0.8Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/8GONRG-1982: Update documentation2023-08-18T11:04:01ZYan Sushchynski (EPAM)GONRG-1982: Update documentationUpdate documentation according to new changes in DAGs.Update documentation according to new changes in DAGs.M5 - Release 0.8Laurent DenyLaurent Denyhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/15Fix #35: Remove url encoding of file path2023-08-18T11:03:55ZLaurent DenyFix #35: Remove url encoding of file pathFix for Issue #35 File path in FileSource was wrongly URL encodedFix for Issue #35 File path in FileSource was wrongly URL encodedM6 - Release 0.9Yan Sushchynski (EPAM)Yan Sushchynski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/14GONRG-2399: Fix wrong FileSource2023-08-18T11:03:56ZYan Sushchynski (EPAM)GONRG-2399: Fix wrong FileSourceFix wrong FileSource.
Also there is a problem with with WITSML parser. It returns `%2` instead of `/` ('FileSource': '%2F45e73ec1-706a-4843-8dea-a4d78638a0bd%2F0c06587c47b2437389fa6f3043a55ffe'). @deny please fix this partFix wrong FileSource.
Also there is a problem with with WITSML parser. It returns `%2` instead of `/` ('FileSource': '%2F45e73ec1-706a-4843-8dea-a4d78638a0bd%2F0c06587c47b2437389fa6f3043a55ffe'). @deny please fix this partM6 - Release 0.9Laurent DenyLaurent Denyhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/13Feature/port to r3 schemas2023-08-18T11:03:58ZSiarhei Khaletski (EPAM)Feature/port to r3 schemasM6 - Release 0.9Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/18GONRG-2592: Move DAG to Airflow 2.02023-08-18T11:03:50ZYan Sushchynski (EPAM)GONRG-2592: Move DAG to Airflow 2.0Code changes to run Airflow 2.0Code changes to run Airflow 2.0M7 - Release 0.10https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/17Fix python sdk installation2023-08-18T11:03:51ZYan Sushchynski (EPAM)Fix python sdk installationInstall Python SDK using pip from `PythonSDK`'s package registry.
Also, there are some import fixes.
Closes https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics-osdu-integration/-/issues/37Install Python SDK using pip from `PythonSDK`'s package registry.
Also, there are some import fixes.
Closes https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics-osdu-integration/-/issues/37M7 - Release 0.10Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/16Consolidate copied lib code to single artifact, use dataset optionally, aws c...2023-08-18T11:03:53ZSpencer Suttonsuttonsp@amazon.comConsolidate copied lib code to single artifact, use dataset optionally, aws changesWe need to not be copying lib code across dags but instead referencing them from a single artifact from the python sdk location.
This MR contains those changes.
Additionally, this has a change that will optionally use dataset service i...We need to not be copying lib code across dags but instead referencing them from a single artifact from the python sdk location.
This MR contains those changes.
Additionally, this has a change that will optionally use dataset service if a dataset_id is passed into the dag.M7 - Release 0.10Spencer Suttonsuttonsp@amazon.comSpencer Suttonsuttonsp@amazon.comhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/33Merge branch 'GONRG-3115_gcp_ini_file_generation' into 'master'2021-09-07T16:17:12ZSiarhei Khaletski (EPAM)Merge branch 'GONRG-3115_gcp_ini_file_generation' into 'master'GONRG-3115: add ini file for GCPGONRG-3115: add ini file for GCPM8 - Release 0.11Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/29GONRG-3115: add ini file for GCP2021-09-07T13:38:26ZAleksandr Spivakov (EPAM)GONRG-3115: add ini file for GCPCloses https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/-/issues/226Closes https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/-/issues/226M8 - Release 0.11Siarhei Khaletski (EPAM)Yan Sushchynski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/28Aws update2021-09-07T07:55:34ZSpencer Suttonsuttonsp@amazon.comAws updateM8 - Release 0.11Spencer Suttonsuttonsp@amazon.comSpencer Suttonsuttonsp@amazon.comhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/27GONRG-3165: change imports to follow new code structure2021-08-31T12:33:03ZYan Sushchynski (EPAM)GONRG-3165: change imports to follow new code structureDue to https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/67 we have to change importsDue to https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/67 we have to change importsM8 - Release 0.11Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/26Updated Python SDK version setup2021-08-19T15:25:16ZSiarhei Khaletski (EPAM)Updated Python SDK version setupM8 - Release 0.11Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/25Updated Python SDK version setup2021-08-19T12:50:43ZSiarhei Khaletski (EPAM)Updated Python SDK version setupM8 - Release 0.11Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/merge_requests/24Ibm dockerfile2023-08-18T11:03:43ZGokul NagareIbm dockerfileM8 - Release 0.11Anuj GuptaAnuj Gupta