WITSML Parser issueshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues2024-01-30T14:30:59Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/68User context is not properly used for some record types2024-01-30T14:30:59ZGuillaume CailletUser context is not properly used for some record typesHello,
While testing the user-context ingestion for the WITSML parser, I found that for some records type, the user was not properly set.
I'm using the Postman Collections from [here](https://community.opengroup.org/osdu/platform/pre-sh...Hello,
While testing the user-context ingestion for the WITSML parser, I found that for some records type, the user was not properly set.
I'm using the Postman Collections from [here](https://community.opengroup.org/osdu/platform/pre-shipping/-/tree/main/R3-M22/AWS-M22/Ingestion%20DAG%20WITSML) to manually ingest records with a "real" user (not the service principal)
Well/Wellbore/Tubular works well, when querying the ingested data I can see that the `createUser` is properly set to "admin-main2@testing.com" (my current user)
For WellboreMarker/Log, the created user is still the service principal account.
I checked the pod logs of the parser, and in both case I properly have this line :
`user_id in Context Initialization is admin-main2@testing.com`
So the correct "real" user is properly sent to the pod in both cases. Looks like the bug is inside the specific code related to the paring of these 3 record types.https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/67New Reference data types are not getting ingested using WITSML parser in Pres...2023-07-27T04:50:09ZDurga Prasad Reddy NadavaluriNew Reference data types are not getting ingested using WITSML parser in Preshipping M19 Azure & GCP environment.Currently I am using M19 preshipping Azure & GCP environment to test the below case.
[Well.xml](/uploads/88d51b6a96246fae9fc181bb26cd041b/Well.xml)
I am trying to ingest a well WITSML file (attached to this issue) which has few referen...Currently I am using M19 preshipping Azure & GCP environment to test the below case.
[Well.xml](/uploads/88d51b6a96246fae9fc181bb26cd041b/Well.xml)
I am trying to ingest a well WITSML file (attached to this issue) which has few reference data types along master data. Below I have copied few reference attributes from the XML file which I am trying to ingest.
<WellDatum uid="KB">
<Name>DKelly Bushing</Name>
<Code>Dkelly bushing</Code>
<Elevation uom="ft" datum="SL">78.5</Elevation>
<Crs xsi:type="eml:VerticalUnknownCrs">
<eml:Unknown>unknown</eml:Unknown>
</Crs>
</WellDatum>
<WellDatum uid="SL">
<Name>DSea Level</Name>
<Code>Dmean sea level</Code>
<Crs xsi:type="eml:VerticalUnknownCrs">
<eml:Unknown>unknown</eml:Unknown>
</Crs>
</WellDatum>
When I am trying to ingested this XML file, ideally it should ingest reference data first as per hierarchy and then rest of the master data but here in this example, it searches for reference data and shows error that missing in the instance.
If I use Kelly Bushing instead of DKelly Bushing then ingestion will work and able to see the results in search/storage query. But I use a name which has not ingested so far in the given instance like example : DKelly Bushing, Rotary Table then it will throw error.
Errors which I am seeing in Airflow:
{'id': 'm19:master-data--Well:Durga70636287-98CA-4684-AD49-F65BC1D925FD', 'kind': 'osdu:wks:master-data--Well:1.0.0', 'reason': 'Missing parents: {SRN: m19:reference-data--VerticalMeasurementType:Dkelly%20bushing, SRN: m19:reference-data--VerticalMeasurementType:Dmean%20sea%20level}
Open questions:
1. Do I need to ingest the reference data types from the XML file first before ingesting any of the data?
2. Does anyone faced similar problems when they to ingest a reference data which is not ingested till now?https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/66Tubular data type - adjust to recent change in group type (Wpc -> Master)2023-06-01T12:42:42ZDebasis ChatterjeeTubular data type - adjust to recent change in group type (Wpc -> Master)Please see this issue.
https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/issues/551
TubularAssembly and TubularComponentPlease see this issue.
https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/issues/551
TubularAssembly and TubularComponenthttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/65Well Log record type - does not link to existing Reference data for Curve Types2023-03-13T19:41:17ZDebasis ChatterjeeWell Log record type - does not link to existing Reference data for Curve TypesI noticed that the Parser shows synthetic/random number for Log curve IDs.
It does not attempt to match with existing entries in LogCurve.
"LogCurveTypeID": "namespace:reference-data--LogCurveType:BakerHughesInteq:A08A:",
What ...I noticed that the Parser shows synthetic/random number for Log curve IDs.
It does not attempt to match with existing entries in LogCurve.
"LogCurveTypeID": "namespace:reference-data--LogCurveType:BakerHughesInteq:A08A:",
What are your thoughts? Should this be considered as gap?
I can create issue for tracking.
Copying to others who tested this feature.
Bonus requirements - populate LogCurveFamilyID and LogCurveMainFamilyID.
```
"LogCurveBusinessValueID": "namespace:reference-data--LogCurveBusinessValue:High:",
"LogCurveMainFamilyID": "namespace:reference-data--LogCurveMainFamily:Acoustic:",
"LogCurveFamilyID": "namespace:reference-data--LogCurveFamily:Acoustic%20Amplitude:"
```
Excerpt from WellLog work-product component record created by the Parser.
```
{
"IsProcessed": true,
"LogCurveMainFamilyID": null,
"DateStamp": "2023-03-12T13:14:48.039219+0000",
"LogCurveFamilyID": null,
"CurveID": "92c731a9-ae27-49d8-a246-27ddff7a1ad1",
"TopDepth": 499.0,
"CurveVersion": null,
"InterpreterName": null,
"CurveQuality": null,
"NullValue": null,
"Interpolate": true,
"DepthUnit": "odesprod:reference-data--UnitOfMeasure:m:",
"DepthCoding": null,
"Mnemonic": "ROP",
"BaseDepth": 509.01,
"LogCurveTypeID": null,
"LogCurveBusinessValueID": null,
"CurveUnit": "odesprod:reference-data--UnitOfMeasure:m:"
},
```https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/64Refactor DAG related code2023-04-04T10:49:00ZYan Sushchynski (EPAM)Refactor DAG related code### Introduction
There is DAG related code that is executed in the container during a DAG run. The code is [here](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/blob/master/energistics/src...### Introduction
There is DAG related code that is executed in the container during a DAG run. The code is [here](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/blob/master/energistics/src/witsml_parser/main.py) and [here](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/blob/master/energistics/src/witsml_parser/energistics/libs/create_energistics_manifest.py). And this code looks messy and outdated, and requires some refactoring.
### What should be done?
1. Update the code to make it work with the most recent `osdu-*` Python libs. The dependencies are here https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/blob/master/build/requirements.txt
2. Delete deprecated functionality of processing files by `preload_file_path` [here](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/blob/master/energistics/src/witsml_parser/energistics/libs/create_energistics_manifest.py#L314).
3. Add the static-analysis step in the CI/CD.
4. Add possibility to pass the user's access/id token to the DAG
5. Common refactoring, because the code is messy now (a lot of "ifs" and lines of code in a single function)M17 - Release 0.20Vadzim Kulybaharshit aggarwalWalter Detienne peyssonMarc Burnie [AWS]Vadzim Kulybahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/63WITSML Parser in M16 (Trajectory data type) is looking for WorkProduct schema...2023-03-12T14:30:48ZDebasis ChatterjeeWITSML Parser in M16 (Trajectory data type) is looking for WorkProduct schema version 1.1.0Noticed this in GCP.
[2022-09-06, 16:02:31 CDT] {validate_schema.py:394} WARNING - Resource with kind odesprod:wks:work-product--WorkProduct:1.1.0 was rejected
But as per Data Definition, there is only version 1.0.0 for this entity.
cc...Noticed this in GCP.
[2022-09-06, 16:02:31 CDT] {validate_schema.py:394} WARNING - Resource with kind odesprod:wks:work-product--WorkProduct:1.1.0 was rejected
But as per Data Definition, there is only version 1.0.0 for this entity.
cc @epeysson and @Yan_Sushchynskihttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/62WITSML Parser - SchemaFormatType needs to be updated2023-05-02T20:56:59ZChad LeongWITSML Parser - SchemaFormatType needs to be updatedThe reference data for Energistics SchemaFormatType has been updated in the data definition https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/ReferenceValues/Manifests/reference-data/OPEN/SchemaFormatType.1.0.0.jso...The reference data for Energistics SchemaFormatType has been updated in the data definition https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/ReferenceValues/Manifests/reference-data/OPEN/SchemaFormatType.1.0.0.json to reflect the different WITSML version.
Problem:
The WITSML parser creates a manifest after parsing using the hardcoded value:
https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/blob/master/energistics/src/witsml_parser/energistics/libs/energistics_parsers/parser.py#L446
It needs to be updated to reflect the changes in the data definition.
`osdu:reference-data--SchemaFormatType:EnergisticsWITSML`
to
`osdu:reference-data--SchemaFormatType:Energistics.WITSML.v1.4`https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/61WITSML Parser - Well Trajectory Failure2022-08-31T11:19:18ZVadzim KulybaWITSML Parser - Well Trajectory Failure```
[2022-08-29, 20:47:17 UTC] {validate_schema.py:322} ERROR - Schema validation error. Data field.
[2022-08-29, 20:47:17 UTC] {validate_schema.py:323} ERROR - Manifest kind: opendes:wks:work-product-component--WellboreTrajectory:1.1.0
...```
[2022-08-29, 20:47:17 UTC] {validate_schema.py:322} ERROR - Schema validation error. Data field.
[2022-08-29, 20:47:17 UTC] {validate_schema.py:323} ERROR - Manifest kind: opendes:wks:work-product-component--WellboreTrajectory:1.1.0
[2022-08-29, 20:47:17 UTC] {validate_schema.py:324} ERROR - Error: 'Azi' does not match '^[\\w\\-\\.]+:reference-data\\-\\-TrajectoryStationPropertyType:[\\w\\-\\.\\:\\%]+:[0-9]*$'
Failed validating 'pattern' in schema['properties']['data']['allOf'][3]['properties']['AvailableTrajectoryStationProperties']['items']['properties']['TrajectoryStationPropertyTypeID']:
{'description': 'The reference to a trajectory station property type - '
'of if interpreted as channels, the curve or channel '
'name type, identifying e.g. MD, Inclination, Azimuth. '
'This is a relationship to a '
'reference-data--TrajectoryStationPropertyType record '
'id.',
'example': 'partition-id:reference-data--TrajectoryStationPropertyType:AzimuthTN:',
'pattern': '^[\\w\\-\\.]+:reference-data\\-\\-TrajectoryStationPropertyType:[\\w\\-\\.\\:\\%]+:[0-9]*$',
'title': 'Trajectory Station Property Type ID',
'type': 'string',
'x-osdu-relationship': [{'EntityType': 'TrajectoryStationPropertyType',
'GroupType': 'reference-data'}]}
On instance['data']['AvailableTrajectoryStationProperties'][0]['TrajectoryStationPropertyTypeID']:
'Azi'
```
This is error log from azure DEMO validate_manifest_schema_task, but it is common code issue, because it is repro on gcp (cc @Yan_Sushchynski)
I think the main issue inside parser in this line:
https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/blob/master/energistics/src/witsml_parser/energistics/libs/energistics_parsers/witsml_2_0/trajectory_parser.py#L117
Because `tagname` didn't match this schema pattern (cc @epeysson)M14 - Release 0.17https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/60Implement missing fields following Wellbore Trajectory schema update2022-06-21T18:48:18Zetienne peyssonImplement missing fields following Wellbore Trajectory schema updateFollowing the WEllboare trajectory schema update (1.0.0 -> 1.1.0), we need to add the mapping for the following fields :
```
"AppliedOperations": [
"Example AppliedOperations"
],
"CompanyID": "namespace:master-data--Org...Following the WEllboare trajectory schema update (1.0.0 -> 1.1.0), we need to add the mapping for the following fields :
```
"AppliedOperations": [
"Example AppliedOperations"
],
"CompanyID": "namespace:master-data--Organisation:SomeUniqueOrganisationID:",
```M13 - Release 0.16https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/59Support of V1.4 data files2022-05-07T15:09:47ZDebasis ChatterjeeSupport of V1.4 data filesTime and again we are hearing that most data files in real world still use V1.4. So, there is a need to support older version 1.4 over and above current support of V2.0.
See note from TotalEnergy - 6-May-2022.
TotalEnergies – access to...Time and again we are hearing that most data files in real world still use V1.4. So, there is a need to support older version 1.4 over and above current support of V2.0.
See note from TotalEnergy - 6-May-2022.
TotalEnergies – access to representative test data in WITSML V2.0 format (that is what is supported by the Parser today).
- (most of) our suppliers still use 1.4.1 so we do not have WITSML v2.0 data available
cc - @epeysson , @chad , @Keith_Wall , @jean_francois.rainaudhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/58WITSML Parser is failing with Tubular data2022-06-30T20:14:04ZDebasis ChatterjeeWITSML Parser is failing with Tubular dataTested in Azure R3M11 Preship environment.
I have experienced a failure.
Data file.
[Tubular__witsml-DC.xml](/uploads/9f7618f3cf1825573343cc7a39e6a2bd/Tubular__witsml-DC.xml)
Log
[M11_Azure_WITSML-Tubular-Debasis.txt](/uploads/11ae3e2...Tested in Azure R3M11 Preship environment.
I have experienced a failure.
Data file.
[Tubular__witsml-DC.xml](/uploads/9f7618f3cf1825573343cc7a39e6a2bd/Tubular__witsml-DC.xml)
Log
[M11_Azure_WITSML-Tubular-Debasis.txt](/uploads/11ae3e249b9ddd2bc3b1b62ae902904c/M11_Azure_WITSML-Tubular-Debasis.txt)
cc - @todaiks for informationetienne peyssonetienne peyssonhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/57Lack of documentation of WITSML parser2023-05-22T18:33:25ZAha!Lack of documentation of WITSML parserCurrently there is an ongoing work to migrate the functionality to enhance the capability of WITSML parser to write WITSML log, trajectory data into Wellbore DDMS.
There is a need to refactor the existing code base to be able to ...Currently there is an ongoing work to migrate the functionality to enhance the capability of WITSML parser to write WITSML log, trajectory data into Wellbore DDMS.
There is a need to refactor the existing code base to be able to support that.
There are few issues identified :
- The [energistics-osdu-integration](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics-osdu-integration) repo is extremely big - there is a lot of code in there so it is difficult to work out what it does.
- No documentation is available as confirmed by Energistics devleopers - it would take a significant amount of time and effort for current developers to understand and work out how all of the code in the repo works.
- Lack of working data (current WITSML supports v2.0) but majority of data avaialble now is in v1.4. - despite numerous requests to the forum, we did not get any feedback.
Actions needed:
1. Understand what all of the functional code does in the repo and document it.
2. Identify code that is not used and remove it.
3. Consider splitting up or refactoring the remaining code into components (e.g. the core parser component)
4. Support from public data
Created from Aha! https://osdu.aha.io/features/TICKETS-11https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/56Lack of documentation of WITSML parser2022-04-28T16:51:04ZAha!Lack of documentation of WITSML parserAdding documentation
Created from Aha! https://osdu.aha.io/features/TICKETS-11Adding documentation
Created from Aha! https://osdu.aha.io/features/TICKETS-11M12 - Release 0.15https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/55Adjust WITSML Parser to suitable Reference value2022-05-02T15:22:50ZDebasis ChatterjeeAdjust WITSML Parser to suitable Reference value@epeysson - Please see related issue here.
https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/issues/343
@gehrmann is working on getting these entries added for official "Data Definition" release.
At that t...@epeysson - Please see related issue here.
https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/issues/343
@gehrmann is working on getting these entries added for official "Data Definition" release.
At that time, the parser will need adjustment to the value of version 2.0.
Thank you
cc - @chad , @jean_francois.rainaud for informationM12 - Release 0.15etienne peyssonetienne peyssonhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/54Use Wellbore Trajectory schema version 1.1.02022-06-21T18:48:59ZDebasis ChatterjeeUse Wellbore Trajectory schema version 1.1.0As per my recent discussion with Etienne, we looked at the Station properties array. This is one thing to leverage from the new schema 1.1.0.
(@gehrmann - Can you please point us to "change log" from 1.0.0 to 1.1.0 so that we can ensur...As per my recent discussion with Etienne, we looked at the Station properties array. This is one thing to leverage from the new schema 1.1.0.
(@gehrmann - Can you please point us to "change log" from 1.0.0 to 1.1.0 so that we can ensure that this revision makes use of enriched features of 1.1.0)
[WellboreTrajectory.1.1.0.json](/uploads/cce8064e9b610493c099d0243dec3cff/WellboreTrajectory.1.1.0.json)
This would be good place to capture the list of measurements for each station (similar to list of curves in WellLog work-product component).
```
"AvailableTrajectoryStationProperties": [
{
"TrajectoryStationPropertyTypeID": "partition-id:reference-data--TrajectoryStationPropertyType:AzimuthTN:",
"StationPropertyUnitID": "partition-id:reference-data--UnitOfMeasure:dega:",
"Name": "AzimuthTN"
}
],
```
See below some common measured properties for each station depth.
- MeasuredDepth
- TVD
- Azimuth
- Inclination
- SurfaceX
- SurfaceY
Also see this Reference entity and its value list for an understanding of mapping.
https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/blob/master/E-R/reference-data/TrajectoryStationPropertyType.1.0.0.mdM12 - Release 0.15etienne peyssonetienne peyssonhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/53Code clean up2022-06-17T14:01:12Zetienne peyssonCode clean upAs it is the codebase is a little bit messy.
Some simple clean up could ease the reading...As it is the codebase is a little bit messy.
Some simple clean up could ease the reading...M10 - Release 0.13etienne peyssonetienne peyssonhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/52Witsml parser - Wrong encoding for double colons - CoordinateReferenceSystem2022-08-23T13:29:38Zetienne peyssonWitsml parser - Wrong encoding for double colons - CoordinateReferenceSystemEven though the DAG passes, looking at Xcom logs we see skipped ids :
{'provide_manifest_integrity_task': [{'id': 'odesprod:work-product--WorkProduct:TEST_TRAJECTORY_002', 'kind': 'odesprod:wks:work-product--WorkProduct:1.0.0', 'reason'...Even though the DAG passes, looking at Xcom logs we see skipped ids :
{'provide_manifest_integrity_task': [{'id': 'odesprod:work-product--WorkProduct:TEST_TRAJECTORY_002', 'kind': 'odesprod:wks:work-product--WorkProduct:1.0.0', 'reason': 'Missing parents: {SRN: odesprod:work-product-component--WellboreTrajectory:TEST_TRAJECTORY_002}'}, {'id': 'odesprod:work-product-component--WellboreTrajectory:TEST_TRAJECTORY_002', 'kind': 'odesprod:wks:work-product-component--WellboreTrajectory:1.0.0', 'reason': 'Missing parents: {SRN: odesprod:reference-data--CoordinateReferenceSystem:odesprod%3Areference-data--CoordinateReferenceSystem%3AGeodeticCRS%253A%253AEPSG%253A%253A4230%3A, SRN: odesprod:reference-data--CoordinateReferenceSystem:GeodeticCRS%3A%3AEPSG%3A%3A4230}'}]}
The witsml file is the one for the trajectory and the concerned field the following :
```
<Crs xsi:type="eml:GeodeticLocalAuthorityCrs">
<eml:LocalAuthorityCrsName authority="EPSG">GeodeticCRS::EPSG::4230</eml:LocalAuthorityCrsName>
</Crs>
```
As you can see we need to fill in details about the ID of the reference data but as it contains double colons, it does not pass.M10 - Release 0.13https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/51A proposal to solve "Version" Mapping uncompatibility between Energistics Ab...2022-01-06T14:30:46Zjean-francois RAINAUDA proposal to solve "Version" Mapping uncompatibility between Energistics Abstract Objects and OSDU Id version**DIFFICULTY ENCOUNTERED**
This difficulty was emphazise when we tryed to operate the WITSML parser in the Ingestion Workflow. We constated an inconsistency during integrity checking.
The "Version" is processed differently into OSDU Da...**DIFFICULTY ENCOUNTERED**
This difficulty was emphazise when we tryed to operate the WITSML parser in the Ingestion Workflow. We constated an inconsistency during integrity checking.
The "Version" is processed differently into OSDU Data definition of Id.version and into the Energistics standard on the Uuid.Version and this drives to an uncompatibility between the two ways of managing these "versions".
It looks like In OSDU the version number is an integer which is the "translation" of the "lastUpdate Date" of the Meta data creation of an Object into a number of seconds from 01/01/1900.
In Energistics Common //Eml23, the version is an attribute of AbstractObject. His name is objectVersion: String64. As you can see this is a String which uncompatible with the OSDU Version number. This Attribute is non mandatory.
In this case we cannot execute an easay mapping between these two attributes.
**PROPOSED SOLUTION**
The proposal is to find a way to map another attribute of the Energistics Common // eml23 with the OSDU version number. this attribute : "last Update: Time stamp" is a date , not mandatory in AbstractObject.Citation which could be easilly translated into an OSDU version Number.
If this attribute exists in the "Energistics Entity" when the manifest is created, the the id version number can be calculated from it.
If this attribute does not exist in the "Energistics Entity" it could be generated when the manifest is created and added into the original Energistics XML file before storing it into the persistent store.
By this way we will have a totally consistent management of the id version number between the Energistics Entities and their meta data into the OSDU platform. This will help to solve the inconsistency captured during integrity checking .
If we follow this method we will be able to be totally consistent between the id.version and the uuid."numerical translation of Last update date".https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/50ADR: Add DAG Bootstrap2022-02-07T17:01:03ZYan Sushchynski (EPAM)ADR: Add DAG BootstrapI believe, that we should start following the common approach of bootstrapping DAGs in this project. For this purpose we need to add placeholders to the DAG in order to replace them with each CSP's values within `bootstrap_dag` step.
Th...I believe, that we should start following the common approach of bootstrapping DAGs in this project. For this purpose we need to add placeholders to the DAG in order to replace them with each CSP's values within `bootstrap_dag` step.
This was implemented for other DAGs (Segy->Zgy, CSV)
Now CSP's values are specified in the `yaml`-file, which is parsed as a dictionary and passed as `**kwargs` into KubernetesPodOperator.
(https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics-osdu-integration/-/blob/master/devops/osdu-gcp/airflow_configs.yaml)
MR with adding Bootstrap for GCP:
https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics-osdu-integration/-/merge_requests/47M10 - Release 0.13Vadzim KulybaSiarhei Khaletski (EPAM)Spencer Suttonsuttonsp@amazon.comGokul NagareAleksandr Spivakov (EPAM)Vadzim Kulybahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/49WITSML parser support on Azure2022-08-23T13:14:43ZKishore BattulaWITSML parser support on Azure**Topic**: `WITSML parser support on Azure`
- [x] Code changes need to support WITSML parser on Azure
- [x] Pipeline to deploy WITSML changes to gitlab environment
- [x] Postman E2E tests integration into MR & merge pipelines
- [ ] ADO ...**Topic**: `WITSML parser support on Azure`
- [x] Code changes need to support WITSML parser on Azure
- [x] Pipeline to deploy WITSML changes to gitlab environment
- [x] Postman E2E tests integration into MR & merge pipelines
- [ ] ADO pipelines to deploy into Dev & Demo environments
- [x] Pre-shipping scripts and documentation to load & Validate WITSML parser on pre-shipping environmentM13 - Release 0.16Vadzim KulybaVadzim Kulyba