WITSML Parser issueshttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues2022-08-23T13:29:37Zhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/36WITSML ingestion integration with Wellbore DDMS2022-08-23T13:29:37ZChris ZhangWITSML ingestion integration with Wellbore DDMSTo support End to end wellbore DDMS workflow
Today WITSML just stores the file, doesn’t extract trajectories and logs or use by applications thru wellbore DDMS. This issue is track the work to integrate WITSML ingestion with Wellbore DDM...To support End to end wellbore DDMS workflow
Today WITSML just stores the file, doesn’t extract trajectories and logs or use by applications thru wellbore DDMS. This issue is track the work to integrate WITSML ingestion with Wellbore DDMS to enable end to end workflow.M13 - Release 0.16https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/21Need common AirFlow GitLab repo directory structure2021-09-07T16:10:18ZJana ScheyNeed common AirFlow GitLab repo directory structureProgress is hindered by the lack of a common repository structure for Airflow components. That is, code for DAGs, operators, parsers called by operators and libraries shared by the parsers.
Currently, each cloud provider is creating its...Progress is hindered by the lack of a common repository structure for Airflow components. That is, code for DAGs, operators, parsers called by operators and libraries shared by the parsers.
Currently, each cloud provider is creating its own repo which is not synchronized with the common publicly-available one. This makes it impossible to test changes made either by CSP or Energistics developers without propagating changes manually.Jay HollingsworthDania Kodeih (Microsoft)Wladmir FrazaoJoeLaurent DenyAsh SathyaseelanDmitriy RudkoJay Hollingsworthhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/34Reimplement work with File Service2021-07-19T19:52:00ZYan Sushchynski (EPAM)Reimplement work with File ServiceThere are changes required in this file: https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics-osdu-integration/-/blob/feature/Port_to_R3_schemas/energistics/src/witsml_parser/create_energistics_manifest.py
At th...There are changes required in this file: https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics-osdu-integration/-/blob/feature/Port_to_R3_schemas/energistics/src/witsml_parser/create_energistics_manifest.py
At the current moment, the DAG expects `preloadFilePath` (an original XML file path in some bucket or storage).
What the main script (create_energistics_manifest.py) does:
1. gets a file's content from `preloadFilePath` (a file path in a bucket);
1. uploads the file using File Service;
1. creates the file's metarecord using File Service and gets the file's id;
1. gets its SignedUrl by the file's Id;
1. downloads the file's contents and parses it;
1. returns the Manifest.
The main concern is that the first three steps (uploading the file to File Service, creating its metarecord etc.) must be **delegated** to a client.
The DAG should expect only a file id and the script(create_energistics_manifest.py) should implement the last three steps.
What the main script (create_energistics_manifest.py) should do:
1. get the file's SignedUrl by the file's Id;
1. download the file's contents and parse it;
1. return the Manifest.
Expected Workflow Payload after changes:
```
{
"executionContext": {
"Payload": {
"AppKey": "test-app",
"data-partition-id": <string>
},
"Context": {
"fileId": <string>
}
}
}
```
Current Workflow Payload:
```
{
"executionContext": {
"Payload": {
"AppKey": "test-app",
"data-partition-id": <string>
},
"Context": {
"acl": {
"viewers": [
<string>
],
"owners": [
<string>
]
},
"legal": {
"legaltags": [
<string>
],
"otherRelevantDataCountries": [
<string>
],
"status": "compliant"
},
"kind": <string>,
"file_name": <string>,
"preload_file_path": <string>,
"version": 5
}
}
}
```https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/44OSS ETP Server Java Contribution2021-10-07T02:12:45ZWelly Tambunancoach@wellytambunan.comOSS ETP Server Java ContributionHi All,
i'm trying to contribute etp-java, not sure if this repo is the right place to do so.. as this spesifically focusing on resqml etc..
is there any place that i can discuss about etp contributor? thanks
here's the initial repo....Hi All,
i'm trying to contribute etp-java, not sure if this repo is the right place to do so.. as this spesifically focusing on resqml etc..
is there any place that i can discuss about etp contributor? thanks
here's the initial repo.. it's just a skeleton for now.
https://github.com/weltam/etp-java
cc @debasisc @Siarhei_Khaletski
cheershttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/47Integration E2E Tests for WITSML Parser - IBM2021-11-08T19:44:26ZChris ZhangIntegration E2E Tests for WITSML Parser - IBMThis is to track the IBM team's work for Integration E2E Tests for WITSML Parser.
Related to issue 41 https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics-osdu-integration/-/issues/41This is to track the IBM team's work for Integration E2E Tests for WITSML Parser.
Related to issue 41 https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics-osdu-integration/-/issues/41M10 - Release 0.13Shrikant Gargjingdong sunShrikant Garghttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/51A proposal to solve "Version" Mapping uncompatibility between Energistics Ab...2022-01-06T14:30:46Zjean-francois RAINAUDA proposal to solve "Version" Mapping uncompatibility between Energistics Abstract Objects and OSDU Id version**DIFFICULTY ENCOUNTERED**
This difficulty was emphazise when we tryed to operate the WITSML parser in the Ingestion Workflow. We constated an inconsistency during integrity checking.
The "Version" is processed differently into OSDU Da...**DIFFICULTY ENCOUNTERED**
This difficulty was emphazise when we tryed to operate the WITSML parser in the Ingestion Workflow. We constated an inconsistency during integrity checking.
The "Version" is processed differently into OSDU Data definition of Id.version and into the Energistics standard on the Uuid.Version and this drives to an uncompatibility between the two ways of managing these "versions".
It looks like In OSDU the version number is an integer which is the "translation" of the "lastUpdate Date" of the Meta data creation of an Object into a number of seconds from 01/01/1900.
In Energistics Common //Eml23, the version is an attribute of AbstractObject. His name is objectVersion: String64. As you can see this is a String which uncompatible with the OSDU Version number. This Attribute is non mandatory.
In this case we cannot execute an easay mapping between these two attributes.
**PROPOSED SOLUTION**
The proposal is to find a way to map another attribute of the Energistics Common // eml23 with the OSDU version number. this attribute : "last Update: Time stamp" is a date , not mandatory in AbstractObject.Citation which could be easilly translated into an OSDU version Number.
If this attribute exists in the "Energistics Entity" when the manifest is created, the the id version number can be calculated from it.
If this attribute does not exist in the "Energistics Entity" it could be generated when the manifest is created and added into the original Energistics XML file before storing it into the persistent store.
By this way we will have a totally consistent management of the id version number between the Energistics Entities and their meta data into the OSDU platform. This will help to solve the inconsistency captured during integrity checking .
If we follow this method we will be able to be totally consistent between the id.version and the uuid."numerical translation of Last update date".https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/55Adjust WITSML Parser to suitable Reference value2022-05-02T15:22:50ZDebasis ChatterjeeAdjust WITSML Parser to suitable Reference value@epeysson - Please see related issue here.
https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/issues/343
@gehrmann is working on getting these entries added for official "Data Definition" release.
At that t...@epeysson - Please see related issue here.
https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/issues/343
@gehrmann is working on getting these entries added for official "Data Definition" release.
At that time, the parser will need adjustment to the value of version 2.0.
Thank you
cc - @chad , @jean_francois.rainaud for informationM12 - Release 0.15etienne peyssonetienne peyssonhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/57Lack of documentation of WITSML parser2023-05-22T18:33:25ZAha!Lack of documentation of WITSML parserCurrently there is an ongoing work to migrate the functionality to enhance the capability of WITSML parser to write WITSML log, trajectory data into Wellbore DDMS.
There is a need to refactor the existing code base to be able to ...Currently there is an ongoing work to migrate the functionality to enhance the capability of WITSML parser to write WITSML log, trajectory data into Wellbore DDMS.
There is a need to refactor the existing code base to be able to support that.
There are few issues identified :
- The [energistics-osdu-integration](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics-osdu-integration) repo is extremely big - there is a lot of code in there so it is difficult to work out what it does.
- No documentation is available as confirmed by Energistics devleopers - it would take a significant amount of time and effort for current developers to understand and work out how all of the code in the repo works.
- Lack of working data (current WITSML supports v2.0) but majority of data avaialble now is in v1.4. - despite numerous requests to the forum, we did not get any feedback.
Actions needed:
1. Understand what all of the functional code does in the repo and document it.
2. Identify code that is not used and remove it.
3. Consider splitting up or refactoring the remaining code into components (e.g. the core parser component)
4. Support from public data
Created from Aha! https://osdu.aha.io/features/TICKETS-11https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/58WITSML Parser is failing with Tubular data2022-06-30T20:14:04ZDebasis ChatterjeeWITSML Parser is failing with Tubular dataTested in Azure R3M11 Preship environment.
I have experienced a failure.
Data file.
[Tubular__witsml-DC.xml](/uploads/9f7618f3cf1825573343cc7a39e6a2bd/Tubular__witsml-DC.xml)
Log
[M11_Azure_WITSML-Tubular-Debasis.txt](/uploads/11ae3e2...Tested in Azure R3M11 Preship environment.
I have experienced a failure.
Data file.
[Tubular__witsml-DC.xml](/uploads/9f7618f3cf1825573343cc7a39e6a2bd/Tubular__witsml-DC.xml)
Log
[M11_Azure_WITSML-Tubular-Debasis.txt](/uploads/11ae3e249b9ddd2bc3b1b62ae902904c/M11_Azure_WITSML-Tubular-Debasis.txt)
cc - @todaiks for informationetienne peyssonetienne peyssonhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/59Support of V1.4 data files2022-05-07T15:09:47ZDebasis ChatterjeeSupport of V1.4 data filesTime and again we are hearing that most data files in real world still use V1.4. So, there is a need to support older version 1.4 over and above current support of V2.0.
See note from TotalEnergy - 6-May-2022.
TotalEnergies – access to...Time and again we are hearing that most data files in real world still use V1.4. So, there is a need to support older version 1.4 over and above current support of V2.0.
See note from TotalEnergy - 6-May-2022.
TotalEnergies – access to representative test data in WITSML V2.0 format (that is what is supported by the Parser today).
- (most of) our suppliers still use 1.4.1 so we do not have WITSML v2.0 data available
cc - @epeysson , @chad , @Keith_Wall , @jean_francois.rainaudhttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/60Implement missing fields following Wellbore Trajectory schema update2022-06-21T18:48:18Zetienne peyssonImplement missing fields following Wellbore Trajectory schema updateFollowing the WEllboare trajectory schema update (1.0.0 -> 1.1.0), we need to add the mapping for the following fields :
```
"AppliedOperations": [
"Example AppliedOperations"
],
"CompanyID": "namespace:master-data--Org...Following the WEllboare trajectory schema update (1.0.0 -> 1.1.0), we need to add the mapping for the following fields :
```
"AppliedOperations": [
"Example AppliedOperations"
],
"CompanyID": "namespace:master-data--Organisation:SomeUniqueOrganisationID:",
```M13 - Release 0.16https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/61WITSML Parser - Well Trajectory Failure2022-08-31T11:19:18ZVadzim KulybaWITSML Parser - Well Trajectory Failure```
[2022-08-29, 20:47:17 UTC] {validate_schema.py:322} ERROR - Schema validation error. Data field.
[2022-08-29, 20:47:17 UTC] {validate_schema.py:323} ERROR - Manifest kind: opendes:wks:work-product-component--WellboreTrajectory:1.1.0
...```
[2022-08-29, 20:47:17 UTC] {validate_schema.py:322} ERROR - Schema validation error. Data field.
[2022-08-29, 20:47:17 UTC] {validate_schema.py:323} ERROR - Manifest kind: opendes:wks:work-product-component--WellboreTrajectory:1.1.0
[2022-08-29, 20:47:17 UTC] {validate_schema.py:324} ERROR - Error: 'Azi' does not match '^[\\w\\-\\.]+:reference-data\\-\\-TrajectoryStationPropertyType:[\\w\\-\\.\\:\\%]+:[0-9]*$'
Failed validating 'pattern' in schema['properties']['data']['allOf'][3]['properties']['AvailableTrajectoryStationProperties']['items']['properties']['TrajectoryStationPropertyTypeID']:
{'description': 'The reference to a trajectory station property type - '
'of if interpreted as channels, the curve or channel '
'name type, identifying e.g. MD, Inclination, Azimuth. '
'This is a relationship to a '
'reference-data--TrajectoryStationPropertyType record '
'id.',
'example': 'partition-id:reference-data--TrajectoryStationPropertyType:AzimuthTN:',
'pattern': '^[\\w\\-\\.]+:reference-data\\-\\-TrajectoryStationPropertyType:[\\w\\-\\.\\:\\%]+:[0-9]*$',
'title': 'Trajectory Station Property Type ID',
'type': 'string',
'x-osdu-relationship': [{'EntityType': 'TrajectoryStationPropertyType',
'GroupType': 'reference-data'}]}
On instance['data']['AvailableTrajectoryStationProperties'][0]['TrajectoryStationPropertyTypeID']:
'Azi'
```
This is error log from azure DEMO validate_manifest_schema_task, but it is common code issue, because it is repro on gcp (cc @Yan_Sushchynski)
I think the main issue inside parser in this line:
https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/blob/master/energistics/src/witsml_parser/energistics/libs/energistics_parsers/witsml_2_0/trajectory_parser.py#L117
Because `tagname` didn't match this schema pattern (cc @epeysson)M14 - Release 0.17https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/62WITSML Parser - SchemaFormatType needs to be updated2023-05-02T20:56:59ZChad LeongWITSML Parser - SchemaFormatType needs to be updatedThe reference data for Energistics SchemaFormatType has been updated in the data definition https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/ReferenceValues/Manifests/reference-data/OPEN/SchemaFormatType.1.0.0.jso...The reference data for Energistics SchemaFormatType has been updated in the data definition https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/ReferenceValues/Manifests/reference-data/OPEN/SchemaFormatType.1.0.0.json to reflect the different WITSML version.
Problem:
The WITSML parser creates a manifest after parsing using the hardcoded value:
https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/blob/master/energistics/src/witsml_parser/energistics/libs/energistics_parsers/parser.py#L446
It needs to be updated to reflect the changes in the data definition.
`osdu:reference-data--SchemaFormatType:EnergisticsWITSML`
to
`osdu:reference-data--SchemaFormatType:Energistics.WITSML.v1.4`https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/63WITSML Parser in M16 (Trajectory data type) is looking for WorkProduct schema...2023-03-12T14:30:48ZDebasis ChatterjeeWITSML Parser in M16 (Trajectory data type) is looking for WorkProduct schema version 1.1.0Noticed this in GCP.
[2022-09-06, 16:02:31 CDT] {validate_schema.py:394} WARNING - Resource with kind odesprod:wks:work-product--WorkProduct:1.1.0 was rejected
But as per Data Definition, there is only version 1.0.0 for this entity.
cc...Noticed this in GCP.
[2022-09-06, 16:02:31 CDT] {validate_schema.py:394} WARNING - Resource with kind odesprod:wks:work-product--WorkProduct:1.1.0 was rejected
But as per Data Definition, there is only version 1.0.0 for this entity.
cc @epeysson and @Yan_Sushchynskihttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/64Refactor DAG related code2023-04-04T10:49:00ZYan Sushchynski (EPAM)Refactor DAG related code### Introduction
There is DAG related code that is executed in the container during a DAG run. The code is [here](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/blob/master/energistics/src...### Introduction
There is DAG related code that is executed in the container during a DAG run. The code is [here](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/blob/master/energistics/src/witsml_parser/main.py) and [here](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/blob/master/energistics/src/witsml_parser/energistics/libs/create_energistics_manifest.py). And this code looks messy and outdated, and requires some refactoring.
### What should be done?
1. Update the code to make it work with the most recent `osdu-*` Python libs. The dependencies are here https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/blob/master/build/requirements.txt
2. Delete deprecated functionality of processing files by `preload_file_path` [here](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/blob/master/energistics/src/witsml_parser/energistics/libs/create_energistics_manifest.py#L314).
3. Add the static-analysis step in the CI/CD.
4. Add possibility to pass the user's access/id token to the DAG
5. Common refactoring, because the code is messy now (a lot of "ifs" and lines of code in a single function)M17 - Release 0.20Vadzim Kulybaharshit aggarwalWalter Detienne peyssonMarc Burnie [AWS]Vadzim Kulybahttps://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/65Well Log record type - does not link to existing Reference data for Curve Types2023-03-13T19:41:17ZDebasis ChatterjeeWell Log record type - does not link to existing Reference data for Curve TypesI noticed that the Parser shows synthetic/random number for Log curve IDs.
It does not attempt to match with existing entries in LogCurve.
"LogCurveTypeID": "namespace:reference-data--LogCurveType:BakerHughesInteq:A08A:",
What ...I noticed that the Parser shows synthetic/random number for Log curve IDs.
It does not attempt to match with existing entries in LogCurve.
"LogCurveTypeID": "namespace:reference-data--LogCurveType:BakerHughesInteq:A08A:",
What are your thoughts? Should this be considered as gap?
I can create issue for tracking.
Copying to others who tested this feature.
Bonus requirements - populate LogCurveFamilyID and LogCurveMainFamilyID.
```
"LogCurveBusinessValueID": "namespace:reference-data--LogCurveBusinessValue:High:",
"LogCurveMainFamilyID": "namespace:reference-data--LogCurveMainFamily:Acoustic:",
"LogCurveFamilyID": "namespace:reference-data--LogCurveFamily:Acoustic%20Amplitude:"
```
Excerpt from WellLog work-product component record created by the Parser.
```
{
"IsProcessed": true,
"LogCurveMainFamilyID": null,
"DateStamp": "2023-03-12T13:14:48.039219+0000",
"LogCurveFamilyID": null,
"CurveID": "92c731a9-ae27-49d8-a246-27ddff7a1ad1",
"TopDepth": 499.0,
"CurveVersion": null,
"InterpreterName": null,
"CurveQuality": null,
"NullValue": null,
"Interpolate": true,
"DepthUnit": "odesprod:reference-data--UnitOfMeasure:m:",
"DepthCoding": null,
"Mnemonic": "ROP",
"BaseDepth": 509.01,
"LogCurveTypeID": null,
"LogCurveBusinessValueID": null,
"CurveUnit": "odesprod:reference-data--UnitOfMeasure:m:"
},
```https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics/witsml-parser/-/issues/66Tubular data type - adjust to recent change in group type (Wpc -> Master)2023-06-01T12:42:42ZDebasis ChatterjeeTubular data type - adjust to recent change in group type (Wpc -> Master)Please see this issue.
https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/issues/551
TubularAssembly and TubularComponentPlease see this issue.
https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/issues/551
TubularAssembly and TubularComponent