Wellbore DDMS Data Loader issueshttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues2024-03-18T04:26:16Zhttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/51Well Log loader from DLIS format2024-03-18T04:26:16ZDebasis ChatterjeeWell Log loader from DLIS formatSimilar to existing LAS loaderSimilar to existing LAS loaderhttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/62Unable to ingest las file due to "timed out"2023-09-25T09:50:24ZDo DangUnable to ingest las file due to "timed out"I have tried to load a smal las file [7004_a1501_1978_comp.las](/uploads/17208d94f2a737114ca873bd672ab186/7004_a1501_1978_comp.las),
however, It runs into a timed out error.
```
New wellbore IDs: ['osdu:master-data--Wellbore:d6f31332097b...I have tried to load a smal las file [7004_a1501_1978_comp.las](/uploads/17208d94f2a737114ca873bd672ab186/7004_a1501_1978_comp.las),
however, It runs into a timed out error.
```
New wellbore IDs: ['osdu:master-data--Wellbore:d6f31332097b4ce48a0e07fea9c16216']
Recognizing the curve families
New welllog IDs: ['osdu:work-product-component--WellLog:cefeedec213c4cbbbd89665f9e223756']
Ingest failed: 7004_a1501_1978_comp.las (see summary for details)
SUMMARY - files not ingested:
7004_a1501_1978_comp.las (timed out)
```
My deployment
```
"service": "Wellbore DDMS OSDU",
"version": "0.2",
"buildNumber": "local",
"cloudEnvironment": "anthos",
"release": "M20"
```
How can I increase the timeout for loading a las file? or any other solution for this problem?https://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/37Trajectory data loader from IOGP P7/17 format2023-09-08T17:42:31ZDebasis ChatterjeeTrajectory data loader from IOGP P7/17 formatPlease check details about this data exchange format
https://www.iogp.org/bookstore/product/p7-17-wellbore-positioning-data-exchange-format/
This program may work similar to "LAS to Wellbore DDMS" tool.Please check details about this data exchange format
https://www.iogp.org/bookstore/product/p7-17-wellbore-positioning-data-exchange-format/
This program may work similar to "LAS to Wellbore DDMS" tool.https://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/58wbdutil - use associated LAS file that is already available in cloud (OSDU DP...2023-03-06T14:12:42ZDebasis Chatterjeewbdutil - use associated LAS file that is already available in cloud (OSDU DP) for case when we use existing WellLog work-product-componentC:\TEMP>wbdutil ingest data --welllog_id osdu:work-product-component--WellLog:3762da4efdbc44c093cca48c597fb3dc **-p "C:\TEMP\7556_l0102_1984_comp.las"** -t %OSDUTOKEN% -c %CONFIGPATH%
For the above use case, the WellLog WPC will have an...C:\TEMP>wbdutil ingest data --welllog_id osdu:work-product-component--WellLog:3762da4efdbc44c093cca48c597fb3dc **-p "C:\TEMP\7556_l0102_1984_comp.las"** -t %OSDUTOKEN% -c %CONFIGPATH%
For the above use case, the WellLog WPC will have an associated LAS file.
In order to be consistent, it would be useful to retrieve that and use as source LAS file. Hence no need for "-p" file part above.
Then things would be more consistent.
cc @chadhttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/13Creating custom schema for LAS headers2022-12-13T08:25:27ZChad LeongCreating custom schema for LAS headers## Introduction
In OSDU, the canonical data model is typically encapsulated and defined by the official OSDU data definition [schema model](https://community.opengroup.org/osdu/platform/system/schema-service/-/tree/master/deployments/sh...## Introduction
In OSDU, the canonical data model is typically encapsulated and defined by the official OSDU data definition [schema model](https://community.opengroup.org/osdu/platform/system/schema-service/-/tree/master/deployments/shared-schemas/osdu) team. These schemas are then deployed as part of the OSDU instance deployment by the CSPs. However, in some cases, it is always possible to for the OSDU adopters to use a data model in ways different from the data modelers intended.
## Objective
As a user, when I have LAS headers that are not modeled in the OSDU canonical model, I want to be able to define a custom source schema and capture the fidelity of the headers – so this can be searched and consumed.
We want to be able to include additional attributes that are not present in the [WellLog 1.1.0 schema](https://community.opengroup.org/osdu/platform/system/schema-service/-/blob/master/deployments/shared-schemas/osdu/work-product-component/WellLog.1.1.0.json)
- Here is an example of a custom schema
[Wellbore-Source_schema_v2.json](/uploads/dc54475c9f9b788fb220d31db68b0873/Wellbore-Source_schema_v2.json)
- The custom attributes are located under definitions/wellboreData/properties/XXXGreg HarrisGreg Harrishttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/47Soft mapping for OSDU Welllog to las file2022-12-13T08:25:26ZGreg HarrisSoft mapping for OSDU Welllog to las fileThis is the complementary change to #45 .
This refactor will allow welllog data downloaded from OSDU to be mapped to a LAS file, using a soft mapping as is now done fro LAS to OSDU data.This is the complementary change to #45 .
This refactor will allow welllog data downloaded from OSDU to be mapped to a LAS file, using a soft mapping as is now done fro LAS to OSDU data.https://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/44Exception display improvement2022-12-13T08:25:26ZChad LeongException display improvementWe need to display the error in more user-friendly way to show that this is a 404 error.
```
C:\Project\OSDU\CodeBase\data-ingestion\LAS\wellbore-ddms-data-loader\venv\Scripts\python.exe C:/Project/OSDU/CodeBase/data-ingestion/LAS/wellb...We need to display the error in more user-friendly way to show that this is a 404 error.
```
C:\Project\OSDU\CodeBase\data-ingestion\LAS\wellbore-ddms-data-loader\venv\Scripts\python.exe C:/Project/OSDU/CodeBase/data-ingestion/LAS/wellbore-ddms-data-loader/demo/LAS_Loader_WellboreDDMS.py
Successfully authenticated
(1/6) Parsing Las File for header info...
(2/6) Creating wellbore record from LAS data...
Traceback (most recent call last):
File "C:\Project\OSDU\CodeBase\data-ingestion\LAS\wellbore-ddms-data-loader\demo\LAS_Loader_WellboreDDMS.py", line 92, in <module>
wellbore_id = res.json()["recordIds"][0]
File "C:\Project\OSDU\CodeBase\data-ingestion\LAS\wellbore-ddms-data-loader\venv\lib\site-packages\httpx\_models.py", line 1514, in json
Response: b'<html>\r\n<head><title>404 Not Found</title></head>\r\n<body>\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>openresty</center>\r\n</body>\r\n</html>\r\n'
return jsonlib.loads(self.content.decode(encoding), **kwargs)
File "C:\Users\0023OI744\AppData\Local\Programs\Python\Python39\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Users\0023OI744\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\0023OI744\AppData\Local\Programs\Python\Python39\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
```
Process finished with exit code 1https://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/35Investigate WITSML parser capability2022-11-29T20:55:12ZChad LeongInvestigate WITSML parser capabilityInvestigate the extensibility of current WITSML parser vs a new parser for WITSML files that support Wellbore DDMS
The parser is here - https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics-osdu-integration/-/tre...Investigate the extensibility of current WITSML parser vs a new parser for WITSML files that support Wellbore DDMS
The parser is here - https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics-osdu-integration/-/tree/master/energisticsNiall McDaidNiall McDaidhttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/42API definition for developers2022-08-23T15:55:56ZGreg HarrisAPI definition for developersWe need some documentation that defines an Application Programming Interface that can be used by other development teams to enable them to use our package within their applications. This is primarily of interest when we come to [integrat...We need some documentation that defines an Application Programming Interface that can be used by other development teams to enable them to use our package within their applications. This is primarily of interest when we come to [integrate with osdu-cli](https://community.opengroup.org/osdu/platform/data-flow/data-loading/wellbore-ddms-data-loader/-/issues/29).
We will also need to update the code base to conform with this API.https://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/29Integration strategy with osdu-cli2022-08-23T15:55:55ZChad LeongIntegration strategy with osdu-cliAs a user, I want to be able to use a single utility to load all my data (reference, master, welllogs, etc) . This single utility is presently available as (https://community.opengroup.org/osdu/platform/data-flow/data-loading/osdu-cli).
...As a user, I want to be able to use a single utility to load all my data (reference, master, welllogs, etc) . This single utility is presently available as (https://community.opengroup.org/osdu/platform/data-flow/data-loading/osdu-cli).
We want to be able to merge this project into the osdu-cli. What is the best way/least friction way to achieve this ?https://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/30Loading WITSML Log data into Wellbore DDMS2022-08-23T13:29:48ZChad LeongLoading WITSML Log data into Wellbore DDMS# Introduction
WITSML Well Log data is another form of logging data format apart from LAS that is developed by Energistics. We want to be able to load this data into the Wellbore DDMS.
# Objective
As a user, I want to be able to load ...# Introduction
WITSML Well Log data is another form of logging data format apart from LAS that is developed by Energistics. We want to be able to load this data into the Wellbore DDMS.
# Objective
As a user, I want to be able to load a [WITSML log data](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics-osdu-integration/-/blob/master/energistics/witsml_data/Log.xml) into the Wellbore DDMS.
- The data loading workflow for loading a WITSML should be identical to loading a LAS.
1. Parse WITSML - We can use integrate this [parser](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics-osdu-integration/-/blob/master/energistics/src/witsml_parser/energistics/libs/energistics_parsers/witsml_2_0/witsml_2_0_xsd_log.py) available under Energistics.
2. Create Wellbore record if not exists
3. Create Welllog record
4. Write bulk data to Welllog record idhttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/31Retreiving WITSML Log data from Wellbore DDMS2022-08-23T10:50:58ZChad LeongRetreiving WITSML Log data from Wellbore DDMShttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/39Launch LAS loader (into Wellbore DDMS) from DAG (Airflow)2022-08-23T10:50:56ZDebasis ChatterjeeLaunch LAS loader (into Wellbore DDMS) from DAG (Airflow)Can you consider creating DAG component for just the LOAD aspect?
Use case -
1. User has already uploaded LAS file into Cloud storage, created wp, wpc, dataaset/file for LAS file and WellLog WPC.
2. The intent now is to launch the LAS-...Can you consider creating DAG component for just the LOAD aspect?
Use case -
1. User has already uploaded LAS file into Cloud storage, created wp, wpc, dataaset/file for LAS file and WellLog WPC.
2. The intent now is to launch the LAS-into-WDMS-loader with all required parameters in body of "trigger workflow".
Also, as catalog records have already been created, this should not attempt to create new Wellbore, nor new WellLog work-product component record.https://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/50Define format for line endings in code2022-05-31T15:15:40ZNiall McDaidDefine format for line endings in codeThere are two types of line endings - "LF" (line feed), "CRLF" (carriage return, line feed)
LF is the default on Unix systems, CRLF is the default on Windows. Our codebase currently uses both and this is causing issues with merge reques...There are two types of line endings - "LF" (line feed), "CRLF" (carriage return, line feed)
LF is the default on Unix systems, CRLF is the default on Windows. Our codebase currently uses both and this is causing issues with merge requests (lots of changes are being shown when in fact the only difference is line endings because different developers have a different setup).
We should choose one format to use, document this in the SMG and if possible, find a way to automate the control of this (so developers don't have to think about it).