Wellbore DDMS Data Loader issueshttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues2024-01-25T11:16:40Zhttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/60Unable to ingest a LAS file size of 5 MB using wellbore DDMS - Getting error ...2024-01-25T11:16:40ZDurga Prasad Reddy NadavaluriUnable to ingest a LAS file size of 5 MB using wellbore DDMS - Getting error "413 Request Entity Too Large"When I am trying to ingest a welllog LAS file (size of 4-5 MB) using wellbore DDMS API in Azure M14 environment, I am getting error as "413 Request Entity Too Large". In general log las sizes vary from 2-8 MB.
Is there a size limit to i...When I am trying to ingest a welllog LAS file (size of 4-5 MB) using wellbore DDMS API in Azure M14 environment, I am getting error as "413 Request Entity Too Large". In general log las sizes vary from 2-8 MB.
Is there a size limit to ingesting las files and if yes, how can it be extended?
Attached a dummy LAS file to this issue.
[Dummy_LAS.las](/uploads/87f92bb71570c0fac93b0be6e41c3c59/Dummy_LAS.las)https://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/59LAS loader (Wellbore DDMS) - run time messages2023-06-25T11:24:02ZDebasis ChatterjeeLAS loader (Wellbore DDMS) - run time messages@chad - great improvement with messaging.
I made one pass successfully for existing WellLog WPC record.
A few comments -
1. Fix typo error "Begining" -> :Beginning"
2. Mention rowcount - as read from LAS file for curve data and the same...@chad - great improvement with messaging.
I made one pass successfully for existing WellLog WPC record.
A few comments -
1. Fix typo error "Begining" -> :Beginning"
2. Mention rowcount - as read from LAS file for curve data and the same for output (how many written in Wellbore DDMS).
3. Bonus - this could serve as "unit test". show curve values at certain depth in the file (not the very first or last, somewhere in between) and show equivalent value as seen from Wellbore DDMS.I myself do that spot check when I run this utility as tester.
```
C:\TEMP>wbdutil ingest data --welllog_id osdu:work-product-component--WellLog:b43d82d4b6fa4c7caaf0402f85d9c517 -p "C:\TEMP\7556_l0102_1984_comp.las" -t %OSDUTOKEN% -c %CONFIGPATH%
Begining ingest of LAS file: C:\TEMP\7556_l0102_1984_comp.las
Retrieving well log record and associated information.
Validating LAS file well name and curves against existing well log record.
LAS well name: 7023
Well log record well name: 7023
LAS curves: ['DEPT', 'GR', 'DT', 'RHOB', 'NPHI']
Well log record curves: ['DEPT', 'GR', 'DT', 'RHOB', 'NPHI']
Ingest complete: Data written from 7556_l0102_1984_comp.las to Well Log (osdu:work-product-component--WellLog:b43d82d4b6fa4c7caaf0402f85d9c517).
C:\TEMP>
```https://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/61Investigate issue from preshipping GC M182023-06-25T11:24:02ZChad LeongInvestigate issue from preshipping GC M18https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/527https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/527https://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/49Failed to ingest LAS data when UWI is empty or not present2023-03-15T19:24:31ZDipika KheraFailed to ingest LAS data when UWI is empty or not presentFailed to ingest LAS data using `wbdutil` when Well Name present but UWI is empty.
the one which is not working -
```
WELL. 15/9-F-15 : WELL
UWI. : UNIQUE WELL ID
```
other which is working -
```
WELL . ...Failed to ingest LAS data using `wbdutil` when Well Name present but UWI is empty.
the one which is not working -
```
WELL. 15/9-F-15 : WELL
UWI. : UNIQUE WELL ID
```
other which is working -
```
WELL . NO_15/9-F-5: WELL
UWI . 15/9-F-5: UNIQUE WELL ID
```
**Error Logs:**
![image](/uploads/ac43b2c384455580edfb7bb627ad7a9f/image.png)
FYI: @chadGreg HarrisNiall McDaidGreg Harrishttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/56Well log loader from LAS into Wellbore DDMS - please make error message more ...2023-03-15T18:19:02ZDebasis ChatterjeeWell log loader from LAS into Wellbore DDMS - please make error message more verbose to help troubleshootingWhen running, this gave me the following message.
C:\TEMP>wbdutil ingest data --welllog_id osdu:work-product-component--WellLog:77874e51358347b9a5a8a4309ef50bfc -p "C:\TEMP\7556_l0102_1984_comp-WDMS.las" -t %OSDUTOKEN% -c %CONFIGPATH%
D...When running, this gave me the following message.
C:\TEMP>wbdutil ingest data --welllog_id osdu:work-product-component--WellLog:77874e51358347b9a5a8a4309ef50bfc -p "C:\TEMP\7556_l0102_1984_comp-WDMS.las" -t %OSDUTOKEN% -c %CONFIGPATH%
Data not ingested - LAS file validation failed: **Well name associated with well log record does not match well name in LAS file**.
Ingest complete: Data written from 7556_l0102_1984_comp-WDMS.las to Well Log (osdu:work-product-component--WellLog:77874e51358347b9a5a8a4309ef50bfc).
C:\TEMP>
It would be helpful if it shows both values.
1. What is obtained from work-product-component WellLog record.
2. What is obtained from LAS header
We also know that there is issue with convention LAS header says "**Well**" whereas in OSDU DP wpc WellLog is linked to parent **Wellbore**.
So, even that part can be clarified in error message.
cc @chad for informationhttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/57wbdutil - status message misleading after failure2023-03-15T18:19:01ZDebasis Chatterjeewbdutil - status message misleading after failureSee for example this - The job failed. Yet it says "Ingest complete".
Propose saying "Ingest failed" instead of "Ingest complete".
```
C:\TEMP>wbdutil ingest data --welllog_id osdu:work-product-component--WellLog:77874e51358347b9a5a8a43...See for example this - The job failed. Yet it says "Ingest complete".
Propose saying "Ingest failed" instead of "Ingest complete".
```
C:\TEMP>wbdutil ingest data --welllog_id osdu:work-product-component--WellLog:77874e51358347b9a5a8a4309ef50bfc -p "C:\TEMP\7556_l0102_1984_comp-WDMS.las" -t %OSDUTOKEN% -c %CONFIGPATH%
Data not ingested - LAS file validation failed: Well name associated with well log record does not match well name in LAS file.
Ingest complete: Data written from 7556_l0102_1984_comp-WDMS.las to Well Log (osdu:work-product-component--WellLog:77874e51358347b9a5a8a4309ef50bfc).
```
Another example
```
C:\TEMP>wbdutil ingest data --welllog_id osdu:work-product-component--WellLog:77874e51358347b9a5a8a4309ef50bfc -p "C:\TEMP\7556_l0102_1984_comp-WDMS.las" -t %OSDUTOKEN% -c %CONFIGPATH%
Data not ingested: HTTP error: 401. Calling: 'https://r3m15.preshiptesting.osdu.aws/api/os-wellbore-ddms/ddms/v3/welllogs/osdu:work-product-component--WellLog:77874e51358347b9a5a8a4309ef50bfc'. {"reason":"Access denied","message":"The user is not authorized to perform this action. Invalid token.","status":401}
Ingest complete: Data written from 7556_l0102_1984_comp-WDMS.las to Well Log (osdu:work-product-component--WellLog:77874e51358347b9a5a8a4309ef50bfc).
```https://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/53Not able to load Welllog Data from LAS file using wbdutil, because of NumberO...2023-01-02T09:32:20ZKamlesh TodaiNot able to load Welllog Data from LAS file using wbdutil, because of NumberOfColumns field for the curve data has been changed to required field while creating the log header.
**wbdutil ingest data --welllog_id "odesprod:work-product-component--WellLog:9aab0457b54b44f3b1f0811aab118561" -p "D:\OSDU\dev\LoadLogData\wellbore-ddms-data-loader-main\test-data\1013_akm11_1978_comp.las" -c %CONFIGPATH% -t %OSDUTOKEN%...
**wbdutil ingest data --welllog_id "odesprod:work-product-component--WellLog:9aab0457b54b44f3b1f0811aab118561" -p "D:\OSDU\dev\LoadLogData\wellbore-ddms-data-loader-main\test-data\1013_akm11_1978_comp.las" -c %CONFIGPATH% -t %OSDUTOKEN%**
HTTP error: 400. Calling: 'https://preship.gcp.gnrg-osdu.projects.epam.com/api/os-wellbore-ddms/ddms/v3/welllogs/odesprod:work-product-component--WellLog:9aab0457b54b44f3b1f0811aab118561/data'. {"detail":"The number of columns for curve(s): {'DEPT': 1, 'GR': 1, 'DT': 1, 'RHOB': 1, 'DRHO': 1, 'NPHI': 1} in the bulk data do(es) not match the 'NumberOfColumns' property value in the WellLog record for CurveID: {'DEPT': None, 'GR': None, 'DT': None, 'RHOB': None, 'DRHO': None, 'NPHI': None} ."}
Traceback (most recent call last):
File "C:\Python38\lib\site-packages\knack\cli.py", line 231, in invoke
cmd_result = self.invocation.execute(args)
File "C:\Python38\lib\site-packages\knack\invocation.py", line 223, in execute
cmd_result = parsed_args.func(params)
File "C:\Python38\lib\site-packages\knack\commands.py", line 146, in __call__
return self.handler(*args, **kwargs)
File "C:\Python38\lib\site-packages\knack\commands.py", line 253, in _command_handler
result = op(client, **command_args) if client else op(**command_args)
File "C:\Python38\lib\site-packages\wbdutil\commands\ingest.py", line 80, in welllog_data
service.ingest_welllog_data(input_path, welllog_id)
File "C:\Python38\lib\site-packages\wbdutil\well_service.py", line 149, in ingest_welllog_data
self._client.add_welllog_data(data, welllog_id)
File "C:\Python38\lib\site-packages\wbdutil\osdu_client.py", line 112, in add_welllog_data
self._send_request_json_response("POST", url, payload, None, headers=headers)
File "C:\Python38\lib\site-packages\wbdutil\osdu_client.py", line 181, in _send_request_json_response
res = self._send_request(method, url, content, json, headers)
File "C:\Python38\lib\site-packages\wbdutil\osdu_client.py", line 193, in _send_request
raise DataLoaderWebResponseError(res.status_code, url, res.text)
wbdutil.osdu_client.DataLoaderWebResponseError: HTTP error: 400. Calling: 'https://preship.gcp.gnrg-osdu.projects.epam.com/api/os-wellbore-ddms/ddms/v3/welllogs/odesprod:work-product-component--WellLog:9aab0457b54b44f3b1f0811aab118561/data'. {"detail":"The number of columns for curve(s): {'DEPT': 1, 'GR': 1, 'DT': 1, 'RHOB': 1, 'DRHO': 1, 'NPHI': 1} **in the bulk data do(es) not match the 'NumberOfColumns' property value in the WellLog record for CurveID: {'DEPT': None, 'GR': None, 'DT': None, 'RHOB': None, 'DRHO': None, 'NPHI': None} ."}**
@debasiscChad LeongChad Leonghttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/9Add support for well bore/log download from an OSDU cloud instance.2022-12-13T08:25:27ZGreg HarrisAdd support for well bore/log download from an OSDU cloud instance.Write classes that allow Well Bore and Well Logs to be downloaded from OSDU.
- Get well bore
- Get Well log
- Get bulk log data
Add command line options to support this.
Test by writing data to OSDU and pulling it back.Write classes that allow Well Bore and Well Logs to be downloaded from OSDU.
- Get well bore
- Get Well log
- Get bulk log data
Add command line options to support this.
Test by writing data to OSDU and pulling it back.M9 - Release 0.12https://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/7Support for file download using OSDU SDK.2022-12-13T08:25:27ZGreg HarrisSupport for file download using OSDU SDK.Implement a class that can retrieve LAS files from OSDU storage using the OSDU SDK.
- Add a command line flag to allow users to access this functionality.
- At this stage we will need to tackle authentication against OSDU.
* This assume...Implement a class that can retrieve LAS files from OSDU storage using the OSDU SDK.
- Add a command line flag to allow users to access this functionality.
- At this stage we will need to tackle authentication against OSDU.
* This assumes we have a valid token we can use to call the OSDU API.Greg HarrisGreg Harrishttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/10Multi file parse/load support.2022-12-13T08:25:26ZGreg HarrisMulti file parse/load support.Add support for loading multiple LAS files into OSDU.
- Determine if input path is file or folder.
- List contents of folder.
- Process file in folder one by one, on error log and continue.
- Add some logging capability the Knack packag...Add support for loading multiple LAS files into OSDU.
- Determine if input path is file or folder.
- List contents of folder.
- Process file in folder one by one, on error log and continue.
- Add some logging capability the Knack package has logging support built in.M9 - Release 0.12Niall McDaidNiall McDaidhttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/18Acceptance test2022-12-13T08:25:26ZChad LeongAcceptance testTo test the end to end workflow for
1. Data Loading
2. Data retrievalTo test the end to end workflow for
1. Data Loading
2. Data retrievalM9 - Release 0.12Chad LeongChad Leonghttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/3Add Class to map LAS data object to Wellbore object.2022-12-13T08:25:26ZGreg HarrisAdd Class to map LAS data object to Wellbore object.Add a class to covert the LAS data object output by LASIO to a OSDU Well Bore object.
Add unit tests for the class.Add a class to covert the LAS data object output by LASIO to a OSDU Well Bore object.
Add unit tests for the class.M9 - Release 0.12Niall McDaidNiall McDaidhttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/2Add class that loads and validate LAS file.2022-12-13T08:25:26ZGreg HarrisAdd class that loads and validate LAS file.Add a class that uses the LASIO parser to load the las file.
https://lasio.readthedocs.io/en/latest/
Add basic validation to ensure that the well name is populated.
Add unit tests for the new functionality.Add a class that uses the LASIO parser to load the las file.
https://lasio.readthedocs.io/en/latest/
Add basic validation to ensure that the well name is populated.
Add unit tests for the new functionality.M9 - Release 0.12Niall McDaidNiall McDaidhttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/15Writing bulk data into wellbore DDMS2022-12-13T08:25:26ZChad LeongWriting bulk data into wellbore DDMSAs a user, I want to be able to write the curve data into the Wellbore DDMS.
This can be done using the following endpoint where `welllog_id` is the record created through parsing of the LAS file or user supplied.
`POST {BaseUrl}/api/o...As a user, I want to be able to write the curve data into the Wellbore DDMS.
This can be done using the following endpoint where `welllog_id` is the record created through parsing of the LAS file or user supplied.
`POST {BaseUrl}/api/os-wellbore-ddms/ddms/v3/welllogs/{welllog_id}/data`M9 - Release 0.12Greg HarrisGreg Harrishttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/4Wrapper for web client calls to insert data into OSDU storage.2022-12-13T08:25:26ZGreg HarrisWrapper for web client calls to insert data into OSDU storage.The wrapper class is intended to help support execution of the LAS uploader using OSDU cloud storage or writing to a local file system. It should also enable writing end to end unit tests.
This could include a simple write to file syst...The wrapper class is intended to help support execution of the LAS uploader using OSDU cloud storage or writing to a local file system. It should also enable writing end to end unit tests.
This could include a simple write to file system class.M9 - Release 0.12Greg HarrisGreg Harrishttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/8Support for well bore and logs upload using OSDU SDK.2022-12-13T08:25:26ZGreg HarrisSupport for well bore and logs upload using OSDU SDK.Write classes that use the OSDU SDK to upload Well Bore and Well log objects to an OSDU cloud instance.
Add a command line option to support this.Write classes that use the OSDU SDK to upload Well Bore and Well log objects to an OSDU cloud instance.
Add a command line option to support this.M9 - Release 0.12Greg HarrisGreg Harrishttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/11Support install and deployment of the application2022-12-13T08:25:26ZGreg HarrisSupport install and deployment of the applicationNot sure at this stage whether we will need to compile to an executable or create a python installer.
Plan to duplicate the deployment approach of https://github.com/equinor/osdu-cliNot sure at this stage whether we will need to compile to an executable or create a python installer.
Plan to duplicate the deployment approach of https://github.com/equinor/osdu-cliM9 - Release 0.12https://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/1Setup CI automated testing and linting (flake8) and add a simple unit test.2022-12-13T08:25:26ZGreg HarrisSetup CI automated testing and linting (flake8) and add a simple unit test.This story is to setup the basic development infrastructure for the LAS data loader. It includes
- Creating the folder structure within the repository. (eg src and test folders)
- Setting up pipenv.
- Creating a simple unit test.
- Addin...This story is to setup the basic development infrastructure for the LAS data loader. It includes
- Creating the folder structure within the repository. (eg src and test folders)
- Setting up pipenv.
- Creating a simple unit test.
- Adding an automated test pipeline to gitlab that runs the simple unit test.
- Adding static analysis (linting) to the automated build pipeline (flake8)M9 - Release 0.12Greg HarrisGreg Harrishttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/16LAS Output from bulk data ingested in wellbore DDMS2022-12-13T08:25:26ZChad LeongLAS Output from bulk data ingested in wellbore DDMSAs a user, I want to be able to retrieve the ingested bulk data from the wellbore DDMS and write it out as .LAS file.
User will provide an array of curves that he wishes to fetch.
`curves=DEPT,DT,PHI`
This can be done with the followi...As a user, I want to be able to retrieve the ingested bulk data from the wellbore DDMS and write it out as .LAS file.
User will provide an array of curves that he wishes to fetch.
`curves=DEPT,DT,PHI`
This can be done with the following endpoint:
`GET {BaseUrl}/api/os-wellbore-ddms/ddms/v3/welllogs/{welllog_id}/data?describe=false{curves}`
Once the data is retrieved, the data can be output as a LAS file through lasio.LASFile() method.
Headers information needs to be constructed based on the well log record.M9 - Release 0.12Greg HarrisGreg Harrishttps://community.opengroup.org/osdu/ui/data-loading/wellbore-ddms-data-loader/-/issues/14Add Class to map LAS data object to Well Log object2022-12-13T08:25:25ZNiall McDaidAdd Class to map LAS data object to Well Log objectAdd a class to covert the LAS data object output by LASIO to one or more OSDU Well Log objects.
Add unit tests for the class.Add a class to covert the LAS data object output by LASIO to one or more OSDU Well Log objects.
Add unit tests for the class.M9 - Release 0.12Niall McDaidNiall McDaid