Commit 7e0fa7e8 authored by Chad Leong's avatar Chad Leong
Browse files

Merge branch '41-rename-lasloader-package-and-any-associated-references-in-repo' into 'main'

Resolve "Rename 'lasloader' package and any associated references in repo"

Closes #41

See merge request !35
parents 6bc962d6 fa82cb14
Pipeline #96377 passed with stages
in 53 seconds
dist/
env/
__pycache__/
.vs/
......
......@@ -13,26 +13,26 @@
### Before installation:
- Please ensure your Python version is 3.8 or newer and that Python has been added to the PATH (in your system's environment variables).
- Confirm you have `pip` installed by running the following command (it should be installed automatically when you installed Python) - `pip --version`
- If installing `lasloader` within a virtual environment, first install `pipenv` using the following command - `pip install pipenv`
- Note: There is a bug in `pipenv` version 2021.11.23 which will cause the `lasloader` installation command to fail. Ensure your version number is different to this (e.g. `pip install pipenv=="2021.11.15"`).
- If installing `wbdutil` within a virtual environment, first install `pipenv` using the following command - `pip install pipenv`
- Note: There is a bug in `pipenv` version 2021.11.23 which will cause the `wbdutil` installation command to fail. Ensure your version number is different to this (e.g. `pip install pipenv=="2021.11.15"`).
### Installation:
The simplest way to install `lasloader` is from the community package registry.
The simplest way to install `wbdutil` is from the community package registry.
**Method 1:** To download and install `lasloader` on your machine run:
**Method 1:** To download and install `wbdutil` on your machine run:
```
pip install lasloader --extra-index-url https://community.opengroup.org/api/v4/projects/801/packages/pypi/simple
pip install wbdutil --extra-index-url https://community.opengroup.org/api/v4/projects/801/packages/pypi/simple
```
**Method 2:** Alternatively, you may install the `lasloader` package within a virtual environment using `pipenv`. If you have `pipenv` installed, simply run:
**Method 2:** Alternatively, you may install the `wbdutil` package within a virtual environment using `pipenv`. If you have `pipenv` installed, simply run:
```
pipenv install lasloader --extra-index-url https://community.opengroup.org/api/v4/projects/801/packages/pypi/simple --skip-lock
pipenv install wbdutil --extra-index-url https://community.opengroup.org/api/v4/projects/801/packages/pypi/simple --skip-lock
```
## Usage
The `lasloader` package has a command line interface called `lascli` which has the general syntax:
The `wbdutil` package has a command line interface of the same name `wbdutil` which has the general syntax:
```
lascli <group> <command> options
wbdutil <group> <command> options
```
Help for any group or command can be obtained by using the `-h` option.
......@@ -45,7 +45,7 @@ There are several groups:
* `search`: Search for a wellbore given a well name.
* `update`: Update the existing bulk data for a given welllog
The `lascli` reads configuration information from a [configuration file](#config-file).
The `wbdutil` reads configuration information from a [configuration file](#config-file).
The path to this file must either be provided in the environment variable `CONFIGPATH` or as a command line option.
All the commands that connect to an OSDU instance require a bearer token.
......@@ -61,7 +61,7 @@ This must be provided either in the environment variable `OSDUTOKEN` or as a com
### Config file
The LASCLI requires a configuration file that has the following JSON structure:
The wbdutil requires a configuration file that has the following JSON structure:
```
{
"base_url": "https://osdu-ship.msft-osdu-test.org",
......@@ -86,7 +86,7 @@ The `base_url`, `data_partition_id` and `acl_domain` must be correct for the OSD
## Development
The following instructions are provided as guidance for setting up a development environment for lasloader. For development work we use a virtual environment managed by pipenv. To execute the application and run the tests you will need to install Python 3.8+ and pipenv.
The following instructions are provided as guidance for setting up a development environment for wbdutil. For development work we use a virtual environment managed by pipenv. To execute the application and run the tests you will need to install Python 3.8+ and pipenv.
The source code and unit test are stored in the `src` folder.
To download the package dependencies for the virtual environment change to the src folder and run:
......@@ -125,14 +125,14 @@ pipenv install package_name
### Install from repository source
The lasloader module can be installed in edit mode on the command line by running the following command in the root folder of the repository:
The wbdutil module can be installed in edit mode on the command line by running the following command in the root folder of the repository:
```
pip install -e .
```
The LAS CLI can then be accessed using the command:
The wbdutil CLI can then be accessed using the command:
```
lascli
wbdutil
```
Or alternatively using a virtual environment, install with:
......@@ -142,15 +142,15 @@ pipenv install -e .
This installs the package in a virtual environment. The CLI can then be run using:
```
pipenv run lascli
pipenv run wbdutil
```
### Publish to the Package Registry
Packages are published to the community package registry via the CI pipeline. Package publish is a manually triggered job. To run it navigate to the CI/CD jobs page of the `lasloader` module:
Packages are published to the community package registry via the CI pipeline. Package publish is a manually triggered job. To run it navigate to the CI/CD jobs page of the `wbdutil` module:
[https://community.opengroup.org/osdu/platform/data-flow/data-loading/wellbore-ddms-las-loader/-/jobs](https://community.opengroup.org/osdu/platform/data-flow/data-loading/wellbore-ddms-las-loader/-/jobs).
Run the publish job for the build you want to publish. It will build and upload `lasloader` with a specific package version number. The package version number is the `__VERSION__` found in the `__init__.py` file for the module followed by a build tag. The build tag is based upon the CI build_id:
Run the publish job for the build you want to publish. It will build and upload `wbdutil` with a specific package version number. The package version number is the `__VERSION__` found in the `__init__.py` file for the module followed by a build tag. The build tag is based upon the CI build_id:
1. The default build tag is an alpha release of the form 1.2.3a456
1. If the repo branch has been tagged with a string starting with `BETA` then the beta release form is used 1.2.3b456.
1. If the repo branch has been tagged with a string starting with `RELEASE` then the primary release form is used 1.2.3.456.
......@@ -7,6 +7,8 @@ from setuptools import setup
with open("README.md", "r", encoding="utf-8") as fh:
long_description = fh.read()
_BASE_MODULE_NAME="wbdutil"
_CLI_NAME="wbdutil"
def read(fname):
"""Local read helper function for long documentation"""
......@@ -14,7 +16,7 @@ def read(fname):
return open(join(osducli_path, fname)).read()
version_file = read(os.path.join("src", "lasloader", "__init__.py"))
version_file = read(os.path.join("src", _BASE_MODULE_NAME, "__init__.py"))
__VERSION__ = re.search(
r'^__VERSION__\s*=\s*[\'"]([^\'"]*)[\'"]', version_file, re.MULTILINE
).group(1)
......@@ -44,7 +46,7 @@ def prepare_version():
setup(
name="lasloader",
name=_BASE_MODULE_NAME,
version=prepare_version(),
author="BP",
author_email="greg.harris1@bp.com",
......@@ -64,11 +66,11 @@ setup(
],
keywords="osdu",
python_requires=">=3.8",
packages=["lasloader", "lasloader.commands"],
packages=[_BASE_MODULE_NAME, f"{_BASE_MODULE_NAME}.commands"],
package_dir={"": "src"},
py_modules=[splitext(basename(path))[0] for path in glob("src/*.py")],
include_package_data=True,
entry_points={"console_scripts": ["lascli=lasloader.__main__:main"]},
entry_points={"console_scripts": [f"{_CLI_NAME}={_BASE_MODULE_NAME}.__main__:main"]},
install_requires=[
"anyio==3.4.0; python_full_version >= '3.6.2'",
"argcomplete==1.12.3",
......
from lasloader.lascli import main
if __name__ == "__main__":
main()
# The name of the module to test
TARGET_MODULE_NAME = "wbdutil"
import json
import importlib
from unittest.mock import Mock
from lasloader.configuration import Configuration
from lasloader.file_loader import IFileLoader
from . import TARGET_MODULE_NAME
# Test target module import
Configuration = importlib.import_module(f"{TARGET_MODULE_NAME}.configuration").Configuration
IFileLoader = importlib.import_module(f"{TARGET_MODULE_NAME}.file_loader").IFileLoader
class TestConfiguration:
......
import os
import pytest
from lasloader.environment_var_helper import EnvironmentVarHelper, MissingArgumentError
import importlib
from . import TARGET_MODULE_NAME
# Test target module import
environment_var_helper = importlib.import_module(f"{TARGET_MODULE_NAME}.environment_var_helper")
EnvironmentVarHelper = environment_var_helper.EnvironmentVarHelper
MissingArgumentError = environment_var_helper.MissingArgumentError
class TestGetToken:
......
from lasloader.file_loader import LasParser, FileValidationError, LocalFileLoader
import lasio
import pytest
from importlib import import_module
from . import TARGET_MODULE_NAME
# Test target module import
file_loader = import_module(f"{TARGET_MODULE_NAME}.file_loader")
LasParser = file_loader.LasParser
FileValidationError = file_loader.FileValidationError
LocalFileLoader = file_loader.LocalFileLoader
class TestLasParser:
......
import unittest
import json
import os.path
from lasloader.json_writer import JsonToFile
import pytest
from importlib import import_module
from . import TARGET_MODULE_NAME
# Test target module import
JsonToFile = import_module(f"{TARGET_MODULE_NAME}.json_writer").JsonToFile
class TestJsonToFile(unittest.TestCase):
......
import pytest
import httpx
from pandas import DataFrame
from lasloader.osdu_client import OsduClient, LasLoaderWebResponseError
from lasloader.record_mapper import Record
from importlib import import_module
from . import TARGET_MODULE_NAME
# Test target module import
Record = import_module(f"{TARGET_MODULE_NAME}.record_mapper").Record
osdu_client = import_module(f"{TARGET_MODULE_NAME}.osdu_client")
DataLoaderWebResponseError = osdu_client.DataLoaderWebResponseError
OsduClient = osdu_client.OsduClient
class TestOsduClient:
......@@ -88,7 +94,7 @@ class TestOsduClient:
client = OsduClient(base_url, self._access_token, self._data_partition_id)
# Act
with pytest.raises(LasLoaderWebResponseError) as ex:
with pytest.raises(DataLoaderWebResponseError) as ex:
client.create_wellbore(record)
# Assert
......@@ -123,7 +129,7 @@ class TestOsduClient:
client = OsduClient(base_url, self._access_token, self._data_partition_id)
# Act
with pytest.raises(LasLoaderWebResponseError) as ex:
with pytest.raises(DataLoaderWebResponseError) as ex:
client.post_welllog(record)
# Assert
......@@ -145,7 +151,7 @@ class TestOsduClient:
client = OsduClient(base_url, self._access_token, self._data_partition_id)
# Act
with pytest.raises(LasLoaderWebResponseError) as ex:
with pytest.raises(DataLoaderWebResponseError) as ex:
client.add_welllog_data(dataframe, welllog_id)
# Assert
......@@ -198,7 +204,7 @@ class TestOsduClient:
client = OsduClient(base_url, self._access_token, self._data_partition_id)
# Act
with pytest.raises(LasLoaderWebResponseError) as ex:
with pytest.raises(DataLoaderWebResponseError) as ex:
client.get_wellbore_record(wellbore_id)
# Assert
......@@ -232,7 +238,7 @@ class TestOsduClient:
client = OsduClient(base_url, self._access_token, self._data_partition_id)
# Act
with pytest.raises(LasLoaderWebResponseError) as ex:
with pytest.raises(DataLoaderWebResponseError) as ex:
client.get_welllog_record(welllog_id)
# Assert
......
......@@ -3,10 +3,24 @@ import urllib
from pandas import DataFrame
from lasio import LASFile
from unittest.mock import Mock
from lasloader.record_mapper import Record, AttributeBuilder, LasToRecordMapper, WellLogRecord, MappingUtilities
from lasloader.record_mapper import MapWellLogToLas, WellboreRecord
from lasloader.file_loader import LasParser, LocalFileLoader
from lasloader.configuration import Configuration
from importlib import import_module
from . import TARGET_MODULE_NAME
# Test target module import
record_mapper = import_module(f"{TARGET_MODULE_NAME}.record_mapper")
Record = record_mapper.Record
AttributeBuilder = record_mapper.AttributeBuilder
LasToRecordMapper = record_mapper.LasToRecordMapper
WellLogRecord = record_mapper.WellLogRecord
MappingUtilities = record_mapper.MappingUtilities
MapWellLogToLas = record_mapper.MapWellLogToLas
WellboreRecord = record_mapper.WellboreRecord
file_loader = import_module(f"{TARGET_MODULE_NAME}.file_loader")
LasParser = file_loader.LasParser
LocalFileLoader = file_loader.LocalFileLoader
Configuration = import_module(f"{TARGET_MODULE_NAME}.configuration").Configuration
@pytest.fixture
......
......@@ -2,9 +2,24 @@ from typing import List
from pandas.core.frame import DataFrame
import pytest
from unittest.mock import Mock, ANY
from lasloader.well_service import WellBoreService, WellLogService, LasLoaderConflictError
from lasloader.osdu_client import OsduClient, LasLoaderWebResponseError
from lasloader.record_mapper import LasToRecordMapper, Record, WellLogRecord, WellboreRecord
from importlib import import_module
from . import TARGET_MODULE_NAME
# Test target module import
well_service = import_module(f"{TARGET_MODULE_NAME}.well_service")
WellBoreService = well_service.WellBoreService
WellLogService = well_service.WellLogService
DataLoaderConflictError = well_service.DataLoaderConflictError
osdu_client = import_module(f"{TARGET_MODULE_NAME}.osdu_client")
DataLoaderWebResponseError = osdu_client.DataLoaderWebResponseError
OsduClient = osdu_client.OsduClient
record_mapper = import_module(f"{TARGET_MODULE_NAME}.record_mapper")
LasToRecordMapper = record_mapper.LasToRecordMapper
Record = record_mapper.Record
WellLogRecord = record_mapper.WellLogRecord
WellboreRecord = record_mapper.WellboreRecord
class TestWellLogService:
......@@ -58,7 +73,7 @@ class TestWellLogService:
# Assemble
client = Mock(spec=OsduClient)
client.post_log_recognition.side_effect = LasLoaderWebResponseError('POST', "URL", "BOOM!")
client.post_log_recognition.side_effect = DataLoaderWebResponseError('POST', "URL", "BOOM!")
subject = WellLogService(client)
# Act
......@@ -341,7 +356,7 @@ class TestWellBoreService:
subject = WellBoreService(mock_client, mock_well_log_service)
# Act
with pytest.raises(LasLoaderConflictError):
with pytest.raises(DataLoaderConflictError):
subject.file_ingest(mock_mapper, data_partition_id, False)
# Assert
......@@ -361,7 +376,7 @@ class TestWellBoreService:
subject = WellBoreService(mock_client, mock_well_log_service)
# Act
with pytest.raises(LasLoaderConflictError):
with pytest.raises(DataLoaderConflictError):
subject._get_wellbore_by_name("WellBoreName")
# Assert
......
# The version number, in the form major.minor.patch
__VERSION__ = "0.0.1"
# The name of the module
MODULE_NAME = "wbdutil"
# The name of the command line interface.
CLI_NAME = "wbdutil"
from importlib import import_module
from . import CLI_NAME, MODULE_NAME
# Test target module import
main = import_module(f"{MODULE_NAME}.{CLI_NAME}").main
if __name__ == "__main__":
main()
from typing import List
from knack.log import get_logger
from lasloader.environment_var_helper import EnvironmentVarHelper
from lasloader.osdu_client import OsduClient
from lasloader.configuration import Configuration
from lasloader.file_loader import LocalFileLoader
from lasloader.well_service import WellLogService
from ..environment_var_helper import EnvironmentVarHelper
from ..osdu_client import OsduClient
from ..configuration import Configuration
from ..file_loader import LocalFileLoader
from ..well_service import WellLogService
logger = get_logger(__name__)
......@@ -20,7 +20,7 @@ def download_las(welllog_id: str,
:param str welllog_id: The well bore id of the record to retrieve
:param str outfile: The path of the output file
:param str token: a valid bearer token that is used to authenticate against the OSDU instance
:param str config_path: Path to the las metadata file
:param str config_path: Path to the configuration file
:param List[str] curves: The curves to retrieve, use None to get all curves
"""
token = EnvironmentVarHelper.get_token(token)
......
from pathlib import Path
from knack.log import get_logger
from lasloader.environment_var_helper import EnvironmentVarHelper
from lasloader.file_loader import LasParser, LocalFileLoader, FileUtilities
from lasloader.record_mapper import LasToRecordMapper
from lasloader.configuration import Configuration
from lasloader.json_writer import JsonToFile
from ..environment_var_helper import EnvironmentVarHelper
from ..file_loader import LasParser, LocalFileLoader, FileUtilities
from ..record_mapper import LasToRecordMapper
from ..configuration import Configuration
from ..json_writer import JsonToFile
logger = get_logger(__name__)
......@@ -29,7 +29,7 @@ def convert(input_path: str, wellbore_id: str, config_path: str = None):
Convert a LAS file to Wellbore and Well Log and write to json files.
:param str input_path: Path and filename of a LAS file or folder containing LAS files
:param str wellbore_id: The wellbore id
:param str config_path: Path to the LAS metadata file
:param str config_path: Path to the configuration file
"""
config_path = EnvironmentVarHelper.get_config_path(config_path)
......
......@@ -2,12 +2,12 @@ import json
from pathlib import Path
from ntpath import basename
from knack.log import get_logger
from lasloader.environment_var_helper import EnvironmentVarHelper
from lasloader.file_loader import LasParser, LocalFileLoader, FileUtilities
from lasloader.record_mapper import LasToRecordMapper
from lasloader.osdu_client import OsduClient
from lasloader.configuration import Configuration
from lasloader.well_service import WellBoreService, WellLogService, LasLoaderConflictError
from ..environment_var_helper import EnvironmentVarHelper
from ..file_loader import LasParser, LocalFileLoader, FileUtilities
from ..record_mapper import LasToRecordMapper
from ..osdu_client import OsduClient
from ..configuration import Configuration
from ..well_service import WellBoreService, WellLogService, DataLoaderConflictError
logger = get_logger(__name__)
......@@ -43,7 +43,7 @@ def wellbore(
las_mapper = LasToRecordMapper(las_data, config)
service.file_ingest(las_mapper, config.data_partition_id, no_recognize)
logger.warning(f"Ingest complete: {basename(file_path)}")
except LasLoaderConflictError as e:
except DataLoaderConflictError as e:
logger.error(f"Ingest failed: {basename(file_path)} (see summary for details)")
ids = json.dumps(e.ids, indent=4, sort_keys=True)
logger.debug(f"{basename(file_path)} ({str(e)} Matching wellbore ids {ids})")
......@@ -68,7 +68,7 @@ def welllog_data(
:param str welllog_id: ID of well log to be updated.
:param str input_path: Path and filename of a LAS file containing data to be written to existing well log.
:param str token: A valid bearer token that is used to authenticate the update request.
:param str config_path: Path to the LAS metadata file.
:param str config_path: Path to the configuration file.
"""
token = EnvironmentVarHelper.get_token(token)
config_path = EnvironmentVarHelper.get_config_path(config_path)
......
import json
from typing import List
from knack.log import get_logger
from lasloader.environment_var_helper import EnvironmentVarHelper
from lasloader.osdu_client import OsduClient
from lasloader.configuration import Configuration
from lasloader.file_loader import LocalFileLoader
from ..environment_var_helper import EnvironmentVarHelper
from ..osdu_client import OsduClient
from ..configuration import Configuration
from ..file_loader import LocalFileLoader
logger = get_logger(__name__)
......@@ -17,7 +17,7 @@ def wellbore(wellbore_id: str,
Retrieve and print wellbore record from an OSDU instance
:param str wellbore_id: The well bore id of the record to retrieve
:param str token: a valid bearer token that is used to authenticate against the OSDU instance
:param str config_path: Path to the las metadata file
:param str config_path: Path to the configuration file
"""
token = EnvironmentVarHelper.get_token(token)
config_path = EnvironmentVarHelper.get_config_path(config_path)
......
import json
from knack.log import get_logger
from lasloader.environment_var_helper import EnvironmentVarHelper
from lasloader.osdu_client import OsduClient
from lasloader.configuration import Configuration
from lasloader.file_loader import LocalFileLoader
from lasloader.well_service import WellBoreService, WellLogService
from ..environment_var_helper import EnvironmentVarHelper
from ..osdu_client import OsduClient
from ..configuration import Configuration
from ..file_loader import LocalFileLoader
from ..well_service import WellBoreService, WellLogService
logger = get_logger(__name__)
......@@ -17,7 +17,7 @@ def wellbore_search(wellbore_name: str,
Retrieve and print the ids of wellbores that match the specified name
:param str wellbore_name: The well bore name to search for
:param str token: a valid bearer token that is used to authenticate against the OSDU instance
:param str config_path: Path to the las metadata file
:param str config_path: Path to the configuration file
"""
token = EnvironmentVarHelper.get_token(token)
config_path = EnvironmentVarHelper.get_config_path(config_path)
......
from knack.log import get_logger
from lasloader.environment_var_helper import EnvironmentVarHelper
from lasloader.configuration import Configuration
from lasloader.file_loader import LocalFileLoader
from lasloader.osdu_client import LasLoaderWebResponseError, OsduClient
from lasloader.well_service import WellLogService
from ..environment_var_helper import EnvironmentVarHelper
from ..configuration import Configuration
from ..file_loader import LocalFileLoader
from ..osdu_client import DataLoaderWebResponseError, OsduClient
from ..well_service import WellLogService
logger = get_logger(__name__)
......@@ -17,10 +17,10 @@ def welllog(welllog_id: str,
Update an existing well log record.
:param str welllog_id: ID of the well log to be updated.
:param str token: A valid bearer token that is used to authenticate the update request.
:param str config_path: Path to the LAS metadata file.
:param str config_path: Path to the configuration file.
:param str curve_families: If true, recognize and update the curve families, otherwise don't.
"""
# This command currently only containes functionality to update the curve families of a well log.
# This command currently only contains functionality to update the curve families of a well log.
# It is set up in such a way that it should be possible to extend to include other update operations in future if needed.
token = EnvironmentVarHelper.get_token(token)
config_path = EnvironmentVarHelper.get_config_path(config_path)
......@@ -33,7 +33,7 @@ def welllog(welllog_id: str,
logger.info("Recognizing curve families and updating well log record.")
try:
service.update_log_family(welllog_id, config.data_partition_id)
except LasLoaderWebResponseError as ex:
except DataLoaderWebResponseError as ex:
logger.error(f"Error - record not updated: {str(ex)}")
return
logger.warning(f"Curve families recognized and updated for well log: {welllog_id}")
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment