Commit 91297019 authored by Chad Leong's avatar Chad Leong
Browse files

Merge branch '27-setup-distribution-via-package-registries-look-left' into 'main'

Resolve "Setup distribution via Package & registries"

Closes #27

See merge request osdu/platform/data-flow/data-loading/wellbore-ddms-las-loader!28
parents 6b45d6b4 c649a6d0
Pipeline #77317 passed with stages
in 1 minute and 28 seconds
default:
image: python:3
variables:
BUILD_ID: $CI_PIPELINE_IID
BUILD_TAG: $CI_COMMIT_TAG
before_script:
- python --version
- cd src
......@@ -16,4 +20,19 @@ run-flake8:
run-test:
stage: test
script:
- pipenv run pytest
\ No newline at end of file
- pipenv run pytest
run-test-package:
stage: test
script:
- cd ../
- python setup.py sdist bdist_wheel
publish:
stage: deploy
script:
- cd ../
- pip install twine
- python setup.py sdist bdist_wheel
- TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --verbose --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/*
when: manual
......@@ -8,9 +8,74 @@
2. This is a wrapper behind the Wellbore DDMS [API](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/blob/master/spec/generated/openapi.json)
## Install from Package Registry
The simplest way to install `lasloader` is from the community package registry. If you have `pip` installed simply run:
```
pip install lasloader --extra-index-url https://community.opengroup.org/api/v4/projects/801/packages/pypi/simple
```
This will download and install `lasloader` on your machine.
## Usage
The `lasloader` package has a command line interface called `lascli` which has the general syntax:
```
lascli <group> <command> options
```
Help for any group or command can be obtained by using the `-h` option.
There are several groups:
* `download`: Download a welllog and curve data to a las format file.
* `fileload`: Dry run of the ingest command, instead of uploading data to the OSDU instance it creates json files for the wellbore and welllog.
* `ingest`: Upload a wellbore, welllog and/or bulk data to an OSDU instance
* `list`: List the data held in OSDU for a given wellbore or welllog id.
* `search`: Search for a wellbore given a well name.
* `update`: Update the existing bulk data for a given welllog
The `lascli` reads configuration information from a [configuration file](#config-file).
The path to this file must either be provided in the environment variable `CONFIGPATH` or as a command line option.
All the commands that connect to an OSDU instance require a bearer token.
This must be provided either in the environment variable `OSDUTOKEN` or as a command line option.
### Environment variables
| Varaible name | overriding command option | Comment |
| ------------- | --------------------------| ------- |
| OSDUTOKEN | `--token` `-t` | The JWT required to authenticate against an OSDU instance. |
| CONFIGPATH | `--config_path` `-c` | The path to the configuration file |
### Config file
The LASCLI requires a configuration file that has the following JSON structure:
```
{
"base_url": "https://osdu-ship.msft-osdu-test.org",
"data_partition_id": "opendes",
"acl_domain": "contoso.com",
"legal":
{
"legaltags": ["opendes-public-usa-dataset-7643990"],
"otherRelevantDataCountries": ["US"],
"status": "compliant"
},
"data": {
"default": {
"viewers": [ "data.default.viewers@opendes.contoso.com" ],
"owners": [ "data.default.owners@opendes.contoso.com" ]
}
}
}
```
The `base_url`, `data_partition_id` and `acl_domain` must be correct for the OSDU instance you want to connect to. An example configuration file that is setup for the preship OSDU instance is given in `src/example_opendes_configuration.json`.
## Development
The application uses a virtual environment managed by pipenv. To execute the application and run the tests you will need to install Python 3.9 and pipenv.
The following instructions are provided as guidance for setting up a development environment for lasloader. For development work we use a virtual environment managed by pipenv. To execute the application and run the tests you will need to install Python 3.8+ and pipenv.
The source code and unit test are stored in the `src` folder.
To download the package dependencies for the virtual environment change to the src folder and run:
......@@ -45,36 +110,6 @@ Production packages should be installed using:
```
pipenv install package_name
```
### Usage via the command line interface
To invoke the CLI using the virtual environment and from within the `src` folder:
```
pipenv run python -m lasloader.lascli <group> <command> <options>
```
Available groups are `fileload`, `ingest`, `list`, `update` and `download`. Help can be obtained with the `-h` option.
```
pipenv run python -m lasloader.lascli -h
pipenv run python -m lasloader.lascli fileload -h
pipenv run python -m lasloader.lascli fileload convert -h
```
A token is required to access an OSDU instance. The `-t/--token` option is used to specify the token, this is a string. As tokens are long strings we recommend that the token is stored in an environment variable.
For example, on a Windows PowerShell prompt:
```
set OSDUTOKEN <token>
pipenv run python -m lasloader.lascli ingest wellbore -t $OSDUTOKEN
```
Or Windows command prompt:
```
set OSDUTOKEN=<token>
pipenv run python -m lasloader.lascli ingest wellbore -t %OSDUTOKEN%
```
Many commands require a bearer token and filepath to a configuration file. If these are set as environment variables (named
'OSDUTOKEN' and 'CONFIGPATH' respectively), it is not necessary to provide them as arguments for commands (the CLI defaults to
use the environment variable).
## Deploy
### Install from repository source
......@@ -99,49 +134,12 @@ This installs the package in a virtual environment. The CLI can then be run usin
pipenv run lascli
```
Before build and deploy ensure that the setup.py and the pipfile.lock dependencies synced, by running:
```
pipenv-setup sync
```
Check that the version number in the `src/lasloader/__init__.py` file is correct.
### Creating distribution archives
To create a distribution run:
```
python setup.py sdist bdist_wheel
```
This will create both source and binary wheel distributions. The package can then be deployed using:
```
pip install .
```
### Publish the package distribution
### Publish to the Package Registry
To be determined...
Packages are published to the community package registry via the CI pipeline. Package publish is a manually triggered job. To run it navigate to the CI/CD jobs page of the `lasloader` module:
[https://community.opengroup.org/osdu/platform/data-flow/data-loading/wellbore-ddms-las-loader/-/jobs](https://community.opengroup.org/osdu/platform/data-flow/data-loading/wellbore-ddms-las-loader/-/jobs).
## Config file
The LASCLI requires a configuration file that has the following JSON structure:
```
{
"base_url": "https://osdu-ship.msft-osdu-test.org",
"data_partition_id": "opendes",
"acl_domain": "contoso.com",
"legal":
{
"legaltags": ["opendes-public-usa-dataset-7643990"],
"otherRelevantDataCountries": ["US"],
"status": "compliant"
},
"data": {
"default": {
"viewers": [ "data.default.viewers@opendes.contoso.com" ],
"owners": [ "data.default.owners@opendes.contoso.com" ]
}
}
}
```
Run the publish job for the build you want to publish. It will build and upload `lasloader` with a specific package version number. The package version number is the `__VERSION__` found in the `__init__.py` file for the module followed by a build tag. The build tag is based upon the CI build_id:
1. The default build tag is an alpha release of the form 1.2.3a456
1. If the repo branch has been tagged with a string starting with `BETA` then the beta release form is used 1.2.3b456.
1. If the repo branch has been tagged with a string starting with `RELEASE` then the primary release form is used 1.2.3.456.
......@@ -20,9 +20,32 @@ __VERSION__ = re.search(
).group(1)
def prepare_version():
# Construct the version number
build_tag = os.getenv("BUILD_TAG")
build_id = os.getenv("BUILD_ID")
if build_id is None:
# local dev build (non-CI)
version = f"{__VERSION__}.dev"
elif build_tag is not None and build_tag.lower().startswith('release'):
# Tagged as release
version = f"{__VERSION__}.{build_id}"
elif build_tag is not None and build_tag.lower().startswith('beta'):
# Tagged as beta release
version = f"{__VERSION__}.beta{build_id}"
else:
# Standard CI release.
version = f"{__VERSION__}.alpha{build_id}"
print(f"Packaging release: {version}")
return version
setup(
name="lasloader",
version=__VERSION__,
version=prepare_version(),
author="BP",
author_email="greg.harris1@bp.com",
description="OSDU LAS file ingest - command line interface",
......
# Gets the version number
# The version number, in the form major.minor.patch
__VERSION__ = "0.0.1"
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment