README.md 6.31 KB
Newer Older
Chad Leong's avatar
Chad Leong committed
1
2
# LAS Loader utility for Wellbore DDMS

3
## :exclamation: This is currently in development
Chad Leong's avatar
Chad Leong committed
4

Chad Leong's avatar
Chad Leong committed
5
6
7
8
9
## Description

1. This will be the OSDU LAS loader utility to support Wellbore DDMS.

2. This is a wrapper behind the Wellbore DDMS [API](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/blob/master/spec/generated/openapi.json)
10

11
12
## Install from Package Registry

13
14
15
### Before installation:
- Please ensure your Python version is 3.8 or newer and that Python has been added to the PATH (in your system's environment variables).
- Confirm you have `pip` installed by running the following command (it should be installed automatically when you installed Python) - `pip --version`
16
17
- If installing `wbdutil` within a virtual environment, first install `pipenv` using the following command - `pip install pipenv`
  - Note: There is a bug in `pipenv` version 2021.11.23 which will cause the `wbdutil` installation command to fail. Ensure your version number is different to this (e.g. `pip install pipenv=="2021.11.15"`).
18

19
### Installation:
20
The simplest way to install `wbdutil` is from the community package registry.
21

22
**Method 1:** To download and install `wbdutil` on your machine run:
23
```
24
pip install wbdutil --extra-index-url https://community.opengroup.org/api/v4/projects/801/packages/pypi/simple
25
```
26
**Method 2:** Alternatively, you may install the `wbdutil` package within a virtual environment using `pipenv`. If you have `pipenv` installed, simply run:
27
```
28
pipenv install wbdutil --extra-index-url https://community.opengroup.org/api/v4/projects/801/packages/pypi/simple --skip-lock
29
```
30
31
32

## Usage

33
The `wbdutil` package has a command line interface of the same name `wbdutil` which has the general syntax:
34
```
35
wbdutil <group> <command> options
36
37
38
39
40
41
42
43
44
45
46
47
```

Help for any group or command can be obtained by using the `-h` option.

There are several groups:
* `download`: Download a welllog and curve data to a las format file.
* `fileload`: Dry run of the ingest command, instead of uploading data to the OSDU instance it creates json files for the wellbore and welllog.
* `ingest`: Upload a wellbore, welllog and/or bulk data to an OSDU instance
* `list`: List the data held in OSDU for a given wellbore or welllog id.
* `search`: Search for a wellbore given a well name.
* `update`: Update the existing bulk data for a given welllog

48
The `wbdutil` reads configuration information from a [configuration file](#config-file).
Gregory Harris's avatar
Gregory Harris committed
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
The path to this file must either be provided in the environment variable `CONFIGPATH` or as a command line option.

All the commands that connect to an OSDU instance require a bearer token. 
This must be provided either in the environment variable `OSDUTOKEN` or as a command line option.

### Environment variables

| Varaible name | overriding command option | Comment |
| ------------- | --------------------------| ------- |
| OSDUTOKEN     | `--token` `-t`            | The JWT required to authenticate against an OSDU instance. |
| CONFIGPATH    | `--config_path` `-c`      | The path to the configuration file |


### Config file

64
The wbdutil requires a configuration file that has the following JSON structure:
Gregory Harris's avatar
Gregory Harris committed
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
```
{
    "base_url": "https://osdu-ship.msft-osdu-test.org",
    "data_partition_id": "opendes",
    "acl_domain": "contoso.com",
    "legal":
    {
        "legaltags": ["opendes-public-usa-dataset-7643990"],
        "otherRelevantDataCountries": ["US"],
        "status": "compliant"    
    },
    "data": {
        "default": {
            "viewers": [ "data.default.viewers@opendes.contoso.com" ],
            "owners": [ "data.default.owners@opendes.contoso.com" ]
        }
    }
}
```

The `base_url`, `data_partition_id` and `acl_domain` must be correct for the OSDU instance you want to connect to. An example configuration file that is setup for the preship OSDU instance is given in `src/example_opendes_configuration.json`.

87
88
## Development

89
The following instructions are provided as guidance for setting up a development environment for wbdutil. For development work we use a virtual environment managed by pipenv. To execute the application and run the tests you will need to install Python 3.8+ and pipenv.
90
91
92
The source code and unit test are stored in the `src` folder.

To download the package dependencies for the virtual environment change to the src folder and run:
93

94
95
96
97
98
```
pipenv install --dev
```

To run the tests and flake8 linting:
99

100
101
```
pipenv run flake8
Gregory Harris's avatar
Gregory Harris committed
102
pipenv run pytest
103
104
```

Greg Harris's avatar
Greg Harris committed
105
Alternatively, an interactive shell in the virtual environment can be created by running:
106

107
108
109
110
111
112
113
```
pipenv shell
```

### Adding package dependencies

Package dependencies are added using the pipenv command. Dependencies required for only the unit tests should be added to the `[dev-packages]` of the `Pipfile` by using:
114

115
116
117
```
pipenv install --dev package_name
```
118

119
Production packages should be installed using:
120

121
122
```
pipenv install package_name
123
```
124
125
## Deploy

Gregory Harris's avatar
Gregory Harris committed
126
### Install from repository source
127

128
The wbdutil module can be installed in edit mode on the command line by running the following command in the root folder of the repository:
129
130
131
```
pip install -e .
```
Gregory Harris's avatar
Gregory Harris committed
132

133
The wbdutil CLI can then be accessed using the command:
134
```
135
wbdutil
136
137
138
139
140
141
142
143
144
```

Or alternatively using a virtual environment, install with:
```
pipenv install -e .
```

This installs the package in a virtual environment. The CLI can then be run using:
```
145
pipenv run wbdutil
146
147
```

148
### Publish to the Package Registry
149

150
Packages are published to the community package registry via the CI pipeline. Package publish is a manually triggered job. To run it navigate to the CI/CD jobs page of the `wbdutil` module:
151
[https://community.opengroup.org/osdu/platform/data-flow/data-loading/wellbore-ddms-las-loader/-/jobs](https://community.opengroup.org/osdu/platform/data-flow/data-loading/wellbore-ddms-las-loader/-/jobs).
152

153
Run the publish job for the build you want to publish. It will build and upload `wbdutil` with a specific package version number. The package version number is the `__VERSION__` found in the `__init__.py` file for the module followed by a build tag. The build tag is based upon the CI build_id:
154
1. The default build tag is an alpha release of the form 1.2.3a456
Gregory Harris's avatar
Gregory Harris committed
155
156
1. If the repo branch has been tagged with a string starting with `BETA` then the beta release form is used 1.2.3b456.
1. If the repo branch has been tagged with a string starting with `RELEASE` then the primary release form is used 1.2.3.456.