README.md 5.4 KB
Newer Older
Chad Leong's avatar
Chad Leong committed
1
2
# LAS Loader utility for Wellbore DDMS

3
## :exclamation: This is currently in development
Chad Leong's avatar
Chad Leong committed
4

Chad Leong's avatar
Chad Leong committed
5
6
7
8
9
## Description

1. This will be the OSDU LAS loader utility to support Wellbore DDMS.

2. This is a wrapper behind the Wellbore DDMS [API](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/blob/master/spec/generated/openapi.json)
10

11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
## Install from Package Registry

The simplest way to install `lasloader` is from the community package registry. If you have `pip` installed simply run:
```
pip install lasloader --extra-index-url https://community.opengroup.org/api/v4/projects/801/packages/pypi/simple
```

This will download and install `lasloader` on your machine.

## Usage

The `lasloader` package has a command line interface called `lascli` which has the general syntax:
```
lascli <group> <command> options
```

Help for any group or command can be obtained by using the `-h` option.

There are several groups:
* `download`: Download a welllog and curve data to a las format file.
* `fileload`: Dry run of the ingest command, instead of uploading data to the OSDU instance it creates json files for the wellbore and welllog.
* `ingest`: Upload a wellbore, welllog and/or bulk data to an OSDU instance
* `list`: List the data held in OSDU for a given wellbore or welllog id.
* `search`: Search for a wellbore given a well name.
* `update`: Update the existing bulk data for a given welllog

Gregory Harris's avatar
Gregory Harris committed
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
The `lascli` reads configuration information from a [configuration file](#config-file).
The path to this file must either be provided in the environment variable `CONFIGPATH` or as a command line option.

All the commands that connect to an OSDU instance require a bearer token. 
This must be provided either in the environment variable `OSDUTOKEN` or as a command line option.

### Environment variables

| Varaible name | overriding command option | Comment |
| ------------- | --------------------------| ------- |
| OSDUTOKEN     | `--token` `-t`            | The JWT required to authenticate against an OSDU instance. |
| CONFIGPATH    | `--config_path` `-c`      | The path to the configuration file |


### Config file

The LASCLI requires a configuration file that has the following JSON structure:
```
{
    "base_url": "https://osdu-ship.msft-osdu-test.org",
    "data_partition_id": "opendes",
    "acl_domain": "contoso.com",
    "legal":
    {
        "legaltags": ["opendes-public-usa-dataset-7643990"],
        "otherRelevantDataCountries": ["US"],
        "status": "compliant"    
    },
    "data": {
        "default": {
            "viewers": [ "data.default.viewers@opendes.contoso.com" ],
            "owners": [ "data.default.owners@opendes.contoso.com" ]
        }
    }
}
```

The `base_url`, `data_partition_id` and `acl_domain` must be correct for the OSDU instance you want to connect to. An example configuration file that is setup for the preship OSDU instance is given in `src/example_opendes_configuration.json`.

76
77
## Development

Gregory Harris's avatar
Gregory Harris committed
78
The following instructions are provided as guidance for setting up a development environment for lasloader. For development work we use a virtual environment managed by pipenv. To execute the application and run the tests you will need to install Python 3.8+ and pipenv.
79
80
81
The source code and unit test are stored in the `src` folder.

To download the package dependencies for the virtual environment change to the src folder and run:
82

83
84
85
86
87
```
pipenv install --dev
```

To run the tests and flake8 linting:
88

89
90
```
pipenv run flake8
Gregory Harris's avatar
Gregory Harris committed
91
pipenv run pytest
92
93
```

Greg Harris's avatar
Greg Harris committed
94
Alternatively, an interactive shell in the virtual environment can be created by running:
95

96
97
98
99
100
101
102
```
pipenv shell
```

### Adding package dependencies

Package dependencies are added using the pipenv command. Dependencies required for only the unit tests should be added to the `[dev-packages]` of the `Pipfile` by using:
103

104
105
106
```
pipenv install --dev package_name
```
107

108
Production packages should be installed using:
109

110
111
```
pipenv install package_name
112
```
113
114
## Deploy

Gregory Harris's avatar
Gregory Harris committed
115
### Install from repository source
116

Gregory Harris's avatar
Gregory Harris committed
117
The lasloader module can be installed in edit mode on the command line by running the following command in the root folder of the repository:
118
119
120
```
pip install -e .
```
Gregory Harris's avatar
Gregory Harris committed
121

122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
The LAS CLI can then be accessed using the command:
```
lascli
```

Or alternatively using a virtual environment, install with:
```
pipenv install -e .
```

This installs the package in a virtual environment. The CLI can then be run using:
```
pipenv run lascli
```

137
### Publish to the Package Registry
138

139
140
Packages are published to the community package registry via the CI pipeline. Package publish is a manually triggered job. To run it navigate to the CI/CD jobs page of the `lasloader` module:
[https://community.opengroup.org/osdu/platform/data-flow/data-loading/wellbore-ddms-las-loader/-/jobs](https://community.opengroup.org/osdu/platform/data-flow/data-loading/wellbore-ddms-las-loader/-/jobs).
141

142
143
Run the publish job for the build you want to publish. It will build and upload `lasloader` with a specific package version number. The package version number is the `__VERSION__` found in the `__init__.py` file for the module followed by a build tag. The build tag is based upon the CI build_id:
1. The default build tag is an alpha release of the form 1.2.3a456
Gregory Harris's avatar
Gregory Harris committed
144
145
1. If the repo branch has been tagged with a string starting with `BETA` then the beta release form is used 1.2.3b456.
1. If the repo branch has been tagged with a string starting with `RELEASE` then the primary release form is used 1.2.3.456.