README.md 5.69 KB
Newer Older
Luc Yriarte's avatar
Luc Yriarte committed
1
# Python package for OSDU Azure cloud implementation
2

Luc Yriarte's avatar
Luc Yriarte committed
3
## Introduction
4

Luc Yriarte's avatar
Luc Yriarte committed
5
This project creates the OSDU Azure python package that implements intefrace `osdu-core-lib-python`.
6

Luc Yriarte's avatar
Luc Yriarte committed
7
## Prerequisites
8

Luc Yriarte's avatar
Luc Yriarte committed
9
10
11
12
* Python 3.7.x
* [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest)
* [Azurite npm](https://www.npmjs.com/package/azurite) or [Azurite docker image](https://hub.docker.com/_/microsoft-azure-storage-azurite) to run locally, [for more information](https://dev.azure.com/slb-des-ext-collaboration/open-data-ecosystem/_git/open-data-ecosystem.wiki?version=GBwikiMaster&path=%2FOSDU-(C)%2FWellbore-DMS%2FDesigns%2FAzure-Blob-Storage-Test-Design.md&_a=preview)
* Follow the [Wellbore project set up steps](https://dev.azure.com/slb-des-ext-collaboration/open-data-ecosystem/_git/os-wellbore-ddms?path=%2FREADME.md&_a=preview&anchor=project-startup) to ensure the osdu dependencies can be installed successfully.
13

Luc Yriarte's avatar
Luc Yriarte committed
14
## Dependencies
15
16
17

`pip install -r requirements.txt`

Luc Yriarte's avatar
Luc Yriarte committed
18
19
20
21
22
23
24
## **Required variables to run service**

| name | value | description | sensitive? | source |
| ---  | ---   | ---         | ---        | ---    |
| KEYVAULT_URL | ex `https://foo-keyvault.vault.azure.net` | URL of shared KeyVault that holds application and partitions secrets | no | infrastructure deployment |
| SERVICE_HOST_PARTITION | ex `http://partition.osdu/api/partition` | Partition Service endpoint | no | infrastructure deployment |
| USE_PARTITION_SERVICE | `enabled`/`disabled` - defaults to `enabled` | Enabled when Partition Service is available in the environment/Disabled when not available and need to use fake Partition Service | no | infrastructure deployment |
25
26
27

## Tests

Luc Yriarte's avatar
Luc Yriarte committed
28
29
30
### Test Dependencies

The tests use the [pytest](https://docs.pytest.org/en/stable/) framework. Other dependencies are [pytest-asyncio](https://pypi.org/project/pytest-asyncio/) for testing asyncio code and [pytest-cov](https://pypi.org/project/pytest-cov/) to produce coverage report.
31

Luc Yriarte's avatar
Luc Yriarte committed
32
`pip install -r requirements_dev.txt`
33

Luc Yriarte's avatar
Luc Yriarte committed
34
Install and run Azurite for local tests:
35

Luc Yriarte's avatar
Luc Yriarte committed
36
Option 1: using npm:
37

Luc Yriarte's avatar
Luc Yriarte committed
38
39
40
41
42
43
44
45
46
47
48
```bash
# Install Azurite
npm install -g azurite

# Create azurite directory
mkdir c:/azurite

# Launch Azurite for Windows
azurite --silent --location c:\azurite --debug c:\azurite\debug.log
```

Luc Yriarte's avatar
Luc Yriarte committed
49
50
51
52
53
54
55
56
Note: To avoid sudo mode on linux, install Azurite locally:

```bash
npm install azurite
mkdir azurite
./node_modules/azurite/dist/src/azurite.js --silent --location ./azurite --debug ./azurite/debug.log
```

Luc Yriarte's avatar
Luc Yriarte committed
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
Option 2: using docker

```bash
docker run -p 10000:10000 mcr.microsoft.com/azure-storage/azurite azurite-blob --blobHost 0.0.0.0
```

The tests run against an Azure storage emulator using Azurite. Azurite storage account and container are parameterized via environment.

variables:

| Variable | default value |
| --- | --- |
| TESTING_AZ_STORAGE_ACCOUNT | `http://127.0.0.1:10000/devstoreaccount1` |
| TESTING_AZ_CREDENTIALS | `Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==` |
| TESTING_AZ_CONTAINER | `wdms-osdu` |

73
74
75
76
77
78
**Note1: value for TESTING_AZ_CREDENTIALS is default value for Azurite, it's not a private key*

**Note2: The version of Azurite requires to be compatible with the azure SDK python libs installed. For instance, 
azure-storage-blob 12.6+ requires Azurite 3.10. In case of invalid version, calls will fail with an error about invalid
header `x-ms-version` value. For more information see Microsoft
[Versioning for the Azure Storage services](https://docs.microsoft.com/en-us/rest/api/storageservices/versioning-for-the-azure-storage-services).*
Luc Yriarte's avatar
Luc Yriarte committed
79
80

### How to run locally using DefaultAzureCredential and the Azure CLI
81

Luc Yriarte's avatar
Luc Yriarte committed
82
The authentication relies on [DefaultAzureCredential](https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/identity/azure-identity#defaultazurecredential). If a user has signed in via the Azure CLI `az login` command, DefaultAzureCredential will authenticate as that user. In order to have Read/Write access to the storage account and successfully run the tests, the user must be assigned the [Storage Blob Data Contributor](https://docs.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#storage-blob-data-contributor) role.
83

Luc Yriarte's avatar
Luc Yriarte committed
84
85
```bash
az login
86

Luc Yriarte's avatar
Luc Yriarte committed
87
88
# Set the subscription where the storage account is
subscription=<subscription id>
89

Luc Yriarte's avatar
Luc Yriarte committed
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
# Set your default subscription
az account set --subscription $subscription

# Validate your default subscription is set correctly
az account show
```

### Run Tests

```bash
python -m pytest ./tests

# Run all tests and produce coverage reports
python -m pytest --junit-xml=unit_tests_report.xml --cov=osdu_az --cov-report=html --cov-report=xml ./tests
```

### Generate and upload dev package

108
Dev package can be generated and upload to the feed by manually create Merge Request. Each new pipeline triggered from dev branch (not master) and
Luc Yriarte's avatar
Luc Yriarte committed
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
check on the pipeline parameter 'generate dev package'.

The version number is automatically setup following the format:

```bash
<CURRENT_VERSION>.dev<BUILD_NUMBER> # example: _0.0.15.dev20102119_
 ```

In order to select a dev package, command line or in requirement file must explicitly specify the full version:

```bash
pip install osdu-core-lib-python-azure==0.0.15.dev20102116
```

or use the option `--pre` on `pip install` command (see [pip install documentation](https://pip.pypa.io/en/stable/reference/pip_install/#install-pre)).
Otherwise pip only finds stable versions and ignore any pre-release and development versions.
125
126

## License
Luc Yriarte's avatar
Luc Yriarte committed
127

128
129
Apache version 2.0 (see LICENSE.TXT)

Luc Yriarte's avatar
Luc Yriarte committed
130
131
132
## Additional Notes

* The version number must be updated manually for now. Increment by one the version patch part in file `osdu_az/__init__.py`.