Commit 9ea2774c authored by Christophe Lallement's avatar Christophe Lallement
Browse files

Merge branch 'doc-sanitize' into 'master'

Sanitize documentation for OSDU

See merge request !249
parents cbf02914 53ba8ca9
Pipeline #68960 passed with stages
in 76 minutes and 33 seconds
......@@ -349,8 +349,7 @@ This example runs basic tests using the local filesystem for blob storage and st
First, create the temp storage folders and run the service.
```bash
mkdir -p tmpstorage
mkdir -p tmpblob
mkdir -p tmpstorage tmpblob
python main.py -e USE_INTERNAL_STORAGE_SERVICE_WITH_PATH $(pwd)/tmpstorage -e USE_LOCALFS_BLOB_STORAGE_WITH_PATH $(pwd)/tmpblob -e CLOUD_PROVIDER local
```
......@@ -378,6 +377,11 @@ If you want to work with other requirements file, you can specify them
pip-sync requirements.txt requirements_dev.txt
```
**Note:** On a Windows workstation, platform-specific modules such as `pywin32` are also needed. In this case don't use `pip-sync` but `pip install` instead.
```bash
pip install -r requirements.txt -r requirements_dev.txt
```
If you want to update `requirements.txt` to retrieve the most recent version, respecting bounds set in `requirements.in`, you can use:
```bash
......@@ -390,6 +394,8 @@ If you want to update the version of only one dependency, for instance fastapi:
pip-compile --upgrade-package fastapi
```
**Note:** On a Windows workstation, **don't** commit the `pywin32` back to the `requirements.txt` file, that will cause CICD to fail.
For more information: https://github.com/jazzband/pip-tools/
### Debugging:
......
# Indexation schemas
Indexation schemas are required to be able to query on some custom properties. Otherwise these fields are not indexed
and only generic one are. Schema must be registered once for each data partition. This will eventually
be part of the data provisioning but for the time being they are 'manually' registered.
## schema
* [log.json](./log.json)
* [logSet.json](./logSet.json)
* [marker.json](./marker.json)
* [trajectory.json](./trajectory.json)
* [wellbore.json](./wellbore.json)
* [dipSet.json](./dipSet.json)
All schemas but dipSet come from [data-management/wke-schema repository](https://slb-swt.visualstudio.com/data-management/_git/wke-schema?path=%2Fdomains%2Fwell%2Fjson_schema)
and were put here manually (no sync). The dipSet has been created for the needs of WDMS v2.
We may update them to be adapted to wdms v2 needs (for instance, bulk reference instead of DELFI logstore id, or bulk at
logset level ...). Potentially we'll adopt OSDU schemas instead of WKS defined by Schlumberger ([OSDU WellLog.json](https://gitlab.opengroup.org/osdu/json-schemas/-/blob/master/Generated/work-product-component/WellLog.json)).
**WARNING**: The "kind" inside the json should be updated to correspond to the data partition:
e.g.
```
{
"kind": "DATA_PARTITION_TAG:wks:log:1.0.5",
"schema": ...
}
```
in case of data partition = `opendes` is must be updated to
```
{
"kind": "opendes:wks:log:1.0.5",
"schema": ...
}
```
## Commands
given TOKEN, BASE_URL and DATA_PARTITION
### Fetch a schema
```
curl \
--location \
--request GET "$BASE_URL/api/storage/v2/schemas/$DATA_PARTITION:wks:wellbore:1.0.6' \
--header "accept: application/json" \
--header "data-partition-id: $DATA_PARTITION" \
--header "Authorization: Bearer $TOKEN"
```
### Register a schema
```
curl \
--location \
--request POST "$BASE_URL/api/storage/v2/schemas' \
--header 'Content-Type: application/json' \
--header "data-partition-id: $DATA_PARTITION" \
--header "Authorization: Bearer $TOKEN"
--data-raw '{
...
}'
```
The `loadschema.sh` script is a convenient script to register all schemas (to be run from within this directory), and can be used the following way:
```bash
#set the following variable
BASE_URL=TBD
DATA_PARTITION=TBD
TOKEN=TBD
./loadSchema.sh $BASE_URL $DATA_PARTITION $TOKEN
```
{
"kind": "DATA_PARTITION_TAG:wks:dipSet:1.0.0",
"schema": [
{
"path": "azimuthReference",
"kind": "string"
},
{
"path": "classification",
"kind": "string"
},
{
"path": "dateCreated",
"kind": "datetime"
},
{
"path": "dateModified",
"kind": "datetime"
},
{
"path": "elevationReference.elevationFromMsl.unitKey",
"kind": "string"
},
{
"path": "elevationReference.elevationFromMsl.value",
"kind": "double"
},
{
"path": "elevationReference.name",
"kind": "string"
},
{
"path": "externalIds",
"kind": "[]link"
},
{
"path": "name",
"kind": "string"
},
{
"path": "operation",
"kind": "string"
},
{
"path": "reference.absentValue",
"kind": "string"
},
{
"path": "reference.dataType",
"kind": "string"
},
{
"path": "reference.dimension",
"kind": "int"
},
{
"path": "reference.family",
"kind": "string"
},
{
"path": "reference.familyType",
"kind": "string"
},
{
"path": "reference.format",
"kind": "string"
},
{
"path": "reference.logstoreId",
"kind": "long"
},
{
"path": "reference.longName",
"kind": "string"
},
{
"path": "reference.mnemonic",
"kind": "string"
},
{
"path": "reference.name",
"kind": "string"
},
{
"path": "reference.source",
"kind": "string"
},
{
"path": "reference.unitKey",
"kind": "string"
},
{
"path": "referenceType",
"kind": "string"
},
{
"path": "relationships.well.confidence",
"kind": "double"
},
{
"path": "relationships.well.id",
"kind": "link"
},
{
"path": "relationships.well.name",
"kind": "string"
},
{
"path": "relationships.well.version",
"kind": "long"
},
{
"path": "relationships.wellbore.confidence",
"kind": "double"
},
{
"path": "relationships.wellbore.id",
"kind": "link"
},
{
"path": "relationships.wellbore.name",
"kind": "string"
},
{
"path": "relationships.wellbore.version",
"kind": "long"
},
{
"path": "relationships.wellboreSection.confidence",
"kind": "double"
},
{
"path": "relationships.wellboreSection.id",
"kind": "link"
},
{
"path": "relationships.wellboreSection.name",
"kind": "string"
},
{
"path": "relationships.wellboreSection.version",
"kind": "long"
},
{
"path": "relationships.trueDipAzimuthLog.confidence",
"kind": "double"
},
{
"path": "relationships.trueDipAzimuthLog.id",
"kind": "link"
},
{
"path": "relationships.trueDipAzimuthLog.name",
"kind": "string"
},
{
"path": "relationships.trueDipAzimuthLog.version",
"kind": "long"
},
{
"path": "relationships.trueDipInclinationLog.confidence",
"kind": "double"
},
{
"path": "relationships.trueDipInclinationLog.id",
"kind": "link"
},
{
"path": "relationships.trueDipInclinationLog.name",
"kind": "string"
},
{
"path": "relationships.trueDipInclinationLog.version",
"kind": "long"
},
{
"path": "relationships.XCoordinateLog.confidence",
"kind": "double"
},
{
"path": "relationships.XCoordinateLog.id",
"kind": "link"
},
{
"path": "relationships.XCoordinateLog.name",
"kind": "string"
},
{
"path": "relationships.XCoordinateLog.version",
"kind": "long"
},
{
"path": "relationships.YCoordinateLog.confidence",
"kind": "double"
},
{
"path": "relationships.YCoordinateLog.id",
"kind": "link"
},
{
"path": "relationships.YCoordinateLog.name",
"kind": "string"
},
{
"path": "relationships.YCoordinateLog.version",
"kind": "long"
},
{
"path": "relationships.ZCoordinateLog.confidence",
"kind": "double"
},
{
"path": "relationships.ZCoordinateLog.id",
"kind": "link"
},
{
"path": "relationships.ZCoordinateLog.name",
"kind": "string"
},
{
"path": "relationships.ZCoordinateLog.version",
"kind": "long"
},
{
"path": "relationships.quality.confidence",
"kind": "double"
},
{
"path": "relationships.quality.id",
"kind": "link"
},
{
"path": "relationships.quality.name",
"kind": "string"
},
{
"path": "relationships.quality.version",
"kind": "long"
},
{
"path": "start.unitKey",
"kind": "string"
},
{
"path": "start.value",
"kind": "double"
},
{
"path": "step.unitKey",
"kind": "string"
},
{
"path": "step.value",
"kind": "double"
},
{
"path": "stop.unitKey",
"kind": "string"
},
{
"path": "stop.value",
"kind": "double"
}
],
"ext": null
}
\ No newline at end of file
#!/bin/bash
#This is a convenient script for loading all the schemas
set -e
if [ $# -ne 3 ]
then
echo "Usage ./loadSchema.sh $BASE_URL $DATA_PARTITION $TOKEN"
exit -1
fi
BASE_URL=$1
DATA_PARTITION=$2
TOKEN=$3
echo "Loading Schemas on '$BASE_URL', for DataPartition '$DATA_PARTITION'"
schemaFiles=$(ls *.json)
for schemaFile in $schemaFiles
do
echo "loading $schemaFile: "
schema=$(sed "s/DATA_PARTITION_TAG/${DATA_PARTITION}/" ${schemaFile})
echo $schema | head -c 100
echo "..."
curl \
--location \
--request POST "$BASE_URL/api/storage/v2/schemas" \
--header "Content-Type: application/json" \
--header "data-partition-id: $DATA_PARTITION" \
--header "Authorization: Bearer $TOKEN" \
--data-raw "${schema}"
echo ""
echo "---"
done
echo "Done!"
{
"kind": "DATA_PARTITION_TAG:wks:log:1.0.5",
"schema": [
{
"path": "azimuthReference",
"kind": "string"
},
{
"path": "dateCreated",
"kind": "datetime"
},
{
"path": "dateModified",
"kind": "datetime"
},
{
"path": "elevationReference.elevationFromMsl.unitKey",
"kind": "string"
},
{
"path": "elevationReference.elevationFromMsl.value",
"kind": "double"
},
{
"path": "elevationReference.name",
"kind": "string"
},
{
"path": "externalIds",
"kind": "[]link"
},
{
"path": "log.columnNames",
"kind": "[]string"
},
{
"path": "log.dataType",
"kind": "string"
},
{
"path": "log.dimension",
"kind": "int"
},
{
"path": "log.family",
"kind": "string"
},
{
"path": "log.familyType",
"kind": "string"
},
{
"path": "log.format",
"kind": "string"
},
{
"path": "log.logstoreId",
"kind": "long"
},
{
"path": "log.longName",
"kind": "string"
},
{
"path": "log.mnemonic",
"kind": "string"
},
{
"path": "log.name",
"kind": "string"
},
{
"path": "log.source",
"kind": "string"
},
{
"path": "log.unitKey",
"kind": "string"
},
{
"path": "name",
"kind": "string"
},
{
"path": "operation",
"kind": "string"
},
{
"path": "reference.columnNames",
"kind": "[]string"
},
{
"path": "reference.dataType",
"kind": "string"
},
{
"path": "reference.dimension",
"kind": "int"
},
{
"path": "reference.family",
"kind": "string"
},
{
"path": "reference.familyType",
"kind": "string"
},
{
"path": "reference.format",
"kind": "string"
},
{
"path": "reference.logstoreId",
"kind": "long"
},
{
"path": "reference.longName",
"kind": "string"
},
{
"path": "reference.mnemonic",
"kind": "string"
},
{
"path": "reference.name",
"kind": "string"
},
{
"path": "reference.source",
"kind": "string"
},
{
"path": "reference.unitKey",
"kind": "string"
},
{
"path": "referenceType",
"kind": "string"
},
{
"path": "relationships.logSet.confidence",
"kind": "double"
},
{
"path": "relationships.logSet.id",
"kind": "link"
},
{
"path": "relationships.logSet.name",
"kind": "string"
},
{
"path": "relationships.logSet.version",
"kind": "long"
},
{
"path": "relationships.timeDepthRelation.confidence",
"kind": "double"
},
{
"path": "relationships.timeDepthRelation.id",
"kind": "link"
},
{
"path": "relationships.timeDepthRelation.name",
"kind": "string"
},
{
"path": "relationships.timeDepthRelation.version",
"kind": "long"
},
{
"path": "relationships.well.confidence",
"kind": "double"
},
{
"path": "relationships.well.id",
"kind": "link"
},
{
"path": "relationships.well.name",
"kind": "string"
},
{
"path": "relationships.well.version",
"kind": "long"
},
{
"path": "relationships.wellbore.confidence",
"kind": "double"
},
{
"path": "relationships.wellbore.id",