Skip to content
Snippets Groups Projects
Commit 5a73d1e7 authored by Shane Hutchins's avatar Shane Hutchins
Browse files

Documentation update and publish to pages

parent 52d29609
No related branches found
No related tags found
1 merge request!448Documentation update and publish to pages
Pipeline #249631 failed
Stage: review
Stage: build
Stage: coverage
Stage: containerize
Stage: scan
Stage: deploy
Stage: integration
Stage: publish
Hide whitespace changes
Inline Side-by-side
.gitlab-ci.yml
+ 1
− 0
View file @ 5a73d1e7
......@@ -48,6 +48,7 @@ include:
file: "cloud-providers/azure.yml"
- local: "devops/gc/pipeline/override-stages.yml"
- local: "/publish.yml"
.maven:
image: maven:3.9.3-eclipse-temurin-17
......
docs/tutorial/HowToBecomeADDMS.md → docs/docs/HowToBecomeADDMS.md
+ 21
− 32
View file @ 5a73d1e7
## How to become a DDMS
* [Introduction](#introduction)
* [Register as a DDMS](#register)
* [Create Schema](#create-schema)
* [Expose legal tag and ACLs through your APIs](#expose-legal)
* [Optional: Expose compliance for derivative data](#expose-derivative)
* [Ingest data and create a shadow record](#shadow-record)
* [Perform compliance and ACL checks using shadow records](#validate)
* [Client retrieves the bulk data](#retrieve)
* [Client retrieves Single Entity id data](#retrieve-single-id)
## Introduction <a name="introduction"></a>
# How to become a DDMS
## Introduction
A Domain Data Management Service (DDMS) can be seen as any source of truth for data that manages the data life cycle, satisfies given mandatory data access concerns, and makes its data globally discoverable and retrievable through the OSDU. It could be a standalone service dedicated to a specific data type or a subcomponent of an application or platform. It simply enables its data to be retrieved outside of its regular scope.
A DDMS needs to enforce the common concerns of
......@@ -24,7 +14,7 @@ OSDU solves these concerns primarily using Storage records. A Storage record is
The following is the preferred method of using Records to enable these concerns for a DDMS.
## Register as a DDMS <a name="register"></a>
## Register as a DDMS
The first step is to register as a DDMS. This makes your DDMS discoverable to clients and presents them with an API definition that tells them how to retrieve the bulk data when a record from their DDMS is discovered.
......@@ -32,7 +22,7 @@ The only API that needs to be defined is the one that tells them how to retrieve
Note that you can register as much of your API specification as you like. You only need to define the method clients should use to retrieve the bulk data using the custom property x-ddms-retrieve-entity: true.
<details><summary>curl</summary>
<details><summary>Curl Post</summary>
```
......@@ -118,7 +108,7 @@ Note that you can register as much of your API specification as you like. You on
</details>
## Create a DDMS schema <a name="create-schema"></a>
## Create a DDMS schema
It is up to the bulk data store to determine what properties of the bulk data they want to push into a Storage record and to make discoverable within the OSDU.
......@@ -128,7 +118,7 @@ When deploying your service, you should do the one-time operation of publishing
You can find the schema definition in [shared schemas](https://community.opengroup.org/osdu/platform/system/schema-service/-/tree/master/deployments/shared-schemas) for reference.
<details><summary>curl</summary>
<details><summary>Curl Post Schema Service</summary>
```
......@@ -175,12 +165,12 @@ Notice, we are also declaring 3 properties:
These act as well-known properties that should be added to the record by your DDMS. Clients can then use this information to retrieve the bulk data after discovery using the DDMS registration APIs. Every schema created should declare these properties to use this pattern of ingestion.
## Expose legal tag and ACLs through your APIs <a name="expose-legal"></a>
## Expose legal tag and ACLs through your APIs
Unless you have a scenario where you know what legal tag and ACL should be applied to the data you are ingesting, you will need to expose the legal tag and ACL in your ingestion APIs. This allows your clients to supply the legal tag and ACL themselves.
You can expose the same interface as the Storage records API, allowing you to assign them to the record you create.
<details>
<details><summary>JSON</summary>
```
......@@ -194,7 +184,7 @@ You can expose the same interface as the Storage records API, allowing you to as
```
</details>
## Optionally expose derivative compliance through your APIs <a name="expose-derivative"></a>
## Optionally expose derivative compliance through your APIs
If you expect derivative data to be stored in your DDMS, you need to expose 2 more properties through your APIs that can be appended to your Storage record.
Again, you can expose the same interface as the Storage records API, allowing you to assign them to the record directly. The 2 properties are:
......@@ -204,7 +194,7 @@ Again, you can expose the same interface as the Storage records API, allowing yo
If a derivative is being created then a legal tag does not need to be assigned as it inherits this from its parents.
<details><summary>curl</summary>
<details><summary>JSON</summary>
```
......@@ -218,7 +208,7 @@ If a derivative is being created then a legal tag does not need to be assigned a
```
</details>
## Ingest data and create a shadow record <a name="shadow-record"></a>
## Ingest data and create a shadow record
Whenever bulk data is ingested, you need to create a shadow record within Storage. DDMS may have the feature to create the record and ingest it automatically such as `api/os-wellbore-ddms/ddms/v3/wellbores`. This shadow record represents the specific bulk data instance in a 1:1 relationship and makes each instance globally discoverable.
......@@ -228,7 +218,7 @@ First, you should store the bulk data, and then create the shadow Record. This w
Remember, you should append your DDMS Id, entityType and the bulk data’s local id to the Storage record.
<details><summary>curl - storage</summary>
<details><summary>Curl Put Storage</summary>
```
......@@ -260,7 +250,7 @@ Remember, you should append your DDMS Id, entityType and the bulk data’s local
</details>
<details><summary>curl - ddms</summary>
<details><summary>Curl Put DDMS</summary>
```
......@@ -374,13 +364,13 @@ Remember, you should append your DDMS Id, entityType and the bulk data’s local
</details>
## Perform compliance and ACL checks using shadow records <a name="validate"></a>
## Perform compliance and ACL checks using shadow records
As mentioned, a DDMS should create a shadow record for every instance of bulk data ingested into their data store. This can have advantages beyond global discover-ability. Whenever you request a storage record, both compliance and entitlements are checked before returning the data. A DDMS can use this to their advantage.
By forwarding on any request by the client to retrieve the record, you can delegate these responsibilities to the Storage service. If OSDU returns the Record, the client can access both this and the bulk data, and so you can return the same to the client or only the Record.
<details><summary>curl</summary>
<details><summary>Curl Post Storage Query Records</summary>
```
......@@ -407,7 +397,7 @@ By forwarding on any request by the client to retrieve the record, you can deleg
In this scenario, you also don’t need to store the ACL or legal tag information in your DDMS because those are being retrieved directly from the OSDU in this request. However, you need to either store or be able to generate the Storage record ID needed to retrieve the record for the bulk data requested.
## Client retrieves the bulk data <a name="retrieve"></a>
## Client retrieves the bulk data
Imagine the client discovered a record with the following data
......@@ -523,7 +513,7 @@ Currently, this is a manual step for the user and so they have to understand you
Using the returned specification and data we discovered, the resulting API call the user would be expected to make to retrieve the bulk data would be
<details><summary>curl</summary>
<details><summary>Curl</summary>
```
curl --request GET \
......@@ -539,13 +529,13 @@ Using the returned specification and data we discovered, the resulting API call
Where 123456 is the localId stored in the records data and the URL is defined in the API spec in the server, and path sections of the operation that has the property x-ddms-retrieve-entity.
## Client retrieves Single Entity id data <a name="retrieve-single-id"></a>
## Client retrieves Single Entity id data
Client should have already some data/record id for the ddms, I.E. `opendes:master-data--Wellbore:NPD-3180`.
You can retrieve them directly with register service Single Entity retrieval.
<details><summary>curl</summary>
<details><summary>Curl Get Register</summary>
```text
curl --request GET \
......@@ -656,5 +646,4 @@ x-envoy-upstream-service-time: 16
}
}
```
</details>
</details>
\ No newline at end of file
docs/tutorial/ActionService.md → docs/docs/api.md
+ 15
− 52
View file @ 5a73d1e7
## Action Service
# Action APIs
## Table of Contents <a name="TOC"></a>
- [Introduction](#introduction)
- [Action APIs](#action-apis)
* [Registering an Action](#register-action)
* [Get an Action by ID](#get-action)
* [Retrieve Actions](#retrieve-action)
* [Delete an Action by ID](#delete-action)
* [Validate action](#regex-test)
* [Version info endpoint](#version-info-endpoint)
- [Current Limitations](#limitation)
## Introduction <a name="introduction"></a>
The high level design of this service can conceptually be thought of similar to the 'command' design pattern. Essentially this pattern decouples a trigger from an action. This is often used in UIs where a trigger is perhaps a user clicking a button and the action is a program function that is triggered by the click. There is often an optional context which can provide the action with data to use in the function, as well as to enable / disable the action for the user (perhaps if the data is not relevant to the action in question).
This service will allow an application to register an action (the function to be triggered). It will expect data (context) to come from OSDU to enable the action, and the application can register a filter (enable/disable) to say what data can be used with this action.
[Back to Table of Contents](#TOC)
## Action APIs <a name="action-apis"></a>
### Registering an Action <a name="register-action"></a>
### Registering an Action
This API allows registering an action in the form of a GET HTTPS URL and a filter. The filter specifies what data the action can be applied to.
> It is recommended that Admins first use the [Validate action](#regex-test) API, to make sure the action is acceptable and the output of the action with a test payload is as expected.
......@@ -28,7 +8,7 @@ This API allows registering an action in the form of a GET HTTPS URL and a filte
```
POST /api/register/v1/action
```
<details><summary>curl</summary>
<details><summary>Curl Post Register Action</summary>
```
curl --request POST \
......@@ -74,14 +54,12 @@ Because the filter of the action matches the Record (petrel and regularheightfie
https://myapp.osdu.com/action/common:doc:123456789/myPetrelProj
```
[Back to Table of Contents](#TOC)
## Get an Action by ID <a name="get-action"></a>
## Get an Action by ID
This API allows getting an action with a given Id.
```
GET /api/register/v1/action/{id}
```
<details><summary>curl</summary>
<details><summary>Curl Get an Action ID</summary>
```
curl --request GET \
......@@ -92,9 +70,8 @@ curl --request GET \
```
</details>
[Back to Table of Contents](#TOC)
## Retrieve Actions <a name="retrieve-action"></a>
## Retrieve Actions
This API allows retrieving all actions that match a given filter.
```
POST /api/register/v1/action:retrieve
......@@ -119,7 +96,7 @@ So imagine you have a Record retrieved from OSDU:
```
And make the following call to retrieve actions API:
<details><summary>curl</summary>
<details><summary>Curl Post Action Retrieve</summary>
```
curl --request POST\
......@@ -175,15 +152,13 @@ Then the response returns all matching actions with the substituted parameters s
]
```
[Back to Table of Contents](#TOC)
## Delete an Action by ID <a name="delete-action"></a>
## Delete an Action by ID
This API allows deleting an action with a given id.
```
DELETE /api/register/v1/action/{id}
```
<details><summary>curl</summary>
<details><summary>Curl Delete Action ID</summary>
```
curl --request DELETE \
......@@ -194,9 +169,7 @@ curl --request DELETE \
```
</details>
[Back to Table of Contents](#TOC)
## Validate action <a name="regex-test"></a>
## Validate action
This API is a helper API method that allows users to validate their action is working as expected, including template and regular expression usage, before they create an action in the system.
```
......@@ -205,7 +178,7 @@ POST /api/register/v1/action:test
Let's consider the following payload for [Register an Action](#register-action) request example
<details><summary>curl</summary>
<details><summary>Curl Post Register Action</summary>
```
curl --request POST \
......@@ -238,7 +211,7 @@ onto the data.uri properties value.
This regular expression attempts to extract the path segment out of a URI. But before we register this action, we want to be sure that the specified regular expression is correct and return the expected value from expected payload for a kind. This can be achieved by following API call:
<details><summary>curl</summary>
<details><summary>Curl Post Register Action</summary>
```
curl --request POST\
......@@ -301,8 +274,6 @@ Here are some regular expression registration examples:
The regular expression match per record field is capped at a maximum of 2 seconds for performance reasons. Please take a look at the error message for detailed response.
[Back to Table of Contents](#TOC)
## Version info endpoint
For deployment available public `/info` endpoint, which provides build and git related information.
#### Example response:
......@@ -331,18 +302,10 @@ For deployment available public `/info` endpoint, which provides build and git r
]
}
```
This endpoint takes information from files, generated by `spring-boot-maven-plugin`,
`git-commit-id-plugin` plugins. Need to specify paths for generated files to matching
properties:
- `version.info.buildPropertiesPath`
- `version.info.gitPropertiesPath`
[Back to table of contents](#TOC)
## Current Limitations <a name="limitation"></a>
There are mainly 2 limitations currently:
- Users need to individually register for each data partition they want the action enabled from.
- We don't support a wildcard in the action URL for the partition id. Something like myaction.com/kind?dpid={data-partition-id} may be supported in the future so that the request can be data partition aware.
[Back to Table of Contents](#TOC)
- `version.info.buildPropertiesPath`
- `version.info.gitPropertiesPath`
\ No newline at end of file
docs/docs/index.md 0 → 100644
+ 14
− 0
View file @ 5a73d1e7
# Register Service
## Introduction
The high level design of this service can conceptually be thought of similar to the 'command' design pattern. Essentially this pattern decouples a trigger from an action. This is often used in UIs where a trigger is perhaps a user clicking a button and the action is a program function that is triggered by the click. There is often an optional context which can provide the action with data to use in the function, as well as to enable / disable the action for the user (perhaps if the data is not relevant to the action in question).
This service will allow an application to register an action (the function to be triggered). It will expect data (context) to come from OSDU to enable the action, and the application can register a filter (enable/disable) to say what data can be used with this action.
The Register Service is a Maven multi-module project with each cloud implementation placed in its submodule.
## Current Limitations
There are mainly 2 limitations currently:
- Users need to individually register for each data partition they want the action enabled from.
- We don't support a wildcard in the action URL for the partition id. Something like myaction.com/kind?dpid={data-partition-id} may be supported in the future so that the request can be data partition aware.
\ No newline at end of file
docs/docs/running.md 0 → 100644
+ 28
− 0
View file @ 5a73d1e7
# Running Register service
The Register Service is a Maven multi-module project with each cloud implementation placed in its submodule.
## Running integration tests
See Integration-Testing [Readme](https://community.opengroup.org/osdu/platform/system/register/-/blob/master/testing/README.md).
## Open API 3.0 - Swagger
- Swagger UI : https://host/context-path/swagger (will redirect to https://host/context-path/swagger-ui/index.html)
- api-docs (JSON) : https://host/context-path/api-docs
- api-docs (YAML) : https://host/context-path/api-docs.yaml
All the Swagger and OpenAPI related common properties are managed here [swagger.properties](https://community.opengroup.org/osdu/platform/system/register/-/blob/master/register-core/src/main/resources/swagger.properties)
#### Server Url(full path vs relative path) configuration
- `api.server.fullUrl.enabled=true` It will generate full server url in the OpenAPI swagger
- `api.server.fullUrl.enabled=false` It will generate only the contextPath only
- default value is false (Currently only in Azure it is enabled)
[Reference]:(https://springdoc.org/faq.html#_how_is_server_url_generated)
## AWS
Instructions for running and testing this service can be found [here](https://community.opengroup.org/osdu/platform/system/register/-/blob/master/provider/register-aws/README.md)
## Azure
Instructions for running and testing this service can be found [here](https://community.opengroup.org/osdu/platform/system/register/-/blob/master/provider/register-azure/README.md)
## Google Cloud
Instructions for running the Google Cloud implementation locally can be found [here](https://community.opengroup.org/osdu/platform/system/register/-/blob/master/provider/register-gc/README.md).
docs/mkdocs.yml 0 → 100644
+ 38
− 0
View file @ 5a73d1e7
site_name: OSDU Register Documentation
site_description: OSDU Register Service
site_author: Shane Hutchins
# Repository
repo_url: https://community.opengroup.org/osdu/platform/system/register
repo_name: register
# Copyright
copyright: Copyright &copy; 2024 Open Subsurface Data Universe Software
theme:
name: material
palette:
- scheme: default
primary: indigo
accent: indigo
toggle:
icon: material/brightness-7
name: Switch to dark mode
- scheme: slate
primary: indigo
accent: indigo
toggle:
icon: material/brightness-4
name: Switch to light mode
font:
text: Roboto
code: Roboto Mono
markdown_extensions:
- admonition
- sane_lists
- tables
plugins:
- search
- git-revision-date
publish.yml 0 → 100644
+ 30
− 0
View file @ 5a73d1e7
run-test-pages:
stage: build
image: python:latest
script:
- cd docs
- pip install mkdocs-material
- pip install mkdocs-git-revision-date-plugin
- mkdocs build --site-dir ../test
allow_failure: true
artifacts:
paths:
- test
when: manual
pages:
stage: publish
image: python:latest
tags: ["osdu-medium"]
script:
- cd docs
- pip install --default-timeout=1000 mkdocs-material
- pip install mkdocs-git-revision-date-plugin
- mkdocs build --site-dir ../public
allow_failure: true
needs: []
artifacts:
paths:
- public
# rules:
# - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH'
\ No newline at end of file
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment