|
|
The Essence of Common Code
|
|
|
==========================
|
|
|
|
|
|
![Common_Code](uploads/98626af166fd52280050890dc2bab706/Common_Code.png)
|
|
|
|
|
|
OSDU Data Platform
|
|
|
=======================
|
|
|
These are the data platform services for OSDU that form the core set of services for all OSDU data workflows.
|
|
|
|
|
|
|
|
|
## The structure of a repository
|
|
|
|
|
|
![Code_Repo](uploads/0bf56fa172bb2d0d9ca4283db79ab358/Cross_Cloud_SPI.png))
|
|
|
|
|
|
## The Services and APIs
|
|
|
|
|
|
| **Service**| **Open API Spec** | **Documentation** | **Code Repository**| **AWS**| **Azure** | **GCP** | **IBM**
|
|
|
| ------- | ------- | -------- | ------ | -------- | -------- |-------- |-------- |
|
|
|
| Legal | [API](https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/blob/master/docs/api/compliance_openapi.yaml) | [How To](https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/blob/master/docs/tutorial/ComplianceService.md) | [GitLab Legal ](https://community.opengroup.org/osdu/platform/security-and-compliance/legal) | | [Gitlab Azure Legal](https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/tree/master/provider/legal-azure) | | |
|
|
|
| Entitlements | [API](/Releases/R2.0/Services/Entitlements/Swagger) | [How To](/Releases/R2.0/Services/Entitlements/Tutorial) | Entitlements is Cloud Specific | [GitLab AWS Entitlements](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements-aws) | [GitLab Azure Entitlements](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements-azure) | [GitLab GCP Entitlements](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements-gcp) | [GitLab IBM Entitlements](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements-ibm)
|
|
|
| Indexer | Not a Public Service | [How To](https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/master/docs/tutorial/IndexerService.md) | [GitLab Indexer](https://community.opengroup.org/osdu/platform/system/indexer-service) <br /> [GitLab indexer queue ](https://community.opengroup.org/osdu/platform/system/indexer-queue) | | [Gitlab Azure Indexer](https://community.opengroup.org/osdu/platform/system/indexer-service/-/tree/master/provider/indexer-azure) <br /> [Gitlab Azure Indexer Queue](https://community.opengroup.org/osdu/platform/system/indexer-queue/-/tree/master/indexer-queue-azure) | | |
|
|
|
| Storage | [API](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/api/storage_openapi.yaml) | [How To](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md) |[GitLab Storage ](https://community.opengroup.org/osdu/platform/system/storage) | | [Gitlab Azure Storage Impl](https://community.opengroup.org/osdu/platform/system/storage/-/tree/master/provider/storage-azure)
|
|
|
| Search | [API](https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/docs/api/search_openapi.yaml) | [How To](https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/docs/tutorial/SearchService.md) |[GitLab Search ](https://community.opengroup.org/osdu/platform/system/search-service) | | [Gitlab Azure Search](https://community.opengroup.org/osdu/platform/system/search-service/-/tree/master/provider/search-azure) | | |
|
|
|
|Delivery | [API](https://community.opengroup.org/osdu/platform/system/delivery/-/blob/master/docs/api/delivery.yaml) | [Notes](https://community.opengroup.org/osdu/documentation/-/wikis/OSDU-(C)/Design-and-Implementation/Ingestion-and-Enrichment-Detail/R2-Delivery) | [GitLab Delivery](https://community.opengroup.org/osdu/platform/system/delivery) | | The Azure Delivery Implementation is an additional endpoint to os-storage which will be migrated in R3 and can be found [here](https://community.opengroup.org/osdu/platform/system/storage/-/tree/master/provider/storage-azure/src/main/java/org/opengroup/osdu/storage/provider/azure/simpledelivery) (see caveats below)| | |
|
|
|
|
|
|
OpenVDS Libraries
|
|
|
=======================
|
|
|
This is the repository for OpenVDS:
|
|
|
- [OpenVDS](https://community.opengroup.org/osdu/open-vds)
|
|
|
|
|
|
OSDU Data Platform Libraries
|
|
|
=======================
|
|
|
These are the convenience libraries that the data platform uses across different services. There is a core set of libraries (common behavior) as well as cloud platform specific versions that facilitate integration in their environment:
|
|
|
|
|
|
- [GitLab Core Common](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common)
|
|
|
- [GitLab AWS Lib ](https://community.opengroup.org/osdu/platform/system/lib/cloud/aws/os-core-lib-aws)
|
|
|
- [GitLab Azure Lib](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure)
|
|
|
- [GitLab GCP Lib](https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/os-core-lib-gcp)
|
|
|
- [GitLab IBM Lib ](https://community.opengroup.org/osdu/platform/system/lib/cloud/ibm/os-core-lib-ibm)
|
|
|
|
|
|
OSDU Ingestion Framework Prototype
|
|
|
==================================
|
|
|
- [OSDU R2 Protoype](https://github.com/google/framework-for-osdu/tree/master/osdu-r2)
|
|
|
- [OpenAPI Endpoint Specifications](https://amer-demo28-osdudev.apigee.io/apis)
|
|
|
- [Ingestion Design](https://community.opengroup.org/osdu/documentation/-/wikis/OSDU-(C)/Design-and-Implementation/Ingestion-and-Enrichment-Detail/R2-Ingestion-Workflow-Orchestration-non-Spike)
|
|
|
- [Implementation Design](https://github.com/google/framework-for-osdu/tree/master/osdu-r2)
|
|
|
|
|
|
Provider Specific Caveats in R2
|
|
|
================================
|
|
|
The Azure implementation of delivery follows a different implementation pattern as a result of provider dependency which will be addressed in R3. The Delivery function posts to a Delivery endpoint, but is hosted in the os-storage microservice.
|
|
|
The current API endpoint supported for delivery on Azure is as follows: https://\<main_service_URI\>/delivery/GetFileSignedURL
|
|
|
Agreed input/output and SPI
|
|
|
|
|
|
The main API call called for delivery is getFileSignedURLs(). Passing in the list of SRNs, it expects a list of file signed URLs in the JSON format below (output). We're generalizing using a String for the response (for now) to avoid pitfalls of strongly typing the response and reduce re-work when more functionality is added.
|
|
|
|
|
|
|
|
|
#### Delivery::GetFileSignedURL::Post - Input
|
|
|
```
|
|
|
{
|
|
|
srns: ["srn1", "srn2", "srn3"] < R2: required
|
|
|
}
|
|
|
```
|
|
|
#### Delivery::GetFileSignedURL::Post - Output
|
|
|
```
|
|
|
{
|
|
|
"unprocessed": ["srn1", "srn2"],
|
|
|
"processed": {
|
|
|
"srn3": {
|
|
|
"signedUrl": "https://...", < R2: required
|
|
|
"unsignedUrl": "...", < R2: Not required. Future extension point
|
|
|
"kind": "..." < R2: Not required. Future extension point
|
|
|
}
|
|
|
}
|
|
|
}
|
|
|
```
|
|
|
#### SPI
|
|
|
```
|
|
|
public interface ISignedURLService {
|
|
|
String getFileSignedURLS(List<String> srns); < R2: JSON representation of the API Output
|
|
|
}
|
|
|
```
|
|
|
|
|
|
We propose attaching the delivery API to the storage service as another endpoint.
|
|
|
|
|
|
Example query Implemented in this Story:
|
|
|
```
|
|
|
{
|
|
|
"SRNS": [ "srn:file/csv:a88797521dd111eab9ceff4274137c5c:1", … ]
|
|
|
}
|
|
|
```
|
|
|
Example query phase 2 (used for reference):
|
|
|
```
|
|
|
{
|
|
|
"SRNS": ["srn:master-data/Well:8438:",
|
|
|
"srn:work-product-component/WellboreMarker:8438_csv:",
|
|
|
"srn:file/csv:a88797521dd111eab9ceff4274137c5c:1"]
|
|
|
}
|
|
|
``` |