Create Core Services Overview authored by Dania Kodeih (Microsoft)'s avatar Dania Kodeih (Microsoft)
The Essence of Common Code
==========================
![Common_Code](uploads/98626af166fd52280050890dc2bab706/Common_Code.png)
OSDU Data Platform
=======================
These are the data platform services for OSDU that form the core set of services for all OSDU data workflows.
## The structure of a repository
![Code_Repo](uploads/0bf56fa172bb2d0d9ca4283db79ab358/Cross_Cloud_SPI.png))
## The Services and APIs
| **Service**| **Open API Spec** | **Documentation** | **Code Repository**| **AWS**| **Azure** | **GCP** | **IBM**
| ------- | ------- | -------- | ------ | -------- | -------- |-------- |-------- |
| Legal | [API](https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/blob/master/docs/api/compliance_openapi.yaml) | [How To](https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/blob/master/docs/tutorial/ComplianceService.md) | [GitLab Legal ](https://community.opengroup.org/osdu/platform/security-and-compliance/legal) | | [Gitlab Azure Legal](https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/tree/master/provider/legal-azure) | | |
| Entitlements | [API](/Releases/R2.0/Services/Entitlements/Swagger) | [How To](/Releases/R2.0/Services/Entitlements/Tutorial) | Entitlements is Cloud Specific | [GitLab AWS Entitlements](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements-aws) | [GitLab Azure Entitlements](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements-azure) | [GitLab GCP Entitlements](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements-gcp) | [GitLab IBM Entitlements](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements-ibm)
| Indexer | Not a Public Service | [How To](https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/master/docs/tutorial/IndexerService.md) | [GitLab Indexer](https://community.opengroup.org/osdu/platform/system/indexer-service) <br /> [GitLab indexer queue ](https://community.opengroup.org/osdu/platform/system/indexer-queue) | | [Gitlab Azure Indexer](https://community.opengroup.org/osdu/platform/system/indexer-service/-/tree/master/provider/indexer-azure) <br /> [Gitlab Azure Indexer Queue](https://community.opengroup.org/osdu/platform/system/indexer-queue/-/tree/master/indexer-queue-azure) | | |
| Storage | [API](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/api/storage_openapi.yaml) | [How To](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md) |[GitLab Storage ](https://community.opengroup.org/osdu/platform/system/storage) | | [Gitlab Azure Storage Impl](https://community.opengroup.org/osdu/platform/system/storage/-/tree/master/provider/storage-azure)
| Search | [API](https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/docs/api/search_openapi.yaml) | [How To](https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/docs/tutorial/SearchService.md) |[GitLab Search ](https://community.opengroup.org/osdu/platform/system/search-service) | | [Gitlab Azure Search](https://community.opengroup.org/osdu/platform/system/search-service/-/tree/master/provider/search-azure) | | |
|Delivery | [API](https://community.opengroup.org/osdu/platform/system/delivery/-/blob/master/docs/api/delivery.yaml) | [Notes](https://community.opengroup.org/osdu/documentation/-/wikis/OSDU-(C)/Design-and-Implementation/Ingestion-and-Enrichment-Detail/R2-Delivery) | [GitLab Delivery](https://community.opengroup.org/osdu/platform/system/delivery) | | The Azure Delivery Implementation is an additional endpoint to os-storage which will be migrated in R3 and can be found [here](https://community.opengroup.org/osdu/platform/system/storage/-/tree/master/provider/storage-azure/src/main/java/org/opengroup/osdu/storage/provider/azure/simpledelivery) (see caveats below)| | |
OpenVDS Libraries
=======================
This is the repository for OpenVDS:
- [OpenVDS](https://community.opengroup.org/osdu/open-vds)
OSDU Data Platform Libraries
=======================
These are the convenience libraries that the data platform uses across different services. There is a core set of libraries (common behavior) as well as cloud platform specific versions that facilitate integration in their environment:
- [GitLab Core Common](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common)
- [GitLab AWS Lib ](https://community.opengroup.org/osdu/platform/system/lib/cloud/aws/os-core-lib-aws)
- [GitLab Azure Lib](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure)
- [GitLab GCP Lib](https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/os-core-lib-gcp)
- [GitLab IBM Lib ](https://community.opengroup.org/osdu/platform/system/lib/cloud/ibm/os-core-lib-ibm)
OSDU Ingestion Framework Prototype
==================================
- [OSDU R2 Protoype](https://github.com/google/framework-for-osdu/tree/master/osdu-r2)
- [OpenAPI Endpoint Specifications](https://amer-demo28-osdudev.apigee.io/apis)
- [Ingestion Design](https://community.opengroup.org/osdu/documentation/-/wikis/OSDU-(C)/Design-and-Implementation/Ingestion-and-Enrichment-Detail/R2-Ingestion-Workflow-Orchestration-non-Spike)
- [Implementation Design](https://github.com/google/framework-for-osdu/tree/master/osdu-r2)
Provider Specific Caveats in R2
================================
The Azure implementation of delivery follows a different implementation pattern as a result of provider dependency which will be addressed in R3. The Delivery function posts to a Delivery endpoint, but is hosted in the os-storage microservice.
The current API endpoint supported for delivery on Azure is as follows: https://\<main_service_URI\>/delivery/GetFileSignedURL
Agreed input/output and SPI
The main API call called for delivery is getFileSignedURLs(). Passing in the list of SRNs, it expects a list of file signed URLs in the JSON format below (output). We're generalizing using a String for the response (for now) to avoid pitfalls of strongly typing the response and reduce re-work when more functionality is added.
#### Delivery::GetFileSignedURL::Post - Input
```
{
srns: ["srn1", "srn2", "srn3"] < R2: required
}
```
#### Delivery::GetFileSignedURL::Post - Output
```
{
"unprocessed": ["srn1", "srn2"],
"processed": {
"srn3": {
"signedUrl": "https://...", < R2: required
"unsignedUrl": "...", < R2: Not required. Future extension point
"kind": "..." < R2: Not required. Future extension point
}
}
}
```
#### SPI
```
public interface ISignedURLService {
String getFileSignedURLS(List<String> srns); < R2: JSON representation of the API Output
}
```
We propose attaching the delivery API to the storage service as another endpoint.
Example query Implemented in this Story:
```
{
"SRNS": [ "srn:file/csv:a88797521dd111eab9ceff4274137c5c:1", … ]
}
```
Example query phase 2 (used for reference):
```
{
"SRNS": ["srn:master-data/Well:8438:",
"srn:work-product-component/WellboreMarker:8438_csv:",
"srn:file/csv:a88797521dd111eab9ceff4274137c5c:1"]
}
```