Changes

Page history
Update Core Services Overview authored by Chad Leong's avatar Chad Leong
Hide whitespace changes
Inline Side-by-side
Core-Services-Overview.md
View page @ 476e238b
# The Essence of Common Code
We are moving to this documentation to this new location
![Common Code](uploads/65eec5eb5a4cd91e08f0f86077954884/image.png)
# OSDU Data Platform
These are the data platform services for OSDU that form the core set of services for all OSDU data workflows.
## The structure of a repository
![Code Repo](uploads/fbf6825b59f4e965377de0760e6e235b/image.png)
## Core Services and APIs
*Refer [here](https://community.opengroup.org/osdu/documentation/-/wikis/Core-Services-API-Docs) to the latest Open API doc
| **Service** | * **Open API Spec** | **Documentation** | **Code Repository** |
|-------------|-------------------|-------------------|---------------------|
| Dataset | [~~API~~](https://community.opengroup.org/osdu/platform/system/dataset/-/blob/master/docs/dataset.swagger.yaml) | [Docs](https://community.opengroup.org/osdu/platform/system/dataset/-/tree/master) | [Dataset](https://community.opengroup.org/osdu/platform/system/dataset) |
| Entitlements | [~~API~~](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/blob/master/docs/api/entitlements_openapi.yaml) | [Docs](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements/-/tree/master/docs/tutorial) | [Entitlement](https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements) |
| File | [~~API~~](https://community.opengroup.org/osdu/platform/system/file/-/blob/master/docs/file-service_openapi.yaml) | [Docs](https://community.opengroup.org/osdu/platform/system/file/-/blob/master/README.md) | [File](https://community.opengroup.org/osdu/platform/system/file) |
| Indexer | Not a Public Service | [Docs](https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/master/docs/tutorial/IndexerService.md) | [Indexer](https://community.opengroup.org/osdu/platform/system/indexer-service) <br>[Indexer Queue](https://community.opengroup.org/osdu/platform/system/indexer-queue) |
| Legal | [~~API~~](https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/blob/master/docs/api/compliance_openapi.yaml) | [Docs](https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/blob/master/docs/tutorial/ComplianceService.md) | [Legal](https://community.opengroup.org/osdu/platform/security-and-compliance/legal) |
| Notification | [~~API~~](https://community.opengroup.org/osdu/platform/system/notification/-/blob/master/docs/api/notification_openapi.yaml) | [Docs](https://community.opengroup.org/osdu/platform/system/notification/-/blob/master/docs/tutorial/DataNotification.md) | [Notification](https://community.opengroup.org/osdu/platform/system/notification) |
| Partition | [~~API~~](https://community.opengroup.org/osdu/platform/system/partition/-/blob/master/docs/api/partition_openapi.yaml) | [Docs](https://community.opengroup.org/osdu/platform/system/partition/-/blob/master/docs/tutorial/Partition.md) | [Partition](https://community.opengroup.org/osdu/platform/system/partition) |
| Policy | [~~API~~](https://community.opengroup.org/osdu/platform/security-and-compliance/policy/-/blob/master/docs/openapi.yaml) | [Docs](https://osdu.pages.opengroup.org/platform/security-and-compliance/policy/) | [Policy](https://community.opengroup.org/osdu/platform/security-and-compliance/policy) |
| Register | [~~API~~](https://community.opengroup.org/osdu/platform/system/register/-/blob/master/docs/api/register_openapi.yaml) | [Docs](https://community.opengroup.org/osdu/platform/system/register/-/tree/master/docs/tutorial) | [Register](https://community.opengroup.org/osdu/platform/system/register) | (https://community.opengroup.org/osdu/platform/system/notification/-/blob/master/docs/api/notification_openapi.yaml) |
| Schema | [~~API~~](https://community.opengroup.org/osdu/platform/system/schema-service/-/blob/master/docs/api/schema_openapi.yaml) | [Docs](https://community.opengroup.org/osdu/platform/system/schema-service/-/blob/master/docs/SchemaService-OSDU.md) | [Schema](https://community.opengroup.org/osdu/platform/system/schema-service) |
| Search | [~~API~~](https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/docs/api/search_openapi.yaml) | [Docs](https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/docs/tutorial/SearchService.md) | [Search](https://community.opengroup.org/osdu/platform/system/search-service) |
| Secret | [~~API~~](https://community.opengroup.org/osdu/platform/security-and-compliance/secret/-/blob/main/docs/api/secret_openapi.json) | | [Secret](https://community.opengroup.org/osdu/platform/security-and-compliance/secret) |
| Storage | [~~API~~](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/api/storage_openapi.yaml) | [Docs](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md) | [Storage](https://community.opengroup.org/osdu/platform/system/storage) |
## Data Flow Services and DAGs
| Service | Open API Spec | Documentation | Code Repository |
|-------------|-------------------|-------------------|---------------------|
| Ingestion Workflow Service | [API](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/blob/master/docs/api/openapi.workflow.yaml) | [Docs](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/blob/master/README.md) | [Ingestion Workflow](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow) |
| - CSV Ingestion DAG | | [Docs](https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/wikis/home) | [CSV Parser](https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser) |
| - Manifest Ingestion DAG | | [Docs](https://community.opengroup.org/groups/osdu/platform/data-flow/data-loading/-/wikis/home) | [Manifest ingestion](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags) |
| - Manifest Ingestion by Reference DAG | | [Docs](https://community.opengroup.org/groups/osdu/platform/data-flow/data-loading/-/wikis/home) | [Manifest ingestion by reference](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/blob/master/src/osdu_dags/osdu-ingest-r3-by-reference.py?ref_type=heads) |
| - WITSML Ingestion DAG | | [Docs](https://community.opengroup.org/groups/osdu/platform/data-flow/data-loading/-/wikis/home) | [WITSML parser](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics-osdu-integration) |
| - SegY to oZGY Conversion DAG | | [Docs](https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion/-/blob/master/doc/gcp/QUICKSTART.md) | [SegY to oZGY Conversion](https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion) |
| - SegY to oVDS Conversion DAG | | [Docs](https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-vds-conversion/-/blob/master/docs/gcp/QUICKSTART.md) | [SegY to oVDS Conversion](https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-vds-conversion) |
## External Data Services and APIs
| Service | Open API Spec | Documentation | Code Repository |
|-------------|-------------------|-------------------|---------------------|
| External Data Workflow DAG | | - [Docs](https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/extern-data/docs/-/tree/master?ref_type=heads)<br /> - [Wiki](https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/extern-data/home/-/wikis/home) | [Core External Data workflow](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow)
| External Data Sources Data Management Service | [API](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/eds-dms/-/blob/master/doc/api/eds_dms_openapi_v3.yaml?ref_type=heads) | -[Docs](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/eds-dms/-/tree/master/doc?ref_type=heads)<br /> - [Wiki](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/eds-dms/-/wikis/External-Data-Sources-Data-Management-Service) | [EDS DMS](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/eds-dms)
## Reference Systems Services and APIs
*Refer [here](https://community.opengroup.org/osdu/documentation/-/wikis/Core-Services-API-Docs) to the latest Open API doc
| Service | Open API Spec | Documentation | Code Repository |
|-------------|-------------------|-------------------|---------------------|
|CRS Conversion |[~~API v3~~](https://community.opengroup.org/osdu/platform/system/reference/crs-conversion-service/-/blob/master/docs/v3/api_spec/crs_converter_openapi.json)<br /> [~~API v2~~](https://community.opengroup.org/osdu/platform/system/reference/crs-conversion-service/-/blob/master/docs/v2/api_spec/crs_converter_openapi.json) | - [Docs](https://community.opengroup.org/osdu/platform/system/reference/crs-conversion-service/-/blob/master/docs/v3/tutorial/CRS_Convert_Service_howto.md)<br /> - [Wiki](https://community.opengroup.org/osdu/platform/system/reference/crs-conversion-service/-/wikis/home)<br /> - [API Docs](https://community.opengroup.org/osdu/platform/system/reference/crs-conversion-service/-/tree/master/docs) | [CRS Conversion Service](https://community.opengroup.org/osdu/platform/system/reference/crs-conversion-service) |
|CRS Catalog|[~~API v3~~](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/docs/api_spec/crs-catalog-openapi-v3.yaml)<br /> [~~API v2~~](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/docs/api_spec/crs-catalog-openapi-v2.json)| - [Docs](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/docs/tutorial/CRS_Catalog_Service.md)<br /> - [Wiki](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/wikis/home)<br /> - [API Docs](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/tree/master/docs) |[CRS Catalog Service](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service)|
|Unit|[~~API v3~~](https://community.opengroup.org/osdu/platform/system/reference/unit-service/-/blob/master/docs/v3/api_spec/unit_service_openapi_v3.json)<br /> [~~API v2~~](https://community.opengroup.org/osdu/platform/system/reference/unit-service/-/blob/master/docs/v2/api_spec/unit_service_openapi_v2.json) | - [Wiki](https://community.opengroup.org/osdu/platform/system/reference/unit-service/-/wikis/home) <br/> - [API Docs](https://community.opengroup.org/osdu/platform/system/reference/unit-service/-/tree/master/docs) |[Unit Service](https://community.opengroup.org/osdu/platform/system/reference/unit-service)|
## Domain Data Management Services
| Domain | Open API Spec | Documentation | Code Repository |
|--------|---------------|---------------|-----------|
| Seismic | [API](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/seismic-dms-suite/seismic-store-service/-/blob/master/app/sdms/docs/api/openapi.osdu.yaml) | [Docs](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/home/-/tree/master/docs) | [Seismic DDMS](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic) |
| Wellbore | [API](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/blob/master/spec/generated/openapi.json) | [Docs](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/tree/master/docs) | [Wellbore DDMS](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore) |
| Well Delivery | [API](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/well-delivery/well-delivery/-/blob/master/docs/api/swagger.yaml) | [Docs](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/well-delivery/well-delivery/-/tree/master/docs) | [Well delivery DDMS](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/well-delivery/well-delivery) |
| Reservoir | [API](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/reservoir/home/-/blob/main/swagger.json) | [Docs](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/reservoir/home/-/blob/main/ReleaseNotes.md) | [Reservoir DDMS](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/reservoir) |
| Rock and Fluid Samples | [API](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/rock-and-fluid-sample/rafs-ddms-services/-/blob/main/docs/spec/openapi.json) | [Docs](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/rock-and-fluid-sample/rafs-ddms-services/-/blob/main/docs/tutorial/readme.md) | [Rock and Fluid Sample DDMS](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/rock-and-fluid-sample/rafs-ddms-services) |
| Production (WIP) | N/A | [Docs](https://gitlab.opengroup.org/osdu/subcommittees/data-def/projects/production/docs/-/tree/master/Presentations) | [Production DDMS](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/production) |
## Consumption Zone
| Apps | Open API Spec | Documentation | Code Repository |
|----------|-------------------|-------------------|---------------------|
| Geospatial Consumption Zone | [API](https://community.opengroup.org/osdu/platform/consumption/geospatial/-/blob/master/docs/deployment/post_deployment_testing.md) | [Docs](https://community.opengroup.org/osdu/platform/consumption/geospatial/-/tree/master/docs) | [Geospatial Consumption Zone](https://community.opengroup.org/osdu/platform/consumption/geospatial) |
# OSDU Data Platform Libraries
These are the convenience libraries that the data platform uses across different services. There is a core set of libraries (common behavior) as well as cloud platform-specific versions that facilitate integration in their environment:
- [GitLab Core Common](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common)
- [GitLab AWS Lib](https://community.opengroup.org/osdu/platform/system/lib/cloud/aws/os-core-lib-aws)
- [GitLab Azure Lib](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure)
- [GitLab GCP Lib](https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/os-core-lib-gcp)
- [GitLab IBM Lib](https://community.opengroup.org/osdu/platform/system/lib/cloud/ibm/os-core-lib-ibm)
\ No newline at end of file
https://community.opengroup.org/groups/osdu/platform/-/wikis/Core-Services-Overview
\ No newline at end of file