Schema issueshttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues2020-11-09T20:06:02Zhttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/9[System/Core] Schema Service2020-11-09T20:06:02ZStephen Whitley (Invited Expert)[System/Core] Schema ServiceThe schema service is both an implementation of the schema standard defined below and a service to help create, manage and share schemas between different users. It aims to both raise awareness of the different models used by teams to he...The schema service is both an implementation of the schema standard defined below and a service to help create, manage and share schemas between different users. It aims to both raise awareness of the different models used by teams to help reuse both of the data sources but also of the schemas themselves. This will hopefully help reduce the friction obtaining data from multiple sources.
**Schema service goals**:
- To enable easier creation of new schemas
- To discover popular schemas in use today
- To make reuse of existing schemas easier to achieve
**Schema Definition**:
A schema is the model definition for a particular entity. At its core it defines the property names, their types and relationships to other models. It also includes certain qualities to expect about the actual entities and the data its hold.
An entity is a specific instance of a schema. A schema is assigned an entityType which categorizes the data the schema defines. For example 'well' is an example of an entityType.
An entityType is assigned an 'authority'. An authority is a namespace to help differentiate types defined by different groups e.g. it allows for both an slb:well and a chevron:well type to exist.
With schema definitions we aim to define the standards for data models used by a DDMS. This will reduce the friction with different systems data sharing needs helping with our vision to enable data discovery and sharing.
**Principles**:
- Low threshold for Schema Registration
: We should make it as easy as possible to create a new schema and register it, Whether this be through examples, tooling to help auto-generate a schema or validators to give early feedback when invalid markup is used. Also it must be easy to reuse existing schema definitions - DE-schemas and external schemas - using composition.
- Easy to consume
: JSON schema allows for JSON pointers to other schema parts and fragments, which need to be chased to obtain a full schema. A registered DE schema is self-contained, i.e. it contains only internal references "$ref": "#/definitions/../target". This is particularly important for schemas, which originally refer to external URLs. All internal "$ref" are resolvable. Consumers find all the answers inside the single schema document. The schema has a predictable structure.
- Robust
: All Schemas are read-only and versioned. The major version is controlled by the schema producer, the minor version is controlled by the schema service. When a schema is assigned version 1.0.0 or higher no breaking changes can be applied on new schema versions. Breaking changes will be allowed on any schema version with major version 0.X.X.
- Breaking changes are:
- removed properties
- changes to existing properties e.g.changed types, changed required properties
: This gives consumers confidence that they can use data without fear of breaks in the future. However schemas can be deprecated meaning that no new data will be assigned to it.
**Schema attributes**
Schema definitions are based on in the [JSON schema standard](https://json-schema.org/latest/json-schema-core.html) and [OpenAPI 3](https://swagger.io/docs/specification/data-models/). There are subtle differences in the keyword support, which are explained in conjunction with OAS3 [here](https://swagger.io/docs/specification/data-models/keywords/).
- [x] AWS
- [x] Azure
- [x] IBM
- [ ] GCPM1 - Release 0.1Hrvoje MarkovicFerris ArgyleDania Kodeih (Microsoft)Wladmir FrazaoChris ZhangMichael CleminsonHrvoje Markovic2020-10-30https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/155OSDU-DD-Delivery-M23.0 (v0.26.0)2024-03-23T10:12:02ZThomas Gehrmann [slb]OSDU-DD-Delivery-M23.0 (v0.26.0)- [ ] Update to the M23.0 deliverables from OSDU Data Definitions- [ ] Update to the M23.0 deliverables from OSDU Data DefinitionsM23 - Release 0.26Thomas Gehrmann [slb]Thomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/144Schema Bootstrap failure issue2024-01-08T13:43:28ZIsha KumariSchema Bootstrap failure issueThe system schema has experienced substantial growth, expanding from an initial range of 200-300 to approximately 900. Consequently, I've been facing failures in the schema during script execution. Notably, the script is designed to rei...The system schema has experienced substantial growth, expanding from an initial range of 200-300 to approximately 900. Consequently, I've been facing failures in the schema during script execution. Notably, the script is designed to reinitiate the schema upon encountering failures
Additionally, the Bootstrap process is showing failures randomly due to the schema variations. To address this, I have implemented error-catching mechanisms and introduced a retry strategy to mitigate the impact of these intermittent failures.M22 - Release 0.25Isha KumariIsha Kumarihttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/143OSDU-DD-Delivery-M22 (v0.25.0)2023-12-11T06:12:12ZThomas Gehrmann [slb]OSDU-DD-Delivery-M22 (v0.25.0)- [x] Update to the M22 deliverables from OSDU Data Definitions
CC @chad- [x] Update to the M22 deliverables from OSDU Data Definitions
CC @chadThomas Gehrmann [slb]Thomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/142Content-Type returned doesn't match OpenAPI json2024-03-14T16:18:32ZShane HutchinsContent-Type returned doesn't match OpenAPI jsonOne note the Content-Type returned in the following doesn't match OpenAPI json:
* GET /api/schema-service/v1/schema
* PUT /api/schema-service/v1/schema
* POST /api/schema-service/v1/schema
* GET /api/schema-service/v1/schema/{id}
Rece...One note the Content-Type returned in the following doesn't match OpenAPI json:
* GET /api/schema-service/v1/schema
* PUT /api/schema-service/v1/schema
* POST /api/schema-service/v1/schema
* GET /api/schema-service/v1/schema/{id}
Received a response with 'application/json' Content-Type, but it is not declared in the schema.
`Defined content types: */*`
`curl -X GET -H 'Authorization: [Filtered]' -H 'data-partition-id: opendes' 'https://osdu-glab.msft-osdu-test.org/api/schema-service/v1/schema?authority=osdu&source=wks&entityType=wellbore&schemaVersionMajor=1&schemaVersionMinor=1&status=PUBLISHED&scope=INTERNAL&latestVersion=True&limit=10'`
Response status: 200
Response payload: `{"schemaInfos":[],"offset":0,"count":0,"totalCount":0}`M22 - Release 0.25Chad LeongShane HutchinsChad Leonghttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/139Add /liveness_check2023-11-21T19:08:35ZRiabokon Stanislav(EPAM)[GCP]Add /liveness_checkNeed to add the endpoint '/liveness_check' in order to verify the operational status of the Schema Service.Need to add the endpoint '/liveness_check' in order to verify the operational status of the Schema Service.M22 - Release 0.25Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/138OSDU-DD-Delivery-M21 (v0.24.0)2023-11-24T08:44:39ZChad LeongOSDU-DD-Delivery-M21 (v0.24.0)- [x] Update to the M21 deliverables from OSDU Data Definitions- [x] Update to the M21 deliverables from OSDU Data DefinitionsM21 - Release 0.24https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/134OSDU-DD-Delivery-M20 (v0.23.0)2023-08-21T19:12:47ZThomas Gehrmann [slb]OSDU-DD-Delivery-M20 (v0.23.0)- [x] Update to the M20 deliverables from OSDU Data Definitions- [x] Update to the M20 deliverables from OSDU Data DefinitionsM20 - Release 0.23Thomas Gehrmann [slb]Thomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/131OSDU-DD-Delivery-M19 (v0.22.0)2023-08-17T08:30:47ZThomas Gehrmann [slb]OSDU-DD-Delivery-M19 (v0.22.0)- [x] Update to the M19 deliverables from OSDU Data Definitions- [x] Update to the M19 deliverables from OSDU Data DefinitionsM19 - Release 0.22Thomas Gehrmann [slb]Thomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/127OSDU-DD-Delivery-M18 (v0.21.0)2023-05-19T06:04:20ZThomas Gehrmann [slb]OSDU-DD-Delivery-M18 (v0.21.0)- [x] Update to the M18 deliverables from OSDU Data Definitions- [x] Update to the M18 deliverables from OSDU Data DefinitionsM18 - Release 0.21Thomas Gehrmann [slb]Thomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/125OSDU-DD-Delivery-M17 (v0.20.0)2023-03-29T14:04:40ZThomas Gehrmann [slb]OSDU-DD-Delivery-M17 (v0.20.0)- [x] Update to the M17 deliverables from OSDU Data Definitions- [x] Update to the M17 deliverables from OSDU Data DefinitionsM17 - Release 0.20Thomas Gehrmann [slb]Thomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/124Schema service upgrades may be blocked by schemas created in private tenants.2023-10-12T13:30:24ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comSchema service upgrades may be blocked by schemas created in private tenants.Currently, the creation of system schema may be blocked by the private internal schema in a private tenant.
If schema initially were created in the private tenant.
![Untitled_Diagram.drawio](/uploads/990837be38e4c62e3f0dbb257d914eb7/U...Currently, the creation of system schema may be blocked by the private internal schema in a private tenant.
If schema initially were created in the private tenant.
![Untitled_Diagram.drawio](/uploads/990837be38e4c62e3f0dbb257d914eb7/Untitled_Diagram.drawio.png)
**Cons of current flow:**
- Easy to break bootstrapping, it will fail with errors like: `Error with kind osdu:wks:master-data--ConnectedSourceDataJob:1.3.0: Message: Update/Create failed because schema id is present in another tenant`
- It's not possible to fix it through API, since Schema doesn't have a DELETE endpoint.
**Proposal:**
- There is none for now.
**Additional info:**M21 - Release 0.24Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/123Incorrect status is being returned upon creating the schema that already exis...2023-05-31T11:42:03ZKamlesh TodaiIncorrect status is being returned upon creating the schema that already exists in the systemWhen one tries to create the schema that is already existing in the system, one gets the return error code of **400 - Bad request**. As per the API documentation, it is correct. But I think that the error code is misleading. The message ...When one tries to create the schema that is already existing in the system, one gets the return error code of **400 - Bad request**. As per the API documentation, it is correct. But I think that the error code is misleading. The message returned is also misleading. It returns "message": "Update/Create failed because schema id is present in another tenant, this is not true because the schema is present in the same tenant.
This is what one would expect, The return error code should be **409 Conflict** indicating that schema is already present. and the message should be "schema is present".M19 - Release 0.22https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/122OSDU-DD-Delivery-M16 (v0.19.0)2023-02-15T12:59:00ZThomas Gehrmann [slb]OSDU-DD-Delivery-M16 (v0.19.0)- [x] Publish the OSDU Data Definitions schema registration resources for M16 v0.19.0- [x] Publish the OSDU Data Definitions schema registration resources for M16 v0.19.0M16 - Release 0.19Thomas Gehrmann [slb]Thomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/120System endpoint should also be accessible to user with admin roles2023-11-29T21:16:40ZAbhishek Kumar (SLB)System endpoint should also be accessible to user with admin rolesAs per initial design, there was a need for reserved partition to store **SHARED** schemas so that users can share schemas across partitions. For example, boostrapped schemas can be accessed by any user from any partition. This design do...As per initial design, there was a need for reserved partition to store **SHARED** schemas so that users can share schemas across partitions. For example, boostrapped schemas can be accessed by any user from any partition. This design does not restrict users to create and share their schemas directly rather going through the bootstrapping route.
Initially, creation of all the schemas were through _**common PUT/POST endpoint**_ and the decision of whether schema is **SHARED** or **INTERNAL** was decided based on the data-partition-id mentioned in the header. This design had its own challenges:
- Necessity of maintaining dedicated partition which all end users must be aware about
- Due to common endpoint for both **SHARED** and **INTERNAL** schemas, the decision making was driven through the data-partition-id in the header.
- Could not put any governance at the URL level for better access management.
![image](/uploads/44297a56933116a349c97b9f3e718c38/image.png)
Considering above challenges separate system endpoint was introduced to bifurcate **INTERNAL** and **SHARED** schema creation. Which means SHARED schemas can only be created using system endpoint without specifying any data-partition-id. This, however, was a breaking change and hence it was achieved in two phases:
1. **Creation of System partition:**
At this stage, only system partition was introduced without impacting the user experience. As a result, all existing **SHARED** partitions were moved to system partition and new requests were re-directed to system partitions i.e no new SHARED schemas were getting created into reserved partition (default)
![image](/uploads/42e8bb9df4441cbc7d319101f41582ae/image.png)
2. **Introduction of new endpoint** (https://{{domain}}/api/schema-service/v1/schemas/system):
Because of a dedicated endpoint, **SHARED** schemas are created without providing any data-partition-id in the header.
![image](/uploads/e76b6b085b9e4213d7178a5d8cddc8ec/image.png)
But there is a issue with this design, it is always expected to create SHARED schemas only through bootstrap script which uses service principal. As a result of this, users who were previously able to create and share their schemas are impacted. They now only need to go through bootstrapping process.
To fix this, we need to restrict the access to this endpoint through some entitlement service group role like service **schema-service.admin**.https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/118OSDU-DD-Delivery-M15 (v0.18.0)2023-01-26T07:15:21ZThomas Gehrmann [slb]OSDU-DD-Delivery-M15 (v0.18.0)- [x] Publish the OSDU Data Definitions schema registration resources for M15 v0.18.0- [x] Publish the OSDU Data Definitions schema registration resources for M15 v0.18.0M15 - Release 0.18Thomas Gehrmann [slb]Thomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/114Failing to register WellboreMarkerSet.1.2.1.json schema2022-10-18T05:47:03ZDanylo Vanin (EPAM)Failing to register WellboreMarkerSet.1.2.1.json schemaWhile trying to register shared `WellboreMarkerSet.1.2.1.json` [(link)](https://community.opengroup.org/osdu/platform/system/schema-service/-/blob/master/deployments/shared-schemas/osdu/work-product-component/WellboreMarkerSet.1.2.1.json...While trying to register shared `WellboreMarkerSet.1.2.1.json` [(link)](https://community.opengroup.org/osdu/platform/system/schema-service/-/blob/master/deployments/shared-schemas/osdu/work-product-component/WellboreMarkerSet.1.2.1.json) schema using [DeploySharedSchemas.py](https://community.opengroup.org/osdu/platform/system/schema-service/-/blob/master/deployments/scripts/DeploySharedSchemas.py) - registration fails with the following error:
```
Error with kind osdu:wks:work-product-component--WellboreMarkerSet:1.2.1: Message: Patch version validation failed. Changes requiring a minor or major version increment were found; analysed version: 1.2.1 and 1.2.0. Updating the schema version to a higher minor or major version is required.
```
For the same request application logs the following:
```
schema.app: Failed to resolve the schema and find breaking changes. Reason :Patch version validation failed. Changes requiring a minor or major version increment were found; analysed version: {0} and {1}. Updating the schema version to a higher minor or major version is required.
schema.app: Patch version validation failed. Changes requiring a minor or major version increment were found; analysed version: 1.2.0 and 1.2.1. Updating the schema version to a higher minor or major version is required.
```https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/113OSDU-DD-Delivery-M142022-12-13T00:18:55ZThomas Gehrmann [slb]OSDU-DD-Delivery-M14Publish the shared OSDU schemas for M14 deployment.Publish the shared OSDU schemas for M14 deployment.M14 - Release 0.17Thomas Gehrmann [slb]Thomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/108Schema service must not allow creation of schema with different case2022-08-23T15:53:06ZNeelesh ThakurSchema service must not allow creation of schema with different caseSchema service uses `kind` as an identifier for Schema for a data-set. `kind` with casing difference usually belongs to same data-set and don't have any other notion. This creates confusion for the end-user consuming records from Search ...Schema service uses `kind` as an identifier for Schema for a data-set. `kind` with casing difference usually belongs to same data-set and don't have any other notion. This creates confusion for the end-user consuming records from Search service with different case. Moreover consumption service like Search uses kind as index name. It's backend (Elasticsearch) do not honor casing for index names thus creating issues for index creation.
Data Definition does not provide any rules for kind casing and delegates this governance to Schema service. It should not allow creation of schema with different case for kind.
Related Search service issue: [94](https://community.opengroup.org/osdu/platform/system/search-service/-/issues/94)https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/105Remove unwanted logs from IT2022-12-09T13:39:36ZAbhishek Kumar (SLB)Remove unwanted logs from ITAs of today there are too many unwanted logs printed during IT execution which slows down the execution.As of today there are too many unwanted logs printed during IT execution which slows down the execution.Neha SardaNeha Sarda