Schema issueshttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues2021-05-22T06:37:55Zhttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/34Regex for kind2021-05-22T06:37:55ZThomas Gehrmann [slb]Regex for kindFor consistency, please enforce the same rules for `kind` or SchemaId as the Storage service does:
```regex
KIND_REGEX = "^[\\w\\-\\.]+:[\\w\\-\\.]+:[\\w\\-\\.\\/]+:[0-9]+.[0-9]+.[0-9]+$"
as used in regex101 ...For consistency, please enforce the same rules for `kind` or SchemaId as the Storage service does:
```regex
KIND_REGEX = "^[\\w\\-\\.]+:[\\w\\-\\.]+:[\\w\\-\\.\\/]+:[0-9]+.[0-9]+.[0-9]+$"
as used in regex101 ^[\w\-\.]+:[\w\-\.]+:[\w\-\.\/]+:[0-9]+.[0-9]+.[0-9]+$
```
The regex pattern should be part of the API spec.M1 - Release 0.1ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/32[Schema Service] R3 schema snapshot2021-02-02T21:09:41ZThomas Gehrmann [slb][Schema Service] R3 schema snapshot- [x] Copy a snapshot of the R3 Data Definition schemas into [`deployments/osdu-source/R3-json-schema`](https://community.opengroup.org/osdu/platform/system/schema-service/-/tree/master/deployments/osdu-source/R3-json-schema).
- [x] Prep...- [x] Copy a snapshot of the R3 Data Definition schemas into [`deployments/osdu-source/R3-json-schema`](https://community.opengroup.org/osdu/platform/system/schema-service/-/tree/master/deployments/osdu-source/R3-json-schema).
- [x] Prepare the schemas with `ImportFromOSDU`.M1 - Release 0.1Thomas Gehrmann [slb]Thomas Gehrmann [slb]2021-01-29https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/27[Schema-Service] Integration with core services2020-12-15T21:16:50ZRucha Deshpande[Schema-Service] Integration with core servicesWe would like gather information on
- how the Schema Service is positioned within the overall architecture
- the plan to integrate it with the rest of the core services and deprecate the Storage Service schema APIsWe would like gather information on
- how the Schema Service is positioned within the overall architecture
- the plan to integrate it with the rest of the core services and deprecate the Storage Service schema APIsM1 - Release 0.1ethiraj krishnamanaiduJoeRucha Deshpandeethiraj krishnamanaidu2020-12-18https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/18[Schema Service] Integration test issue (PUT api)2020-11-09T20:40:03ZRucha Deshpande[Schema Service] Integration test issue (PUT api)Scenario Outline: Verify that Schema Service's PUT API registers authority, source, entity and creates a private schema correctly with $ref attribute
Given I hit schema service PUT API with "/input_payloads/postSchemaServiceWithRef_posi...Scenario Outline: Verify that Schema Service's PUT API registers authority, source, entity and creates a private schema correctly with $ref attribute
Given I hit schema service PUT API with "/input_payloads/postSchemaServiceWithRef_positiveScenario.json", data-partition-id as "COMMON"
Then service should respond back with "200" and "/output_payloads/SchemaPost_PrivateScope_SuccessfulCreation.json" and scope whould be "SHARED"
Expected Result: 200
Actual Result: 400
```json
{
"error": {
"code": 400,
"message": "Schema Id is already present",
"errors": [
{
"domain": "global",
"reason": "badRequest",
"message": "Schema Id is already present"
}
]
}
}
```
Reason:
The test case's input payload passes a Major and Minor version that does not exist in (SHARED_TENANT)"common", but exists in PRIVATE_TENANT1(opendes).
Details:
Since it does not exist in "common", it throws a "NoSchemaFoundException" resulting in "create schema" code being called which checks
for uniqueness of the id. The isUnique returns false since the id exists for PRIVATE_TENANT1 resulting in a 400 error.
Solution:
In SchemaServiceStepDef_PUT.java
In,
Given("I hit schema service PUT API with {string}, data-partition-id as {string}",
an existing Major and Minor version belonging to "common" must be set to test the PUT functionalityM1 - Release 0.1ethiraj krishnamanaiduJoeRucha DeshpandeMatt Wiseethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/17[Schema Service] ISchemaInfoStore isUnique functionality2020-12-10T21:09:35ZRucha Deshpande[Schema Service] ISchemaInfoStore isUnique functionalityWe would like to get some clarity on uniqueness check.
Is this method supposed to check for uniqueness of the schemaId on only the passed 'tenant' parameter, or does it have to check for uniqueness for both - the passed 'tenant' and 'c...We would like to get some clarity on uniqueness check.
Is this method supposed to check for uniqueness of the schemaId on only the passed 'tenant' parameter, or does it have to check for uniqueness for both - the passed 'tenant' and 'common'.
This functionality will drive whether schemas with the same major and minor versions can/cannot be created in different tenants.M1 - Release 0.1ethiraj krishnamanaiduJoeRucha DeshpandeMatt Wiseethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/16[Schema Service] Integration test and pipeline issue2020-10-09T14:40:15ZRucha Deshpande[Schema Service] Integration test and pipeline issue- pom.xml in schema-test-core points to the wrong parent folder
- ci pipeline is not running the integration tests and so build failure is not observed- pom.xml in schema-test-core points to the wrong parent folder
- ci pipeline is not running the integration tests and so build failure is not observedM1 - Release 0.1JoeRucha DeshpandeAman VermaJoehttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/9[System/Core] Schema Service2020-11-09T20:06:02ZStephen Whitley (Invited Expert)[System/Core] Schema ServiceThe schema service is both an implementation of the schema standard defined below and a service to help create, manage and share schemas between different users. It aims to both raise awareness of the different models used by teams to he...The schema service is both an implementation of the schema standard defined below and a service to help create, manage and share schemas between different users. It aims to both raise awareness of the different models used by teams to help reuse both of the data sources but also of the schemas themselves. This will hopefully help reduce the friction obtaining data from multiple sources.
**Schema service goals**:
- To enable easier creation of new schemas
- To discover popular schemas in use today
- To make reuse of existing schemas easier to achieve
**Schema Definition**:
A schema is the model definition for a particular entity. At its core it defines the property names, their types and relationships to other models. It also includes certain qualities to expect about the actual entities and the data its hold.
An entity is a specific instance of a schema. A schema is assigned an entityType which categorizes the data the schema defines. For example 'well' is an example of an entityType.
An entityType is assigned an 'authority'. An authority is a namespace to help differentiate types defined by different groups e.g. it allows for both an slb:well and a chevron:well type to exist.
With schema definitions we aim to define the standards for data models used by a DDMS. This will reduce the friction with different systems data sharing needs helping with our vision to enable data discovery and sharing.
**Principles**:
- Low threshold for Schema Registration
: We should make it as easy as possible to create a new schema and register it, Whether this be through examples, tooling to help auto-generate a schema or validators to give early feedback when invalid markup is used. Also it must be easy to reuse existing schema definitions - DE-schemas and external schemas - using composition.
- Easy to consume
: JSON schema allows for JSON pointers to other schema parts and fragments, which need to be chased to obtain a full schema. A registered DE schema is self-contained, i.e. it contains only internal references "$ref": "#/definitions/../target". This is particularly important for schemas, which originally refer to external URLs. All internal "$ref" are resolvable. Consumers find all the answers inside the single schema document. The schema has a predictable structure.
- Robust
: All Schemas are read-only and versioned. The major version is controlled by the schema producer, the minor version is controlled by the schema service. When a schema is assigned version 1.0.0 or higher no breaking changes can be applied on new schema versions. Breaking changes will be allowed on any schema version with major version 0.X.X.
- Breaking changes are:
- removed properties
- changes to existing properties e.g.changed types, changed required properties
: This gives consumers confidence that they can use data without fear of breaks in the future. However schemas can be deprecated meaning that no new data will be assigned to it.
**Schema attributes**
Schema definitions are based on in the [JSON schema standard](https://json-schema.org/latest/json-schema-core.html) and [OpenAPI 3](https://swagger.io/docs/specification/data-models/). There are subtle differences in the keyword support, which are explained in conjunction with OAS3 [here](https://swagger.io/docs/specification/data-models/keywords/).
- [x] AWS
- [x] Azure
- [x] IBM
- [ ] GCPM1 - Release 0.1Hrvoje MarkovicFerris ArgyleDania Kodeih (Microsoft)Wladmir FrazaoChris ZhangMichael CleminsonHrvoje Markovic2020-10-30https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/65M8 Data Definitions content2021-09-22T20:31:55ZThomas Gehrmann [slb]M8 Data Definitions content- [ ] Publish the new types and (decorative) changes from the data Definitions member GitLab to the bootstrap resources.- [ ] Publish the new types and (decorative) changes from the data Definitions member GitLab to the bootstrap resources.M9 - Release 0.12Thomas Gehrmann [slb]Thomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/61Support for version upgrade during schema-validation2022-02-25T14:13:08ZAbhishek Kumar (SLB)Support for version upgrade during schema-validationThere is intensive use of `$ref` to schema fragments, which are incoming as schema-IDs like e.g. `"$ref": "osdu:wks:AbstractSpatialLocation:1.0.0"`. These fragments use semantic versioning as well.
As a consequence, these ids should be v...There is intensive use of `$ref` to schema fragments, which are incoming as schema-IDs like e.g. `"$ref": "osdu:wks:AbstractSpatialLocation:1.0.0"`. These fragments use semantic versioning as well.
As a consequence, these ids should be validated also during schema-validation:
- $ref target versions for patch increments can only refer to higher patch versions - if they refer to a higher minor (or even major version) validation must fail.
<br>**Example:**
| Sl No| Type | Initial Version | New Version | Status |
| ------ | ------ | ------ | ------ |------ |
| 1| Base Schema | osdu:wks:AbstractSpatialLocation:1.0.0 | osdu:wks:AbstractSpatialLocation:1.0.1 | |
| 2| $ref | osdu:wks:AbstractCommonResources:2.1.1 | osdu:wks:AbstractCommonResources:2.1.2 | valid |
| 3| $ref | osdu:wks:AbstractCommonResources:2.1.1 | osdu:wks:AbstractCommonResources:2.2.1 | invalid |
| 4| $ref | osdu:wks:AbstractCommonResources:2.1.1 | osdu:wks:AbstractCommonResources:2.0.1 | invalid |
| 5| $ref | osdu:wks:AbstractCommonResources:2.1.1 | osdu:wks:AbstractCommonResources:2.1.0 | invalid |
| 6| $ref | osdu:wks:AbstractCommonResources:2.1.1 | osdu:wks:AbstractCommonResources:3.1.1 | invalid |
| 7| $ref | osdu:wks:AbstractCommonResources:2.1.1 | osdu:wks:AbstractCommonResources-Changed:2.1.1 | invalid |
- Similarly $ref target versions for higher minor versions can only refer to higher minor versions - higher major version references must be rejected by the validation.
<br>**Example:**
| Sl No| Type | Initial Version | New Version | Status |
| ------ | ------ | ------ | ------ |------ |
| 1| Base Schema | osdu:wks:AbstractSpatialLocation:1.0.0 | osdu:wks:AbstractSpatialLocation:1.1.0 | |
| 2| $ref | osdu:wks:AbstractCommonResources:2.1.1 | osdu:wks:AbstractCommonResources:2.1.2 | valid |
| 3| $ref | osdu:wks:AbstractCommonResources:2.1.1 | osdu:wks:AbstractCommonResources:2.2.1 | valid |
| 4| $ref | osdu:wks:AbstractCommonResources:2.1.1 | osdu:wks:AbstractCommonResources:2.0.1 | invalid |
| 5| $ref | osdu:wks:AbstractCommonResources:2.1.1 | osdu:wks:AbstractCommonResources:2.1.0 | invalid |
| 6| $ref | osdu:wks:AbstractCommonResources:2.1.1 | osdu:wks:AbstractCommonResources:3.1.1 | invalid |
| 7| $ref | osdu:wks:AbstractCommonResources:2.1.1 | osdu:wks:AbstractCommonResources-Changed:2.1.1 | invalid |M9 - Release 0.12Abhishek Kumar (SLB)Abhishek Kumar (SLB)https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/80Refresh DDSC schemas2022-01-05T11:34:34ZThomas Gehrmann [slb]Refresh DDSC schemasCreate a MR to update the shared OSDU standard schemas based on the DDSC master state November 23rd.
- [x] VirtualProperty definitions in schemas
- [x] Updated `x-osdu-indexing` tag values to improvde searchCreate a MR to update the shared OSDU standard schemas based on the DDSC master state November 23rd.
- [x] VirtualProperty definitions in schemas
- [x] Updated `x-osdu-indexing` tag values to improvde searchM10 - Release 0.13Thomas Gehrmann [slb]Thomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/95x-osdu-indexing changes are breaking2022-10-13T11:08:05ZThomas Gehrmann [slb]x-osdu-indexing changes are breaking# Context:
Indexing hints in the OSDU schemas are considered decorations and not taken into account when schemas versions are
validated for 'breaking changes'.
Downstream indexing changes from any state to any other state are considere...# Context:
Indexing hints in the OSDU schemas are considered decorations and not taken into account when schemas versions are
validated for 'breaking changes'.
Downstream indexing changes from any state to any other state are considered breaking changes:
* Breaking changes for the indexer: changes from `flattened` to `nested` require the re-indexing of the kind in
question.
* Consuming applications must use a different query syntax.
# How it's done today:
The process depends on human interaction (assuming OSDU well-known schemas here, but this is no different for custom
schemas):
* Stakeholders ask for an indexing behavior change, OSDU Data Definition reacts by changing the `x-osdu-indexing`
extension tag values in the schema.
* OSDU Data Definition Release notes identify the kinds, which are to be re-indexed.
* In M10 virtually all kinds had to be re-indexed
* In M11 type `reference-data--QualityDataRuleSet` requires re-indexing
* During deployment the records for the affected kinds must be re-indexed.
# Issue with current design:
Upon deployment of a new milestone (or custom schemas),
1. for all involved data-partitions, delete the index for the changed kind and trigger re-indexing. This can take -
depending on the number of records per kind - a very log time and cause serious down-time.
2. Applications have no good way of understanding that the query syntax has changed. Applications may no longer find
data if they depended on queries into data structures affected by the change.
# Proposal:
## `PUBLISHED` Schema Status
1. For schemas with state `PUBLISHED` treat changes to the `x-osdu-indexing` extension tag values in the schema as **_
breaking changes_**.
2. Breaking changes require an incremented major schema version number.
3. Schema Validation Changes during schema creation:
* Changes to the `x-osdu-indexing` extension tag values in `PUBLISHED` schemas with same major schema version
numbers will be **_rejected_**. I.e., the attempted registration of such schema will fail with error.
## `DEVELOPMENT` Schema Status
1. The validation for `DEVELOPMENT` status schemas for incremental versions on top of or between existing minor or patch
versions follows the same rules as for `PUBLISHED` schemas. Attempts to change the `x-osdu-indexing` extension tag
values will be **_rejected_** by the Schema service.
2. For 'single' version schemas in `DEVELOPMENT`, the updates of the `x-osdu-indexing` extension tag values are
permitted.
* It is the responsibility of the schema authors to communicate the impact to deployment and consumers. This is
expected to be acceptable during the development phase.
CC @nthakur @ChrisZhang @chad @pbehedeM12 - Release 0.15https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/113OSDU-DD-Delivery-M142022-12-13T00:18:55ZThomas Gehrmann [slb]OSDU-DD-Delivery-M14Publish the shared OSDU schemas for M14 deployment.Publish the shared OSDU schemas for M14 deployment.M14 - Release 0.17Thomas Gehrmann [slb]Thomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/118OSDU-DD-Delivery-M15 (v0.18.0)2023-01-26T07:15:21ZThomas Gehrmann [slb]OSDU-DD-Delivery-M15 (v0.18.0)- [x] Publish the OSDU Data Definitions schema registration resources for M15 v0.18.0- [x] Publish the OSDU Data Definitions schema registration resources for M15 v0.18.0M15 - Release 0.18Thomas Gehrmann [slb]Thomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/122OSDU-DD-Delivery-M16 (v0.19.0)2023-02-15T12:59:00ZThomas Gehrmann [slb]OSDU-DD-Delivery-M16 (v0.19.0)- [x] Publish the OSDU Data Definitions schema registration resources for M16 v0.19.0- [x] Publish the OSDU Data Definitions schema registration resources for M16 v0.19.0M16 - Release 0.19Thomas Gehrmann [slb]Thomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/125OSDU-DD-Delivery-M17 (v0.20.0)2023-03-29T14:04:40ZThomas Gehrmann [slb]OSDU-DD-Delivery-M17 (v0.20.0)- [x] Update to the M17 deliverables from OSDU Data Definitions- [x] Update to the M17 deliverables from OSDU Data DefinitionsM17 - Release 0.20Thomas Gehrmann [slb]Thomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/127OSDU-DD-Delivery-M18 (v0.21.0)2023-05-19T06:04:20ZThomas Gehrmann [slb]OSDU-DD-Delivery-M18 (v0.21.0)- [x] Update to the M18 deliverables from OSDU Data Definitions- [x] Update to the M18 deliverables from OSDU Data DefinitionsM18 - Release 0.21Thomas Gehrmann [slb]Thomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/131OSDU-DD-Delivery-M19 (v0.22.0)2023-08-17T08:30:47ZThomas Gehrmann [slb]OSDU-DD-Delivery-M19 (v0.22.0)- [x] Update to the M19 deliverables from OSDU Data Definitions- [x] Update to the M19 deliverables from OSDU Data DefinitionsM19 - Release 0.22Thomas Gehrmann [slb]Thomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/123Incorrect status is being returned upon creating the schema that already exis...2023-05-31T11:42:03ZKamlesh TodaiIncorrect status is being returned upon creating the schema that already exists in the systemWhen one tries to create the schema that is already existing in the system, one gets the return error code of **400 - Bad request**. As per the API documentation, it is correct. But I think that the error code is misleading. The message ...When one tries to create the schema that is already existing in the system, one gets the return error code of **400 - Bad request**. As per the API documentation, it is correct. But I think that the error code is misleading. The message returned is also misleading. It returns "message": "Update/Create failed because schema id is present in another tenant, this is not true because the schema is present in the same tenant.
This is what one would expect, The return error code should be **409 Conflict** indicating that schema is already present. and the message should be "schema is present".M19 - Release 0.22https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/134OSDU-DD-Delivery-M20 (v0.23.0)2023-08-21T19:12:47ZThomas Gehrmann [slb]OSDU-DD-Delivery-M20 (v0.23.0)- [x] Update to the M20 deliverables from OSDU Data Definitions- [x] Update to the M20 deliverables from OSDU Data DefinitionsM20 - Release 0.23Thomas Gehrmann [slb]Thomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/138OSDU-DD-Delivery-M21 (v0.24.0)2023-11-24T08:44:39ZChad LeongOSDU-DD-Delivery-M21 (v0.24.0)- [x] Update to the M21 deliverables from OSDU Data Definitions- [x] Update to the M21 deliverables from OSDU Data DefinitionsM21 - Release 0.24