Schema issueshttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues2022-09-29T13:41:56Zhttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/39Add APIs to perform syntax and content validation2022-09-29T13:41:56ZAlan HensonAdd APIs to perform syntax and content validationIngestion workflows perform two types of validation as it relates to schema definitions:
- Is the content syntactically correct according to the specified `kind`'s schema definition
- Is the intent of content correct (e.g., does cited da...Ingestion workflows perform two types of validation as it relates to schema definitions:
- Is the content syntactically correct according to the specified `kind`'s schema definition
- Is the intent of content correct (e.g., does cited data exist)
The implementation of these validations currently exists within Python code as libraries or Airflow DAG operators. This logic is better suited as a set of API endpoints that can perform this type of validation enabling external toolsets easy access to this functionality. The types of validation that occur can be found within this [guide](https://community.opengroup.org/osdu/platform/testing/-/blob/master/R3%20Pre%20Ship/Manifest%20Ingestion/R3%20Manifest%20Ingestion%20Quick%20Test%20Guide.docx) under the `How the Manifest Ingestion workflow functions` section.
The validation logic for content adhering to schema definitions should exist as synchronous API endpoints.https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/96Add /validate endpoint to help non-core-platform developers2022-09-29T13:41:07ZEric SchoenAdd /validate endpoint to help non-core-platform developersOSDU application developers working outside of the OSDU core platform do not necessarily have access to OSDU's schema toolchain to ensure that their schemas are correct, consistent with OSDU abstract types, and usable by the indexer.
...OSDU application developers working outside of the OSDU core platform do not necessarily have access to OSDU's schema toolchain to ensure that their schemas are correct, consistent with OSDU abstract types, and usable by the indexer.
- When working with DEVELOPMENT status schema, there is no validation enforced by the schema service in the POST or PUT /schema endpoint. This allows an invalid schema to be installed into OSDU.
- Subsequently, the storage service PUT /records endpoint will accept data for that (possibly invalid) schema kind without complaint.
- The application is not aware that the data can't be indexed by the indexer due to invalid schema, and an attempt to reindex data of the invalid schema kind will produce a 500 error from the indexer with no explanation.
If the schema service provided a /validate endpoint that is effectively a dry-run for the POST/PUT /schema endpoint, a lot of uncertainty could be reduced, and lost development time from simple errors could be saved.https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/51ADR: New API to handle system schemas2023-02-22T14:24:03ZAman VermaADR: New API to handle system schemas## Status
* [ ] Proposed
* [ ] Trialing
* [ ] Under review
* [x] Approved
* [ ] Retired
Context:
===
Public schemas are schemas which are available to all the tenants out of the box. Today the public schema includes schema from OSDU or...## Status
* [ ] Proposed
* [ ] Trialing
* [ ] Under review
* [x] Approved
* [ ] Retired
Context:
===
Public schemas are schemas which are available to all the tenants out of the box. Today the public schema includes schema from OSDU or SLB and are located in Schema service repository here: https://community.opengroup.org/osdu/platform/system/schema-service/-/tree/master/deployments/shared-schemas
How it's done today:
===
- Deployment expects existence of a special partition (generally termed as `common`) which will be used to load public schemas
- Schema service has only one endpoint today `/schema`. Based on the `data-partition-id` passed in the request headers, it's decided whether the schema is public or private.
- If the data-partition-id passed is "special partition", then the created schema is public, otherwise it's private. This is indicated by a field named `SCOPE` in crated schema.
Issue with current design:
===
1. The API behavior of `/schema` endpoint changes based on the data-partition-id header and can be confusing to users. Users might end up unknowingly creating public schemas by passing data-partition-id of special partition.
2. The special partition is expected to be provisioned for provisioning public schemas.
Proposal:
===
There are two types of schemas in the system, public (or system) schemas and private schema. The proposal is to create a dedicated API to created/ update system schemas. Hence-
- The new API shall be termed as **`/schemas/system`**
- **To create/ updated public schema**- `/schemas/system` endpoint shall be used
- **To read public schema**- existing `GET /schema` endpoint shall be used (same as current behaviour)
- The new API shall not accept `data-partition-id` as a header. Service would be aware where the public schemas are located.
- The authorization of new end point shall be different from exiting schema.editor/viewer role.
**Sequence diagram for creating public schemas**
![image](/uploads/d90a62b4d9a1aae2c6c7a18db4cb66e4/image.png)
**Sequence diagram for reading public schema**
![image](/uploads/53d3e00c194f5ed3e0889682f8184a8f/image.png)M9 - Release 0.12Aman VermaAman Vermahttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/79ADR: Registering Schema Extensions2022-08-31T15:33:29ZParesh BehedeADR: Registering Schema Extensions**Context & Scope**
Here we are proposing a new set of APIs on Schema service on OSDU that allow for the creation, update retrieval and deletion of extensions on existing Schemas.
This ADR carries on from the following ADR [here](https...**Context & Scope**
Here we are proposing a new set of APIs on Schema service on OSDU that allow for the creation, update retrieval and deletion of extensions on existing Schemas.
This ADR carries on from the following ADR [here](https://community.opengroup.org/osdu/platform/system/search-service/-/issues/69) to allow the x-osdu-virtual-property defined in schemas to also be applied as an extension to schemas by services and applications running on top of the OSDU.
**Trade-off Analysis**
An important assumption we make is that a provider of an extension is the only concrete consumer of that extension. We shall call the provider the schema extension authority. This authority defines the scope it can be governed at.
The alternative is to assign versioning onto extensions. However this then faces potential problems of schema bloat if many versions of an extension could exist for a schema and also increases the friction for adoption by client applications where the extensions are trying to give them the freedom and flexibility to use OSDU schemas in their own context.
Allowing virtual properties to be assigned as extensions services like indexer/search are loosely coupled with these extensions as they need to use them in a generic way e.g. to index an extension property but no consumer will have a hard coupling to the specific properties provided unless they choose to. This is important because it allows the extension authority to change their extensions when they need to and not worry about breaking changes outside their own scope.
This also gives the extension authority flexibility and not enforce the same versioning semantics on an extension that is in the schema.
**Decision**
A new set of APIs will be be developed in the schema service that allows clients to register their own extensions on top of existing schemas.
For now this will only allow users to add in their own _"x-osdu-virtual-properties"_ extensions to existing schemas. However theoretically it could be extended for any extension use case.
The extension APIs can append new virtual properties only, it cannot override or change existing virtual properties defined in the schema.
When clients retrieve schemas from the existing schema service the schema service appends the extensions into the schema transparently to the client.
For example imagine the schema osdu:wks:master-data--Well:1.0.0 contains the following virtual property definition.
```json
{
"x-osdu-virtual-properties":{
"data.VirtualProperties.DefaultLocation": {
"type": "object",
"priority": [
{ "path": "data.ProjectedBottomHoleLocation" },
{ "path": "data.GeographicBottomHoleLocation" },
{ "path": "data.SpatialLocation" }
]}
}
}
```
Then in the schema extensions we provide an extension like below on the new extensions _POST_ API
```json
"kind": "osdu:wks:master-data--Well:1.0.0",
"authority": "MyApplication"
"x-osdu-virtual-properties":{
"data.VirtualProperties.MyDefaultName": {
"type": "string",
"priority": [
{ "path": "data.FacilityName", "isType": "string" }
]}
}
```
A client requesting the schema _"osdu:wks:master-data--Well:1.0.0"_ would then get a result that contained the following.
```json
"schema" {
...
...
{
"x-osdu-virtual-properties":{
"data.VirtualProperties.DefaultLocation": {
"type": "object",
"priority": [
{ "path": "data.ProjectedBottomHoleLocation" },
{ "path": "data.GeographicBottomHoleLocation" },
{ "path": "data.SpatialLocation" }
]}
}
...
...
"x-osdu-extensions": {
"MyApplication": {
"x-osdu-virtual-properties": {
"data.VirtualProperties.MyDefaultName": {
"type": "string",
"priority": [{
"path": "data.FacilityName",
"isType": "string"
}]
}
}
}
}
}
```
Where an extensions object is added to the schema and grouped within the `authority`
However if someone tried to register the following to extensions
```json
"kind": osdu:wks:master-data--Well:1.0.0
"x-osdu-virtual-properties":{
"data.VirtualProperties.DefaultLocation": {
"type": "object",
"priority": [
{ "path": "data.ProjectedBottomHoleLocation" }
]}
}
```
It would fail as 'DefaultLocation' is already declared as a virtual property in the schema.
The storage of extensions supplied is always system wide. This means any extensions registered apply to all partitions. However they are scoped by the schema they are applied to.
**API Spec**
Below is the API spec for the schema extensions
```yaml
paths:
"/schema/extensions":
post:
tags:
- Schema
summary: Adds a schema extension to the schema repository.
description: Adds a schema extension to the schema repository. The extension is identified by a combination of the 'kind' and 'authority' properties assigned on the request and must be unique. Scope of an extension is always SHARED. Required roles 'users.datalake.editors' or 'users.datalake.admins' groups to create schema.
operationId: Create Schema extension
parameters:
- $ref: "#/components/parameters/data-partition-id"
requestBody:
content:
application/json:
schema:
$ref: "#/components/schemas/SchemaExtensionRequest"
required: true
responses:
"201":
description: "Schema extension created"
headers:
location:
description: "Path of newly created schema extension."
schema:
type: "string"
content:
application/json:
schema:
$ref: "#/components/schemas/SchemaExtensionResponse"
"400":
description: "Bad request"
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponseFormat'
"401":
description: "Unauthorized"
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponseFormat'
"403":
description: "Forbidden"
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponseFormat'
"409":
description: "Extension with the same 'kind' and 'authority' already created"
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponseFormat'
security:
- bearer: []
- appkey: []
deprecated: false
put:
tags:
- Schema
summary: Adds a schema extension to the schema repository.
description: Updates a schema extension. The extensions is identified by a combination of the 'kind' and 'authority' properties assigned on the request and must be unique. Scope of an extension is always SHARED. Required roles 'users.datalake.editors' or 'users.datalake.admins' groups to create schema.
operationId: Update Schema extension
parameters:
- $ref: "#/components/parameters/data-partition-id"
requestBody:
content:
application/json:
schema:
$ref: "#/components/schemas/SchemaExtensionRequest"
required: true
responses:
"200":
description: "Schema extension updated"
headers:
location:
description: "Path of updated created schema extension."
schema:
type: "string"
content:
application/json:
schema:
$ref: "#/components/schemas/SchemaExtensionResponse"
"400":
description: "Bad request"
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponseFormat'
"401":
description: "Unauthorized"
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponseFormat'
"403":
description: "Forbidden"
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponseFormat'
security:
- bearer: []
- appkey: []
deprecated: false
get:
tags:
- Schema
summary: "Gets schema extensions from the schema repository."
description: "Gets schema extensions. Required roles 'users.datalake.viewers' or 'users.datalake.editors' or 'users.datalake.admins' groups."
operationId: Gets Schema extensions
parameters:
- in: query
name: kind
schema:
type: string
required: false
description: "The kind to retrieve extensions for"
- in: query
name: authority
schema:
type: string
required: false
description: "The authority to retrieve extensions for"
- in: query
name: id
schema:
type: string
required: false
description: "The ID to retrieve the extension for"
responses:
"200":
description: "OK"
content:
application/json:
schema:
type: array
items:
$ref: "#/components/schemas/SchemaExtensionResponse"
"400":
description: "Bad request"
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponseFormat'
"401":
description: "Unauthorized"
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponseFormat'
"403":
description: "Forbidden"
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponseFormat'
security:
- bearer: []
- appkey: []
deprecated: false
delete:
tags:
- Schema
summary: "Deletes a schema extensions"
description: "Deletes a schema extensions. Required roles 'users.datalake.admins' groups."
operationId: Deletes a schema extension
parameters:
- in: query
name: id
schema:
type: string
required: true
description: "The ID to delete the extension for"
responses:
"204":
description: "No content"
"400":
description: "Bad request"
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponseFormat'
"401":
description: "Unauthorized"
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponseFormat'
"403":
description: "Forbidden"
content:
application/json:
schema:
$ref: '#/components/schemas/ErrorResponseFormat'
security:
- bearer: []
- appkey: []
deprecated: false
schemas:
SchemaExtensionRequest:
type: object
title: SchemaExtensionRequest
properties:
kind:
type: string
example: "osdu:wks:master-data--Well:1.0.0"
description: "The schema kind(s) the extension applies to. Can use explicit 'authority','source','type','version' values for one or more parts."
authority:
type: string
example: "OSDU"
description: "The authority who supplied the extension. Can be used to identify who should use the extension. A authority can supply one or more extensions but only one for for each unique kind."
x-osdu-extensions:
type: object
example: {}
description: "The extensions to apply to the given kind"
example:
kind: "osdu:wks:master-data--Well:1.0.0"
authority: "GIS"
x-osdu-extensions:
x-osdu-virtual-properties:
data.VirtualProperties.DefaultPosition:
type: "object"
priority: [{ "path": "data.ProjectedBottomHoleLocation" },{"path": "data.GeographicBottomHoleLocation", "type": "GeoJson" },{"path": "data.SpatialLocation"} ]
SchemaExtensionResponse:
type: object
title: SchemaExtensionRequest
properties:
id:
type: string
description: "The unique identifier for the extension"
example:
id: "e4erg55677hfhrrbe5erveer4=="
kind: "osdu:wks:master-data--Well:1.0.0"
authority: "GIS"
x-osdu-extensions:
x-osdu-virtual-properties:
data.VirtualProperties.DefaultPosition:
type: "object"
priority: [{ "path": "data.ProjectedBottomHoleLocation" },{"path": "data.GeographicBottomHoleLocation", "type": "GeoJson" },{"path": "data.SpatialLocation"} ]
```
**Example Use case: Adding and consuming an extensions data property**
Up to this point we have looked at how we can map multiple properties to a virtual property to enable the use case of discovery across different kinds.
However another use case for schema extensions is for different consumers to add new properties into schemas and have them also become discoverable.
To enable this we can make use of the new ```x-osdu-virtual-properties```. For example when an application ingests data they include the properties from the OSDU schema that it relates. However they could choose to add their own properties as well that aren't mentioned in the schema.
This data is then kept in the storage records but is not indexed or known by other consumption services as it is not represented in the schema.
So if I ingest a record from my application like below
```
{
"id": "p1:wks:master-data--Well:1234",
"kind": "osdu:wks:master-data--Well:1.0.0",
...
...
"data": {
"Name": "1234-abc,
"ExtensionProperties": {
"petrel-project": "sim1.pet"
}
...
'''
}
}
```
The property ```Name``` is indexed as it is part of the referenced schema but the property ```petrel-project``` is not part of the schema so is not indexed.
However I could use the new ```x-osdu-virtual-properties``` to assign it as a virtual property so it is indexed.
```
"kind": "osdu:wks:master-data--Well:1.0.0",
"x-osdu-extensions" : {
"authority": "SLB",
"x-osdu-virtual-properties":{
"data.ExtensionProperties.petrel-project": {
"type": "object",
"priority": [
{
"path": "data.ExtensionProperties.petrel-project",
"isType":"string"
}
]}
}
}
```
As I am using the same virtual property key ```data.ExtensionProperties.petrel-project``` as the path to the property in the storage record this means it is indexed on the same path.
e.g. to search for this property
```
{
"kind": "osdu:wks:master-data--Well:1.0.0",
"query": "data.ExtensionProperties.petrel-project:'sim1.pet'"
}
```
**Consequences**
- Schema service needs to be extended to support creating, updating and deleting extensions
- Schema service needs to be updated to add an extensions object to schemas returned to clients
- Schema service needs to send notifications when extensions change
- Indexer needs to support schemas with extensions containing `"x-osdu-virtual-properties"`Paresh BehedeParesh Behedehttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/67API description for schemas/system API2022-11-24T11:39:48ZAman VermaAPI description for schemas/system API[[_TOC_]]
# What are system schemas
System schemas are schemas which are available to all the data partitions within an OSDU deployment out of the box.
# How do you identify a system schema
The scope field is set to `SHARED` for syste...[[_TOC_]]
# What are system schemas
System schemas are schemas which are available to all the data partitions within an OSDU deployment out of the box.
# How do you identify a system schema
The scope field is set to `SHARED` for system schemas in the schemaInfo section. Sample schemaInfo for system schemas:
```
{
"schemaInfo": {
"schemaIdentity": {
"authority": "osdu",
"source": "wks",
"entityType": "work-product-component--WellLog",
"schemaVersionMajor": 1,
"schemaVersionMinor": 0,
"schemaVersionPatch": 0,
"id": "osdu:wks:work-product-component--WellLog:1.0.0"
},
"createdBy": "OSDU Data Definition Group",
"scope": "SHARED",
"status": "DEVELOPMENT"
},
"schema": {
}
}
```
# How to create system schema
You can create system schemas by (to be introduced) `schemas/system` endpoint in your schema service. Find more details in the ADR: https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/51
The `schemas/system` endpoint differs from the regular `/schema` endpoint in two ways:
1. It does not accept the `data-partition-id` header. Since the schemas being created belong to no data-partition in particular, the API does not allow users to pass data-partition-id header. If you do pass the header, the request would fail with error indicating `data-partition-id` header should not be supplied.
2. The API works on service principal auth only. The authorization mechanism is similar to what we have in partition service related APIs. Refer example https://community.opengroup.org/osdu/platform/system/partition/-/blob/master/partition-core/src/main/java/org/opengroup/osdu/partition/api/PartitionApi.java#L49Aman VermaAman Vermahttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/129AWS M18 GET /api/schema-service/v1/schema/{id}2023-06-15T13:06:31ZShane HutchinsAWS M18 GET /api/schema-service/v1/schema/{id}Received a response with 5xx status code: 500
{"error":{"code":500,"message":"Internal server error","errors":[{"domain":"global","reason":"internalError","message":"Internal server error"}]}}
I was really expecting a 4xx error, like a ...Received a response with 5xx status code: 500
{"error":{"code":500,"message":"Internal server error","errors":[{"domain":"global","reason":"internalError","message":"Internal server error"}]}}
I was really expecting a 4xx error, like a 404.
Run this curl command to reproduce this failure:
curl -X GET -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: osdu' https://osdu.r3m18.preshiptesting.osdu.aws/api/schema-service/v1/schema/%1F
I could produce this in AWS. I could not reproduce this issue in Azure.https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/142Content-Type returned doesn't match OpenAPI json2024-03-14T16:18:32ZShane HutchinsContent-Type returned doesn't match OpenAPI jsonOne note the Content-Type returned in the following doesn't match OpenAPI json:
* GET /api/schema-service/v1/schema
* PUT /api/schema-service/v1/schema
* POST /api/schema-service/v1/schema
* GET /api/schema-service/v1/schema/{id}
Rece...One note the Content-Type returned in the following doesn't match OpenAPI json:
* GET /api/schema-service/v1/schema
* PUT /api/schema-service/v1/schema
* POST /api/schema-service/v1/schema
* GET /api/schema-service/v1/schema/{id}
Received a response with 'application/json' Content-Type, but it is not declared in the schema.
`Defined content types: */*`
`curl -X GET -H 'Authorization: [Filtered]' -H 'data-partition-id: opendes' 'https://osdu-glab.msft-osdu-test.org/api/schema-service/v1/schema?authority=osdu&source=wks&entityType=wellbore&schemaVersionMajor=1&schemaVersionMinor=1&status=PUBLISHED&scope=INTERNAL&latestVersion=True&limit=10'`
Response status: 200
Response payload: `{"schemaInfos":[],"offset":0,"count":0,"totalCount":0}`M22 - Release 0.25Chad LeongShane HutchinsChad Leonghttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/128DRAFT ADR: New API to delete private schemas2023-12-13T16:20:03ZAndrei Dalhikh [EPAM/GC]DRAFT ADR: New API to delete private schemas
## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
End users and QA engineers often need to delete individual schemas, which contain errors or were uploaded by mistake. Current ...
## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
End users and QA engineers often need to delete individual schemas, which contain errors or were uploaded by mistake. Current Schema service API does not have such method and these schemas are deleted manually.
# Tradeoff Analysis
The important thing to consider before deleting any schema from a private context is that data may already exist for that schema. Therefore, delete should only be successful when schema with same id/body is recreated into system partition as SHARED.
**TODO:** edge cases to be elaborated and documented here
## Decision
1. Introduce a delete endpoint to have an ability to delete individual private schema. This endpoint should be governed by some admin role to cleanup schemas from private partitions only.
1. Schemas in SHARED partition should still not have delete support.
1. Create new API method as below:
![Schema_delete_API](/uploads/c787d54444f164938246c2be132e8809/Schema_delete_API.jpg)
OpenAPI spec for this API change [schema_openapi.yaml](https://community.opengroup.org/osdu/platform/system/schema-service/-/blob/Delete_schema_api_draft/docs/api/schema_openapi.yaml)
## Consequences
1. This will help users to keep clean set of schemas in their environments
2. This will reduce the number of support requests for DevOpsM22 - Release 0.25https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/107Have an IT for nested $ref version upgrade2022-08-24T11:29:09ZAbhishek Kumar (SLB)Have an IT for nested $ref version upgradeThere is no IT present to validate nested $ref version upgrade.There is no IT present to validate nested $ref version upgrade.Neha SardaNeha Sardahttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/119If abstract reference schema is DEVELOPMENT than kind schema cannot be set to...2022-11-22T11:04:42ZNeelesh ThakurIf abstract reference schema is DEVELOPMENT than kind schema cannot be set to PUBLISHEDIf a Kind schema is using abstract reference schema (using $ref) and these abstract reference schema status is `DEVELOPMENT` than Schema service shouldn't allow changing the status of Kind schema to 'PUBLISHED'.
This can lead to breakin...If a Kind schema is using abstract reference schema (using $ref) and these abstract reference schema status is `DEVELOPMENT` than Schema service shouldn't allow changing the status of Kind schema to 'PUBLISHED'.
This can lead to breaking change in Kind schema as abstract reference schema can still be changed and breaking changes are allowed on `DEVELOPMENT`.https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/123Incorrect status is being returned upon creating the schema that already exis...2023-05-31T11:42:03ZKamlesh TodaiIncorrect status is being returned upon creating the schema that already exists in the systemWhen one tries to create the schema that is already existing in the system, one gets the return error code of **400 - Bad request**. As per the API documentation, it is correct. But I think that the error code is misleading. The message ...When one tries to create the schema that is already existing in the system, one gets the return error code of **400 - Bad request**. As per the API documentation, it is correct. But I think that the error code is misleading. The message returned is also misleading. It returns "message": "Update/Create failed because schema id is present in another tenant, this is not true because the schema is present in the same tenant.
This is what one would expect, The return error code should be **409 Conflict** indicating that schema is already present. and the message should be "schema is present".M19 - Release 0.22https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/69Intermittent GET Schema endpoint response with 4042023-06-28T08:35:38ZNeelesh ThakurIntermittent GET Schema endpoint response with 404Indexer/Search consumers have reported issues that data block is not indexed for certain records.
While triaging the issue, we have found Schema service is responding with 404 when it receives multiple concurrent request in this case (p...Indexer/Search consumers have reported issues that data block is not indexed for certain records.
While triaging the issue, we have found Schema service is responding with 404 when it receives multiple concurrent request in this case (please take a look at logs below).
![Schema_Error](/uploads/e2eb86fb1697c46dc09fa3283e0133f1/Schema_Error.jpg)Aman VermaAman Vermahttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/106Make each IT test to run in isolation2022-08-24T11:28:19ZAbhishek Kumar (SLB)Make each IT test to run in isolationAs of today there are some test cases which depends on successful execution of previous testcase.
This should not be the case and each test-case should be able to execute independently.As of today there are some test cases which depends on successful execution of previous testcase.
This should not be the case and each test-case should be able to execute independently.Neha SardaNeha Sardahttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/89Need improved exception handling for different scenarios2022-09-12T14:26:24ZAbhishek Kumar (SLB)Need improved exception handling for different scenariosAt present Not Found exceptions are thrown without checking the actual error codes e.g
![image](/uploads/74fe1073d4b4e3341351d09788bf47a8/image.png)
```java
public Boolean checkAndRegisterSystemAuthorityIfNotPresent(String authorityId) ...At present Not Found exceptions are thrown without checking the actual error codes e.g
![image](/uploads/74fe1073d4b4e3341351d09788bf47a8/image.png)
```java
public Boolean checkAndRegisterSystemAuthorityIfNotPresent(String authorityId) {
try {
Authority authority = new Authority();
authority.setAuthorityId(authorityId);
authorityStore.createSystemAuthority(authority);
} catch (ApplicationException e) {
return false;
} catch (BadRequestException ex) {
return true;
}
```
There should be check for specific error code:
eg.
```java
BadRequestException(MessageFormat.format(SchemaConstants.AUTHORITY_EXISTS_EXCEPTION,
authority.getAuthorityId()));
```https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/78One should be able to delete the created schema2022-05-05T08:30:39ZKamlesh TodaiOne should be able to delete the created schemaAt present, the API seems to be missing the delete functionality. Prior to schema service when storage API was used there was an option to delete the schema. But in Schema Service that option is not present. So when creating the custom s...At present, the API seems to be missing the delete functionality. Prior to schema service when storage API was used there was an option to delete the schema. But in Schema Service that option is not present. So when creating the custom schemas for testing, it is difficult to clean after the testhttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/155OSDU-DD-Delivery-M23.0 (v0.26.0)2024-03-23T10:12:02ZThomas Gehrmann [slb]OSDU-DD-Delivery-M23.0 (v0.26.0)- [ ] Update to the M23.0 deliverables from OSDU Data Definitions- [ ] Update to the M23.0 deliverables from OSDU Data DefinitionsM23 - Release 0.26Thomas Gehrmann [slb]Thomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/99Poor performance of schema info list endpoint and uniqueness check2022-08-24T11:00:46ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comPoor performance of schema info list endpoint and uniqueness checkPerformance issue related to get schema info list request and uniqueness check in the schema creation process:
~~~
curl --location --request GET 'localhost:8080/api/schema-service/v1/schema?authority=SchemaSanityTest' \
--header 'Data-P...Performance issue related to get schema info list request and uniqueness check in the schema creation process:
~~~
curl --location --request GET 'localhost:8080/api/schema-service/v1/schema?authority=SchemaSanityTest' \
--header 'Data-Partition-Id: osdu'
~~~
This request can use the `offset` and `limit` parameters, it is ok when these parameters are used at the data access layer,
but in the Schema service they were used at the core level by design:
https://community.opengroup.org/osdu/platform/system/schema-service/-/blob/master/schema-core/src/main/java/org/opengroup/osdu/schema/service/serviceimpl/SchemaService.java#L305
Also, this logic is used during schema creation, the same methods used to verify schema uniqueness and whether breaking changes are present or not.
This leads to loading a lot of unwanted data, for example, the query presented in the example will fetch over 6500 schema information from the GCP dev env, but by default, they will be discarded in the core service and only 100 records will be returned in the response.
Previously issues were spotted at GCP and Azure envs, to fix GCP we manually delete schemas created by IT's, my guess is that the Azure team does the same:
https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/76
https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/70
Suggestion for the fix is to pass `limit` and `offset` parameters to the provider level and use them directly.Andrei Dalhikh [EPAM/GC]Andrei Dalhikh [EPAM/GC]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/82Provide validation of schema and avoid surprise from indexed data (lacking fi...2023-09-28T14:43:13ZDebasis ChatterjeeProvide validation of schema and avoid surprise from indexed data (lacking fields)Because of gap in schema definition script, this is one case of user experience.
We can successfully populate record. Retrieve data by using Storage service. But Search service (query) misses many fields.
Even if we do usual troublesho...Because of gap in schema definition script, this is one case of user experience.
We can successfully populate record. Retrieve data by using Storage service. But Search service (query) misses many fields.
Even if we do usual troubleshooting (Storage - Get - id and index), there is no obvious clue anywhere about the failure.
See related issue https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser/-/issues/64
Hence we propose that strict validation is added to avoid future surprise.
cc @nthakur for informationhttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/71Queuing of Schema build2021-09-16T12:25:18ZAbhishek Kumar (SLB)Queuing of Schema buildCurrently, schema builds are triggered concurrently. This creates a problem with:
- Integration testing
- Bootstrap process
The above two processes make calls to the actual endpoint of the service and hence it is important that these st...Currently, schema builds are triggered concurrently. This creates a problem with:
- Integration testing
- Bootstrap process
The above two processes make calls to the actual endpoint of the service and hence it is important that these steps are triggered on the right image that is deployed.
There have been instances where the bootstrap or IT stage has failed because a different build is triggered and original image is replaced.https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/105Remove unwanted logs from IT2022-12-09T13:39:36ZAbhishek Kumar (SLB)Remove unwanted logs from ITAs of today there are too many unwanted logs printed during IT execution which slows down the execution.As of today there are too many unwanted logs printed during IT execution which slows down the execution.Neha SardaNeha Sarda