Schema issueshttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues2023-02-22T14:24:03Zhttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/51ADR: New API to handle system schemas2023-02-22T14:24:03ZAman VermaADR: New API to handle system schemas## Status
* [ ] Proposed
* [ ] Trialing
* [ ] Under review
* [x] Approved
* [ ] Retired
Context:
===
Public schemas are schemas which are available to all the tenants out of the box. Today the public schema includes schema from OSDU or...## Status
* [ ] Proposed
* [ ] Trialing
* [ ] Under review
* [x] Approved
* [ ] Retired
Context:
===
Public schemas are schemas which are available to all the tenants out of the box. Today the public schema includes schema from OSDU or SLB and are located in Schema service repository here: https://community.opengroup.org/osdu/platform/system/schema-service/-/tree/master/deployments/shared-schemas
How it's done today:
===
- Deployment expects existence of a special partition (generally termed as `common`) which will be used to load public schemas
- Schema service has only one endpoint today `/schema`. Based on the `data-partition-id` passed in the request headers, it's decided whether the schema is public or private.
- If the data-partition-id passed is "special partition", then the created schema is public, otherwise it's private. This is indicated by a field named `SCOPE` in crated schema.
Issue with current design:
===
1. The API behavior of `/schema` endpoint changes based on the data-partition-id header and can be confusing to users. Users might end up unknowingly creating public schemas by passing data-partition-id of special partition.
2. The special partition is expected to be provisioned for provisioning public schemas.
Proposal:
===
There are two types of schemas in the system, public (or system) schemas and private schema. The proposal is to create a dedicated API to created/ update system schemas. Hence-
- The new API shall be termed as **`/schemas/system`**
- **To create/ updated public schema**- `/schemas/system` endpoint shall be used
- **To read public schema**- existing `GET /schema` endpoint shall be used (same as current behaviour)
- The new API shall not accept `data-partition-id` as a header. Service would be aware where the public schemas are located.
- The authorization of new end point shall be different from exiting schema.editor/viewer role.
**Sequence diagram for creating public schemas**
![image](/uploads/d90a62b4d9a1aae2c6c7a18db4cb66e4/image.png)
**Sequence diagram for reading public schema**
![image](/uploads/53d3e00c194f5ed3e0889682f8184a8f/image.png)M9 - Release 0.12Aman VermaAman Vermahttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/95x-osdu-indexing changes are breaking2022-10-13T11:08:05ZThomas Gehrmann [slb]x-osdu-indexing changes are breaking# Context:
Indexing hints in the OSDU schemas are considered decorations and not taken into account when schemas versions are
validated for 'breaking changes'.
Downstream indexing changes from any state to any other state are considere...# Context:
Indexing hints in the OSDU schemas are considered decorations and not taken into account when schemas versions are
validated for 'breaking changes'.
Downstream indexing changes from any state to any other state are considered breaking changes:
* Breaking changes for the indexer: changes from `flattened` to `nested` require the re-indexing of the kind in
question.
* Consuming applications must use a different query syntax.
# How it's done today:
The process depends on human interaction (assuming OSDU well-known schemas here, but this is no different for custom
schemas):
* Stakeholders ask for an indexing behavior change, OSDU Data Definition reacts by changing the `x-osdu-indexing`
extension tag values in the schema.
* OSDU Data Definition Release notes identify the kinds, which are to be re-indexed.
* In M10 virtually all kinds had to be re-indexed
* In M11 type `reference-data--QualityDataRuleSet` requires re-indexing
* During deployment the records for the affected kinds must be re-indexed.
# Issue with current design:
Upon deployment of a new milestone (or custom schemas),
1. for all involved data-partitions, delete the index for the changed kind and trigger re-indexing. This can take -
depending on the number of records per kind - a very log time and cause serious down-time.
2. Applications have no good way of understanding that the query syntax has changed. Applications may no longer find
data if they depended on queries into data structures affected by the change.
# Proposal:
## `PUBLISHED` Schema Status
1. For schemas with state `PUBLISHED` treat changes to the `x-osdu-indexing` extension tag values in the schema as **_
breaking changes_**.
2. Breaking changes require an incremented major schema version number.
3. Schema Validation Changes during schema creation:
* Changes to the `x-osdu-indexing` extension tag values in `PUBLISHED` schemas with same major schema version
numbers will be **_rejected_**. I.e., the attempted registration of such schema will fail with error.
## `DEVELOPMENT` Schema Status
1. The validation for `DEVELOPMENT` status schemas for incremental versions on top of or between existing minor or patch
versions follows the same rules as for `PUBLISHED` schemas. Attempts to change the `x-osdu-indexing` extension tag
values will be **_rejected_** by the Schema service.
2. For 'single' version schemas in `DEVELOPMENT`, the updates of the `x-osdu-indexing` extension tag values are
permitted.
* It is the responsibility of the schema authors to communicate the impact to deployment and consumers. This is
expected to be acceptable during the development phase.
CC @nthakur @ChrisZhang @chad @pbehedeM12 - Release 0.15https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/130Schema service whitesource issue2024-01-11T11:53:42ZSudesh TagadpallewarSchema service whitesource issueThere are vulnerabilities exist in schema service. I have fixed vulnerabilities relevant libraries. I have skipped upgrading libraries which needs Java 17+. Right now pipeline is failing at gc-deploy and gc-baremetal-deploy. We need to f...There are vulnerabilities exist in schema service. I have fixed vulnerabilities relevant libraries. I have skipped upgrading libraries which needs Java 17+. Right now pipeline is failing at gc-deploy and gc-baremetal-deploy. We need to fix these vulnerabilities.
MR link - https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/504
![image](/uploads/dfa39bf12a986d682cf04e05da9e4896/image.png)M19 - Release 0.22vikas ranavikas ranahttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/123Incorrect status is being returned upon creating the schema that already exis...2023-05-31T11:42:03ZKamlesh TodaiIncorrect status is being returned upon creating the schema that already exists in the systemWhen one tries to create the schema that is already existing in the system, one gets the return error code of **400 - Bad request**. As per the API documentation, it is correct. But I think that the error code is misleading. The message ...When one tries to create the schema that is already existing in the system, one gets the return error code of **400 - Bad request**. As per the API documentation, it is correct. But I think that the error code is misleading. The message returned is also misleading. It returns "message": "Update/Create failed because schema id is present in another tenant, this is not true because the schema is present in the same tenant.
This is what one would expect, The return error code should be **409 Conflict** indicating that schema is already present. and the message should be "schema is present".M19 - Release 0.22https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/124Schema service upgrades may be blocked by schemas created in private tenants.2023-10-12T13:30:24ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comSchema service upgrades may be blocked by schemas created in private tenants.Currently, the creation of system schema may be blocked by the private internal schema in a private tenant.
If schema initially were created in the private tenant.
![Untitled_Diagram.drawio](/uploads/990837be38e4c62e3f0dbb257d914eb7/U...Currently, the creation of system schema may be blocked by the private internal schema in a private tenant.
If schema initially were created in the private tenant.
![Untitled_Diagram.drawio](/uploads/990837be38e4c62e3f0dbb257d914eb7/Untitled_Diagram.drawio.png)
**Cons of current flow:**
- Easy to break bootstrapping, it will fail with errors like: `Error with kind osdu:wks:master-data--ConnectedSourceDataJob:1.3.0: Message: Update/Create failed because schema id is present in another tenant`
- It's not possible to fix it through API, since Schema doesn't have a DELETE endpoint.
**Proposal:**
- There is none for now.
**Additional info:**M21 - Release 0.24Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/142Content-Type returned doesn't match OpenAPI json2024-03-14T16:18:32ZShane HutchinsContent-Type returned doesn't match OpenAPI jsonOne note the Content-Type returned in the following doesn't match OpenAPI json:
* GET /api/schema-service/v1/schema
* PUT /api/schema-service/v1/schema
* POST /api/schema-service/v1/schema
* GET /api/schema-service/v1/schema/{id}
Rece...One note the Content-Type returned in the following doesn't match OpenAPI json:
* GET /api/schema-service/v1/schema
* PUT /api/schema-service/v1/schema
* POST /api/schema-service/v1/schema
* GET /api/schema-service/v1/schema/{id}
Received a response with 'application/json' Content-Type, but it is not declared in the schema.
`Defined content types: */*`
`curl -X GET -H 'Authorization: [Filtered]' -H 'data-partition-id: opendes' 'https://osdu-glab.msft-osdu-test.org/api/schema-service/v1/schema?authority=osdu&source=wks&entityType=wellbore&schemaVersionMajor=1&schemaVersionMinor=1&status=PUBLISHED&scope=INTERNAL&latestVersion=True&limit=10'`
Response status: 200
Response payload: `{"schemaInfos":[],"offset":0,"count":0,"totalCount":0}`M22 - Release 0.25Chad LeongShane HutchinsChad Leonghttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/128DRAFT ADR: New API to delete private schemas2023-12-13T16:20:03ZAndrei Dalhikh [EPAM/GC]DRAFT ADR: New API to delete private schemas
## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
End users and QA engineers often need to delete individual schemas, which contain errors or were uploaded by mistake. Current ...
## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context & Scope
End users and QA engineers often need to delete individual schemas, which contain errors or were uploaded by mistake. Current Schema service API does not have such method and these schemas are deleted manually.
# Tradeoff Analysis
The important thing to consider before deleting any schema from a private context is that data may already exist for that schema. Therefore, delete should only be successful when schema with same id/body is recreated into system partition as SHARED.
**TODO:** edge cases to be elaborated and documented here
## Decision
1. Introduce a delete endpoint to have an ability to delete individual private schema. This endpoint should be governed by some admin role to cleanup schemas from private partitions only.
1. Schemas in SHARED partition should still not have delete support.
1. Create new API method as below:
![Schema_delete_API](/uploads/c787d54444f164938246c2be132e8809/Schema_delete_API.jpg)
OpenAPI spec for this API change [schema_openapi.yaml](https://community.opengroup.org/osdu/platform/system/schema-service/-/blob/Delete_schema_api_draft/docs/api/schema_openapi.yaml)
## Consequences
1. This will help users to keep clean set of schemas in their environments
2. This will reduce the number of support requests for DevOpsM22 - Release 0.25https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/155OSDU-DD-Delivery-M23.0 (v0.26.0)2024-03-23T10:12:02ZThomas Gehrmann [slb]OSDU-DD-Delivery-M23.0 (v0.26.0)- [ ] Update to the M23.0 deliverables from OSDU Data Definitions- [ ] Update to the M23.0 deliverables from OSDU Data DefinitionsM23 - Release 0.26Thomas Gehrmann [slb]Thomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/136Service Schema Loading Fails with `InaccessibleObjectException`2023-11-24T11:12:06ZPaweł GrudzieńService Schema Loading Fails with `InaccessibleObjectException`This issue should belong here: https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/322
TLDR:
The schama PUT ends with `InaccessibleObjectException`. See attached logs on the original...This issue should belong here: https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/322
TLDR:
The schama PUT ends with `InaccessibleObjectException`. See attached logs on the original bug.
This is my latest log trying to PUT a new schema:
```
2023-09-25 10:26:10.548 INFO schema-5fd5f874-47k6z --- [p-nio-80-exec-9] o.o.o.a.l.Slf4JLogger correlation-id=09de3213-25f6-4e59-97ec-e858c7d6f93f data-partition-id=opendes: schema.audit {auditLog={resources=[{"x-osdu-inheriting-from-kind":[],"x-osdu-license":"Copyright 2022, The Open Group \\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 . Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.","$schema":"http://json-schema.org/draft-07/schema#","x-osdu-schema-source":"osdu:wks:AbstractAccessControlList:1.0.0","description":"The access control tags associated with this entity. This structure is included by the SystemProperties \"acl\", which is part of all OSDU records. Not extensible.","additionalProperties":false,"title":"Access Control List","type":"object","properties":{"viewers":{"description":"The list of viewers to which this data record is accessible/visible/discoverable formatted as an email (core.common.model.storage.validation.ValidationDoc.EMAIL_REGEX).","title":"List of Viewers","type":"array","items":{"pattern":"^[a-zA-Z0-9_+&*-]+(?:\\.[a-zA-Z0-9_+&*-]+)*@(?:[a-zA-Z0-9-]+\\.)+[a-zA-Z]{2,7}$","type":"string"}},"owners":{"description":"The list of owners of this data record formatted as an email (core.common.model.storage.validation.ValidationDoc.EMAIL_REGEX).","title":"List of Owners","type":"array","items":{"pattern":"^[a-zA-Z0-9_+&*-]+(?:\\.[a-zA-Z0-9_+&*-]+)*@(?:[a-zA-Z0-9-]+\\.)+[a-zA-Z]{2,7}$","type":"string"}}},"required":["owners","viewers"],"$id":"https://schema.osdu.opengroup.org/json/abstract/AbstractAccessControlList.1.0.0.json"}], action=READ, actionId=SC002, message=Schema retrieved success, user=ed5bc0d2-a29c-4d2c-8fae-ec445f5e397b, status=SUCCESS}} {correlation-id=09de3213-25f6-4e59-97ec-e858c7d6f93f, data-partition-id=opendes}
2023-09-25 10:26:10.704 INFO schema-5fd5f874-47k6z --- [p-nio-80-exec-9] o.o.o.a.l.Slf4JLogger correlation-id=09de3213-25f6-4e59-97ec-e858c7d6f93f data-partition-id=opendes: schema.app Latest patch version is not found so trying to find the nearest matching major version {correlation-id=09de3213-25f6-4e59-97ec-e858c7d6f93f, data-partition-id=opendes}
2023-09-25 10:26:10.795 INFO schema-5fd5f874-47k6z --- [p-nio-80-exec-9] o.o.o.a.l.Slf4JLogger correlation-id=09de3213-25f6-4e59-97ec-e858c7d6f93f data-partition-id=opendes: schema.app Schema info updated in schema info store {correlation-id=09de3213-25f6-4e59-97ec-e858c7d6f93f, data-partition-id=opendes}
2023-09-25 10:26:10.810 INFO schema-5fd5f874-47k6z --- [p-nio-80-exec-9] o.o.o.a.l.Slf4JLogger correlation-id=09de3213-25f6-4e59-97ec-e858c7d6f93f data-partition-id=opendes: schema.app Schmema saved in schema store {correlation-id=09de3213-25f6-4e59-97ec-e858c7d6f93f, data-partition-id=opendes}
2023-09-25 10:26:10.810 INFO schema-5fd5f874-47k6z --- [p-nio-80-exec-9] Generating event of type {} correlation-id=09de3213-25f6-4e59-97ec-e858c7d6f93f data-partition-id=opendes: schema.app update {correlation-id=09de3213-25f6-4e59-97ec-e858c7d6f93f, data-partition-id=opendes}
2023-09-25 10:26:10.810 INFO schema-5fd5f874-47k6z --- [p-nio-80-exec-9] o.o.o.a.l.Slf4JLogger correlation-id=09de3213-25f6-4e59-97ec-e858c7d6f93f data-partition-id=opendes: schema.app Schema event created. {correlation-id=09de3213-25f6-4e59-97ec-e858c7d6f93f, data-partition-id=opendes}
2023-09-25 10:26:11.520 INFO schema-5fd5f874-47k6z --- [ Thread-25] c.a.i.ClientSecretCredential correlation-id= data-partition-id=: Azure Identity => getToken() result for scopes [https://vault.azure.net/.default]: SUCCESS
2023-09-25 10:26:11.520 INFO schema-5fd5f874-47k6z --- [ Thread-25] c.a.i.DefaultAzureCredential correlation-id= data-partition-id=: Azure Identity => Attempted credential EnvironmentCredential returns a token
2023-09-25 10:26:11.544 INFO schema-5fd5f874-47k6z --- [p-nio-80-exec-9] o.o.o.a.KeyVaultFacade correlation-id=09de3213-25f6-4e59-97ec-e858c7d6f93f data-partition-id=opendes: Successfully retrieved opendes-eventgrid-schemachangedtopic-accesskey.
2023-09-25 10:26:11.604 INFO schema-5fd5f874-47k6z --- [p-nio-80-exec-9] o.o.o.a.KeyVaultFacade correlation-id=09de3213-25f6-4e59-97ec-e858c7d6f93f data-partition-id=opendes: Successfully retrieved opendes-eventgrid-schemachangedtopic.
2023-09-25 10:26:11.605 ERROR schema-5fd5f874-47k6z --- [p-nio-80-exec-9] o.o.o.a.l.Slf4JLogger correlation-id=09de3213-25f6-4e59-97ec-e858c7d6f93f data-partition-id=opendes: schema.app Internal server error {correlation-id=09de3213-25f6-4e59-97ec-e858c7d6f93f, data-partition-id=opendes}
2023-09-25 10:26:11.606 WARN schema-5fd5f874-47k6z --- [p-nio-80-exec-9] o.o.o.a.l.Slf4JLogger correlation-id=09de3213-25f6-4e59-97ec-e858c7d6f93f data-partition-id=opendes: schema.app Internal server error {correlation-id=09de3213-25f6-4e59-97ec-e858c7d6f93f, data-partition-id=opendes}
java.lang.reflect.InaccessibleObjectException: Unable to make field private static final long java.util.ArrayList.serialVersionUID accessible: module java.base does not "opens java.util" to unnamed module @69c532af
at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
at com.microsoft.rest.Validator.validateClass(Validator.java:77)
at com.microsoft.rest.Validator.validate(Validator.java:64)
at com.microsoft.azure.eventgrid.implementation.EventGridClientImpl.publishEventsWithServiceResponseAsync(EventGridClientImpl.java:259)
at com.microsoft.azure.eventgrid.implementation.EventGridClientImpl.publishEvents(EventGridClientImpl.java:208)
at org.opengroup.osdu.azure.eventgrid.EventGridTopicStore.publishToEventGridTopic(EventGridTopicStore.java:87)
at org.opengroup.osdu.azure.publisherFacade.EventGridPublisher.publishToEventGrid(EventGridPublisher.java:69)
at org.opengroup.osdu.azure.publisherFacade.MessagePublisher.publishMessage(MessagePublisher.java:46)
at org.opengroup.osdu.schema.azure.impl.messagebus.MessageBusImpl.publishSchemaEventToAzure(MessageBusImpl.java:135)
at org.opengroup.osdu.schema.azure.impl.messagebus.MessageBusImpl.publishMessage(MessageBusImpl.java:68)
at org.opengroup.osdu.schema.service.serviceimpl.SchemaService.updateSchemaInternal(SchemaService.java:230)
at org.opengroup.osdu.schema.service.serviceimpl.SchemaService.updateSchema(SchemaService.java:203)
at org.opengroup.osdu.schema.service.serviceimpl.SchemaService.upsertSchemaInternal(SchemaService.java:331)
at org.opengroup.osdu.schema.service.serviceimpl.SchemaService.upsertSchema(SchemaService.java:320)
at org.opengroup.osdu.schema.api.SchemaController.upsertSchema(SchemaController.java:150)
at org.opengroup.osdu.schema.api.SchemaController$$FastClassBySpringCGLIB$$38d8ac68.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:67)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698)
at org.opengroup.osdu.schema.api.SchemaController$$EnhancerBySpringCGLIB$$aa8168cf.upsertSchema(<generated>)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:920)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:684)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.opengroup.osdu.schema.security.ResponseHeaderFIlter.doFilter(ResponseHeaderFIlter.java:69)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.opengroup.osdu.azure.filters.TransactionLogFilter.doFilter(TransactionLogFilter.java:74)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:111)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:337)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:122)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:116)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:109)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346)
at org.opengroup.osdu.schema.security.AzureIstioSecurityFilter.doFilterInternal(AzureIstioSecurityFilter.java:84)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346)
at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:112)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:82)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:221)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.opengroup.osdu.azure.filters.Slf4jMDCFilter.doFilter(Slf4jMDCFilter.java:54)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.servlet.v3_1.OpenTelemetryHandlerMappingFilter.doFilter(OpenTelemetryHandlerMappingFilter.java:83)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:97)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at com.microsoft.applicationinsights.web.internal.WebRequestTrackingFilter.doFilter(WebRequestTrackingFilter.java)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:769)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:895)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1722)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:833)
2023-09-25 10:26:11.606 INFO schema-5fd5f874-47k6z --- [p-nio-80-exec-9] TxnLogger correlation-id=09de3213-25f6-4e59-97ec-e858c7d6f93f data-partition-id=opendes: schema.app End Web-API PUT /schema/ Headers: {correlation-id:09de3213-25f6-4e59-97ec-e858c7d6f93f,content-type:application/json} status=500 time=1441 ms {correlation-id=09de3213-25f6-4e59-97ec-e858c7d6f93f, data-partition-id=opendes}
```
This is the body ot the PUT (which worked on previous verion 0.20.0 of the OSDU:
```
{
"schemaInfo": {
"schemaIdentity": {
"authority": "opendes",
"source": "wks",
"entityType": "master-data--site",
"schemaVersionMajor": 0,
"schemaVersionMinor": 0,
"schemaVersionPatch": 0,
"id": "osdu:wks:master-data--site:0.0.0"
},
"createdBy": "OSDU Data Definition Group",
"scope": "SHARED",
"status": "DEVELOPMENT"
},
"schema": {
"x-osdu-license": "Copyright 2021",
"$id": "",
"$schema": "http://json-schema.org/draft-07/schema#",
"x-osdu-schema-source": "osdu:wks:master-data--site:0.0.0",
"title": "Site",
"description": "Site entity.",
"type": "object",
"properties": {
"id": {
"description": "Previously called ResourceID or SRN which identifies this OSDU resource object without version.",
"title": "Entity ID",
"type": "string",
"pattern": "^[\\w\\-\\.]+:master-data\\-\\-site:[\\w\\-\\.\\:\\%]+$",
"example": "namespace:master-data--site:6c60ceb0-3521-57b7-9bd8-e1d7c9f66230"
},
"kind": {
"description": "The schema identification for the OSDU resource object following the pattern {Namespace}:{Source}:{Type}:{VersionMajor}.{VersionMinor}.{VersionPatch}. The versioning scheme follows the semantic versioning, https://semver.org/.",
"title": "Entity Kind",
"type": "string",
"pattern": "^[\\w\\-\\.]+:[\\w\\-\\.]+:[\\w\\-\\.]+:[0-9]+.[0-9]+.[0-9]+$",
"example": "osdu:wks:master-data--site:0.0.0"
},
"version": {
"description": "The version number of this OSDU resource; set by the framework.",
"title": "Version Number",
"type": "integer",
"format": "int64",
"example": 1562066009929332
},
"acl": {
"description": "The access control tags associated with this entity.",
"title": "Access Control List",
"$ref": "osdu:wks:AbstractAccessControlList:1.0.0"
},
"legal": {
"description": "The entity's legal tags and compliance status. The actual contents associated with the legal tags is managed by the Compliance Service.",
"title": "Legal Tags",
"$ref": "osdu:wks:AbstractLegalTags:1.0.0"
},
"tags": {
"title": "Tag Dictionary",
"description": "A generic dictionary of string keys mapping to string value. Only strings are permitted as keys and values.",
"type": "object",
"additionalProperties": {
"type": "string"
},
"example": {
"NameOfKey": "String value"
}
},
"createTime": {
"description": "Timestamp of the time at which initial version of this OSDU resource object was created. Set by the System. The value is a combined date-time string in ISO-8601 given in UTC.",
"title": "Resource Object Creation DateTime",
"type": "string",
"format": "date-time",
"example": "2020-12-16T11:46:20.163Z"
},
"createUser": {
"title": "Resource Object Creation User Reference",
"description": "The user reference, which created the first version of this resource object. Set by the System.",
"type": "string",
"example": "some-user@some-company-cloud.com"
},
"modifyTime": {
"description": "Timestamp of the time at which this version of the OSDU resource object was created. Set by the System. The value is a combined date-time string in ISO-8601 given in UTC.",
"title": "Resource Object Version Creation DateTime",
"type": "string",
"format": "date-time",
"example": "2020-12-16T11:52:24.477Z"
},
"modifyUser": {
"title": "Resource Object Version Creation User Reference",
"description": "The user reference, which created this version of this resource object. Set by the System.",
"type": "string",
"example": "some-user@some-company-cloud.com"
},
"data": {
"allOf": [
{
"type": "object",
"properties": {
"Id": {
"title": "Id",
"type": "integer"
},
"SiteName": {
"title": "SiteName",
"type": "string"
},
"Longitude": {
"title": "Longitude",
"type": "number"
},
"Latitude": {
"title": "Latitude",
"type": "number"
}
}
}
]
}
},
"required": [
"kind",
"acl",
"legal"
],
"additionalProperties": false,
"x-osdu-inheriting-from-kind": []
}
}
```https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/135Schema bootstrap script failure2023-09-05T06:13:32ZSachin JaiswalSchema bootstrap script failureAnyone tries to deploy M19 on fresh environment would run into below issue because of incorrect sequence in load_sequence file.
`
Error with kind osdu:wks:work-product-component--SeismicTraceData:1.5.0: Message: Invalid input, osdu:wks:...Anyone tries to deploy M19 on fresh environment would run into below issue because of incorrect sequence in load_sequence file.
`
Error with kind osdu:wks:work-product-component--SeismicTraceData:1.5.0: Message: Invalid input, osdu:wks:AbstractWPCGroupType:1.2.0 not registered but provided as reference
Error with kind osdu:wks:work-product-component--SoilGasMonitoring:1.1.0: Message: Invalid input, osdu:wks:AbstractWPCGroupType:1.2.0 not registered but provided as reference
Error with kind osdu:wks:master-data--StorageFacility:1.2.0: Message: Invalid input, osdu:wks:AbstractMaster:1.2.0 not registered but provided as reference
Error with kind osdu:wks:work-product-component--StratigraphicColumn:1.2.0: Message: Invalid input, osdu:wks:AbstractWPCGroupType:1.2.0 not registered but provided as reference
Error with kind osdu:wks:work-product-component--StratigraphicColumnRankInterpretation:1.3.0: Message: Invalid input, osdu:wks:AbstractWPCGroupType:1.2.0 not registered but provided as reference
Error with kind osdu:wks:work-product-component--StratigraphicUnitInterpretation:1.2.0: Message: Invalid input, osdu:wks:AbstractWPCGroupType:1.2.0 not registered but provided as reference
Error with kind osdu:wks:work-product-component--StructuralOrganizationInterpretation:1.2.0: Message: Invalid input, osdu:wks:AbstractWPCGroupType:1.2.0 not registered but provided as reference
Error with kind osdu:wks:work-product-component--SubRepresentation:1.2.0: Message: Invalid input, osdu:wks:AbstractWPCGroupType:1.2.0 not registered but provided as reference
Error with kind osdu:wks:master-data--SurveyProgram:1.2.0: Message: Invalid input, osdu:wks:AbstractMaster:1.2.0 not registered but provided as reference
Error with kind osdu:wks:work-product-component--TimeSeries:1.2.0: Message: Invalid input, osdu:wks:AbstractWPCGroupType:1.2.0 not registered but provided as reference
Error with kind osdu:wks:work-product-component--TubularAssembly:1.3.0: Message: Invalid input, osdu:wks:AbstractWPCGroupType:1.2.0 not registered but provided as reference
Error with kind osdu:wks:work-product-component--TubularComponent:1.3.0: Message: Invalid input, osdu:wks:AbstractWPCGroupType:1.2.0 not registered but provided as reference
Error with kind osdu:wks:work-product-component--TubularExternalComponent:1.1.0: Message: Invalid input, osdu:wks:AbstractWPCGroupType:1.2.0 not registered but provided as reference
Error with kind osdu:wks:work-product-component--TubularUmbilical:1.2.0: Message: Invalid input, osdu:wks:AbstractWPCGroupType:1.2.0 not registered but provided as reference
Error with kind osdu:wks:work-product-component--UnsealedSurfaceFramework:1.3.0: Message: Invalid input, osdu:wks:AbstractWPCGroupType:1.2.0 not registered but provided as reference
Error with kind osdu:wks:work-product-component--UnstructuredColumnLayerGridRepresentation:1.2.0: Message: Invalid input, osdu:wks:AbstractWPCGroupType:1.2.0 not registered but provided as reference
Error with kind osdu:wks:work-product-component--UnstructuredGridRepresentation:1.2.0: Message: Invalid input, osdu:wks:AbstractWPCGroupType:1.2.0 not registered but provided as reference
Error with kind osdu:wks:work-product-component--VelocityModeling:1.3.0: Message: Invalid input, osdu:wks:AbstractWPCGroupType:1.2.0 not registered but provided as reference
Error with kind osdu:wks:work-product-component--VoidageGroupInterpretation:1.2.0: Message: Invalid input, osdu:wks:AbstractWPCGroupType:1.2.0 not registered but provided as reference
Error with kind osdu:wks:master-data--Well:1.3.0: Message: Invalid input, osdu:wks:AbstractMaster:1.2.0 not registered but provided as reference
Error with kind osdu:wks:master-data--WellActivity:1.2.0: Message: Invalid input, osdu:wks:AbstractMaster:1.2.0 not registered but provided as reference
Error with kind osdu:wks:master-data--WellActivityProgram:1.1.0: Message: Invalid input, osdu:wks:AbstractMaster:1.2.0 not registered but provided as reference
Error with kind osdu:wks:master-data--WellBarrierElementTest:1.1.0: Message: Invalid input, osdu:wks:AbstractMaster:1.2.0 not registered but provided as reference
Error with kind osdu:wks:work-product-component--WellLog:1.4.0: Message: Invalid input, osdu:wks:AbstractWPCGroupType:1.2.0 not registered but provided as reference
Error with kind osdu:wks:master-data--WellPlanningWell:1.1.0: Message: Invalid input, osdu:wks:AbstractMaster:1.2.0 not registered but provided as reference
Error with kind osdu:wks:master-data--WellPlanningWellbore:1.1.0: Message: Invalid input, osdu:wks:AbstractMaster:1.2.0 not registered but provided as reference
Error with kind osdu:wks:master-data--Wellbore:1.4.0: Message: Invalid input, osdu:wks:AbstractMaster:1.2.0 not registered but provided as reference
Error with kind osdu:wks:master-data--WellboreArchitecture:1.1.0: Message: Invalid input, osdu:wks:AbstractMaster:1.2.0 not registered but provided as reference
Error with kind osdu:wks:work-product-component--WellboreIntervalSet:1.2.0: Message: Invalid input, osdu:wks:AbstractWPCGroupType:1.2.0 not registered but provided as reference
Error with kind osdu:wks:work-product-component--WellboreMarkerSet:1.4.0: Message: Invalid input, osdu:wks:AbstractWPCGroupType:1.2.0 not registered but provided as reference
Error with kind osdu:wks:master-data--WellboreOpening:1.2.0: Message: Invalid input, osdu:wks:AbstractMaster:1.2.0 not registered but provided as reference
Error with kind osdu:wks:work-product-component--WellboreTrajectory:1.3.0: Message: Invalid input, osdu:wks:AbstractWPCGroupType:1.2.0 not registered but provided as reference
##[error]Bash exited with code '1'.
`Sachin JaiswalSachin Jaiswalhttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/133Restrict high limit value on get schema endpoint2023-08-07T09:54:59ZAbhishek Kumar (SLB)Restrict high limit value on get schema endpointAt present there is no upper limit put by the api on the limit parameter to get the list of Schema Info.
The default value is 100 when limit is not explicitly provided, likewise, there should be some upper bound to restrict abuse case fo...At present there is no upper limit put by the api on the limit parameter to get the list of Schema Info.
The default value is 100 when limit is not explicitly provided, likewise, there should be some upper bound to restrict abuse case for this endpoint.https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/129AWS M18 GET /api/schema-service/v1/schema/{id}2023-06-15T13:06:31ZShane HutchinsAWS M18 GET /api/schema-service/v1/schema/{id}Received a response with 5xx status code: 500
{"error":{"code":500,"message":"Internal server error","errors":[{"domain":"global","reason":"internalError","message":"Internal server error"}]}}
I was really expecting a 4xx error, like a ...Received a response with 5xx status code: 500
{"error":{"code":500,"message":"Internal server error","errors":[{"domain":"global","reason":"internalError","message":"Internal server error"}]}}
I was really expecting a 4xx error, like a 404.
Run this curl command to reproduce this failure:
curl -X GET -H 'Authorization: Bearer TOKEN' -H 'data-partition-id: osdu' https://osdu.r3m18.preshiptesting.osdu.aws/api/schema-service/v1/schema/%1F
I could produce this in AWS. I could not reproduce this issue in Azure.https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/120System endpoint should also be accessible to user with admin roles2023-11-29T21:16:40ZAbhishek Kumar (SLB)System endpoint should also be accessible to user with admin rolesAs per initial design, there was a need for reserved partition to store **SHARED** schemas so that users can share schemas across partitions. For example, boostrapped schemas can be accessed by any user from any partition. This design do...As per initial design, there was a need for reserved partition to store **SHARED** schemas so that users can share schemas across partitions. For example, boostrapped schemas can be accessed by any user from any partition. This design does not restrict users to create and share their schemas directly rather going through the bootstrapping route.
Initially, creation of all the schemas were through _**common PUT/POST endpoint**_ and the decision of whether schema is **SHARED** or **INTERNAL** was decided based on the data-partition-id mentioned in the header. This design had its own challenges:
- Necessity of maintaining dedicated partition which all end users must be aware about
- Due to common endpoint for both **SHARED** and **INTERNAL** schemas, the decision making was driven through the data-partition-id in the header.
- Could not put any governance at the URL level for better access management.
![image](/uploads/44297a56933116a349c97b9f3e718c38/image.png)
Considering above challenges separate system endpoint was introduced to bifurcate **INTERNAL** and **SHARED** schema creation. Which means SHARED schemas can only be created using system endpoint without specifying any data-partition-id. This, however, was a breaking change and hence it was achieved in two phases:
1. **Creation of System partition:**
At this stage, only system partition was introduced without impacting the user experience. As a result, all existing **SHARED** partitions were moved to system partition and new requests were re-directed to system partitions i.e no new SHARED schemas were getting created into reserved partition (default)
![image](/uploads/42e8bb9df4441cbc7d319101f41582ae/image.png)
2. **Introduction of new endpoint** (https://{{domain}}/api/schema-service/v1/schemas/system):
Because of a dedicated endpoint, **SHARED** schemas are created without providing any data-partition-id in the header.
![image](/uploads/e76b6b085b9e4213d7178a5d8cddc8ec/image.png)
But there is a issue with this design, it is always expected to create SHARED schemas only through bootstrap script which uses service principal. As a result of this, users who were previously able to create and share their schemas are impacted. They now only need to go through bootstrapping process.
To fix this, we need to restrict the access to this endpoint through some entitlement service group role like service **schema-service.admin**.https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/119If abstract reference schema is DEVELOPMENT than kind schema cannot be set to...2022-11-22T11:04:42ZNeelesh ThakurIf abstract reference schema is DEVELOPMENT than kind schema cannot be set to PUBLISHEDIf a Kind schema is using abstract reference schema (using $ref) and these abstract reference schema status is `DEVELOPMENT` than Schema service shouldn't allow changing the status of Kind schema to 'PUBLISHED'.
This can lead to breakin...If a Kind schema is using abstract reference schema (using $ref) and these abstract reference schema status is `DEVELOPMENT` than Schema service shouldn't allow changing the status of Kind schema to 'PUBLISHED'.
This can lead to breaking change in Kind schema as abstract reference schema can still be changed and breaking changes are allowed on `DEVELOPMENT`.https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/109Schema log optimization2022-08-24T11:29:49ZAbhishek Kumar (SLB)Schema log optimizationWe are printing the entire schemas into log which seems to be unnecessary.We are printing the entire schemas into log which seems to be unnecessary.Abhishek Kumar (SLB)Abhishek Kumar (SLB)https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/108Schema service must not allow creation of schema with different case2022-08-23T15:53:06ZNeelesh ThakurSchema service must not allow creation of schema with different caseSchema service uses `kind` as an identifier for Schema for a data-set. `kind` with casing difference usually belongs to same data-set and don't have any other notion. This creates confusion for the end-user consuming records from Search ...Schema service uses `kind` as an identifier for Schema for a data-set. `kind` with casing difference usually belongs to same data-set and don't have any other notion. This creates confusion for the end-user consuming records from Search service with different case. Moreover consumption service like Search uses kind as index name. It's backend (Elasticsearch) do not honor casing for index names thus creating issues for index creation.
Data Definition does not provide any rules for kind casing and delegates this governance to Schema service. It should not allow creation of schema with different case for kind.
Related Search service issue: [94](https://community.opengroup.org/osdu/platform/system/search-service/-/issues/94)https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/107Have an IT for nested $ref version upgrade2022-08-24T11:29:09ZAbhishek Kumar (SLB)Have an IT for nested $ref version upgradeThere is no IT present to validate nested $ref version upgrade.There is no IT present to validate nested $ref version upgrade.Neha SardaNeha Sardahttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/106Make each IT test to run in isolation2022-08-24T11:28:19ZAbhishek Kumar (SLB)Make each IT test to run in isolationAs of today there are some test cases which depends on successful execution of previous testcase.
This should not be the case and each test-case should be able to execute independently.As of today there are some test cases which depends on successful execution of previous testcase.
This should not be the case and each test-case should be able to execute independently.Neha SardaNeha Sardahttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/105Remove unwanted logs from IT2022-12-09T13:39:36ZAbhishek Kumar (SLB)Remove unwanted logs from ITAs of today there are too many unwanted logs printed during IT execution which slows down the execution.As of today there are too many unwanted logs printed during IT execution which slows down the execution.Neha SardaNeha Sardahttps://community.opengroup.org/osdu/platform/system/schema-service/-/issues/99Poor performance of schema info list endpoint and uniqueness check2022-08-24T11:00:46ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comPoor performance of schema info list endpoint and uniqueness checkPerformance issue related to get schema info list request and uniqueness check in the schema creation process:
~~~
curl --location --request GET 'localhost:8080/api/schema-service/v1/schema?authority=SchemaSanityTest' \
--header 'Data-P...Performance issue related to get schema info list request and uniqueness check in the schema creation process:
~~~
curl --location --request GET 'localhost:8080/api/schema-service/v1/schema?authority=SchemaSanityTest' \
--header 'Data-Partition-Id: osdu'
~~~
This request can use the `offset` and `limit` parameters, it is ok when these parameters are used at the data access layer,
but in the Schema service they were used at the core level by design:
https://community.opengroup.org/osdu/platform/system/schema-service/-/blob/master/schema-core/src/main/java/org/opengroup/osdu/schema/service/serviceimpl/SchemaService.java#L305
Also, this logic is used during schema creation, the same methods used to verify schema uniqueness and whether breaking changes are present or not.
This leads to loading a lot of unwanted data, for example, the query presented in the example will fetch over 6500 schema information from the GCP dev env, but by default, they will be discarded in the core service and only 100 records will be returned in the response.
Previously issues were spotted at GCP and Azure envs, to fix GCP we manually delete schemas created by IT's, my guess is that the Azure team does the same:
https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/76
https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/70
Suggestion for the fix is to pass `limit` and `offset` parameters to the provider level and use them directly.Andrei Dalhikh [EPAM/GC]Andrei Dalhikh [EPAM/GC]