Schema merge requestshttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests2023-08-18T21:58:22Zhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/91Azure cutting over to entitlements v22023-08-18T21:58:22ZJasonAzure cutting over to entitlements v2Updating Azure endpoint for entitlements to point to new entitlements v2 service. Upgrading os-core-common.Updating Azure endpoint for entitlements to point to new entitlements v2 service. Upgrading os-core-common.M5 - Release 0.8JasonJasonhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/92Adding multi tenancy tests2021-11-19T12:00:30ZAshay BansodAdding multi tenancy testsAdding multi-tenancy test cases to schema service integration tests.
These are not to run in gitlab using tag feature as only single tenant infra is available.
When ever any consumer would have infra to execute this, it will have to run...Adding multi-tenancy test cases to schema service integration tests.
These are not to run in gitlab using tag feature as only single tenant infra is available.
When ever any consumer would have infra to execute this, it will have to run specific tag 'NotToRunInGitlab' along with 'SchemaService' tag.
Removed pre and post integration file (runner, step-def along with pom changes) as they are not configured for schema service anyways. Setup part of schema service is being already handled using background feature of BDD. So, no impact of current integration setup.https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/93Committing tutorial2023-08-18T21:58:21ZAbhishek Kumar (SLB)Committing tutorialM5 - Release 0.8Abhishek Kumar (SLB)Abhishek Kumar (SLB)https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/94Switching the dependencies to release versions2023-08-18T21:58:19ZDavid Diederichd.diederich@opengroup.orgSwitching the dependencies to release versionsThis changes the library dependencies to use released versions of the core libraries.
They were previously depending on SNAPSHOT versions, which are less stable versions.
More importantly, the SNAPSHOT versions are periodically purged fr...This changes the library dependencies to use released versions of the core libraries.
They were previously depending on SNAPSHOT versions, which are less stable versions.
More importantly, the SNAPSHOT versions are periodically purged from the system to save disk space -- this happened recently.
Since these libraries no longer exist on community, building becomes difficult.
This MR moves those dependencies to a release version, which is better going forward and allows FOSSA to do the build and get good dependency information.
For the GCP change, I assert that there are no substantial changes between the SNAPSHOT version I moved from and the latest release version that I moved to.
It's difficult to know which commit the SNAPSHOT dependency linked to, since it moved many times, but here are the differences from the last time the SNAPSHOT dependency was listed and the one commit that has the release version (0.7.0).
All of these changes were from me, updating versions and references as part of the release process.
- [GCP Differences from v0.6.1-SNAPSHOT to 0.7.0](https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/os-core-lib-gcp/-/compare/ff52818d929b7a32e491b75743285026c4c0a9b4...v0.7.0)
For the IBM changes, version 0.3.23-SNAPSHOT of os-core-lib-ibm doesn't exist in the repository.
As far as I can tell, it was either built on a local system and uploaded directly, or was part of a CI build that was squashed on merge.
Either way, I have no reference to know what changes have occurred since then.
There were code changes between the latest version of v0.3.6-SNAPSHOT and 0.7.0, but they appear to be additions only.
I don't think this will impact compatibility.
- [IBM Differences from v0.3.6-SNAPSHOT to 0.7.0](https://community.opengroup.org/osdu/platform/system/lib/cloud/ibm/os-core-lib-ibm/-/compare/030d7bdb47271193eebfe85515fc83542daa9a55...v0.7.0)
Separately, since I was working with FOSSA, I updated the configuration file and the corresponding NOTICE changes resulting from the new module.M5 - Release 0.8David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/95BugFix: Fixing Custom Dimensions and Spring App Name2023-08-18T21:58:17ZAman VermaBugFix: Fixing Custom Dimensions and Spring App Name## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it? YES
* [YES] Does the MR contain pipeline/ helm chart related changes? NO
* [NA] ...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it? YES
* [YES] Does the MR contain pipeline/ helm chart related changes? NO
* [NA] I have updated the documentation accordingly. NA
* [NA] I have added tests to cover my changes. NA
* [YES] All new and existing tests passed. YES
* [YES] My code follows the code style of this project. YES
* [NA] I ran lint checks locally prior to submission. NA
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
- The `correlation id` in custom dimensions was not getting populated automatically for schema service as of now.
- The spring app name was set to `schema` whereas container name was set to `schema-service`. Due to this the dashboards started failing as we pass in container name in query.
High level design:
The latest `core-lib-azure` library has logic in place to populate the correlation id in custom dimensions. Updating the core-lib-azure version should fix it. Using consistent naming for both spring app name as well as container name will solve the dashboard failure.
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
- Updated the core-lib-azure version
- Made spring app name same as container name.
Validated the changes by running service locally. Logs in AI have correlation id populated now. As well the spring app name is also same as container name.
Sample log
```
Schema.app End Web-API GET /schema Headers: {correlation-id:db59025f-219f-4c97-b714-b91edc7118d8,content-type:application/json} status=200 time=41179 ms {correlation-id=db59025f-219f-4c97-b714-b91edc7118d8, data-partition-id=opendes}
```
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
NA
## Does this introduce a breaking change?
-------------------------------------
- [NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
NONE
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->
cc: @polavishnu @kibattulM5 - Release 0.8Aman VermaAman Vermahttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/96Resolve "[Schema Service] Refresh R3 schemas from DD Feb 25/reference-data Ma...2023-08-18T21:58:16Zdevesh bajpaiResolve "[Schema Service] Refresh R3 schemas from DD Feb 25/reference-data Mar 16"Closes #38Closes #38M5 - Release 0.8https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/97Dont print bearer token2023-08-18T21:58:14ZMatt WiseDont print bearer tokenM5 - Release 0.8ethiraj krishnamanaiduRucha Deshpandeethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/98update aws bootstrap to match new params in DeploySharedSchema.py2023-08-18T21:58:13ZMatt Wiseupdate aws bootstrap to match new params in DeploySharedSchema.pyM5 - Release 0.8Matt WiseMatt Wisehttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/99Correcting the parameter list for `DeploySharedSchemas` step2021-04-14T06:08:24ZAman VermaCorrecting the parameter list for `DeploySharedSchemas` step**Issue**
Azure_bootstrap step fails with error
```
DeploySharedSchemas.py: error: unrecognized arguments: -l load_sequence.1.0.0.json
```
**Cause**
Previously the `DeploySharedSchemas.py` used to accept two arguments - load sequenc...**Issue**
Azure_bootstrap step fails with error
```
DeploySharedSchemas.py: error: unrecognized arguments: -l load_sequence.1.0.0.json
```
**Cause**
Previously the `DeploySharedSchemas.py` used to accept two arguments - load sequence and URL
`$AZURE_DEPLOYMENTS_SCRIPTS_SUBDIR/DeploySharedSchemas.py -l load_sequence.1.0.0.json -u $AZURE_SCHEMA_URL`. The script has been modified to accept these values from environment variables in this commit https://community.opengroup.org/osdu/platform/system/schema-service/-/commit/28729d8eb5018ef9c9481a5332d51e6f1471d08b . The `-l` parameter is no longer supported.
**Fix**
Taking reactionary changes in azure related scripts. Now the script call looks like this:
`$AZURE_DEPLOYMENTS_SCRIPTS_SUBDIR/DeploySharedSchemas.py -u $AZURE_SCHEMA_URL`
No load sequence is being passed since default value works in our case.
cc: @kibattul , @harshit283Aman VermaAman Vermahttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/100Container Benchmarking2021-09-30T02:59:14ZAbhishek PatilContainer Benchmarking## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it? YES
* [YES] Does the MR contain pipeline/ helm chart related changes? NO
* [NA] ...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it? YES
* [YES] Does the MR contain pipeline/ helm chart related changes? NO
* [NA] I have updated the documentation accordingly. NA
* [NA] I have added tests to cover my changes. NA
* [YES] All new and existing tests passed. YES
* [YES] My code follows the code style of this project. YES
* [NA] I ran lint checks locally prior to submission. NA
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
- Schema service's pod does not have container limits defined which causes it to use all the available cpu and memory.
- [Tomcat's MBean registry is disabled by default](https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-2.2-Release-Notes#tomcats-mbean-registry) in the spring boot's version used for schema service.
High level design:
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
- Set the limits for cpu and memory.
- Enabled the Tomcat's MBean registry in order to make tomcat metrics available for monitoring.
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
NA
## Does this introduce a breaking change?
-------------------------------------
- [NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
NONE
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->Abhishek PatilAbhishek Patilhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/101Deploy shared schema changes incorporated2023-08-18T21:58:11ZMANISH KUMARDeploy shared schema changes incorporatedM5 - Release 0.8MANISH KUMARMANISH KUMARhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/102Partion service as tenantinfo provider(GONRG-2196)2023-08-18T21:58:10ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comPartion service as tenantinfo provider(GONRG-2196)# Description:
Added optional service that replace TenantInfoFactory with implementation that utilize Partition Service<br/>
# How to test:
Functionality was tested with Partition service, Schema service successfully consume TenantInfo...# Description:
Added optional service that replace TenantInfoFactory with implementation that utilize Partition Service<br/>
# How to test:
Functionality was tested with Partition service, Schema service successfully consume TenantInfo from Partition Service
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [x] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [x] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.
# Other comments:
Dependencies:<br/>
https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/os-core-lib-gcp/-/merge_requests/12 <br/>
https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/73 <br/>M6 - Release 0.9Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/103Schema Notification2023-08-18T21:58:08ZAbhishek Kumar (SLB)Schema NotificationWith schema notification changes there will be a notification triggered on the event-grid topic.
There is a new contract introduced in the core module `IMessageBus` with the below signature:
```
public interface IMessageBus {
void publi...With schema notification changes there will be a notification triggered on the event-grid topic.
There is a new contract introduced in the core module `IMessageBus` with the below signature:
```
public interface IMessageBus {
void publishMessage(String schemaId, String eventType);
}
```
This is a breaking change for the rest of the vendors but to avoid any run time issues below stub is added which is expected to implemented by each vendor.
```
@Component
public class MessageBusImpl implements IMessageBus{
@Inject
private JaxRsDpsLog logger;
@Override
public void publishMessage(String schemaId, String eventType) {
// TODO Auto-generated method stub
logger.warning("publish message not implemented ye");
}
}
```
## Runtime variables:
By default, notification feature is turned off using the below parameter:
```
azure.eventGrid.enabled=${event_grid_enabled}
```
Value of this parameter can be found at: `\schema-service\devops\azure\chart\templates\deployment.yaml`
```
- name: event_grid_enabled
value: "false"
```M6 - Release 0.9Paresh BehedeParesh Behedehttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/104Resolve "Indexer Hints"2023-08-18T22:42:29ZThomas Gehrmann [slb]Resolve "Indexer Hints"Closes #41Closes #41M6 - Release 0.9ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/105Resolving ref in definition section2023-08-18T21:58:05ZAbhishek Kumar (SLB)Resolving ref in definition section`$ref` value inside `definitions` section is not getting resolved correctly.
**Before Fix**
Ref value at /definitions/projectedPosition/elevationFromMsl is not resolved correctly
```
{
"$schema":"http://json-schema.org/draft-07/schem...`$ref` value inside `definitions` section is not getting resolved correctly.
**Before Fix**
Ref value at /definitions/projectedPosition/elevationFromMsl is not resolved correctly
```
{
"$schema":"http://json-schema.org/draft-07/schema#",
"$id":"https://schema.osdu.opengroup.org/json/dataset/schema-with-id-added.json",
"$comment":"This is a test schema",
"type":"object",
"title":"My schema - Changed",
"definitions":{
"projectedPosition":{
"type":"object",
"properties":{
"elevationFromMsl":{
"title":"Elevation from MSL",
"description":"Elevation from Mean Seal Level, downwards negative. The unit definition is found via 'elevationFromMsl.unitKey' in 'frameOfReference.units' dictionary.",
"$ref":"osdu:wks:valueWithUnit:1.0.0"
}
}
},
"osdu:wks:valueWithUnit:1.0.0":{
"type":"object"
}
},
"properties":{
"AddressLine1":{
"type":"string",
"$comment":"This is a mandatory attribute"
},
"AddressLine2":{
"type":"string"
},
"City":{
"type":"string"
}
}
}
```
**After Fix**
```
{
"$schema":"http://json-schema.org/draft-07/schema#",
"$id":"https://schema.osdu.opengroup.org/json/dataset/schema-with-id-added.json",
"$comment":"This is a test schema",
"type":"object",
"title":"My schema - Changed",
"definitions":{
"projectedPosition":{
"type":"object",
"properties":{
"elevationFromMsl":{
"title":"Elevation from MSL",
"description":"Elevation from Mean Seal Level, downwards negative. The unit definition is found via 'elevationFromMsl.unitKey' in 'frameOfReference.units' dictionary.",
"$ref":"#/definitions/osdu:wks:valueWithUnit:1.0.0"
}
}
},
"osdu:wks:valueWithUnit:1.0.0":{
"type":"object"
}
},
"properties":{
"AddressLine1":{
"type":"string",
"$comment":"This is a mandatory attribute"
},
"AddressLine2":{
"type":"string"
},
"City":{
"type":"string"
}
}
}
```M7 - Release 0.10Abhishek Kumar (SLB)Abhishek Kumar (SLB)https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/106Check invalid refences in schema2023-08-18T21:58:03Zdevesh bajpaiCheck invalid refences in schemaM6 - Release 0.9https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/107Multitenant/Data Split2023-08-18T21:58:02ZMatt WiseMultitenant/Data Splitcommit edb8dacc
Author: Matt Wise <wsmatth@amazon.com>
Date: Wed May 05 2021 11:45:54 GMT-0500 (Central Daylight Time)
code cleanup
commit f42a82ca
Author: Matt Wise <wsmatth@amazon.com>
Date: Fri Apr 30 2021 09:54:22 GMT-050...commit edb8dacc
Author: Matt Wise <wsmatth@amazon.com>
Date: Wed May 05 2021 11:45:54 GMT-0500 (Central Daylight Time)
code cleanup
commit f42a82ca
Author: Matt Wise <wsmatth@amazon.com>
Date: Fri Apr 30 2021 09:54:22 GMT-0500 (Central Daylight Time)
fix unit tests
commit db32c1e5
Author: Matt Wise <wsmatth@amazon.com>
Date: Thu Apr 29 2021 15:33:49 GMT-0500 (Central Daylight Time)
schema multi-tenant implM6 - Release 0.9Matt WiseMatt Wisehttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/108Update NOTICE2023-08-18T21:58:00ZMatt WiseUpdate NOTICEM6 - Release 0.9Matt WiseMatt Wisehttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/109Schema validation changes2022-09-16T09:02:44ZAbhishek Kumar (SLB)Schema validation changes**Schema Validation Changes**
There are new schema validations going to be performed everytime there is a create or update request. These changes are driven by below rules:
[SchemaChanges](https://gitlab.opengroup.org/osdu/subcommittee...**Schema Validation Changes**
There are new schema validations going to be performed everytime there is a create or update request. These changes are driven by below rules:
[SchemaChanges](https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/schema/-/blob/master/_doc/SchemaChanges.md)
[[_TOC_]]
# Schema Changes
Schema modifications have to be done with great care as schemas can be
used to generate code and API specifications. Changes may or may not break
clients consuming the schemas.
The solution to this is the concept of [semantic versioning](https://semver.org).
General rules have been defined during the
[R3 cycle](R3-changes/SchemaCompositionAndVersioning.md) in this summary:
* _Given a version number `MAJOR.MINOR.PATCH`, increment the:_
* _`MAJOR` version when you make **incompatible schema** changes,_
* _`MINOR` version when you add functionality **or contents** in a backwards
compatible manner, and_
* _`PATCH` version when you make backwards compatible bug fixes **or documentation/decoration changes**._
# Schema Service
New schemas are registered with the Schema service. The Schema service checks
the incoming schema definition against the previous minor and/or patch version
should they exist.
## Permitted Changes for PATCH Version Increments
Changes to the values (text) of the following JSON tags:
1. `title`
1. `description`
1. `example`
1. `pattern`
1. `$id`
1. `$comment`
1. any JSON extension tag starting with `x-osdu`
## Permitted Changes for MINOR Version Increments
In addition to all permitted changes in PATCH versions, the following
actions are permitted:
1. adding properties to existing data and nested structures
1. adding object structures to
1. `allOf` arrays and
1. `oneOf` arrays.
Explicitly not permitted is changing
1. the list of `required` properties
1. the state of `additionalProperties`.
It is permitted to declare existing properties as deprecated, preferable with
instructions how to migrate from the previous to the next version. The documentation
will automatically mark properties with ~~strike-through~~ if the description starts
with `"DEPRECATED: "`.
## Permitted Changes for MAJOR Version Increments
Any changes are permitted, specifically
* removal of properties
* removal of structures in `allOf`, `oneOf`
* changing the list of `required` properties
* changing the state of `additionalProperties`.M7 - Release 0.10Abhishek Kumar (SLB)Abhishek Kumar (SLB)https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/110Limit on open select for schema list added2023-08-18T21:57:58ZManish SinghLimit on open select for schema list addedThe limit was causing a truncated search and that has been fixed with a configurable limit for open select.The limit was causing a truncated search and that has been fixed with a configurable limit for open select.M6 - Release 0.9Anuj GuptaAnuj Gupta