Schema merge requestshttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests2023-08-18T21:58:28Zhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/85Logging Enhancements for GCP modules (GONRG-1735, GONRG-1778)2023-08-18T21:58:28ZArtem Dobrynin (EPAM)Logging Enhancements for GCP modules (GONRG-1735, GONRG-1778)## Type of change
- [ ] Bug Fix
- [X] Feature
Some enhancements for GCP logging system. Supplies the logs with user id, data-partition-id and correlation id
https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/...## Type of change
- [ ] Bug Fix
- [X] Feature
Some enhancements for GCP logging system. Supplies the logs with user id, data-partition-id and correlation id
https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/-/issues/50 https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/-/issues/49
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
The logs are printed in an unparseable inline format.
## What is the new/expected behavior?
The logs are printed in JSON format and compatible with GCP Logger.
## Have you added/updated Unit Tests and Integration Tests?
No, since it is not required for these changes
## Any other useful information
There are two types of logging: for local Spring profile (`spring.profiles.active=local`) and for any other, incl. default. This was done for better readability while testing locally.
The MR should be merged after https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/os-core-lib-gcp/-/merge_requests/10M4 - Release 0.7Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/87Aws integration only2023-08-18T21:58:27ZMatt WiseAws integration onlyM4 - Release 0.7ethiraj krishnamanaiduSpencer Suttonsuttonsp@amazon.comMatt Wiseethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/88ibm partition check added2023-08-18T21:58:25ZGokul Nagareibm partition check addedM4 - Release 0.7Anuj GuptaAnuj Guptahttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/89CORS Fix Update2023-08-18T21:58:24ZSpencer Suttonsuttonsp@amazon.comCORS Fix UpdateM4 - Release 0.7ethiraj krishnamanaiduDania Kodeih (Microsoft)Wladmir FrazaoJoeDmitriy RudkoMatt Wiseethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/91Azure cutting over to entitlements v22023-08-18T21:58:22ZJasonAzure cutting over to entitlements v2Updating Azure endpoint for entitlements to point to new entitlements v2 service. Upgrading os-core-common.Updating Azure endpoint for entitlements to point to new entitlements v2 service. Upgrading os-core-common.M5 - Release 0.8JasonJasonhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/93Committing tutorial2023-08-18T21:58:21ZAbhishek Kumar (SLB)Committing tutorialM5 - Release 0.8Abhishek Kumar (SLB)Abhishek Kumar (SLB)https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/94Switching the dependencies to release versions2023-08-18T21:58:19ZDavid Diederichd.diederich@opengroup.orgSwitching the dependencies to release versionsThis changes the library dependencies to use released versions of the core libraries.
They were previously depending on SNAPSHOT versions, which are less stable versions.
More importantly, the SNAPSHOT versions are periodically purged fr...This changes the library dependencies to use released versions of the core libraries.
They were previously depending on SNAPSHOT versions, which are less stable versions.
More importantly, the SNAPSHOT versions are periodically purged from the system to save disk space -- this happened recently.
Since these libraries no longer exist on community, building becomes difficult.
This MR moves those dependencies to a release version, which is better going forward and allows FOSSA to do the build and get good dependency information.
For the GCP change, I assert that there are no substantial changes between the SNAPSHOT version I moved from and the latest release version that I moved to.
It's difficult to know which commit the SNAPSHOT dependency linked to, since it moved many times, but here are the differences from the last time the SNAPSHOT dependency was listed and the one commit that has the release version (0.7.0).
All of these changes were from me, updating versions and references as part of the release process.
- [GCP Differences from v0.6.1-SNAPSHOT to 0.7.0](https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/os-core-lib-gcp/-/compare/ff52818d929b7a32e491b75743285026c4c0a9b4...v0.7.0)
For the IBM changes, version 0.3.23-SNAPSHOT of os-core-lib-ibm doesn't exist in the repository.
As far as I can tell, it was either built on a local system and uploaded directly, or was part of a CI build that was squashed on merge.
Either way, I have no reference to know what changes have occurred since then.
There were code changes between the latest version of v0.3.6-SNAPSHOT and 0.7.0, but they appear to be additions only.
I don't think this will impact compatibility.
- [IBM Differences from v0.3.6-SNAPSHOT to 0.7.0](https://community.opengroup.org/osdu/platform/system/lib/cloud/ibm/os-core-lib-ibm/-/compare/030d7bdb47271193eebfe85515fc83542daa9a55...v0.7.0)
Separately, since I was working with FOSSA, I updated the configuration file and the corresponding NOTICE changes resulting from the new module.M5 - Release 0.8David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/95BugFix: Fixing Custom Dimensions and Spring App Name2023-08-18T21:58:17ZAman VermaBugFix: Fixing Custom Dimensions and Spring App Name## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it? YES
* [YES] Does the MR contain pipeline/ helm chart related changes? NO
* [NA] ...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it? YES
* [YES] Does the MR contain pipeline/ helm chart related changes? NO
* [NA] I have updated the documentation accordingly. NA
* [NA] I have added tests to cover my changes. NA
* [YES] All new and existing tests passed. YES
* [YES] My code follows the code style of this project. YES
* [NA] I ran lint checks locally prior to submission. NA
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
- The `correlation id` in custom dimensions was not getting populated automatically for schema service as of now.
- The spring app name was set to `schema` whereas container name was set to `schema-service`. Due to this the dashboards started failing as we pass in container name in query.
High level design:
The latest `core-lib-azure` library has logic in place to populate the correlation id in custom dimensions. Updating the core-lib-azure version should fix it. Using consistent naming for both spring app name as well as container name will solve the dashboard failure.
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
- Updated the core-lib-azure version
- Made spring app name same as container name.
Validated the changes by running service locally. Logs in AI have correlation id populated now. As well the spring app name is also same as container name.
Sample log
```
Schema.app End Web-API GET /schema Headers: {correlation-id:db59025f-219f-4c97-b714-b91edc7118d8,content-type:application/json} status=200 time=41179 ms {correlation-id=db59025f-219f-4c97-b714-b91edc7118d8, data-partition-id=opendes}
```
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
NA
## Does this introduce a breaking change?
-------------------------------------
- [NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
NONE
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->
cc: @polavishnu @kibattulM5 - Release 0.8Aman VermaAman Vermahttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/96Resolve "[Schema Service] Refresh R3 schemas from DD Feb 25/reference-data Ma...2023-08-18T21:58:16Zdevesh bajpaiResolve "[Schema Service] Refresh R3 schemas from DD Feb 25/reference-data Mar 16"Closes #38Closes #38M5 - Release 0.8https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/97Dont print bearer token2023-08-18T21:58:14ZMatt WiseDont print bearer tokenM5 - Release 0.8ethiraj krishnamanaiduRucha Deshpandeethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/98update aws bootstrap to match new params in DeploySharedSchema.py2023-08-18T21:58:13ZMatt Wiseupdate aws bootstrap to match new params in DeploySharedSchema.pyM5 - Release 0.8Matt WiseMatt Wisehttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/101Deploy shared schema changes incorporated2023-08-18T21:58:11ZMANISH KUMARDeploy shared schema changes incorporatedM5 - Release 0.8MANISH KUMARMANISH KUMARhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/102Partion service as tenantinfo provider(GONRG-2196)2023-08-18T21:58:10ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comPartion service as tenantinfo provider(GONRG-2196)# Description:
Added optional service that replace TenantInfoFactory with implementation that utilize Partition Service<br/>
# How to test:
Functionality was tested with Partition service, Schema service successfully consume TenantInfo...# Description:
Added optional service that replace TenantInfoFactory with implementation that utilize Partition Service<br/>
# How to test:
Functionality was tested with Partition service, Schema service successfully consume TenantInfo from Partition Service
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [x] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [x] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.
# Other comments:
Dependencies:<br/>
https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/os-core-lib-gcp/-/merge_requests/12 <br/>
https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/73 <br/>M6 - Release 0.9Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/103Schema Notification2023-08-18T21:58:08ZAbhishek Kumar (SLB)Schema NotificationWith schema notification changes there will be a notification triggered on the event-grid topic.
There is a new contract introduced in the core module `IMessageBus` with the below signature:
```
public interface IMessageBus {
void publi...With schema notification changes there will be a notification triggered on the event-grid topic.
There is a new contract introduced in the core module `IMessageBus` with the below signature:
```
public interface IMessageBus {
void publishMessage(String schemaId, String eventType);
}
```
This is a breaking change for the rest of the vendors but to avoid any run time issues below stub is added which is expected to implemented by each vendor.
```
@Component
public class MessageBusImpl implements IMessageBus{
@Inject
private JaxRsDpsLog logger;
@Override
public void publishMessage(String schemaId, String eventType) {
// TODO Auto-generated method stub
logger.warning("publish message not implemented ye");
}
}
```
## Runtime variables:
By default, notification feature is turned off using the below parameter:
```
azure.eventGrid.enabled=${event_grid_enabled}
```
Value of this parameter can be found at: `\schema-service\devops\azure\chart\templates\deployment.yaml`
```
- name: event_grid_enabled
value: "false"
```M6 - Release 0.9Paresh BehedeParesh Behedehttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/105Resolving ref in definition section2023-08-18T21:58:05ZAbhishek Kumar (SLB)Resolving ref in definition section`$ref` value inside `definitions` section is not getting resolved correctly.
**Before Fix**
Ref value at /definitions/projectedPosition/elevationFromMsl is not resolved correctly
```
{
"$schema":"http://json-schema.org/draft-07/schem...`$ref` value inside `definitions` section is not getting resolved correctly.
**Before Fix**
Ref value at /definitions/projectedPosition/elevationFromMsl is not resolved correctly
```
{
"$schema":"http://json-schema.org/draft-07/schema#",
"$id":"https://schema.osdu.opengroup.org/json/dataset/schema-with-id-added.json",
"$comment":"This is a test schema",
"type":"object",
"title":"My schema - Changed",
"definitions":{
"projectedPosition":{
"type":"object",
"properties":{
"elevationFromMsl":{
"title":"Elevation from MSL",
"description":"Elevation from Mean Seal Level, downwards negative. The unit definition is found via 'elevationFromMsl.unitKey' in 'frameOfReference.units' dictionary.",
"$ref":"osdu:wks:valueWithUnit:1.0.0"
}
}
},
"osdu:wks:valueWithUnit:1.0.0":{
"type":"object"
}
},
"properties":{
"AddressLine1":{
"type":"string",
"$comment":"This is a mandatory attribute"
},
"AddressLine2":{
"type":"string"
},
"City":{
"type":"string"
}
}
}
```
**After Fix**
```
{
"$schema":"http://json-schema.org/draft-07/schema#",
"$id":"https://schema.osdu.opengroup.org/json/dataset/schema-with-id-added.json",
"$comment":"This is a test schema",
"type":"object",
"title":"My schema - Changed",
"definitions":{
"projectedPosition":{
"type":"object",
"properties":{
"elevationFromMsl":{
"title":"Elevation from MSL",
"description":"Elevation from Mean Seal Level, downwards negative. The unit definition is found via 'elevationFromMsl.unitKey' in 'frameOfReference.units' dictionary.",
"$ref":"#/definitions/osdu:wks:valueWithUnit:1.0.0"
}
}
},
"osdu:wks:valueWithUnit:1.0.0":{
"type":"object"
}
},
"properties":{
"AddressLine1":{
"type":"string",
"$comment":"This is a mandatory attribute"
},
"AddressLine2":{
"type":"string"
},
"City":{
"type":"string"
}
}
}
```M7 - Release 0.10Abhishek Kumar (SLB)Abhishek Kumar (SLB)https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/106Check invalid refences in schema2023-08-18T21:58:03Zdevesh bajpaiCheck invalid refences in schemaM6 - Release 0.9https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/107Multitenant/Data Split2023-08-18T21:58:02ZMatt WiseMultitenant/Data Splitcommit edb8dacc
Author: Matt Wise <wsmatth@amazon.com>
Date: Wed May 05 2021 11:45:54 GMT-0500 (Central Daylight Time)
code cleanup
commit f42a82ca
Author: Matt Wise <wsmatth@amazon.com>
Date: Fri Apr 30 2021 09:54:22 GMT-050...commit edb8dacc
Author: Matt Wise <wsmatth@amazon.com>
Date: Wed May 05 2021 11:45:54 GMT-0500 (Central Daylight Time)
code cleanup
commit f42a82ca
Author: Matt Wise <wsmatth@amazon.com>
Date: Fri Apr 30 2021 09:54:22 GMT-0500 (Central Daylight Time)
fix unit tests
commit db32c1e5
Author: Matt Wise <wsmatth@amazon.com>
Date: Thu Apr 29 2021 15:33:49 GMT-0500 (Central Daylight Time)
schema multi-tenant implM6 - Release 0.9Matt WiseMatt Wisehttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/108Update NOTICE2023-08-18T21:58:00ZMatt WiseUpdate NOTICEM6 - Release 0.9Matt WiseMatt Wisehttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/110Limit on open select for schema list added2023-08-18T21:57:58ZManish SinghLimit on open select for schema list addedThe limit was causing a truncated search and that has been fixed with a configurable limit for open select.The limit was causing a truncated search and that has been fixed with a configurable limit for open select.M6 - Release 0.9Anuj GuptaAnuj Guptahttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/111Added new variables required for service2023-08-18T21:57:57ZOleksii Tsyganov (EPAM)Added new variables required for serviceM6 - Release 0.9Oleksii Tsyganov (EPAM)Oleksii Tsyganov (EPAM)