Schema merge requestshttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests2023-08-18T22:01:11Zhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/10Adding azure provider implementation for schema service2023-08-18T22:01:11ZAman VermaAdding azure provider implementation for schema serviceThese set of changes introduce azure implementation of schema service.
*Testing*
Unit tests have been added for required classes.
Integration tests: pending
*What might change in future:*
Data partitioning in azure cosmos db: cur...These set of changes introduce azure implementation of schema service.
*Testing*
Unit tests have been added for required classes.
Integration tests: pending
*What might change in future:*
Data partitioning in azure cosmos db: currently data from all the tenants is being stored in a single cosmos db instance, partitioned by the dataPartitionId of that tenant. This model might change in future.
@danielscholl , @dkodeih FYI.M1 - Release 0.1https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/9updated code, test folder segratation, readme updated, code coverage increased2023-08-18T22:01:12ZSubham Agarwalupdated code, test folder segratation, readme updated, code coverage increasedBelow are the list of changes incorporated in below merge request,
- Increased code coverage
- Updated readme with prerequisites to run the service locally
- Segregated testing folder as per OSDU folder structure
- Resolved bug related t...Below are the list of changes incorporated in below merge request,
- Increased code coverage
- Updated readme with prerequisites to run the service locally
- Segregated testing folder as per OSDU folder structure
- Resolved bug related to two schemasInfo getting returned for latestVersion true instead of returning only one schema
- Integration test scenarios addition
- PUT endpoint enhancement to support create schema for development stage schema
- Using custom logging better debuggingM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/7IBM initial implementation2020-07-20T13:11:55ZAlan BrazIBM initial implementationM1 - Release 0.1ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/5Reducing the log output in Spring tests to be WARN or higher2023-08-18T22:01:14ZDavid Diederichd.diederich@opengroup.orgReducing the log output in Spring tests to be WARN or higherIt seems like the build failures on master were rooted in this problem; at least, they've gone away here.It seems like the build failures on master were rooted in this problem; at least, they've gone away here.M1 - Release 0.1ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/4Update .gitlab-ci.yml2023-08-18T22:01:15Zethiraj krishnamanaiduUpdate .gitlab-ci.ymlM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/3OpenDES ADO code merge2023-08-18T22:01:17Zethiraj krishnamanaiduOpenDES ADO code mergePushed the latest code from ADO to GitLab.Pushed the latest code from ADO to GitLab.M1 - Release 0.1ethiraj krishnamanaiduethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/2ADO latest code merge2023-08-18T22:01:18Zethiraj krishnamanaiduADO latest code mergeM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/78Fix AWS int tests internal failure. Update AWS bootstrap for use with intern...2023-08-18T21:58:33ZMatt WiseFix AWS int tests internal failure. Update AWS bootstrap for use with internal CI.M3 - Release 0.5ethiraj krishnamanaiduRucha DeshpandeMatt Wiseethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/75adding load sequence parameter in bootstrapping scripts2023-08-18T21:58:36ZAman Vermaadding load sequence parameter in bootstrapping scriptsadding the `loadSequence` parameter as prescribed here: https://community.opengroup.org/osdu/platform/system/schema-service/-/blob/master/deployments/shared-schemas/README.md
Issue:
https://community.opengroup.org/osdu/platform/system/s...adding the `loadSequence` parameter as prescribed here: https://community.opengroup.org/osdu/platform/system/schema-service/-/blob/master/deployments/shared-schemas/README.md
Issue:
https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/33M3 - Release 0.5Aman VermaAman Vermahttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/71Fix related to schema resolve logic2023-08-18T21:58:38ZOrsu AkhilFix related to schema resolve logicAt the moment schema references resolution happen recursively for json object type in json schema, through this MR have extended the support to handle schema reference resolutions which may arise in attributes of json object present in a...At the moment schema references resolution happen recursively for json object type in json schema, through this MR have extended the support to handle schema reference resolutions which may arise in attributes of json object present in a json array.
Closes [#35](https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/35)M3 - Release 0.5Orsu AkhilOrsu Akhilhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/69Audit Logger Implementation (GONRG-1547)2023-08-18T21:58:39ZArtem Dobrynin (EPAM)Audit Logger Implementation (GONRG-1547)# Description:
1.Added audit log events according to this issue: https://community.opengroup.org/osdu/platform/deployment-and-operations/audit-and-metrics/-/issues/13
2.Minor fix core libs conflict for gcp provider:
core-lib-gcp(0.1.15...# Description:
1.Added audit log events according to this issue: https://community.opengroup.org/osdu/platform/deployment-and-operations/audit-and-metrics/-/issues/13
2.Minor fix core libs conflict for gcp provider:
core-lib-gcp(0.1.15) uses methods that not present in core-common that leads to
(NoSuchMethodError: org.opengroup.osdu.core.common.http.HttpResponse.getAsJsonObject())
* Update core-lib-gcp from `0.1.15` to `0.3.25`
* Fix related methods & unit tests
# How to test:
Bussines cases (Schema Service API).
# Changes include:
* [ ] Refactor (a non-breaking change that improves code maintainability).
* [x] Bugfix (a non-breaking change that solves an issue).
* [x] New feature (a non-breaking change that adds functionality).
* [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
* [X] Core
* [x] GCP
* [ ] Azure
* [ ] AWS
* [ ] IBM
# Dev Checklist:
* [x] Added Unit Tests, wherever applicable.
* [ ] Updated the Readme, if applicable.
* [x] Existing Tests pass
* [x] Verified functionality locally
* [x] Self Reviewed my code for formatting and complex business logic.M3 - Release 0.5Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/58Slb multipartition for azure2023-08-18T22:00:25ZAbhishek Kumar (SLB)Slb multipartition for azureContributing the changes pertaining to multi-partition support of schema-service.
Now, schema-service depends on partition-service to interact with different partition depending upon the data-partition-id passed in the header.Contributing the changes pertaining to multi-partition support of schema-service.
Now, schema-service depends on partition-service to interact with different partition depending upon the data-partition-id passed in the header.M3 - Release 0.5ethiraj krishnamanaiduKishore Battulaethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/89CORS Fix Update2023-08-18T21:58:24ZSpencer Suttonsuttonsp@amazon.comCORS Fix UpdateM4 - Release 0.7ethiraj krishnamanaiduDania Kodeih (Microsoft)Wladmir FrazaoJoeDmitriy RudkoMatt Wiseethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/88ibm partition check added2023-08-18T21:58:25ZGokul Nagareibm partition check addedM4 - Release 0.7Anuj GuptaAnuj Guptahttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/87Aws integration only2023-08-18T21:58:27ZMatt WiseAws integration onlyM4 - Release 0.7ethiraj krishnamanaiduSpencer Suttonsuttonsp@amazon.comMatt Wiseethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/85Logging Enhancements for GCP modules (GONRG-1735, GONRG-1778)2023-08-18T21:58:28ZArtem Dobrynin (EPAM)Logging Enhancements for GCP modules (GONRG-1735, GONRG-1778)## Type of change
- [ ] Bug Fix
- [X] Feature
Some enhancements for GCP logging system. Supplies the logs with user id, data-partition-id and correlation id
https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/...## Type of change
- [ ] Bug Fix
- [X] Feature
Some enhancements for GCP logging system. Supplies the logs with user id, data-partition-id and correlation id
https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/-/issues/50 https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/pre-shipping/home/-/issues/49
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
The logs are printed in an unparseable inline format.
## What is the new/expected behavior?
The logs are printed in JSON format and compatible with GCP Logger.
## Have you added/updated Unit Tests and Integration Tests?
No, since it is not required for these changes
## Any other useful information
There are two types of logging: for local Spring profile (`spring.profiles.active=local`) and for any other, incl. default. This was done for better readability while testing locally.
The MR should be merged after https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/os-core-lib-gcp/-/merge_requests/10M4 - Release 0.7Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/84CORS Fix, AWS Updates2023-08-18T21:58:30ZSpencer Suttonsuttonsp@amazon.comCORS Fix, AWS UpdatesM4 - Release 0.7ethiraj krishnamanaiduDania Kodeih (Microsoft)Wladmir FrazaoJoeDmitriy Rudkoethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/83Adding exception handling in IsUnique checks2023-08-18T21:58:31ZAman VermaAdding exception handling in IsUnique checks## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it? YES
* [YES/NO] Does the MR contain pipeline/ helm chart related changes? NO
*...## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it? YES
* [YES/NO] Does the MR contain pipeline/ helm chart related changes? NO
* [YES/NO] I have updated the documentation accordingly. NA
* [YES/NO/NA] I have added tests to cover my changes. YES
* [YES/NO/NA] All new and existing tests passed. YES
* [YES/NO/NA] My code follows the code style of this project. YES
* [YES/NO/NA] I ran lint checks locally prior to submission. NA
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
POST calls to create schemas fail with NullPointerException while creating cosmos client
High level design:
In the `IsUnique` check, we iterate over each tenant to see whether the schema that we are trying to create is already present there. If the call to cosmos DB fails for one of the tenants, the entire loop crashed down without graceful handling.
Adding proper exception handling for such cases
PS: Why were the cosmos db calls failing?
There were some test tenants in `tenantInfo` table for which we did not have cosmos end point information populated in partition service. Hence for those test tenants the `cosmosCLient` generation would fail. This won't repro in all the environments because they might not have these incompletely populated tenants present there.
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
- Added a `try catch` block
- Added UTs
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
88%
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO]
NO
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->
cc: @harshit283 @osdu/platform/roles/azure-contributorsM4 - Release 0.7Aman VermaAman Vermahttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/101Deploy shared schema changes incorporated2023-08-18T21:58:11ZMANISH KUMARDeploy shared schema changes incorporatedM5 - Release 0.8MANISH KUMARMANISH KUMARhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/98update aws bootstrap to match new params in DeploySharedSchema.py2023-08-18T21:58:13ZMatt Wiseupdate aws bootstrap to match new params in DeploySharedSchema.pyM5 - Release 0.8Matt WiseMatt Wise