Schema merge requestshttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests2023-08-18T22:00:31Zhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/52[azure] defining istio auth policies in the schema repo2023-08-18T22:00:31ZAman Verma[azure] defining istio auth policies in the schema repo## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it? YES
* [YES/NO] I have updated the documentation accordingly. N/A
* [YES/NO/NA...## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it? YES
* [YES/NO] I have updated the documentation accordingly. N/A
* [YES/NO/NA] I have added tests to cover my changes. N/A
* [YES/NO/NA] All new and existing tests passed. N/A
* [YES/NO/NA] My code follows the code style of this project. N/A
* [YES/NO/NA] I ran lint checks locally prior to submission. N/A
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
Moving and updating the yaml file defining istio auth policies from aure-infra repo to Schema service repository.
High level design:
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
See https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/82
Issue: https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/51
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO]
NO
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
The istio related file for schema service will be removed from azure-infra-provisioning repo post this merge.
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->
cc: @polavishnu , @harshit283 , @kibattulM1 - Release 0.1Aman VermaAman Vermahttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/53Dependency Management & Azure SDK Upgrade2023-08-18T22:00:30ZAbhishek PatilDependency Management & Azure SDK Upgrade## All Submissions:
-------------------------------------
* [YES] Have you followed our code review [guidelines](https://github.com/microsoft/code-with-engineering-playbook/blob/master/pull-requests/code-reviews/readme.md)?
* [YES] Have ...## All Submissions:
-------------------------------------
* [YES] Have you followed our code review [guidelines](https://github.com/microsoft/code-with-engineering-playbook/blob/master/pull-requests/code-reviews/readme.md)?
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES] I have updated the documentation accordingly.
* [NA] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [NO] I ran lint checks locally prior to submission.
## What is the current behavior?
-------------------------------------
- Old version of Azure SDKs are used in service.
- Versions of dependencies which are common across all OSDU services are not consistent/same in all OSDU services.<!-- Please describe the current behavior that you are modifying, or link to a relevant issue. -->
## What is the new behavior?
-------------------------------------
- All Azure SDKs are upgraded to latest stable version.
- Common dependencies are inherited from core-lib-azure as managed dependencies. This will make sure that version info of those dependencies are inherited from core-lib-azure. All OSDU services using same version of core-lib-azure will automatically use same versions for all those dependencies.
- Changes to support new CosmosStore class & azure-cosmos sdk version.
- Changes to make service work for single partition (opendes) and bypass partition service.<!-- Please describe the behavior or changes that are being added by this PR. -->
## Does this introduce a breaking change?
-------------------------------------
- [NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Any relevant logs, error output, etc?
-------------------------------------
(If it’s long, please paste to https://ghostbin.com/ and insert the link here.)
## Other information
-------------------------------------
<!-- Any other information that is important to this PR such as screenshots of how the component looks before and after the change. -->M1 - Release 0.1Abhishek PatilAbhishek Patilhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/54allow usage of os-core-lib-aws from GL2023-08-18T22:00:28ZRucha Deshpandeallow usage of os-core-lib-aws from GLM1 - Release 0.1Rucha DeshpandeRucha Deshpandehttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/56Gonrg 1131 integration tests for schema2023-08-18T22:00:27ZOleksii Tsyganov (EPAM)Gonrg 1131 integration tests for schemaM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/58Slb multipartition for azure2023-08-18T22:00:25ZAbhishek Kumar (SLB)Slb multipartition for azureContributing the changes pertaining to multi-partition support of schema-service.
Now, schema-service depends on partition-service to interact with different partition depending upon the data-partition-id passed in the header.Contributing the changes pertaining to multi-partition support of schema-service.
Now, schema-service depends on partition-service to interact with different partition depending upon the data-partition-id passed in the header.M3 - Release 0.5ethiraj krishnamanaiduKishore Battulaethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/59Update prepare-dist.sh2023-08-18T22:00:24ZMatt WiseUpdate prepare-dist.shM1 - Release 0.1Matt WiseMatt Wisehttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/60Externalizing the shared partition name2023-08-18T22:00:22ZAbhishek Kumar (SLB)Externalizing the shared partition nameNow, shared partition name is no more tightly coupled as "common". Each vendor can now name it differently.Now, shared partition name is no more tightly coupled as "common". Each vendor can now name it differently.M1 - Release 0.1ethiraj krishnamanaiduDania Kodeih (Microsoft)Wladmir FrazaoJoeDmitriy Rudkoethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/61Feat/aws impl2023-08-18T22:00:20ZRucha DeshpandeFeat/aws implRemoved pom files added to schema-test-aws which are not required.
Updated schema-test-core/pom.xml - The integration-test goal is used thrice which ends up running the integration tests thrice.Removed pom files added to schema-test-aws which are not required.
Updated schema-test-core/pom.xml - The integration-test goal is used thrice which ends up running the integration tests thrice.M1 - Release 0.1ethiraj krishnamanaiduJoeRucha Deshpandeethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/62Gcp fix sonar comments (GONRG-1376)2023-08-18T22:00:19ZAnastasiia GelmutGcp fix sonar comments (GONRG-1376)# Description:
SonarLint comments were fixed. Non-breakable refactoring.
# How to test:
mvn clean install
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking chan...# Description:
SonarLint comments were fixed. Non-breakable refactoring.
# How to test:
mvn clean install
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
# Dev Checklist:
- [x] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M1 - Release 0.1Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/64Adding Azure pipelines2023-08-18T22:00:17Zharshit aggarwalAdding Azure pipelinesThis MR is making the changes to add the pipelines for DEV and DEMO environment
All the changes are tested on DEV/DEMO environment and pipeline is passing
Test Run on DEV -> https://dev.azure.com/osdu-demo/OSDU_Rx/_build/results?buildI...This MR is making the changes to add the pipelines for DEV and DEMO environment
All the changes are tested on DEV/DEMO environment and pipeline is passing
Test Run on DEV -> https://dev.azure.com/osdu-demo/OSDU_Rx/_build/results?buildId=64436&view=results
Test Run on DEMO -> https://dev.azure.com/osdu-demo/osdu-mvp/_build/results?buildId=64469&view=resultsM1 - Release 0.1harshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/65OSDU-GCP: parametrize containerization stage2023-08-18T22:00:16ZAliaksandr Ramanovich (EPAM)OSDU-GCP: parametrize containerization stageOSDU-GCP: parametrize containerization stage to remove hardcode in image pathOSDU-GCP: parametrize containerization stage to remove hardcode in image pathM1 - Release 0.1Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/67Fixed Replica Count2023-08-18T21:59:59ZDaniel SchollFixed Replica CountUpdating Replica Count to 2 and Readme clarification.Updating Replica Count to 2 and Readme clarification.M1 - Release 0.1Daniel SchollDaniel Schollhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/68latest ibm changes2023-08-18T21:58:41ZBhushan Radelatest ibm changeslatest ibm changeslatest ibm changesM1 - Release 0.1Anuj GuptaAnuj Guptahttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/69Audit Logger Implementation (GONRG-1547)2023-08-18T21:58:39ZArtem Dobrynin (EPAM)Audit Logger Implementation (GONRG-1547)# Description:
1.Added audit log events according to this issue: https://community.opengroup.org/osdu/platform/deployment-and-operations/audit-and-metrics/-/issues/13
2.Minor fix core libs conflict for gcp provider:
core-lib-gcp(0.1.15...# Description:
1.Added audit log events according to this issue: https://community.opengroup.org/osdu/platform/deployment-and-operations/audit-and-metrics/-/issues/13
2.Minor fix core libs conflict for gcp provider:
core-lib-gcp(0.1.15) uses methods that not present in core-common that leads to
(NoSuchMethodError: org.opengroup.osdu.core.common.http.HttpResponse.getAsJsonObject())
* Update core-lib-gcp from `0.1.15` to `0.3.25`
* Fix related methods & unit tests
# How to test:
Bussines cases (Schema Service API).
# Changes include:
* [ ] Refactor (a non-breaking change that improves code maintainability).
* [x] Bugfix (a non-breaking change that solves an issue).
* [x] New feature (a non-breaking change that adds functionality).
* [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
* [X] Core
* [x] GCP
* [ ] Azure
* [ ] AWS
* [ ] IBM
# Dev Checklist:
* [x] Added Unit Tests, wherever applicable.
* [ ] Updated the Readme, if applicable.
* [x] Existing Tests pass
* [x] Verified functionality locally
* [x] Self Reviewed my code for formatting and complex business logic.M3 - Release 0.5Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/71Fix related to schema resolve logic2023-08-18T21:58:38ZOrsu AkhilFix related to schema resolve logicAt the moment schema references resolution happen recursively for json object type in json schema, through this MR have extended the support to handle schema reference resolutions which may arise in attributes of json object present in a...At the moment schema references resolution happen recursively for json object type in json schema, through this MR have extended the support to handle schema reference resolutions which may arise in attributes of json object present in a json array.
Closes [#35](https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/35)M3 - Release 0.5Orsu AkhilOrsu Akhilhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/75adding load sequence parameter in bootstrapping scripts2023-08-18T21:58:36ZAman Vermaadding load sequence parameter in bootstrapping scriptsadding the `loadSequence` parameter as prescribed here: https://community.opengroup.org/osdu/platform/system/schema-service/-/blob/master/deployments/shared-schemas/README.md
Issue:
https://community.opengroup.org/osdu/platform/system/s...adding the `loadSequence` parameter as prescribed here: https://community.opengroup.org/osdu/platform/system/schema-service/-/blob/master/deployments/shared-schemas/README.md
Issue:
https://community.opengroup.org/osdu/platform/system/schema-service/-/issues/33M3 - Release 0.5Aman VermaAman Vermahttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/77Bump lib version to fix white source issues2023-08-18T21:58:34Zdevesh bajpaiBump lib version to fix white source issuesBump lib version to fix white source issuesBump lib version to fix white source issuesM6 - Release 0.9https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/78Fix AWS int tests internal failure. Update AWS bootstrap for use with intern...2023-08-18T21:58:33ZMatt WiseFix AWS int tests internal failure. Update AWS bootstrap for use with internal CI.M3 - Release 0.5ethiraj krishnamanaiduRucha DeshpandeMatt Wiseethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/83Adding exception handling in IsUnique checks2023-08-18T21:58:31ZAman VermaAdding exception handling in IsUnique checks## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it? YES
* [YES/NO] Does the MR contain pipeline/ helm chart related changes? NO
*...## All Submissions:
-------------------------------------
* [YES/NO] I have added an explanation of what changes in this merge do and why we should include it? YES
* [YES/NO] Does the MR contain pipeline/ helm chart related changes? NO
* [YES/NO] I have updated the documentation accordingly. NA
* [YES/NO/NA] I have added tests to cover my changes. YES
* [YES/NO/NA] All new and existing tests passed. YES
* [YES/NO/NA] My code follows the code style of this project. YES
* [YES/NO/NA] I ran lint checks locally prior to submission. NA
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
POST calls to create schemas fail with NullPointerException while creating cosmos client
High level design:
In the `IsUnique` check, we iterate over each tenant to see whether the schema that we are trying to create is already present there. If the call to cosmos DB fails for one of the tenants, the entire loop crashed down without graceful handling.
Adding proper exception handling for such cases
PS: Why were the cosmos db calls failing?
There were some test tenants in `tenantInfo` table for which we did not have cosmos end point information populated in partition service. Hence for those test tenants the `cosmosCLient` generation would fail. This won't repro in all the environments because they might not have these incompletely populated tenants present there.
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
- Added a `try catch` block
- Added UTs
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
88%
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO]
NO
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->
cc: @harshit283 @osdu/platform/roles/azure-contributorsM4 - Release 0.7Aman VermaAman Vermahttps://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/84CORS Fix, AWS Updates2023-08-18T21:58:30ZSpencer Suttonsuttonsp@amazon.comCORS Fix, AWS UpdatesM4 - Release 0.7ethiraj krishnamanaiduDania Kodeih (Microsoft)Wladmir FrazaoJoeDmitriy Rudkoethiraj krishnamanaidu