Indexer merge requestshttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests2023-11-27T19:07:40Zhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/647AWS - Merge from dev2023-11-27T19:07:40ZGuillaume CailletAWS - Merge from devAWS changes only. No changes on core logic or other CSP.
* Add or update License header
* Upgrade buildspec
* Update security configAWS changes only. No changes on core logic or other CSP.
* Add or update License header
* Upgrade buildspec
* Update security configM22 - Release 0.25Guillaume CailletGuillaume Caillethttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/646Setting default charset2023-11-28T07:44:45ZSabarish K R ESetting default charsetThe encoding depends on the host of the system where indexer is running, and it sometimes is not UTF-8.
This change ensures indexer runs with UTF-8 as the default charset so that it supports characters like ÆThe encoding depends on the host of the system where indexer is running, and it sometimes is not UTF-8.
This change ensures indexer runs with UTF-8 as the default charset so that it supports characters like Æhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/645logs for negative test case scenarios added2023-12-07T16:33:22ZHarshika Dhootlogs for negative test case scenarios addedvalidation for case scenarios in the azure M18 PR: [https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/636](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/636)
Ad...validation for case scenarios in the azure M18 PR: [https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/636](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/636)
Adding logs such that for any 4xx or 5xx calls we are able to trace the flow using correlation id, urls, http methods, which will enable us to perform the debugging faster and also at the places where we are blocked because of no correlation id
master merge pipeline link: [https://community.opengroup.org/osdu/platform/system/indexer-service/-/commit/88f0eb5938b76eb525f247191bd514560cd36056](https://community.opengroup.org/osdu/platform/system/indexer-service/-/commit/88f0eb5938b76eb525f247191bd514560cd36056)
Master parity link without my changes: [https://community.opengroup.org/osdu/platform/system/indexer-service/-/commit/1dcb603e784e03ce5ec825d8267cff254cd1a05a](https://community.opengroup.org/osdu/platform/system/indexer-service/-/commit/1dcb603e784e03ce5ec825d8267cff254cd1a05a)
work item ID: 28385M22 - Release 0.25Harshika DhootHarshika Dhoothttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/642Cherry-pick 'Full Upgrade of First Party Library Dependencies' into release/0.242023-10-19T10:29:28ZChad LeongCherry-pick 'Full Upgrade of First Party Library Dependencies' into release/0.24**Original MR**: !641
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !641
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/indexer-service/-/pipelines/new?ref=cherry-pick-for-641)M21 - Release 0.24David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/641Full Upgrade of First Party Library Dependencies2023-10-19T10:24:24ZChad LeongFull Upgrade of First Party Library DependenciesThis generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep all dependent libraries up to date.
This upgrade can be merged immediately without further approval if the C...This generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep all dependent libraries up to date.
This upgrade can be merged immediately without further approval if the CI pipeline reports success.
If this MR has failed, we need to work with the maintainers and affected provider teams to find a solution.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: 0d7aa6afb27b1b1ee89542eaaac134ddf73139fd
Maven: 0.25.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | -------------------------- | ---------------------- |
| core-lib-azure | 0.23.2 | 0.23.2 |
| core-lib-gc | 0.23.1 | |
| os-core-lib-aws | 0.24.0 | 0.23.0 |
| oqm | 0.23.0 | |
| os-core-common | 0.23.3 | 0.23.3 |
| os-core-lib-ibm | 0.23.0 | 0.23.0 |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.1, 2.13.2.2, 2.13.4.2 | 2.13.2.2, 2.13.2 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.11.1, 2.11.2, 2.13.0 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.11.2, 2.13.0 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.26, 1.27, 2.0 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade-2
SHA: 3f491d13510c3c1ed50e64f3c16ea171743578e5
Maven: 0.25.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | -------------------------- | ---------------------- |
| core-lib-azure | 0.24.0 | 0.24.0 |
| core-lib-gc | 0.24.0 | |
| os-core-lib-aws | 0.24.0 | 0.24.0 |
| oqm | 0.24.0 | |
| os-core-common | 0.24.0 | 0.24.0 |
| os-core-lib-ibm | 0.24.0 | 0.24.0 |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.1, 2.13.2.2, 2.13.4.2 | 2.13.2.2, 2.13.2 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.11.1, 2.11.2, 2.13.0 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.11.2, 2.13.0 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.26, 1.27, 2.0 |M21 - Release 0.24https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/640fix: disabling test cases for indexer extended2023-10-18T22:54:21ZKiran Veerapanenifix: disabling test cases for indexer extendedKiran VeerapaneniKiran Veerapanenihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/638Merged the latest changes on M162023-10-17T20:05:21ZMahsa HanifiMerged the latest changes on M16Mahsa HanifiMahsa Hanifihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/637fix: igonring experimental test cases2023-10-17T15:40:44ZKiran Veerapanenifix: igonring experimental test casesKiran VeerapaneniKiran Veerapanenihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/636logs for negative test case scenarios added2023-12-01T10:28:41ZHarshika Dhootlogs for negative test case scenarios addedAdding logs such that for any 4xx or 5xx calls we are able to trace the flow using correlation id, urls, http methods, which will enable us to perform the debugging faster and also at the places where we are blocked because of no correla...Adding logs such that for any 4xx or 5xx calls we are able to trace the flow using correlation id, urls, http methods, which will enable us to perform the debugging faster and also at the places where we are blocked because of no correlation id
ADME compatibility:
azure/m18-master: 6 failures: ![indexeradme](/uploads/b3c1a137fd8f23dfeac85c9617a24e4b/indexeradme.png)
this branch: 6 failures at similar places and causes, indicating no breaking change from this PR: ![indexeradme-m18](/uploads/0511fa529365e98d1b29b43c30428e31/indexeradme-m18.png)
identation for response and request body:![indexerindentation](/uploads/baf106a79e41d3ce400f0721b5bb2c96/indexerindentation.png)
4xx: ![indexe4xx](/uploads/6ce4df33db1ab2473cceab238753559e/indexe4xx.png)
5xx: ![indexe5xx](/uploads/913c526da87288288898d5914238f51a/indexe5xx.png)Harshika DhootHarshika Dhoothttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/635Remove SNAPSHOT dependencies2023-10-13T23:10:22ZDavid Diederichd.diederich@opengroup.orgRemove SNAPSHOT dependenciesThis automated MR removes usage of `SNAPSHOT` versions in the first party library dependencies.
Since `SNAPSHOT` dependencies change frequently -- by their nature -- usage of them across projects is dangerous and should be avoided.
### ...This automated MR removes usage of `SNAPSHOT` versions in the first party library dependencies.
Since `SNAPSHOT` dependencies change frequently -- by their nature -- usage of them across projects is dangerous and should be avoided.
### Dependency Information Before the Upgrade
```
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Branch: master
SHA: 0bc88510c245349dac7f6d46708c2655afa0021f
Maven: 0.24.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | -------------------------- | ---------------------- |
| core-lib-azure | 0.20.0-rc5 | 0.12.0-rc10 |
| core-lib-gc | 0.21.0 | |
| os-core-lib-aws | 0.24.0-SNAPSHOT | 0.21.0 |
| oqm | 0.21.0 | |
| os-core-common | 0.22.0-rc4 | 0.14.0-rc8 |
| os-core-lib-ibm | 0.16.0-rc1 | 0.13.0 |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.1, 2.13.2.2, 2.13.4.2 | 2.13.2.2, 2.13.2 |
| (3rd Party) net.minidev.json-smart | 2.4.7 | 2.4.6 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.11.1, 2.11.2, 2.13.0 |
| (3rd Party) org.apache.logging.log4j.log4j-core | 2.17.1 | 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-jul | 2.17.1 | 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.11.2, 2.13.0 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.26 |
### Dependency Information After the Upgrade
```
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Branch: dependency-upgrade-2
SHA: 90fde1cb02075f664cb1009879697e7a1908a1a0
Maven: 0.24.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | -------------------------- | ---------------------- |
| core-lib-azure | 0.20.0-rc5 | 0.12.0-rc10 |
| core-lib-gc | 0.21.0 | |
| os-core-lib-aws | 0.24.0 | 0.21.0 |
| oqm | 0.21.0 | |
| os-core-common | 0.22.0-rc4 | 0.14.0-rc8 |
| os-core-lib-ibm | 0.16.0-rc1 | 0.13.0 |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.1, 2.13.2.2, 2.13.4.2 | 2.13.2.2, 2.13.2 |
| (3rd Party) net.minidev.json-smart | 2.4.7 | 2.4.6 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.11.1, 2.11.2, 2.13.0 |
| (3rd Party) org.apache.logging.log4j.log4j-core | 2.17.1 | 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-jul | 2.17.1 | 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.11.2, 2.13.0 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.26 |M21 - Release 0.24https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/634Vulnerability fix for Indexer Service2023-10-13T15:21:53ZKamalika SahaVulnerability fix for Indexer ServiceComponent Governance Link : https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/_componentGovernance/oep-deployment-resources/alert/7626931?typeId=13890661&pipelinesTrackingFilter=0
Failed Pipeline link with CG name : http...Component Governance Link : https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/_componentGovernance/oep-deployment-resources/alert/7626931?typeId=13890661&pipelinesTrackingFilter=0
Failed Pipeline link with CG name : https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/_build/results?buildId=131859&view=logs&j=096205eb-4599-56da-affa-13246cc49eac
PBI link : https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/_workitems/edit/31229
Build Success Screenshot : ![image](/uploads/acb7e0e86ff9a806985536a2c4a52469/image.png)
Service IT Pipeline Link : https://dev.azure.com/OpenEnergyPlatform/Open%20Energy%20Platform/_build/results?buildId=132339&view=logs&j=18352b40-e004-552e-ac5b-401ff1a0f1fe&t=d1676cdd-0015-5643-b478-c4fe68a8be7d
Instance for which the Indexer image was pushed in : https://ms.portal.azure.com/#view/Microsoft_Azure_ContainerService/AksK8ResourceMenuBlade/~/overview-Deployment/aksClusterId/%2Fsubscriptions%2Fc99e2bf3-1777-412b-baba-d823676589c2%2FresourceGroups%2FCompute-rg-it1697053864288-jupwah%2Fproviders%2FMicrosoft.ContainerService%2FmanagedClusters%2Faks-bdftmqisqae2k/resource~/%7B%22kind%22%3A%22Deployment%22%2C%22metadata%22%3A%7B%22name%22%3A%22indexer%22%2C%22namespace%22%3A%22osdu-azure%22%2C%22uid%22%3A%2280490894-8de9-4474-88d0-162fbc485d53%22%7D%2C%22spec%22%3A%7B%22selector%22%3A%7B%22matchLabels%22%3A%7B%22app%22%3A%22indexer%22%7D%7D%7D%7DKamalika SahaKamalika Sahahttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/633[GONRG-7917] added replicas variable in helm2023-10-12T13:22:09ZAleksandr Primachenko [EPAM / GCP][GONRG-7917] added replicas variable in helmM21 - Release 0.24https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/632fix plexus version2023-10-12T18:47:49ZNathan Strelserfix plexus versionNathan StrelserNathan Strelserhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/631merge code to gitlab2023-10-12T10:20:11ZYunhua Koglinmerge code to gitlabadd unit testsadd unit testsM21 - Release 0.24Derek HudsonDerek Hudsonhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/630Change the default setting of trackTotalCount to false in SearchRequest2023-11-27T20:15:12ZZhibin MaiChange the default setting of trackTotalCount to false in SearchRequestChange the default setting of trackTotalCount to false in SearchRequest. Setting trackTotalCount to true could increase the overhead on ElasticSearch and affect the performance.Change the default setting of trackTotalCount to false in SearchRequest. Setting trackTotalCount to true could increase the overhead on ElasticSearch and affect the performance.M21 - Release 0.24Zhibin MaiZhibin Maihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/629fix azure io.netty, vulnerability and exclude document db2023-10-12T06:10:36ZVidyaDharani Lokamfix azure io.netty, vulnerability and exclude document db# Change details
* excluded unused dependency `documentdb-bulkexecutor`
* upgrade `io.netty:netty-bom` version to `4.1.98.Final`
# Changes in:
* [ ] GCP
* [x] Azure
* [ ] AWS
* [ ] IBM# Change details
* excluded unused dependency `documentdb-bulkexecutor`
* upgrade `io.netty:netty-bom` version to `4.1.98.Final`
# Changes in:
* [ ] GCP
* [x] Azure
* [ ] AWS
* [ ] IBMM21 - Release 0.24VidyaDharani LokamVidyaDharani Lokamhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/628resolve the bug of "/reindex/record API returns 500 when there is no not-foun...2023-10-13T11:53:33ZMingyang Zhuresolve the bug of "/reindex/record API returns 500 when there is no not-found records"fix the bug and add test cases to cover the scenariofix the bug and add test cases to cover the scenarioM21 - Release 0.24Mingyang ZhuMingyang Zhuhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/627Full Upgrade of First Party Library Dependencies2023-10-19T10:27:56ZChad LeongFull Upgrade of First Party Library DependenciesThis generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep all dependent libraries up to date.
This upgrade can be merged immediately without further approval if the C...This generated MR upgrades the first party libraries (other OSDU libraries) to utilize the latest release.
The intent is to keep all dependent libraries up to date.
This upgrade can be merged immediately without further approval if the CI pipeline reports success.
If this MR has failed, we need to work with the maintainers and affected provider teams to find a solution.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: 535c51c821fc58850e524e086159de696808ae12
Maven: 0.24.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | -------------------------- | ---------------------- |
| core-lib-azure | 0.20.0-rc5 | 0.12.0-rc10 |
| core-lib-gc | 0.21.0 | |
| os-core-lib-aws | 0.21.0 | 0.21.0 |
| oqm | 0.21.0 | |
| os-core-common | 0.22.0-rc4 | 0.14.0-rc8 |
| os-core-lib-ibm | 0.16.0-rc1 | 0.13.0 |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.1, 2.13.2.2, 2.13.4.2 | 2.13.2.2, 2.13.2 |
| (3rd Party) net.minidev.json-smart | 2.4.7 | 2.4.6 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.11.1, 2.11.2, 2.13.0 |
| (3rd Party) org.apache.logging.log4j.log4j-core | 2.17.1 | 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-jul | 2.17.1 | 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.11.2, 2.13.0 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.26 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade
SHA: 84c217f183243c2c0736eae6ad0f6e57ecf386c5
Maven: 0.24.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | -------------------------- | ---------------------- |
| core-lib-azure | 0.23.2 | 0.23.2 |
| core-lib-gc | 0.23.1 | |
| os-core-lib-aws | 0.23.0 | 0.23.0 |
| oqm | 0.23.0 | |
| os-core-common | 0.23.3 | 0.23.3 |
| os-core-lib-ibm | 0.23.0 | 0.23.0 |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.1, 2.13.2.2, 2.13.4.2 | 2.13.2.2, 2.13.2 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.11.1, 2.11.2, 2.13.0 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.11.2, 2.13.0 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.26, 1.27, 2.0 |M21 - Release 0.24Chad LeongChad Leonghttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/626[GONRG-7820] - Added liveness health check2023-10-05T10:36:31ZAnatolii Hrychaniuk[GONRG-7820] - Added liveness health checkM21 - Release 0.24https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/625Rename some classes and variables to improve readable2023-10-04T07:56:46ZZhibin MaiRename some classes and variables to improve readableThis is part of the efforts to improve the readability of the augmenter related codes. The major changes is to rename some classes and variables via IDE tool. It also includes two minor changes:
- Reduce redundant codes in class Property...This is part of the efforts to improve the readability of the augmenter related codes. The major changes is to rename some classes and variables via IDE tool. It also includes two minor changes:
- Reduce redundant codes in class PropertyPathDeserializer.class
- Use OSDU kind pattern to verify the Code property in class AugmenterConfiguration.classM21 - Release 0.24Zhibin MaiZhibin Mai