Indexer merge requestshttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests2023-08-18T15:18:52Zhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/25increase spring boot version;2023-08-18T15:18:52ZYauheni Lesnikauincrease spring boot version;Updating spring boot version from 2.1.7.RELEASE to 2.1.16.RELEASEUpdating spring boot version from 2.1.7.RELEASE to 2.1.16.RELEASEM1 - Release 0.1ethiraj krishnamanaiduNitin-slbNeelesh ThakurSherman YangAliaksei DarafeyeuYauheni Lesnikauethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/24Added Support for Azure AKS Pipelines2023-08-18T15:18:53ZDaniel SchollAdded Support for Azure AKS PipelinesM1 - Release 0.1Daniel SchollDaniel Schollhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/22Fix vulnerabilities in dependencies found in result of scqp checking2023-08-18T15:18:55ZYauheni LesnikauFix vulnerabilities in dependencies found in result of scqp checkingFix next vulnerabilities in dependencies found in result of scqp checking for core and azure modules
```
CORE:
| Library | ...Fix next vulnerabilities in dependencies found in result of scqp checking for core and azure modules
```
CORE:
| Library | |
|----------------------------------------|----------------------------------------------------------------------------
|
| jackson-databind-2.9.9.jar |
|
| hibernate-validator-6.0.17.Final.jar | |
| snakeyaml-1.23.jar | |
| commons-codec-1.11.jar | |
| tomcat-embed-core-9.0.21.jar | |
| netty-codec-4.1.38.Final.jar | | | commons-beanutils-1.9.3.jar |
AZURE:
| Library | |
|--------------------------------------|----------------------------------------------------------------------------
|
| jackson-databind-2.9.9.jar |
|
| netty-codec-http-4.1.38.Final.jar | |
| reactor-netty-0.8.10.RELEASE.jar | |
| netty-codec-http2-4.1.38.Final.jar | |
| hibernate-validator-6.0.12.Final.jar | |
| snakeyaml-1.23.jar | |
| commons-codec-1.11.jar | |
| tomcat-embed-core-9.0.22.jar | |
| netty-codec-4.1.38.Final.jar | |
| log4j-core-2.11.2.jar | | | bcprov-jdk15on-1.60.jar |
|
| commons-beanutils-1.9.3.jar |
|
| nimbus-jose-jwt-6.0.2.jar |
|
|jackson-mapper-asl-1.9.13.jar |
```M1 - Release 0.1ethiraj krishnamanaiduNitin-slbNeelesh ThakurPavel BachylaAliaksei DarafeyeuYauheni Lesnikauethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/21AWS Ecs impl changes2023-08-18T15:18:57ZMatt WiseAWS Ecs impl changescommit ff2e37a9
Author: Rucha deshpande <deshruch@amazon.com>
Date: Thu Aug 13 2020 12:20:42 GMT-0500 (Central Daylight Time)
Adding appropriate copyright statement
commit 3abe2d51
Author: Rucha deshpande <deshruch@amazon.com>...commit ff2e37a9
Author: Rucha deshpande <deshruch@amazon.com>
Date: Thu Aug 13 2020 12:20:42 GMT-0500 (Central Daylight Time)
Adding appropriate copyright statement
commit 3abe2d51
Author: Rucha deshpande <deshruch@amazon.com>
Date: Mon Aug 10 2020 08:58:06 GMT-0500 (Central Daylight Time)
build command update to build aws impl only
commit b656234a
Author: Rucha deshpande <deshruch@amazon.com>
Date: Sun Aug 09 2020 16:55:19 GMT-0500 (Central Daylight Time)
entitlements on ecs updateM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/19Aws integration2023-08-18T15:18:59ZMatt WiseAws integrationM1 - Release 0.1https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/17Update version of os-core-common and fix broken integration tests2023-08-18T15:19:01ZHong YanUpdate version of os-core-common and fix broken integration tests## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
[[System/Storage] Bulk Update for ACLs on Records](https://community.opengroup.org/osdu/platform/system/storage/...## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
[[System/Storage] Bulk Update for ACLs on Records](https://community.opengroup.org/osdu/platform/system/storage/-/issues/10)
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
The os-core-common lib is outdated
## What is the new/expected behavior?
No behavior changes.
## Have you added/updated Unit Tests and Integration Tests?
os-core-common lib is updated to the newest version. Integration test is updated to include the newly added Access-Control-Allow-Methods "PATCH" in os-core-common.
## Any other useful informationM1 - Release 0.1ethiraj krishnamanaiduNitin-slbNeelesh ThakurAlok Joshiethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/15added possibility to have elastic connectivity over HTTP through config.2023-08-18T15:19:03ZHema Vishnu Pola [Microsoft]added possibility to have elastic connectivity over HTTP through config.## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES] I have updated the documentation accordingly.
* [YES/NO/NA] I have add...## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES] I have updated the documentation accordingly.
* [YES/NO/NA] I have added tests to cover my changes.
* [YES/NO/NA] All new and existing tests passed.
## Current Behavior or Linked Issues
-------------------------------------
Issue details: https://community.opengroup.org/osdu/platform/system/home/-/issues/39
## Does this introduce a breaking change?
-------------------------------------
- [NO]
This doesn't bring in any breaking changes.
## Other information
-------------------------------------
Required core lib to be merged appropriately.
- Tests update to have elastic search connectivity through HTTP.
- POM update to include new azure lib version.
Note: Added the dependent MRs in the "Merge request dependencies" section seen below. (Seen as "Depends on 'n' merge requests..." ----------- Expand below).M1 - Release 0.1Hema Vishnu Pola [Microsoft]Hema Vishnu Pola [Microsoft]https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/13fix: don't log stack traces for missing record properties2023-08-18T15:19:04ZBrandt Bealfix: don't log stack traces for missing record propertiesWhen indexer looks for properties on a storage record and a property can't be found an exception is thrown which is logged. Since the log statement includes the original exception the stack trace is unwound creating a performance issue (...When indexer looks for properties on a storage record and a property can't be found an exception is thrown which is logged. Since the log statement includes the original exception the stack trace is unwound creating a performance issue (cpu/network) which essentially kills underpowered containers. The exception stack trace doesn't provide any additional information and by removing it fixes the performance issue for current data ingestion workloads.
`o.o.o.c.common.logging.DefaultLogWriter : indexer.app: record-id: opendes:doc:b55f7ee408354a7b9b48cd2b443fbf16 | error fetching property: Data.IndividualTypeProperties.Curves.InterpreterName | error: Unknown property 'InterpreterName' on class 'class java.util.ArrayList' {correlation-id=16b8f17d-3ded-462d-9c14-83204bf45fb9, data-partition-id=opendes}`M1 - Release 0.1Dania Kodeih (Microsoft)Brandt BealDaniel SchollDmitriy RudkoAlan BrazDania Kodeih (Microsoft)https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/11Azure - (4/4) - Adding missing context path for indexer2020-07-16T23:03:22ZDuvelis CaraoAzure - (4/4) - Adding missing context path for indexer**This only impacts Azure implementation.**
This MR is part of group of MRs with dependencies between them
0. Update infrastructure (Azure / osdu-infrastructure) to have reference to the new URLs (https://github.com/Azure/osdu-infr...**This only impacts Azure implementation.**
This MR is part of group of MRs with dependencies between them
0. Update infrastructure (Azure / osdu-infrastructure) to have reference to the new URLs (https://github.com/Azure/osdu-infrastructure/pull/23)
1. Adding missing context path for entitlements.
2. Adding missing context path for legal.
3. Adding missing context path for storage.
4. Adding missing context path for indexer
After merging this MR, the new indexer URL will be:
https://{host}**/api/indexer/v2/**
Before merging this MR:
- Please make sure the dependent MR is done: https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/33
--------
This PR is related to 6 other requests for changing a URL structure and adding in context paths for services.
1. osdu-infrastructure 1/6 PR https://github.com/Azure/osdu-infrastructure/pull/23
1. CI-CD Pipelines 2/6 This PR
1. Entitlements Azure 3/6 https://community.opengroup.org/osdu/platform/security-and-compliance/entitlements-azure/-/merge_requests/11
1. Legal 4/6 https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/merge_requests/32
1. Storage 5/6 https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/33
1. Indexer 6/6 https://community.opengroup.org/osdu/platform/system/indexer-service
They are all required and any one but not the other will break builds and tests across the entire ecosystem for Azure.M1 - Release 0.1ethiraj krishnamanaiduDania Kodeih (Microsoft)Daniel Schollethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/9OpenDES GA code merge2023-08-18T15:19:06Zethiraj krishnamanaiduOpenDES GA code merge* Enable special character search
* Fix Racing condition
* Add support for indexing primitives array
* Query API not finding all records
* Fix primitive array parsing
* 5XX high error rate triggers on search regularly* Enable special character search
* Fix Racing condition
* Add support for indexing primitives array
* Query API not finding all records
* Fix primitive array parsing
* 5XX high error rate triggers on search regularlyM1 - Release 0.1ethiraj krishnamanaiduBrandt BealDaniel SchollDmitriy RudkoAlan Brazethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/8Add Azure back in2023-08-18T15:19:08ZDavid Diederichd.diederich@opengroup.orgAdd Azure back inThis was a botched merge as part of !5, which brought in the AWS changes.This was a botched merge as part of !5, which brought in the AWS changes.M1 - Release 0.1https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/7Add IBM Integration Tests to the pipeline2023-08-18T15:19:09ZDavid Diederichd.diederich@opengroup.orgAdd IBM Integration Tests to the pipelineSquashed commits from trusted-ibm to avoid clutter in the repository
This follows on from !6Squashed commits from trusted-ibm to avoid clutter in the repository
This follows on from !6M1 - Release 0.1https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/5Incorporate AWS Integration Tests2023-08-18T15:19:11ZDavid Diederichd.diederich@opengroup.orgIncorporate AWS Integration TestsThis adds the trusted-aws branch, which is where the integration tests have been developed for Indexer.
I had only a minor hand in this -- mostly related to the pipeline configuration near the end. Note the usage pattern of the separate...This adds the trusted-aws branch, which is where the integration tests have been developed for Indexer.
I had only a minor hand in this -- mostly related to the pipeline configuration near the end. Note the usage pattern of the separate runner; this runner is protected and locked to a few projects, and runs within the AWS VPC to have access to the elastic search services.M1 - Release 0.1https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/3Enable Azure Cloud Provider Builds and Integration Tests for service2023-08-18T15:19:13ZDaniel SchollEnable Azure Cloud Provider Builds and Integration Tests for serviceThis change enables a successful build, deploy and test for the Azure Cloud Provider.
1. Change the gitlab-ci.yml file to add in azure. (Ref maintained as master to allow for easier flipping to isolated branches as other changes come o...This change enables a successful build, deploy and test for the Azure Cloud Provider.
1. Change the gitlab-ci.yml file to add in azure. (Ref maintained as master to allow for easier flipping to isolated branches as other changes come online.
2. NOTICE -- New notice generated
3. Azure Pipeline -- Changes to the azure pipeline to support ADO builds.
4. README.md. -- Changes to Azure Provider Readme for deployments.
5. Community Maven Settings -- Add in server for Azure DeploymentM1 - Release 0.1Daniel SchollDaniel Schollhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/87GCP release/0.5 - Fix containerize CICD step2023-08-18T15:09:00ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comGCP release/0.5 - Fix containerize CICD stepM3 - Release 0.5Dmitriy RudkoRiabokon Stanislav(EPAM)[GCP]Dmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/83Fix Int tests failing due to new storage record ID validation rules2023-08-18T15:09:03ZMatt WiseFix Int tests failing due to new storage record ID validation rulesM3 - Release 0.5ethiraj krishnamanaiduMatt Wiseethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/79Azure: removing tokens from logs2023-08-18T15:09:07ZJasonAzure: removing tokens from logsCurrently Azure test step has logging set to DEBUG level. This leads to tokens being logged. This MR sets logging to INFO level, which means that tokens no longer will get logged.Currently Azure test step has logging set to DEBUG level. This leads to tokens being logged. This MR sets logging to INFO level, which means that tokens no longer will get logged.M3 - Release 0.5JasonJasonhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/76Gcp full reindex (GONRG-1534)2023-08-18T15:09:12ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comGcp full reindex (GONRG-1534)# Description:
As a part of the migration to a new Elastic 7 we need some functionality <br/>
that will allow us to do full reindex of Records registered in Storage service.<br/>
Added new endpoint: PATCH /api/indexer/v2/reindex?force_cl...# Description:
As a part of the migration to a new Elastic 7 we need some functionality <br/>
that will allow us to do full reindex of Records registered in Storage service.<br/>
Added new endpoint: PATCH /api/indexer/v2/reindex?force_clean=false<br/>
- Get all kind from Storage
- Clean all old indexes in Elastic
- Iterate over kinds and do reindex
# How to test:
Functionality was tested locally, with local elastic(v6 & v7)
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [x] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [x] Azure
- [x] AWS
- [x] IBM
# Dev Checklist:
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.
# Other comments:
Functionality was implemented with reusing existing logic for reindex of a single kind,
to reduce common code changes, thus due to architectural features, request cannot be executed
asynchronously, and response may be detained.M3 - Release 0.5Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/74Audit Logger Implementation (GONRG-1546)2023-08-18T15:09:14ZArtem Dobrynin (EPAM)Audit Logger Implementation (GONRG-1546)# Description:
Added audit log events according to this issue: https://community.opengroup.org/osdu/platform/deployment-and-operations/audit-and-metrics/-/issues/6
# How to test:
Bussines cases (Indexer Service API).
# Changes includ...# Description:
Added audit log events according to this issue: https://community.opengroup.org/osdu/platform/deployment-and-operations/audit-and-metrics/-/issues/6
# How to test:
Bussines cases (Indexer Service API).
# Changes include:
* [ ] Refactor (a non-breaking change that improves code maintainability).
* [ ] Bugfix (a non-breaking change that solves an issue).
* [x] New feature (a non-breaking change that adds functionality).
* [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
* [X] Core
* [ ] GCP
* [ ] Azure
* [ ] AWS
* [ ] IBM
# Dev Checklist:
* [x] Added Unit Tests, wherever applicable.
* [ ] Updated the Readme, if applicable.
* [x] Existing Tests pass
* [x] Verified functionality locally
* [x] Self Reviewed my code for formatting and complex business logic.M3 - Release 0.5Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/113Azure || Pre-Scale Indexer to match Storage2023-08-18T15:08:39ZKrishna Nikhil VedurumudiAzure || Pre-Scale Indexer to match Storage## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provi...## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [x] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
The min of 2 containers are not enough to handle the load that 10 storage service containers can handle.
## What is the new/expected behavior?
Pre-scaling the Indexer service to 10 containers.
Note: The values can be fine-tuned once container bench marking is done for Indexer Service
## Have you added/updated Unit Tests and Integration Tests?
N/A
## Any other useful informationM4 - Release 0.7