Indexer merge requestshttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests2024-03-26T09:46:33Zhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/725Indexer service should not reschedule failed events when the schema is not va...2024-03-26T09:46:33ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comIndexer service should not reschedule failed events when the schema is not valid.(GONRG-9505)# Description:
Schema not valid exception will cause an event to be skipped instead of retried since retrying such events is ineffective and wastes resources because schema cannot be updated automatically, schema issue should be fixed fi...# Description:
Schema not valid exception will cause an event to be skipped instead of retried since retrying such events is ineffective and wastes resources because schema cannot be updated automatically, schema issue should be fixed first.
# How to test:
IT
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GC
- [x] Community Implementation
# Dev Checklist:
- [x] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.
# Other comments:
Any comments to approvers hereM23 - Release 0.26Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/723[GONRG-9422]: Remove springProfilesActive explicit setting2024-03-20T12:37:25ZMykola Ronik [EPAM / GCP][GONRG-9422]: Remove springProfilesActive explicit settingM23 - Release 0.26Mykola Ronik [EPAM / GCP]Mykola Ronik [EPAM / GCP]https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/721[GONRG-9211] Added labels to gc helm2024-03-19T17:11:08ZDanylo Vanin (EPAM)[GONRG-9211] Added labels to gc helmM23 - Release 0.26Danylo Vanin (EPAM)Danylo Vanin (EPAM)https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/719[#MSCOSDU-35931] upgrade : core-lib-azure Java 17 version2024-03-06T14:13:23ZThulasi Dass Subramanian[#MSCOSDU-35931] upgrade : core-lib-azure Java 17 version## Change details
* For Azure only: upgraded `core-lib-azure` version to `0.26.0-rc5` for Java 17 updates both in source & testing modules
## Changes in:
* [ ] GCP
* [x] Azure
* [ ] AWS
* [ ] IBM## Change details
* For Azure only: upgraded `core-lib-azure` version to `0.26.0-rc5` for Java 17 updates both in source & testing modules
## Changes in:
* [ ] GCP
* [x] Azure
* [ ] AWS
* [ ] IBMM23 - Release 0.26Thulasi Dass SubramanianThulasi Dass Subramanianhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/718Added feature for GC Int tests2024-03-05T12:29:44ZSiarhei Poliak [EPAM / GCP]Added feature for GC Int testsM23 - Release 0.26Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/716Enhance the Augmenter documentation2024-03-08T16:48:00ZZhibin MaiEnhance the Augmenter documentationUpdate Augmenter Document from the following aspects:
* Update introduction section
* Update use cases
* Remove all the `data.` prefix in ValuePath
* Make sure the `Policy` and `RelationshipDirection` are set correctlyUpdate Augmenter Document from the following aspects:
* Update introduction section
* Update use cases
* Remove all the `data.` prefix in ValuePath
* Make sure the `Policy` and `RelationshipDirection` are set correctlyM23 - Release 0.26Zhibin MaiZhibin Maihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/715Aws side code quality2024-02-26T20:24:57ZBruce JinAws side code qualityThis MR will improve AWS code quality, fix some codesmellThis MR will improve AWS code quality, fix some codesmellM23 - Release 0.26Bruce JinBruce Jinhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/714Added some methods for int tests2024-02-26T16:33:53ZRiabokon Stanislav(EPAM)[GCP]Added some methods for int testsAdded some methods for int testsAdded some methods for int testsM23 - Release 0.26Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/713Fix the issue that augmenter throws null pointer exception when casting the r...2024-02-26T14:54:23ZZhibin MaiFix the issue that augmenter throws null pointer exception when casting the related object ids retrieved from the GeoContextThe bug was introduced in M21. For the detail of the issue, please refer to issue [151](https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/151).The bug was introduced in M21. For the detail of the issue, please refer to issue [151](https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/151).M23 - Release 0.26Zhibin MaiZhibin Maihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/711Update os-core-lib-aws2024-03-19T17:11:09ZRucha DeshpandeUpdate os-core-lib-awsUpdate os-core-lib-awsUpdate os-core-lib-awsM23 - Release 0.26https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/710Vulnerbility fix on AWS side2024-03-19T17:11:10ZBruce JinVulnerbility fix on AWS sidethis will fix some AWS internal vulnerabilitiesthis will fix some AWS internal vulnerabilitiesM23 - Release 0.26Bruce JinBruce Jinhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/709[MS-34266] fix jetty vulnerability2024-02-12T15:17:57ZVidyaDharani Lokam[MS-34266] fix jetty vulnerability# Change details
* issue reference: https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/145
https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/144
* updated `jetty` dependency to `6.1...# Change details
* issue reference: https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/145
https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/144
* updated `jetty` dependency to `6.1.26` remediate vulnerability.
* excluded `jetty` from ibm csp as it is unused in it.
# Changes in:
* [x] Aws
* [x] Azure
* [x] GCP
* [x] IBM
* [x] Common codeM23 - Release 0.26VidyaDharani LokamVidyaDharani Lokamhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/708fix aws entitlement url2024-02-08T15:46:32ZYunhua Koglinfix aws entitlement urlM23 - Release 0.26Yunhua KoglinYunhua Koglinhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/706Avoid unnecessary second call to get the properties from the VirtualPropertie...2024-01-30T14:43:55ZZhibin MaiAvoid unnecessary second call to get the properties from the VirtualProperties cacheIt is minor enhancement of MR [699](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/699):
- Should not skip geometry decimation even the virtual properties does not exist which is possible for cust...It is minor enhancement of MR [699](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/699):
- Should not skip geometry decimation even the virtual properties does not exist which is possible for custom schema. Please note that geometry decimation is not only on virtual properties.
The MR also includes two minor enhancement:
- Avoid calling the cache service twice unnecessarily
- Implement and use partition safe virtual properties cache.M23 - Release 0.26Zhibin MaiZhibin Maihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/705Indexer-core-plus service2024-03-26T10:31:37ZJoshua HindmonIndexer-core-plus serviceCreation of new indexer-core-plus for Community Implementation.
(Work in progress, not ready to merge.)
## Type of change
* [ ] Initial commit (a change for the initial development of a project)
* [x] Refactor (a non-breaking change...Creation of new indexer-core-plus for Community Implementation.
(Work in progress, not ready to merge.)
## Type of change
* [ ] Initial commit (a change for the initial development of a project)
* [x] Refactor (a non-breaking change that improves code maintainability).
* [ ] Bugfix (a non-breaking change that solves an issue).
* [ ] Dependencies upgrade (a version change upgrade for release management)
* [ ] Enhancement (a change on the existing code, increasing the efficiency or effectiveness of the software without necessarily introducing new features to end user, can include approved ADR)
* [ ] New feature (a non-breaking change that adds functionality, must include approved ADR).
* [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality, must include approved ADR).
* [ ] Deprecation (a change for code deprecation, will need to include this in the release notes)
* [ ] DevOps (a change for CICD related / DevOps for release management)
* [ ] Documentation (a change of documention only)
* [ ] Logging (a change for work related to internal audit, logging, reporting improvement)
* [ ] FOSSA Notice (a change for FOSSA notice update)
* [ ] Revert (a change for reverting a previous MR)
* [ ] Testing (a changes for improving unit, integration, etc. testing)
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
[Refactor indexer-gc (#13) · Issues · OSDU Forum / OSDU Project Management Committee - PMC / Community Implementation · GitLab (opengroup.org)](https://gitlab.opengroup.org/osdu/pmc/community-implementation/-/issues/13)
## Does this introduce a change in the common core logic?
* \[YES/**NO**\]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
* [ ] AWS
* [ ] Azure
* [ ] GCP
* [ ] IBM
## What is the current behavior?
## What is the new/expected behavior?
## Have you added/updated Unit Tests and Integration Tests?
## Any other useful informationM23 - Release 0.26Joshua HindmonJoshua Hindmonhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/704Augmenter refresh schema mapping2024-01-26T18:29:03ZZhibin MaiAugmenter refresh schema mappingIssue [136](https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/136)
Please pay attention to the change in initialization of the augmenter integration tests. The step `Then I clean up the index of the extended ...Issue [136](https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/136)
Please pay attention to the change in initialization of the augmenter integration tests. The step `Then I clean up the index of the extended kinds <extendedKinds> in the Elastic Search` was to delete the index in order to enforce the update of schema mapping in ElasticSearch index.
With this MR, the update of the augmenter configuration will automatically update the schema mapping in ElasticSearch index.M23 - Release 0.26Zhibin MaiZhibin Maihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/703Cherry-pick 'Fix parsing string arrays' into release/0.252024-01-24T11:49:37ZChad LeongCherry-pick 'Fix parsing string arrays' into release/0.25**Original MR**: !693
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !693
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/indexer-service/-/pipelines/new?ref=cherry-pick-for-693)M22 - Release 0.25David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/700[MSCOSDU-1894] exclude unused maven-core dependency2024-01-25T17:01:10ZVidyaDharani Lokam[MSCOSDU-1894] exclude unused maven-core dependency# Change details
* reference issues: https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/127 https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/126 https://community.opengroup.org/osd...# Change details
* reference issues: https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/127 https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/126 https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/125 https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/124 https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/123
* excluded unused `maven-core` dependency to remediate vulnerability.
* upgraded `spring-security-core` to `5.8.9`
# Changes in:
* [x] GCP
* [x] Azure
* [x] AWS
* [ ] IBM
* [x] commoncodeM23 - Release 0.26VidyaDharani LokamVidyaDharani Lokamhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/699resiliency fixes on cache miss2024-01-27T14:42:28ZNeelesh Thakurresiliency fixes on cache missresiliency fixes on cache miss
- retry fetching schema from schema service if there is a cache miss, otherwise it will incorrectly index the record.
- skip populating VirtualProperties if there a cache, otherwise it throws NullPointerExc...resiliency fixes on cache miss
- retry fetching schema from schema service if there is a cache miss, otherwise it will incorrectly index the record.
- skip populating VirtualProperties if there a cache, otherwise it throws NullPointerException.
Provide audit log entry for partially successful indexed record. Currently we are populating only for successful or failed request only.M23 - Release 0.26https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/696Mchance/integration test string array2024-01-10T16:33:30ZMark ChanceMchance/integration test string array