Indexer merge requestshttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests2024-03-28T13:39:24Zhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/727Augmenter: Fresh the parent record when a child record is created or deleted...2024-03-28T13:39:24ZZhibin MaiAugmenter: Fresh the parent record when a child record is created or deleted...In issue[694](https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/694), it reports that the parent record was not refreshed when a child record is created or deleted in the parent child relationship.
We analyzed the root...In issue[694](https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/694), it reports that the parent record was not refreshed when a child record is created or deleted in the parent child relationship.
We analyzed the root cause for these two issues:
1. The parent record was not updated to include the child record info when a child record is created.
When a record is sent to ElasticSearch for indexing, there is a few seconds to tens of seconds delay on ElasticSearch for new record to be searchable. The time between a record is sent to ElasticSearch and the Augmenter chasing routing is triggered is very short which is in millisecond level.
In order to update the parent record to include the new child record information, it has two prerequisites:
* The id of the parent record can be retrieved from the child record so a re-index message for the parent record can be sent out.
To do that, `the MR makes use of the existing cache of the new child record`.
* The child record information should be available when the re-index message of the parent record is processed.
To do that, we can't make use of the existing cache of the new child record as the re-index message of the parent record does not have the child id. Augmenter has information about how to search all the child records for given parent record id. However, the time between the child record sent to ElasticSearch and the re-index message of the parent record received is also very short (< 100 ms if the indexer message queue is empty). `To make sure that the augmenter can search all the child records for given parent record id, we can set enough delays for the re-index message of the parent record to be received and processed. In this MR, the delay is set to 30 seconds which is a experienced trade-off between the latency of refresh and reliability of required information`.
2. The parent record was not updated to remove the child record info when a child record is deleted.
In order to remove the new child record information from its parent records, we need to know the ids of the parent records in order to send out the re-index message to re-index the parent records. When the Augmenter chasing routing is called, the deleted child record is not searchable from ElasticSearch anymore. `In this MR, we keep a copy of the deleted child records in order to retrieve the ids of the parent records and send out the re-index message`.M23 - Release 0.26Zhibin MaiZhibin Maihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/726fix link & []link indexer mapping2024-03-28T13:01:54ZNeelesh Thakurfix link & []link indexer mappinglink and []link are legacy types carried over from Storage schema & these types are still supported by Schema service.
Indexing of these types was broken by MR: #[649 ](https://community.opengroup.org/osdu/platform/system/indexer-servi...link and []link are legacy types carried over from Storage schema & these types are still supported by Schema service.
Indexing of these types was broken by MR: #[649 ](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/649)and [commit](https://community.opengroup.org/osdu/platform/system/indexer-service/-/commit/01278148cfd8eea101888b9ff259874033575cf9). It sets the index attribute mapping to type that's not supported by `StorageIndexerPayloadMapper`.
This MR restores behavior and fixes the breaking change.M23 - Release 0.26https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/725Indexer service should not reschedule failed events when the schema is not va...2024-03-26T09:46:33ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comIndexer service should not reschedule failed events when the schema is not valid.(GONRG-9505)# Description:
Schema not valid exception will cause an event to be skipped instead of retried since retrying such events is ineffective and wastes resources because schema cannot be updated automatically, schema issue should be fixed fi...# Description:
Schema not valid exception will cause an event to be skipped instead of retried since retrying such events is ineffective and wastes resources because schema cannot be updated automatically, schema issue should be fixed first.
# How to test:
IT
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GC
- [x] Community Implementation
# Dev Checklist:
- [x] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.
# Other comments:
Any comments to approvers hereM23 - Release 0.26Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/724handle null in nested attributes2024-03-28T15:38:11ZMykyta Savchukhandle null in nested attributesfix NPE during indexing when one of the nested attributes in the array is nullfix NPE during indexing when one of the nested attributes in the array is nullMykyta SavchukMykyta Savchukhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/723[GONRG-9422]: Remove springProfilesActive explicit setting2024-03-20T12:37:25ZMykola Ronik [EPAM / GCP][GONRG-9422]: Remove springProfilesActive explicit settingM23 - Release 0.26Mykola Ronik [EPAM / GCP]Mykola Ronik [EPAM / GCP]https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/722Aws upgrade aws lib2024-03-11T14:07:40ZDerek HudsonAws upgrade aws libM23 - Release 0.26Derek HudsonDerek Hudsonhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/721[GONRG-9211] Added labels to gc helm2024-03-19T17:11:08ZDanylo Vanin (EPAM)[GONRG-9211] Added labels to gc helmM23 - Release 0.26Danylo Vanin (EPAM)Danylo Vanin (EPAM)https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/720Draft: GONRG-9287: separate pipeline for core-plus2024-03-05T15:25:53ZMikhail Piatliou (EPAM)Draft: GONRG-9287: separate pipeline for core-plusMikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/719[#MSCOSDU-35931] upgrade : core-lib-azure Java 17 version2024-03-06T14:13:23ZThulasi Dass Subramanian[#MSCOSDU-35931] upgrade : core-lib-azure Java 17 version## Change details
* For Azure only: upgraded `core-lib-azure` version to `0.26.0-rc5` for Java 17 updates both in source & testing modules
## Changes in:
* [ ] GCP
* [x] Azure
* [ ] AWS
* [ ] IBM## Change details
* For Azure only: upgraded `core-lib-azure` version to `0.26.0-rc5` for Java 17 updates both in source & testing modules
## Changes in:
* [ ] GCP
* [x] Azure
* [ ] AWS
* [ ] IBMM23 - Release 0.26Thulasi Dass SubramanianThulasi Dass Subramanianhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/718Added feature for GC Int tests2024-03-05T12:29:44ZSiarhei Poliak [EPAM / GCP]Added feature for GC Int testsM23 - Release 0.26Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/717Draft: Checking elasticsearch upgrade2024-02-28T13:53:09ZHarshika DhootDraft: Checking elasticsearch upgradehttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/716Enhance the Augmenter documentation2024-03-08T16:48:00ZZhibin MaiEnhance the Augmenter documentationUpdate Augmenter Document from the following aspects:
* Update introduction section
* Update use cases
* Remove all the `data.` prefix in ValuePath
* Make sure the `Policy` and `RelationshipDirection` are set correctlyUpdate Augmenter Document from the following aspects:
* Update introduction section
* Update use cases
* Remove all the `data.` prefix in ValuePath
* Make sure the `Policy` and `RelationshipDirection` are set correctlyM23 - Release 0.26Zhibin MaiZhibin Maihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/715Aws side code quality2024-02-26T20:24:57ZBruce JinAws side code qualityThis MR will improve AWS code quality, fix some codesmellThis MR will improve AWS code quality, fix some codesmellM23 - Release 0.26Bruce JinBruce Jinhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/714Added some methods for int tests2024-02-26T16:33:53ZRiabokon Stanislav(EPAM)[GCP]Added some methods for int testsAdded some methods for int testsAdded some methods for int testsM23 - Release 0.26Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/713Fix the issue that augmenter throws null pointer exception when casting the r...2024-02-26T14:54:23ZZhibin MaiFix the issue that augmenter throws null pointer exception when casting the related object ids retrieved from the GeoContextThe bug was introduced in M21. For the detail of the issue, please refer to issue [151](https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/151).The bug was introduced in M21. For the detail of the issue, please refer to issue [151](https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/151).M23 - Release 0.26Zhibin MaiZhibin Maihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/712Disable listner.schema.event.create by default2024-02-22T15:32:06ZAlok JoshiDisable listner.schema.event.create by defaultCreating an index with a new schema (no records yet) creates unnecessary data in elastic. We don't need to create such empty indexCreating an index with a new schema (no records yet) creates unnecessary data in elastic. We don't need to create such empty indexM23 - Release 0.26Alok JoshiChad LeongAlok Joshihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/711Update os-core-lib-aws2024-03-19T17:11:09ZRucha DeshpandeUpdate os-core-lib-awsUpdate os-core-lib-awsUpdate os-core-lib-awsM23 - Release 0.26https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/710Vulnerbility fix on AWS side2024-03-19T17:11:10ZBruce JinVulnerbility fix on AWS sidethis will fix some AWS internal vulnerabilitiesthis will fix some AWS internal vulnerabilitiesM23 - Release 0.26Bruce JinBruce Jinhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/709[MS-34266] fix jetty vulnerability2024-02-12T15:17:57ZVidyaDharani Lokam[MS-34266] fix jetty vulnerability# Change details
* issue reference: https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/145
https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/144
* updated `jetty` dependency to `6.1...# Change details
* issue reference: https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/145
https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/144
* updated `jetty` dependency to `6.1.26` remediate vulnerability.
* excluded `jetty` from ibm csp as it is unused in it.
# Changes in:
* [x] Aws
* [x] Azure
* [x] GCP
* [x] IBM
* [x] Common codeM23 - Release 0.26VidyaDharani LokamVidyaDharani Lokamhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/708fix aws entitlement url2024-02-08T15:46:32ZYunhua Koglinfix aws entitlement urlM23 - Release 0.26Yunhua KoglinYunhua Koglin