Indexer merge requestshttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests2024-03-28T13:39:24Zhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/727Augmenter: Fresh the parent record when a child record is created or deleted...2024-03-28T13:39:24ZZhibin MaiAugmenter: Fresh the parent record when a child record is created or deleted...In issue[694](https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/694), it reports that the parent record was not refreshed when a child record is created or deleted in the parent child relationship.
We analyzed the root...In issue[694](https://community.opengroup.org/osdu/platform/pre-shipping/-/issues/694), it reports that the parent record was not refreshed when a child record is created or deleted in the parent child relationship.
We analyzed the root cause for these two issues:
1. The parent record was not updated to include the child record info when a child record is created.
When a record is sent to ElasticSearch for indexing, there is a few seconds to tens of seconds delay on ElasticSearch for new record to be searchable. The time between a record is sent to ElasticSearch and the Augmenter chasing routing is triggered is very short which is in millisecond level.
In order to update the parent record to include the new child record information, it has two prerequisites:
* The id of the parent record can be retrieved from the child record so a re-index message for the parent record can be sent out.
To do that, `the MR makes use of the existing cache of the new child record`.
* The child record information should be available when the re-index message of the parent record is processed.
To do that, we can't make use of the existing cache of the new child record as the re-index message of the parent record does not have the child id. Augmenter has information about how to search all the child records for given parent record id. However, the time between the child record sent to ElasticSearch and the re-index message of the parent record received is also very short (< 100 ms if the indexer message queue is empty). `To make sure that the augmenter can search all the child records for given parent record id, we can set enough delays for the re-index message of the parent record to be received and processed. In this MR, the delay is set to 30 seconds which is a experienced trade-off between the latency of refresh and reliability of required information`.
2. The parent record was not updated to remove the child record info when a child record is deleted.
In order to remove the new child record information from its parent records, we need to know the ids of the parent records in order to send out the re-index message to re-index the parent records. When the Augmenter chasing routing is called, the deleted child record is not searchable from ElasticSearch anymore. `In this MR, we keep a copy of the deleted child records in order to retrieve the ids of the parent records and send out the re-index message`.M23 - Release 0.26Zhibin MaiZhibin Maihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/726fix link & []link indexer mapping2024-03-28T13:01:54ZNeelesh Thakurfix link & []link indexer mappinglink and []link are legacy types carried over from Storage schema & these types are still supported by Schema service.
Indexing of these types was broken by MR: #[649 ](https://community.opengroup.org/osdu/platform/system/indexer-servi...link and []link are legacy types carried over from Storage schema & these types are still supported by Schema service.
Indexing of these types was broken by MR: #[649 ](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/649)and [commit](https://community.opengroup.org/osdu/platform/system/indexer-service/-/commit/01278148cfd8eea101888b9ff259874033575cf9). It sets the index attribute mapping to type that's not supported by `StorageIndexerPayloadMapper`.
This MR restores behavior and fixes the breaking change.M23 - Release 0.26https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/724handle null in nested attributes2024-03-28T15:38:11ZMykyta Savchukhandle null in nested attributesfix NPE during indexing when one of the nested attributes in the array is nullfix NPE during indexing when one of the nested attributes in the array is nullMykyta SavchukMykyta Savchukhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/722Aws upgrade aws lib2024-03-11T14:07:40ZDerek HudsonAws upgrade aws libM23 - Release 0.26Derek HudsonDerek Hudsonhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/720Draft: GONRG-9287: separate pipeline for core-plus2024-03-05T15:25:53ZMikhail Piatliou (EPAM)Draft: GONRG-9287: separate pipeline for core-plusMikhail Piatliou (EPAM)Mikhail Piatliou (EPAM)https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/717Draft: Checking elasticsearch upgrade2024-02-28T13:53:09ZHarshika DhootDraft: Checking elasticsearch upgradehttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/707Draft: Vazha/test default value2024-02-09T14:24:31ZAnthony Ittiyera vazhappillyDraft: Vazha/test default valuehttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/698update azure.yml ref. to disable azure deploy for indexer service2024-01-19T11:32:11Zsaketh somarajuupdate azure.yml ref. to disable azure deploy for indexer service- Indexer service's azure/m18-master branches causes issues in GLAB env. due to significant code changes from current master.
- Which in turn causes issues in indexing of newly ingested records in dev. env. effecting other services like ...- Indexer service's azure/m18-master branches causes issues in GLAB env. due to significant code changes from current master.
- Which in turn causes issues in indexing of newly ingested records in dev. env. effecting other services like dags, ddms etc.,
- This change disables azure deploy in azure/m18 master branch and it's respective child branches.
- However, one of the downside of this change is team can't test indexer service (for m18 branch) in glab env.
CC @nursheikh , @lucynliusaketh somarajusaketh somarajuhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/685Use full URL instead of relative path2024-02-04T23:16:15ZLawrence ChanUse full URL instead of relative pathUpdate path for indexer in azure/m18-master branchUpdate path for indexer in azure/m18-master branchhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/657test commit2023-11-30T19:10:03Zpreeti singh[Microsoft]test commitpreeti singh[Microsoft]preeti singh[Microsoft]https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/654dummy commit2023-11-27T14:28:38ZSabarish K R Edummy commitdummy commitdummy commithttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/648Elastic search EOL version upgrade2024-01-17T05:04:14ZHarshika DhootElastic search EOL version upgradeHarshika DhootHarshika Dhoothttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/623Draft: Update the NOTICE file to the one generated in the pipeline2023-09-28T17:51:16ZAdrian SaliDraft: Update the NOTICE file to the one generated in the pipelinehttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/608Draft: Upgrade rest-high-level client to 7.17 to support Elasticsearch 8.2023-10-13T11:34:29ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comDraft: Upgrade rest-high-level client to 7.17 to support Elasticsearch 8.# Description:
Backward-compatible migration to Elastic 8 using client version 7.17.12: https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-compatibility.html
Removed types usage, they were deprecated, ...# Description:
Backward-compatible migration to Elastic 8 using client version 7.17.12: https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-compatibility.html
Removed types usage, they were deprecated, only default type `_doc` were left in v 7.* and completely removed in v.8.*: https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html
No changes are required from the client builder, thanks to the provided `ELASTIC_CLIENT_APIVERSIONING=true` environment variable that determines whether to send the 7.x compatibility header.
With these changes, Indexer is able to work with both ElasticSearch v7.* and v8.*.
# How to test:
Functionality was tested via int tests simultaneously on both envs, with Elasticsearch 8 and Elasticsearch7.
Info endpoint example for two environments that run the same binaries, but different Elasticsearch versions:
~~~
curl --location 'https://community.gcp.gnrg-osdu.projects.epam.com/api/indexer/v2/info'
{
"groupId": "org.opengroup.osdu.indexer",
"artifactId": "indexer-gc",
"version": "0.24.0-SNAPSHOT",
"buildTime": "2023-08-31T14:11:53.626Z",
"branch": "trusted-elastic-8-migration",
"commitId": "88c77d0d6977a9d74c74492aa20c2d9ac995f11c",
"commitMessage": "Merge remote-tracking branch 'origin/elastic-8-migration' into elastic-8-migration",
"connectedOuterServices": [
{
"name": "Redis-RedisCacheBuilder$2",
"version": "7.2.0"
},
{
"name": "Redis-RedisCacheBuilder$2",
"version": "7.2.0"
},
{
"name": "Redis-RedisCacheBuilder$2",
"version": "7.2.0"
},
{
"name": "ElasticSearch-osdu",
"version": "8.9.1"
},
{
"name": "ElasticSearch-second",
"version": "8.9.1"
}
]
}
curl --location 'https://osdu.ref.gcp.gnrg-osdu.projects.epam.com/api/indexer/v2/info'
{
"groupId": "org.opengroup.osdu.indexer",
"artifactId": "indexer-gc",
"version": "0.24.0-SNAPSHOT",
"buildTime": "2023-08-31T11:31:25.185Z",
"branch": "trusted-elastic-8-migration",
"commitId": "88c77d0d6977a9d74c74492aa20c2d9ac995f11c",
"commitMessage": "Merge remote-tracking branch 'origin/elastic-8-migration' into elastic-8-migration",
"connectedOuterServices": [
{
"name": "Redis-RedisCacheBuilder$2",
"version": "7.0.12"
},
{
"name": "Redis-RedisCacheBuilder$2",
"version": "7.0.12"
},
{
"name": "Redis-RedisCacheBuilder$2",
"version": "7.0.12"
},
{
"name": "ElasticSearch-osdu",
"version": "7.17.12"
}
]
}
~~~
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [x] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [x] Azure
- [x] AWS
- [x] IBM
- [x] Common code
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [x] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.
# Other comments:Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/605Draft: Cherrypick/m16 to m182023-08-24T16:41:33ZMahsa HanifiDraft: Cherrypick/m16 to m18https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/603Draft: Jackson databind upgrade2023-08-23T04:31:32Zsagar thapaDraft: Jackson databind upgradehttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/596Fixing Concurrency Issues with TypeMapper2023-08-11T19:57:12ZSabarish K R EFixing Concurrency Issues with TypeMappertest case demonstrating a thread-unsafe operation of IndexerMappingServiceImpl.getIndexMappingFromRecordSchema(..) and the fix for that
# Conflicts:
# indexer-core/src/main/java/org/opengroup/osdu/indexer/service/IndexerMappingServiceIm...test case demonstrating a thread-unsafe operation of IndexerMappingServiceImpl.getIndexMappingFromRecordSchema(..) and the fix for that
# Conflicts:
# indexer-core/src/main/java/org/opengroup/osdu/indexer/service/IndexerMappingServiceImpl.javahttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/590Draft: Az/ss code coverage2023-07-24T09:02:10ZShreya ShahDraft: Az/ss code coverageShreya ShahShreya Shahhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/576tomcat vulnerabilty fix2023-07-04T07:41:44ZAkanksha Prasadtomcat vulnerabilty fixThis is the MR to fix vulnerability due to tomcat version being flagged in component governance stage, changing the version to suggested non-vulnerable version.This is the MR to fix vulnerability due to tomcat version being flagged in component governance stage, changing the version to suggested non-vulnerable version.Akanksha PrasadAkanksha Prasadhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/575tomcat vulnerabilty fix2023-07-04T07:32:22ZAkanksha Prasadtomcat vulnerabilty fixThis is the MR to fix vulnerability due to tomcat version being flagged in component governance stage, changing the version to suggested non-vulnerable version.This is the MR to fix vulnerability due to tomcat version being flagged in component governance stage, changing the version to suggested non-vulnerable version.Akanksha PrasadAkanksha Prasad