Indexer merge requestshttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests2023-07-13T13:26:10Zhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/582Cherry-pick 'Implement Redis cache in Azure for two kinds of object caches th...2023-07-13T13:26:10ZChad LeongCherry-pick 'Implement Redis cache in Azure for two kinds of object caches that are...' into release/0.22**Original MR**: !572
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !572
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/indexer-service/-/pipelines/new?ref=cherry-pick-for-572)M19 - Release 0.22David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/581Add istio sidecar to indexer (GONRG-7438)2023-07-14T11:49:35ZAndrei Skorkin [EPAM / GCP]Add istio sidecar to indexer (GONRG-7438)M19 - Release 0.22Andrei Skorkin [EPAM / GCP]Andrei Skorkin [EPAM / GCP]https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/578Added Reindex topic handler (GONRG-7281)2023-07-05T14:38:39ZDmitrii Novikov (EPAM)Added Reindex topic handler (GONRG-7281)https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/91
## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud pro...https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/91
## Type of change
- [x] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [x] Google Cloud
- [ ] IBM
## Does this introduce a breaking change?
- [No]
## What is the new/expected behavior?
Added new `reindex` topic for retry and reindex tasks to separate messages from `record-changed` topicM19 - Release 0.22Dmitrii Novikov (EPAM)Dmitrii Novikov (EPAM)https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/577tomcat vulnerabilty fix2023-07-04T11:42:27ZAkanksha Prasadtomcat vulnerabilty fixThis is the MR to fix vulnerability due to tomcat version being flagged in component governance stage, changing the version to suggested non-vulnerable version.This is the MR to fix vulnerability due to tomcat version being flagged in component governance stage, changing the version to suggested non-vulnerable version.Akanksha PrasadAkanksha Prasadhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/574tomcat vulnerabilty fix2023-06-30T09:36:57ZAkanksha Prasadtomcat vulnerabilty fixThis is the MR to fix vulnerability due to tomcat version being flagged in component governance stage, changing the version to suggested non-vulnerable version.This is the MR to fix vulnerability due to tomcat version being flagged in component governance stage, changing the version to suggested non-vulnerable version.Akanksha PrasadAkanksha Prasadhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/573GONRG-7392: use non-root user for images2023-06-30T12:25:47ZYauheni Rykhter (EPAM)GONRG-7392: use non-root user for imagesM19 - Release 0.22Yauheni Rykhter (EPAM)Yauheni Rykhter (EPAM)https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/572Implement Redis cache in Azure for two kinds of object caches that are...2023-07-13T08:57:25ZZhibin MaiImplement Redis cache in Azure for two kinds of object caches that are...This is follow-up of the MR [
Index extended properties defined in property configurations](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/465).
The Augmenter/Index Extension require a bunch of cac...This is follow-up of the MR [
Index extended properties defined in property configurations](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/465).
The Augmenter/Index Extension require a bunch of caches to gain performance or overcome the delay of index update in the underline elastic search. Most of them are not required to be synchronized across the indexer service except the two kinds of data (records or record change information between the previous version and the current version) that could be used when doing indexing chasing operations (e.g. index update for a Wellbore record may need to chase the index update for the Wellbore's children records, e.g. WellLog records).
In this MR, we adopt the Redis to caches these two kinds of data for Azure. Other CSPs, e.g. GCP and AWS, should implement similar solution.M19 - Release 0.22Zhibin MaiZhibin Maihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/571Catch the generic exception that might be thrown from FetchServiceHttpRequest...2023-07-04T10:39:38ZZhibin MaiCatch the generic exception that might be thrown from FetchServiceHttpRequest.sendRequest(...) APIwhen indexer receives event(s) to index a record, it will try to get the schema of the record and build the internal format of the schema when the schema of the record is not in the cache. If the augmenter feature is enabled, the indexer...when indexer receives event(s) to index a record, it will try to get the schema of the record and build the internal format of the schema when the schema of the record is not in the cache. If the augmenter feature is enabled, the indexer will call search service to find out whether the given kind has index property configuration so that it can combine the original schema with the schema of the extended properties to create a combined internal schema and use it to build the data block. If the exception from the search service is uncaught, the step to build the internal schema will fail and the indexer won't be able to populate the data block.
This MR is to catch generic exception instead of the exception with type (URISyntaxException) defined in the dependent interface FetchServiceHttpRequest.sendRequest(...).M19 - Release 0.22Zhibin MaiZhibin Maihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/570tomcat vulnerabilty fix2023-06-28T06:13:30ZAkanksha Prasadtomcat vulnerabilty fixThis is the MR to fix vulnerability due to tomcat version being flagged in component governance stage, changing the version to suggested non-vulnerable version.This is the MR to fix vulnerability due to tomcat version being flagged in component governance stage, changing the version to suggested non-vulnerable version.Akanksha PrasadAkanksha Prasadhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/569tomcat vulnerabilty fix2023-07-04T10:38:15ZAkanksha Prasadtomcat vulnerabilty fixThis is the MR to fix vulnerability due to tomcat version being flagged in component governance stage, changing the version to suggested non-vulnerable version.This is the MR to fix vulnerability due to tomcat version being flagged in component governance stage, changing the version to suggested non-vulnerable version.M19 - Release 0.22Akanksha PrasadAkanksha Prasadhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/568Add global variable limitsEnabled (GONRG-7266)2023-06-27T07:28:43ZAndrei Skorkin [EPAM / GCP]Add global variable limitsEnabled (GONRG-7266)M19 - Release 0.22Andrei Skorkin [EPAM / GCP]Andrei Skorkin [EPAM / GCP]https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/567Vulnerabilite resolve2023-06-20T09:27:05ZAkshat JoshiVulnerabilite resolvehttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/565vulnerabilities fixed2023-06-20T06:08:36ZAkshat Joshivulnerabilities fixedhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/564Azure reindex topic2023-06-13T11:15:23ZZhibin MaiAzure reindex topicThis is follow-up of [Update the re-index topic name to 'reindextopic'](https://community.opengroup.org/osdu/platform/deployment-and-operations/helm-charts-azure/-/merge_requests/621). Set env. variables of re-index topic in SLB Azure Sa...This is follow-up of [Update the re-index topic name to 'reindextopic'](https://community.opengroup.org/osdu/platform/deployment-and-operations/helm-charts-azure/-/merge_requests/621). Set env. variables of re-index topic in SLB Azure SaaS deployment.M19 - Release 0.22Zhibin MaiZhibin Maihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/563Vulnerability Fix2023-06-19T07:38:57ZKamalika SahaVulnerability FixKamalika SahaKamalika Sahahttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/562Upgrade Maven dependency version2023-06-15T10:17:42ZShreya ShahUpgrade Maven dependency version# Fixes :
https://community.opengroup.org/osdu/platform/system/indexer-service/-/security/vulnerabilities/582
https://community.opengroup.org/osdu/platform/system/indexer-service/-/security/vulnerabilities/635
# Issue
https://community....# Fixes :
https://community.opengroup.org/osdu/platform/system/indexer-service/-/security/vulnerabilities/582
https://community.opengroup.org/osdu/platform/system/indexer-service/-/security/vulnerabilities/635
# Issue
https://community.opengroup.org/osdu/platform/security-and-compliance/home/-/issues/169M19 - Release 0.22Shreya ShahShreya Shahhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/561Avoid using wildcard search to find the associated children records2023-08-09T13:30:43ZZhibin MaiAvoid using wildcard search to find the associated children recordsThis is follow-up of MR [Index extended properties defined in property configurations](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/465) to replace wildcard search with multi-kind search when sear...This is follow-up of MR [Index extended properties defined in property configurations](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/465) to replace wildcard search with multi-kind search when searching the associated children records. However, it does not come for free. It needs extra call to get the child kinds in order to replace wildcard. But it may be worth given wildcard search is so expensive and using cache mechanism may help reduce the extra calls.
It also switches back to use query_with_cursor in order to search more 10k records.M19 - Release 0.22Zhibin MaiZhibin Maihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/560Kamalika/m162023-06-19T07:19:31ZSabarish K R EKamalika/m16https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/558Remove feature flag for geoshape decimation2023-07-05T18:53:03ZZhibin MaiRemove feature flag for geoshape decimationThis MR is to remove the feature flag for geoshape decimation as geoshape decimation has been deployed since M16 and the feedback is positive:
1. It boosts the GIS map performance 20-30 times (in terms of search and data fetch) after the...This MR is to remove the feature flag for geoshape decimation as geoshape decimation has been deployed since M16 and the feedback is positive:
1. It boosts the GIS map performance 20-30 times (in terms of search and data fetch) after the geoshape is decimated in the client deployments.
2. It speeds up the indexing performance as the overhead on geoshape decimation is offset by the performance improvement on the ElasticSearch indexing (geoshape indexing)M19 - Release 0.22Zhibin MaiZhibin Maihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/557Cherry-pick 'Upgrade First Party Library Dependencies for Release 0.21' into ...2023-05-31T20:24:09ZChad LeongCherry-pick 'Upgrade First Party Library Dependencies for Release 0.21' into release/0.21**Original MR**: !553
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !553
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/indexer-service/-/pipelines/new?ref=cherry-pick-for-553)M18 - Release 0.21David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.org