Indexer merge requestshttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests2023-04-27T14:35:59Zhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/530Added a check for RabbitMQ Retry (GONRG-6705)2023-04-27T14:35:59ZRiabokon Stanislav(EPAM)[GCP]Added a check for RabbitMQ Retry (GONRG-6705)## Type of change
- [X] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] IB...## Type of change
- [X] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the new/expected behavior?
Updated oqm mappers.M18 - Release 0.21Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/529Revert "Merge branch 'GONGR-6885-reduce_requests_according_to_VPA'2023-08-18T15:04:59ZOleksandr Kosse (EPAM)Revert "Merge branch 'GONGR-6885-reduce_requests_according_to_VPA'This reverts merge request !526This reverts merge request !526M18 - Release 0.21https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/528Set FailOnError to false2023-05-03T19:26:32ZYash DholakiaSet FailOnError to falseSet FailOnError to false so build does not fail when dependency-check-maven fails.Set FailOnError to false so build does not fail when dependency-check-maven fails.M18 - Release 0.21Yash DholakiaYash Dholakiahttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/527tweak params to improve reindex performance2023-04-25T14:02:09ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comtweak params to improve reindex performanceIssue: https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/66
# Description:
Increase record ids request by kind from 20 to 1000.
Added scheduled publisher to postpone reindex events with the `force_clean=tru...Issue: https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/66
# Description:
Increase record ids request by kind from 20 to 1000.
Added scheduled publisher to postpone reindex events with the `force_clean=true` param.
# How to test:
Does functionality was tested and how?
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] Google Cloud
- [ ] Azure
- [ ] AWS
- [ ] IBM
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M18 - Release 0.21Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/526GONGR-6885 Reduce requests according to VPA2023-08-18T15:05:01ZAndriy Halka [EPAM / GCP]GONGR-6885 Reduce requests according to VPAM18 - Release 0.21https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/525AWS: Fixes user token timeout when a lengthy reindexing is started2023-08-18T15:05:03ZDerek HudsonAWS: Fixes user token timeout when a lengthy reindexing is startedFixes user token timeout when a lengthy reindexing procedure is initiated by using a service principal rather than the user's more temporary tokens.
Note: this is based on a stale master branch. The pipeline stages for GCP are failing b...Fixes user token timeout when a lengthy reindexing procedure is initiated by using a service principal rather than the user's more temporary tokens.
Note: this is based on a stale master branch. The pipeline stages for GCP are failing because this is based on a stale master branch. The branch this was based on is [here](https://community.opengroup.org/osdu/platform/system/indexer-service/-/pipelines/179562), where the pipeline is also failing for those same issues.M18 - Release 0.21Yong ZengDerek HudsonYong Zenghttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/524OpenAPI 3.0 Documentation using springdoc2023-04-21T12:58:59ZThulasi Dass SubramanianOpenAPI 3.0 Documentation using springdoc**Link to ADR(Architecture Decision Record)** : [Swagger using springdoc-openapi](https://community.opengroup.org/osdu/platform/system/home/-/issues/97)
## OpenAPI 3.0 related changes
* upgraded to latest **springdoc openapi** latest v...**Link to ADR(Architecture Decision Record)** : [Swagger using springdoc-openapi](https://community.opengroup.org/osdu/platform/system/home/-/issues/97)
## OpenAPI 3.0 related changes
* upgraded to latest **springdoc openapi** latest version [1.6.14](https://mvnrepository.com/artifact/org.springdoc/springdoc-openapi-ui/1.6.14)
* used **[@OpenAPIDefinition](https://docs.swagger.io/swagger-core/v2.0.9/apidocs/io/swagger/v3/oas/annotations/OpenAPIDefinition.html)** annotation to populate OpenAPI Object fields *\[info, tags, servers, security\]*
* Documented the below API's with OpenAPI 3.0 **Annotations**
- CleanupIndicies API
- PartitionSetup API
- Reindex API
- RecordIndexer API(Internal API, marked as 'Hidden' for documentation)
- Health Check API
- Info API
* Added the standard HTTP Response(4xx, 5x\*\*\*\*x) for API Responses
* Custom Path for
* **Swagger UI**: https://host/context-path/swagger (will redirect to https://host/context-path/swagger-ui/index.html)
* **api-docs (JSON)** : https://host/context-path/api-docs
* **api-docs (YAML)** : https://host/context-path/api-docs.yaml
* Azure Swagger GLAB(for Reference)
* **Swagger UI**: https://osdu-glab.msft-osdu-test.org/api/indexer/v2/swagger (will redirect to https://osdu-glab.msft-osdu-test.org/api/indexer/v2/swagger-ui/index.html)
* **api-docs (JSON)** : https://osdu-glab.msft-osdu-test.org/api/indexer/v2/api-docs
* **api-docs (YAML)** :https://osdu-glab.msft-osdu-test.org/api/indexer/v2/api-docs.yaml
## Other Changes
- **Configurable** descriptions managed in [swagger.properties](url)
- Deleted HomeController
- Marked the 'WhoamiController' as 'Hidden' for documentation
- Updated Readme for swagger related information
## References
- https://springdoc.org/faq.html#_can_i_use_spring_property_with_swagger_annotations
- https://springdoc.org/migrating-from-springfox.htmlM18 - Release 0.21Thulasi Dass SubramanianThulasi Dass Subramanianhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/523Merge branch 'gc-fix-jackson-improve-error-handle' into 'release/0.20'2023-04-17T09:40:59ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comMerge branch 'gc-fix-jackson-improve-error-handle' into 'release/0.20'fix jackson libs conflict, add Error handling in receiver
See merge request osdu/platform/system/indexer-service!521
(cherry picked from commit 9c66b4b977b8de3b4aa6688ab845a35f3043c57b)
d1a99106 fix jackson libs conflict producing NoS...fix jackson libs conflict, add Error handling in receiver
See merge request osdu/platform/system/indexer-service!521
(cherry picked from commit 9c66b4b977b8de3b4aa6688ab845a35f3043c57b)
d1a99106 fix jackson libs conflict producing NoSuchMethodError, add Error handling in...
0fd13ec4 update NOTICE
31aa1204 add correlation id to logsM17 - Release 0.20Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/522M14 cherrypick to m16 master2023-04-20T19:28:05ZKiran VeerapaneniM14 cherrypick to m16 masterM14 cherrypick to m16 masterM14 cherrypick to m16 masterKiran VeerapaneniKiran Veerapanenihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/521fix jackson libs conflict, add Error handling in receiver2023-04-28T20:05:25ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comfix jackson libs conflict, add Error handling in receiverFix Jackson libs conflict producing NoSuchMethodError, add Error handling in the receiver to prevent silent errors.
The error that was met interrupts the processing of records with geodata:
~~~
Caused by: java.lang.NoSuchMethodError: co...Fix Jackson libs conflict producing NoSuchMethodError, add Error handling in the receiver to prevent silent errors.
The error that was met interrupts the processing of records with geodata:
~~~
Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.annotation.JsonSubTypes.failOnRepeatedNames()Z
at com.fasterxml.jackson.databind.introspect.JacksonAnnotationIntrospector.findSubtypes(JacksonAnnotationIntrospector.java:627)
at com.fasterxml.jackson.databind.jsontype.impl.StdSubtypeResolver._collectAndResolveByTypeId(StdSubtypeResolver.java:296)
at com.fasterxml.jackson.databind.jsontype.impl.StdSubtypeResolver.collectAndResolveSubtypesByTypeId(StdSubtypeResolver.java:212)
at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findTypeDeserializer(BasicDeserializerFactory.java:1793)
at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:654)
at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4861)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4731)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3677)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3645)
at org.opengroup.osdu.indexer.util.parser.GeoShapeParser.parseGeoJson(GeoShapeParser.java:41)
~~~M17 - Release 0.20Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/520Updating NOTICE2023-04-13T14:08:15ZChad LeongUpdating NOTICEFixing FOSSA noticeFixing FOSSA noticeM17 - Release 0.20https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/519Cherry-pick 'Upgrade Dependencies' into release/0.202023-04-11T09:49:20ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Upgrade Dependencies' into release/0.20**Original MR**: !515
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !515
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/indexer-service/-/pipelines/new?ref=cherry-pick-for-515)M17 - Release 0.20David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/517Updating core lib aws to latest rc version2023-04-07T17:37:36ZManish JangidUpdating core lib aws to latest rc versionUpdating core lib aws to latest rc versionUpdating core lib aws to latest rc versionM18 - Release 0.21https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/516ReadOnlyRootFileSystem changes for AWS2023-04-07T18:40:24ZAbhay JoshiReadOnlyRootFileSystem changes for AWScommit 6584a46b
Author: Abhay <bios@amazon.com>
Date: Tue Mar 21 2023 08:40:01 GMT-0700 (Pacific Daylight Time)
removing old ssl stuff
commit d6d80ec7
Author: Abhay <bios@amazon.com>
Date: Tue Mar 21 2023 07:29:05 GMT-0700 (Pacific...commit 6584a46b
Author: Abhay <bios@amazon.com>
Date: Tue Mar 21 2023 08:40:01 GMT-0700 (Pacific Daylight Time)
removing old ssl stuff
commit d6d80ec7
Author: Abhay <bios@amazon.com>
Date: Tue Mar 21 2023 07:29:05 GMT-0700 (Pacific Daylight Time)
Renaming variable
commit a2ef5304
Author: Abhay <bios@amazon.com>
Date: Mon Mar 20 2023 10:44:26 GMT-0700 (Pacific Daylight Time)
adding changes for rootFilesystem
(cherry picked from commit 31267f749fc7b067c4af231273f3e3250056013c)M18 - Release 0.21Okoun-Ola Fabien HouetoAbhay JoshiOkoun-Ola Fabien Houetohttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/515Upgrade Dependencies2023-04-11T09:42:17ZAbhay JoshiUpgrade Dependenciescommit 83ba5afa
Author: Abhay <bios@amazon.com>
Date: Mon Mar 27 2023 10:58:20 GMT-0700 (Pacific Daylight Time)
upgrading dependencies
(cherry picked from commit a2c87e54974a60c90af3bc11ce9da46dc23b905f)commit 83ba5afa
Author: Abhay <bios@amazon.com>
Date: Mon Mar 27 2023 10:58:20 GMT-0700 (Pacific Daylight Time)
upgrading dependencies
(cherry picked from commit a2c87e54974a60c90af3bc11ce9da46dc23b905f)M17 - Release 0.20Okoun-Ola Fabien HouetoAbhay JoshiOkoun-Ola Fabien Houetohttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/514remove legacy code that's not being used by any API2023-04-24T14:30:50ZNeelesh Thakurremove legacy code that's not being used by any APIM18 - Release 0.21https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/513removing vulnerability in indexer2023-04-05T05:59:46ZAkanksha Prasadremoving vulnerability in indexerRemoving vulnerabilities in indexer searchRemoving vulnerabilities in indexer searchAkanksha PrasadAkanksha Prasadhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/512tweak parameters to enhance reindex performance2023-03-29T22:02:22ZNeelesh Thakurtweak parameters to enhance reindex performance- Use max 'limit' param value (1000) to fetch record-ids from Storage for kind.
- Batch 1k record-ids to 50 (same limit as Storage PUT API record-changed event).
With these changes, we see 5x performance improvements on reindex API.- Use max 'limit' param value (1000) to fetch record-ids from Storage for kind.
- Batch 1k record-ids to 50 (same limit as Storage PUT API record-changed event).
With these changes, we see 5x performance improvements on reindex API.M17 - Release 0.20https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/511add validation and ignore out of order message when kind is updated2023-03-29T15:30:50ZNeelesh Thakuradd validation and ignore out of order message when kind is updatedUsers have reported issue with data duplication with following scenario:
1. Trigger a re-index job for kinds: both raw and WKS kind at the same time
2. The triggering of the reindexing on raw kind also triggered WKS service to recreate ...Users have reported issue with data duplication with following scenario:
1. Trigger a re-index job for kinds: both raw and WKS kind at the same time
2. The triggering of the reindexing on raw kind also triggered WKS service to recreate the WKS instances derived from them
3. The target schema mapping for these raw records had been updated (e.g. version 1.0 to 1.1), meaning the new versions of these records created by WKS changed the schema.
4. Index therefore received 2 events for the same WKS record instances at similar times, one to re-index the original and another to create the new one
5. If the one to create the new instance for WKS was processed first that would lead to duplicates appearing.
This changes-set adds validation to avoid duplication by ignoring the event when kind is updated in such cases.M17 - Release 0.20https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/510Vulnerability fix for IBM2023-07-25T07:11:22ZDevdatta SantraVulnerability fix for IBMVulnerability fix for IBM for tomcat and netty libs
| CVE-2022-41881 | https://nvd.nist.gov/vuln/detail/CVE-2022-41881 |
|----------------|-------------------------------------------------|
| CVE-2022-45143 | https://nvd.nist.gov/vuln/d...Vulnerability fix for IBM for tomcat and netty libs
| CVE-2022-41881 | https://nvd.nist.gov/vuln/detail/CVE-2022-41881 |
|----------------|-------------------------------------------------|
| CVE-2022-45143 | https://nvd.nist.gov/vuln/detail/CVE-2022-45143 |M17 - Release 0.20Devdatta SantraDevdatta Santra