Indexer merge requestshttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests2023-04-28T20:05:25Zhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/521fix jackson libs conflict, add Error handling in receiver2023-04-28T20:05:25ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comfix jackson libs conflict, add Error handling in receiverFix Jackson libs conflict producing NoSuchMethodError, add Error handling in the receiver to prevent silent errors.
The error that was met interrupts the processing of records with geodata:
~~~
Caused by: java.lang.NoSuchMethodError: co...Fix Jackson libs conflict producing NoSuchMethodError, add Error handling in the receiver to prevent silent errors.
The error that was met interrupts the processing of records with geodata:
~~~
Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.annotation.JsonSubTypes.failOnRepeatedNames()Z
at com.fasterxml.jackson.databind.introspect.JacksonAnnotationIntrospector.findSubtypes(JacksonAnnotationIntrospector.java:627)
at com.fasterxml.jackson.databind.jsontype.impl.StdSubtypeResolver._collectAndResolveByTypeId(StdSubtypeResolver.java:296)
at com.fasterxml.jackson.databind.jsontype.impl.StdSubtypeResolver.collectAndResolveSubtypesByTypeId(StdSubtypeResolver.java:212)
at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findTypeDeserializer(BasicDeserializerFactory.java:1793)
at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:654)
at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4861)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4731)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3677)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3645)
at org.opengroup.osdu.indexer.util.parser.GeoShapeParser.parseGeoJson(GeoShapeParser.java:41)
~~~M17 - Release 0.20Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/348Fix jackson version2022-09-15T23:22:07ZDmitrii Novikov (EPAM)Fix jackson versionM12 - Release 0.15Dmitrii Novikov (EPAM)Dmitrii Novikov (EPAM)https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/726fix link & []link indexer mapping2024-03-28T13:01:54ZNeelesh Thakurfix link & []link indexer mappinglink and []link are legacy types carried over from Storage schema & these types are still supported by Schema service.
Indexing of these types was broken by MR: #[649 ](https://community.opengroup.org/osdu/platform/system/indexer-servi...link and []link are legacy types carried over from Storage schema & these types are still supported by Schema service.
Indexing of these types was broken by MR: #[649 ](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/649)and [commit](https://community.opengroup.org/osdu/platform/system/indexer-service/-/commit/01278148cfd8eea101888b9ff259874033575cf9). It sets the index attribute mapping to type that's not supported by `StorageIndexerPayloadMapper`.
This MR restores behavior and fixes the breaking change.M23 - Release 0.26https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/159fix multilevel nesting2023-08-18T15:07:56ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comfix multilevel nesting# Description:
Fix for correct handling of multilevel nested properties <br/>
~~~
"NestedTest": {
"type": "nested",
"properties": {
...# Description:
Fix for correct handling of multilevel nested properties <br/>
~~~
"NestedTest": {
"type": "nested",
"properties": {
"DateTimeTest": {
"type": "date"
},
"NestedInnerTest": {
"type": "nested",
"properties": {
"DateTimeInnerTest": {
"type": "date"
},
"NumberInnerTest": {
"type": "double"
},
"StringInnerTest": {
"type": "text"
}
}
},
~~~
Also tag test case enabled int int-testing
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [x] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] Common code
- [x] GCP
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M6 - Release 0.9Riabokon Stanislav(EPAM)[GCP]Rostislav Dublin (EPAM)Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/621fix normalizedKind tag index bug2023-09-28T21:16:08ZMingyang Zhufix normalizedKind tag index bugfix normalizedKind tag index bug. The original code won't index the normalizedKind tag if the storage record has tags
original ADR: https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/78
bug item: https://commun...fix normalizedKind tag index bug. The original code won't index the normalizedKind tag if the storage record has tags
original ADR: https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/78
bug item: https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/115M20 - Release 0.23Mingyang ZhuMingyang Zhuhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/693Fix parsing string arrays2024-02-08T19:27:30ZStanisław BienieckiFix parsing string arraysCloses https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/137
Hotfix for parsing string arrays, as previous fix in handling string introduced error in parsing string arraysCloses https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/137
Hotfix for parsing string arrays, as previous fix in handling string introduced error in parsing string arraysM22 - Release 0.25https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/186fix path in schema parser for attributes marked nested by x-osdu-indexing2023-08-18T15:07:25ZNeelesh Thakurfix path in schema parser for attributes marked nested by x-osdu-indexingAddresses: [61](https://community.opengroup.org/osdu/platform/system/search-service/-/issues/61)
Fix indexer schema parser to handle nested attribute decorated with `x-osdu-indexing`. Currently they only work with one level nesting.
As...Addresses: [61](https://community.opengroup.org/osdu/platform/system/search-service/-/issues/61)
Fix indexer schema parser to handle nested attribute decorated with `x-osdu-indexing`. Currently they only work with one level nesting.
As an example consider following schema fragment and we want to find `geopoliticalContext` attribute schema:
```json
"classification": {
"type": "object",
"title": "File Properties",
"properties": {
"petro": {
"description": "Represents petro",
"title": "petro",
"type": "Object",
"properties": {
"geopoliticalContext": {
"description": "Represents value of geopoliticalContext",
"title": "geopoliticalContext",
"type": "array",
"x-osdu-indexing": {
"type": "nested"
},
"items": {
"type": "object",
"properties": {
"country": {
"title": "country",
"type": "string",
"description": "Represents country"
}
}
}
}
```
should be flattened as:
```json
{
"path": "classification.petro.geopoliticalContext",
"kind": "nested",
"properties": [
{
"path": "country",
"kind": "string"
}
]
}
```
instead of (this is the current behavior)
```json
{
"path": "geopoliticalContext",
"kind": "nested",
"properties": [
{
"path": "country",
"kind": "string"
}
]
}
```M8 - Release 0.11https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/632fix plexus version2023-10-12T18:47:49ZNathan Strelserfix plexus versionNathan StrelserNathan Strelserhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/292Fix record retry for deleted events on deleted index2023-08-18T15:06:06ZNeelesh ThakurFix record retry for deleted events on deleted indexEarlier MR: [282](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/282) added a retry for Index not found error to delegate index creation only by Indexer service. This works well for `create` or `upd...Earlier MR: [282](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/282) added a retry for Index not found error to delegate index creation only by Indexer service. This works well for `create` or `update` event but for `delete` event, it ends up retrying indefinitely if index is not found. (This could happen in integration tests scenario if index is deleted but `delete` event is not processed yet).
This change-set removes generic retry on index not found error and adds retry for `create` or `update` event only. In case of `delete` event if index does not exist than it will not retry.M11 - Release 0.14https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/189fix recurring integration tests failures2021-08-20T20:04:11ZNeelesh Thakurfix recurring integration tests failureshttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/495Fix redis-indexer cpu request2023-08-18T15:05:06ZOleksandr Kosse (EPAM)Fix redis-indexer cpu requestM17 - Release 0.20https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/290Fix retry for deleted records when index not exist2022-03-04T12:18:19ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comFix retry for deleted records when index not exist# Description:
When the indexer receives an event with Op type DELETE, it deletes provided records at elasticsearch.
During integration tests scenario when index completely deleted at elasticsearch are often, and in that case,
if index ...# Description:
When the indexer receives an event with Op type DELETE, it deletes provided records at elasticsearch.
During integration tests scenario when index completely deleted at elasticsearch are often, and in that case,
if index receives DELETE events for specific records in that index, they will always fail:
~~~
{
"data": "[{\"id\":\"osdu:work-product-component--WellboreMarkerSet:60182840a28944bb81d81cc4d4c60136\",\"kind\":\"testtest:wks:work-product-component--WellboreMarkerSet:1.0.0\",\"op\":\"delete\"}]",
"attributes": {
"account-id": "{{data-partition-id}}"
}
}
~~~
But due to NOT_FOUND response from elasticsearch this task will go for the next round of retry
~~~
"{auditLog={resources=[RecordStatus(id=osdu:inttest:164613302639927, kind=osdu:ds:inttest:1.0.1646133014242, operationType=delete, status=FAIL, message=[osdu-ds-inttest-1.0.1646133014242] ElasticsearchException[Elasticsearch exception [type=index_not_found_exception, reason=no such index [osdu-ds-inttest-1.0.1646133014242]]])], action=DELETE, actionId=IN003, message=Failed deleting record in index, user=workload-indexer-gcp@nice-etching-277309.iam.gserviceaccount.com, status=FAILURE}} {account-id=osdu, correlation-id=8729b1d4-17c9-4141-9257-5fd63ec9c373, data-partition-id=osdu}"
~~~
Currently, this forms a cycle at GCP env with endlessly task retry.
# How to test:
Does functionality was tested and how?
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [x] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [ ] Common code
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [ ] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M11 - Release 0.14Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/421Fix spring-boot-maven-plugin2022-11-28T16:06:05ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comFix spring-boot-maven-pluginM15 - Release 0.18Chad LeongChad Leonghttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/659Fix Spring vulneraibilites2023-12-13T18:54:49ZSolomon AyalewFix Spring vulneraibilites* pull latest version os-core-common
* fix CVE-2023-20863,CVE-2023-20883, CVE-2023-34034 and other org.springframework:spring-core/security vulnerabilities* pull latest version os-core-common
* fix CVE-2023-20863,CVE-2023-20883, CVE-2023-34034 and other org.springframework:spring-core/security vulnerabilitiesM22 - Release 0.25https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/175fix swagger2023-08-18T15:07:41ZNeelesh Thakurfix swagger- Remove `slb-frame-of-reference` header from swagger page, this header is not applicable for Indexer API
- Hide cleanup-indicies-api, this API is meant for CRON to clean-up indices based of some pattern
Changes are highlighted in pictu...- Remove `slb-frame-of-reference` header from swagger page, this header is not applicable for Indexer API
- Hide cleanup-indicies-api, this API is meant for CRON to clean-up indices based of some pattern
Changes are highlighted in picture below:
![image](/uploads/5f140ec987594f27902378b40e06fb2a/image.png)M8 - Release 0.11https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/306Fix swagger for indexer service2023-08-18T15:05:56ZMina OtgonboldFix swagger for indexer service[Issue-47](https://community.opengroup.org/osdu/platform/home/-/issues/47)
This MR updates the required changes from the swagger version update from 2.0 to 3.0. Additionally, set the default response headers to avoid default Spring Secu...[Issue-47](https://community.opengroup.org/osdu/platform/home/-/issues/47)
This MR updates the required changes from the swagger version update from 2.0 to 3.0. Additionally, set the default response headers to avoid default Spring Security response headers(https://docs.spring.io/spring-security/reference/features/exploits/headers.html).M12 - Release 0.15https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/713Fix the issue that augmenter throws null pointer exception when casting the r...2024-02-26T14:54:23ZZhibin MaiFix the issue that augmenter throws null pointer exception when casting the related object ids retrieved from the GeoContextThe bug was introduced in M21. For the detail of the issue, please refer to issue [151](https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/151).The bug was introduced in M21. For the detail of the issue, please refer to issue [151](https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/151).M23 - Release 0.26Zhibin MaiZhibin Maihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/69Fix typo in IndexerService.md2021-03-25T17:21:26ZSviatoslav NekhaienkoFix typo in IndexerService.mdFix typoFix typohttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/77fix unexpected conversion from int to double2023-08-18T15:09:10ZNeelesh Thakurfix unexpected conversion from int to doubleFix unexpected conversion from int to double caused by Gson lib. Use Jackson lib to deserialize storage service response instead.Fix unexpected conversion from int to double caused by Gson lib. Use Jackson lib to deserialize storage service response instead.M6 - Release 0.9https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/45fix unexpected conversion from int to double2021-07-01T01:39:28ZSviatoslav Nekhaienkofix unexpected conversion from int to doubleGson parser parses any integer value as Double,
Jackson is smarter, it uses Integer of BigIntegerGson parser parses any integer value as Double,
Jackson is smarter, it uses Integer of BigIntegerSviatoslav NekhaienkoSviatoslav Nekhaienko