Indexer merge requestshttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests2021-12-08T15:32:28Zhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/241Revert "Merge branch 'istioDnsFix' into 'master'"2021-12-08T15:32:28ZAnkur RawatRevert "Merge branch 'istioDnsFix' into 'master'"This reverts merge request !240This reverts merge request !240Ankur RawatAnkur Rawathttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/374Revert "Merge branch 'indexing_virtual_properties' into 'master'"2022-09-01T12:27:12ZMarc Burnie [AWS]Revert "Merge branch 'indexing_virtual_properties' into 'master'"This reverts merge request !357This reverts merge request !357https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/529Revert "Merge branch 'GONGR-6885-reduce_requests_according_to_VPA'2023-08-18T15:04:59ZOleksandr Kosse (EPAM)Revert "Merge branch 'GONGR-6885-reduce_requests_according_to_VPA'This reverts merge request !526This reverts merge request !526M18 - Release 0.21https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/422Revert "Merge branch 'azure-m14-candidate' into azure/m14-master"2022-11-29T16:20:09ZKamalika SahaRevert "Merge branch 'azure-m14-candidate' into azure/m14-master"This reverts merge request !418This reverts merge request !418Kamalika SahaKamalika Sahahttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/606Revert "Merge branch 'augmenter_performance_enhancement' into 'master'"2023-08-25T13:51:52ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRevert "Merge branch 'augmenter_performance_enhancement' into 'master'"This reverts merge request !601This reverts merge request !601Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/541retry upsert when bulk response is 400 and matches error type2023-05-19T08:06:03ZMykyta Savchukretry upsert when bulk response is 400 and matches error typeWhen upserting a record, if the elastic responds with 400 and `mapper_parsing_exception` error type due to the record data, immediately retry upserting the record with empty data.When upserting a record, if the elastic responds with 400 and `mapper_parsing_exception` error type due to the record data, immediately retry upserting the record with empty data.M18 - Release 0.21Mykyta SavchukMykyta Savchukhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/382Retry and call for storage service removed2022-10-04T04:57:57ZHarshika DhootRetry and call for storage service removedHarshika DhootHarshika Dhoothttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/165Resolve "Upgrade Core Common Dependency"2022-02-08T12:52:02ZDavid Diederichd.diederich@opengroup.orgResolve "Upgrade Core Common Dependency"Closes #21
See osdu/platform&9 for more details.
## MR's Raison d'ĂȘtre
This MR updates the core common dependency to use the latest released version of the library. Keeping our library usage at the latest helps avoid subtle incompati...Closes #21
See osdu/platform&9 for more details.
## MR's Raison d'ĂȘtre
This MR updates the core common dependency to use the latest released version of the library. Keeping our library usage at the latest helps avoid subtle incompatibility bugs from creeping in, and ensures that all Data Platform code is staying current with bug fixes and performance improvements in the core libraries.
*Dependency Table, after the application of this MR's changes*
| Maven Dependencies | _Root POM_ | testing/ |
| ------------------------ | ------------------ | ---------- |
| os-core-common | 0.12.0-rc3 | 0.12.0-rc3 |
| core.aws.os-core-lib-aws | 0.11.0 | 0.3.16 |
| core-lib-azure | 0.11.0-rc5 | 0.11.0-rc4 |
| core-lib-gcp | 0.10.0, 0.12.0-rc3 | |
| os-core-lib-ibm | 0.12.0-SNAPSHOT | 0.7.0 |
## Housekeeping Updates
### FOSSA Update
This MR also updates the FOSSA attributions, since these need to be kept up to date on every MR. Changes are expected whenever the library dependencies are updated.
### Update Deprecated Pipeline Includes
This MR also updates the `.gitlab-ci.yml` file to use the newer pipeline includes, rather than relying on deprecated includes that mimic old behavior. These housekeeping updates should be done regularly on MRs when they are discovered so the old CI include files can be deleted and no longer supported.M9 - Release 0.12David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/669Resolve the types of the augmented properties recursively2023-12-15T20:06:39ZZhibin MaiResolve the types of the augmented properties recursivelyIssue [120](https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/120)
The fix can resolve schema of the augmented properties recursively. It also has logic to prevent infinite loop when circular references exist...Issue [120](https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/120)
The fix can resolve schema of the augmented properties recursively. It also has logic to prevent infinite loop when circular references exist. In the unit test examples:
- It travels 3 levels of relationships to get the schema of the augmented properties
- `Wellbore` augments `data.Curves[].Mnemonic` as augmented property `WellLogs` from kind `WellLog` while `WellLog` augments several properties from `Wellbore`.M22 - Release 0.25Zhibin MaiZhibin Maihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/628resolve the bug of "/reindex/record API returns 500 when there is no not-foun...2023-10-13T11:53:33ZMingyang Zhuresolve the bug of "/reindex/record API returns 500 when there is no not-found records"fix the bug and add test cases to cover the scenariofix the bug and add test cases to cover the scenarioM21 - Release 0.24Mingyang ZhuMingyang Zhuhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/591Resolve duplicate index PropertyConfigurations by choosing the last modify one2023-08-01T15:34:55ZZhibin MaiResolve duplicate index PropertyConfigurations by choosing the last modify oneIn current implementation of the index Augmenter, there is no ideal mechanism to merge duplicate Index PropertyConfiguratons for given kind. It can only pick one. And we can't prevent users from creating duplicate Index PropertyConfigura...In current implementation of the index Augmenter, there is no ideal mechanism to merge duplicate Index PropertyConfiguratons for given kind. It can only pick one. And we can't prevent users from creating duplicate Index PropertyConfiguratons (using same code but different record id) using the Storage Service API.
In the latest pre-ship test of index augmenter, a user created duplicate Index PropertyConfiguratons by mistake and forgot that a duplicate Index PropertyConfiguratons record was created. No matter how the user changed the new one, the Index Augmenter did not take effect on the new one. User had no idea what happened as the new configuration is correct.
We found that it could be a common problem, especially for users who are not aware of the limitation - no conflict/merge strategy is applied on duplicate Index PropertyConfiguations.
Before we can come out better strategy to deal with duplicate Index PropertyConfiguratons, we change the augmenter to pick the last modified one if there are more than one Index PropertyConfiguarations for a given kind and log a warning message. The change is simple and should be safe.M20 - Release 0.23Zhibin MaiZhibin Maihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/699resiliency fixes on cache miss2024-01-27T14:42:28ZNeelesh Thakurresiliency fixes on cache missresiliency fixes on cache miss
- retry fetching schema from schema service if there is a cache miss, otherwise it will incorrectly index the record.
- skip populating VirtualProperties if there a cache, otherwise it throws NullPointerExc...resiliency fixes on cache miss
- retry fetching schema from schema service if there is a cache miss, otherwise it will incorrectly index the record.
- skip populating VirtualProperties if there a cache, otherwise it throws NullPointerException.
Provide audit log entry for partially successful indexed record. Currently we are populating only for successful or failed request only.M23 - Release 0.26https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/312Replace printStackTrace() with logging2023-08-18T15:05:51ZMorris EstepaReplace printStackTrace() with loggingM12 - Release 0.15Morris EstepaMorris Estepahttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/508Renaming dependency helm chart repo to osdu-aws and removing versioning const...2023-03-17T19:44:42ZMarc Burnie [AWS]Renaming dependency helm chart repo to osdu-aws and removing versioning constraintsM17 - Release 0.20Marc Burnie [AWS]Marc Burnie [AWS]https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/625Rename some classes and variables to improve readable2023-10-04T07:56:46ZZhibin MaiRename some classes and variables to improve readableThis is part of the efforts to improve the readability of the augmenter related codes. The major changes is to rename some classes and variables via IDE tool. It also includes two minor changes:
- Reduce redundant codes in class Property...This is part of the efforts to improve the readability of the augmenter related codes. The major changes is to rename some classes and variables via IDE tool. It also includes two minor changes:
- Reduce redundant codes in class PropertyPathDeserializer.class
- Use OSDU kind pattern to verify the Code property in class AugmenterConfiguration.classM21 - Release 0.24Zhibin MaiZhibin Maihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/535Renamed domain env variable (GONRG-6912)2023-05-18T08:28:32ZYurii Ruban [EPAM / GCP]Renamed domain env variable (GONRG-6912)# Description:
Renamed the domain environment variable to group id for a better understanding of the origin of the variables
# How to test:
Check unit and int tests.
# Changes include:
- [ ] Refactor (a non-breaking change that impro...# Description:
Renamed the domain environment variable to group id for a better understanding of the origin of the variables
# How to test:
Check unit and int tests.
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
# Changes in:
- [ ] GCPM18 - Release 0.21Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/513removing vulnerability in indexer2023-04-05T05:59:46ZAkanksha Prasadremoving vulnerability in indexerRemoving vulnerabilities in indexer searchRemoving vulnerabilities in indexer searchAkanksha PrasadAkanksha Prasadhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/663Remove SNAPSHOT dependencies2023-12-08T23:57:42ZDavid Diederichd.diederich@opengroup.orgRemove SNAPSHOT dependenciesThis automated MR removes usage of `SNAPSHOT` versions in the first party library dependencies.
Since `SNAPSHOT` dependencies change frequently -- by their nature -- usage of them across projects is dangerous and should be avoided.
### ...This automated MR removes usage of `SNAPSHOT` versions in the first party library dependencies.
Since `SNAPSHOT` dependencies change frequently -- by their nature -- usage of them across projects is dangerous and should be avoided.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: e171f71ff98da301ec0772e5f3bd997bd378aedb
Maven: 0.25.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | -------------------------- | ---------------------- |
| core-lib-azure | 0.24.0 | 0.24.0 |
| core-lib-gc | 0.24.0 | |
| os-core-lib-aws | 0.25.1-SNAPSHOT | 0.24.0 |
| oqm | 0.24.0 | |
| os-core-common | 0.24.0 | 0.24.0 |
| os-core-lib-ibm | 0.24.0 | 0.24.0 |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.1, 2.13.2.2, 2.13.4.2 | 2.13.2.2, 2.13.2 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.11.1, 2.11.2, 2.13.0 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.11.2, 2.13.0 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.26, 1.27, 2.0 |
### Dependency Information After the Upgrade
```
Branch: snapshot-removal
SHA: 6b572913d8ca6efd5325570cd0e020bfe88bade8
Maven: 0.25.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | -------------------------- | ---------------------- |
| core-lib-azure | 0.24.0 | 0.24.0 |
| core-lib-gc | 0.24.0 | |
| os-core-lib-aws | 0.25.0-rc3 | 0.24.0 |
| oqm | 0.24.0 | |
| os-core-common | 0.24.0 | 0.24.0 |
| os-core-lib-ibm | 0.24.0 | 0.24.0 |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.1, 2.13.2.2, 2.13.4.2 | 2.13.2.2, 2.13.2 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.11.1, 2.11.2, 2.13.0 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.11.2, 2.13.0 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.26, 1.27, 2.0 |M22 - Release 0.25https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/635Remove SNAPSHOT dependencies2023-10-13T23:10:22ZDavid Diederichd.diederich@opengroup.orgRemove SNAPSHOT dependenciesThis automated MR removes usage of `SNAPSHOT` versions in the first party library dependencies.
Since `SNAPSHOT` dependencies change frequently -- by their nature -- usage of them across projects is dangerous and should be avoided.
### ...This automated MR removes usage of `SNAPSHOT` versions in the first party library dependencies.
Since `SNAPSHOT` dependencies change frequently -- by their nature -- usage of them across projects is dangerous and should be avoided.
### Dependency Information Before the Upgrade
```
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Branch: master
SHA: 0bc88510c245349dac7f6d46708c2655afa0021f
Maven: 0.24.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | -------------------------- | ---------------------- |
| core-lib-azure | 0.20.0-rc5 | 0.12.0-rc10 |
| core-lib-gc | 0.21.0 | |
| os-core-lib-aws | 0.24.0-SNAPSHOT | 0.21.0 |
| oqm | 0.21.0 | |
| os-core-common | 0.22.0-rc4 | 0.14.0-rc8 |
| os-core-lib-ibm | 0.16.0-rc1 | 0.13.0 |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.1, 2.13.2.2, 2.13.4.2 | 2.13.2.2, 2.13.2 |
| (3rd Party) net.minidev.json-smart | 2.4.7 | 2.4.6 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.11.1, 2.11.2, 2.13.0 |
| (3rd Party) org.apache.logging.log4j.log4j-core | 2.17.1 | 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-jul | 2.17.1 | 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.11.2, 2.13.0 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.26 |
### Dependency Information After the Upgrade
```
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Branch: dependency-upgrade-2
SHA: 90fde1cb02075f664cb1009879697e7a1908a1a0
Maven: 0.24.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | -------------------------- | ---------------------- |
| core-lib-azure | 0.20.0-rc5 | 0.12.0-rc10 |
| core-lib-gc | 0.21.0 | |
| os-core-lib-aws | 0.24.0 | 0.21.0 |
| oqm | 0.21.0 | |
| os-core-common | 0.22.0-rc4 | 0.14.0-rc8 |
| os-core-lib-ibm | 0.16.0-rc1 | 0.13.0 |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.14.1, 2.13.2.2, 2.13.4.2 | 2.13.2.2, 2.13.2 |
| (3rd Party) net.minidev.json-smart | 2.4.7 | 2.4.6 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.11.1, 2.11.2, 2.13.0 |
| (3rd Party) org.apache.logging.log4j.log4j-core | 2.17.1 | 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-jul | 2.17.1 | 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.11.2, 2.13.0 |
| (3rd Party) org.yaml.snakeyaml | 2.0 | 1.26 |M21 - Release 0.24https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/491Remove SNAPSHOT dependencies2023-02-18T07:29:34ZDavid Diederichd.diederich@opengroup.orgRemove SNAPSHOT dependenciesThis automated MR removes usage of `SNAPSHOT` versions in the first party library dependencies.
Since `SNAPSHOT` dependencies change frequently -- by their nature -- usage of them across projects is dangerous and should be avoided.
### ...This automated MR removes usage of `SNAPSHOT` versions in the first party library dependencies.
Since `SNAPSHOT` dependencies change frequently -- by their nature -- usage of them across projects is dangerous and should be avoided.
### Dependency Information Before the Upgrade
```
Branch: master
SHA: 58e2828de09ee79e5a399bb6dbfcf8d11eb030e2
Maven: 0.20.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | --------------------------- | ---------------------- |
| core-lib-azure | 0.18.0-rc3 | 0.12.0-rc10 |
| core-lib-gcp | 0.19.0-rc7 | |
| os-core-lib-aws | 0.18.0 | 0.14.0-rc2 |
| obm | 0.19.0-rc4 | |
| oqm | 0.19.0-rc4 | |
| os-core-common | 0.19.0-SNAPSHOT, 0.19.0-rc3 | 0.14.0-rc8 |
| os-core-lib-ibm | 0.16.0-rc1 | 0.13.0 |
| osm | 0.18.0 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.13.4.2, 2.13.2.2, 2.13.4 | 2.13.2.2, 2.13.2 |
| (3rd Party) net.minidev.json-smart | 2.4.7 | 2.4.6 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.11.1, 2.11.2, 2.13.0 |
| (3rd Party) org.apache.logging.log4j.log4j-core | 2.17.1 | 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-jul | 2.17.1 | 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.11.2, 2.13.0 |
| (3rd Party) org.springframework.spring-webflux | 5.3.22 | |
| (3rd Party) org.springframework.spring-webmvc | 5.3.24, 5.3.22 | 5.3.23 |
### Dependency Information After the Upgrade
```
Branch: dependency-upgrade-4
SHA: 4f1e194e9fad9e6608e1890630925267ea1ef856
Maven: 0.20.0-SNAPSHOT
```
| Maven Dependencies | _Root_ | testing/ |
| ------------------------------------------------------- | -------------------------- | ---------------------- |
| core-lib-azure | 0.18.0-rc3 | 0.12.0-rc10 |
| core-lib-gcp | 0.19.0-rc7 | |
| os-core-lib-aws | 0.18.0 | 0.14.0-rc2 |
| obm | 0.19.0-rc4 | |
| oqm | 0.19.0-rc4 | |
| os-core-common | 0.19.0 | 0.14.0-rc8 |
| os-core-lib-ibm | 0.16.0-rc1 | 0.13.0 |
| osm | 0.18.0 | |
| (3rd Party) com.fasterxml.jackson.core.jackson-databind | 2.13.4.2, 2.13.2.2, 2.13.4 | 2.13.2.2, 2.13.2 |
| (3rd Party) net.minidev.json-smart | 2.4.7 | 2.4.6 |
| (3rd Party) org.apache.logging.log4j.log4j-api | 2.17.1 | 2.11.1, 2.11.2, 2.13.0 |
| (3rd Party) org.apache.logging.log4j.log4j-core | 2.17.1 | 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-jul | 2.17.1 | 2.13.3 |
| (3rd Party) org.apache.logging.log4j.log4j-to-slf4j | 2.17.1 | 2.11.2, 2.13.0 |
| (3rd Party) org.springframework.spring-webflux | 5.3.22 | |
| (3rd Party) org.springframework.spring-webmvc | 5.3.24, 5.3.22 | 5.3.23 |M16 - Release 0.19