Indexer merge requestshttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests2024-02-09T14:24:31Zhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/707Draft: Vazha/test default value2024-02-09T14:24:31ZAnthony Ittiyera vazhappillyDraft: Vazha/test default valuehttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/706Avoid unnecessary second call to get the properties from the VirtualPropertie...2024-01-30T14:43:55ZZhibin MaiAvoid unnecessary second call to get the properties from the VirtualProperties cacheIt is minor enhancement of MR [699](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/699):
- Should not skip geometry decimation even the virtual properties does not exist which is possible for cust...It is minor enhancement of MR [699](https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/699):
- Should not skip geometry decimation even the virtual properties does not exist which is possible for custom schema. Please note that geometry decimation is not only on virtual properties.
The MR also includes two minor enhancement:
- Avoid calling the cache service twice unnecessarily
- Implement and use partition safe virtual properties cache.M23 - Release 0.26Zhibin MaiZhibin Maihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/705Indexer-core-plus service2024-03-26T10:31:37ZJoshua HindmonIndexer-core-plus serviceCreation of new indexer-core-plus for Community Implementation.
(Work in progress, not ready to merge.)
## Type of change
* [ ] Initial commit (a change for the initial development of a project)
* [x] Refactor (a non-breaking change...Creation of new indexer-core-plus for Community Implementation.
(Work in progress, not ready to merge.)
## Type of change
* [ ] Initial commit (a change for the initial development of a project)
* [x] Refactor (a non-breaking change that improves code maintainability).
* [ ] Bugfix (a non-breaking change that solves an issue).
* [ ] Dependencies upgrade (a version change upgrade for release management)
* [ ] Enhancement (a change on the existing code, increasing the efficiency or effectiveness of the software without necessarily introducing new features to end user, can include approved ADR)
* [ ] New feature (a non-breaking change that adds functionality, must include approved ADR).
* [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality, must include approved ADR).
* [ ] Deprecation (a change for code deprecation, will need to include this in the release notes)
* [ ] DevOps (a change for CICD related / DevOps for release management)
* [ ] Documentation (a change of documention only)
* [ ] Logging (a change for work related to internal audit, logging, reporting improvement)
* [ ] FOSSA Notice (a change for FOSSA notice update)
* [ ] Revert (a change for reverting a previous MR)
* [ ] Testing (a changes for improving unit, integration, etc. testing)
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
[Refactor indexer-gc (#13) · Issues · OSDU Forum / OSDU Project Management Committee - PMC / Community Implementation · GitLab (opengroup.org)](https://gitlab.opengroup.org/osdu/pmc/community-implementation/-/issues/13)
## Does this introduce a change in the common core logic?
* \[YES/**NO**\]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
* [ ] AWS
* [ ] Azure
* [ ] GCP
* [ ] IBM
## What is the current behavior?
## What is the new/expected behavior?
## Have you added/updated Unit Tests and Integration Tests?
## Any other useful informationM23 - Release 0.26Joshua HindmonJoshua Hindmonhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/704Augmenter refresh schema mapping2024-01-26T18:29:03ZZhibin MaiAugmenter refresh schema mappingIssue [136](https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/136)
Please pay attention to the change in initialization of the augmenter integration tests. The step `Then I clean up the index of the extended ...Issue [136](https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/136)
Please pay attention to the change in initialization of the augmenter integration tests. The step `Then I clean up the index of the extended kinds <extendedKinds> in the Elastic Search` was to delete the index in order to enforce the update of schema mapping in ElasticSearch index.
With this MR, the update of the augmenter configuration will automatically update the schema mapping in ElasticSearch index.M23 - Release 0.26Zhibin MaiZhibin Maihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/703Cherry-pick 'Fix parsing string arrays' into release/0.252024-01-24T11:49:37ZChad LeongCherry-pick 'Fix parsing string arrays' into release/0.25**Original MR**: !693
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !693
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/indexer-service/-/pipelines/new?ref=cherry-pick-for-693)M22 - Release 0.25David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/702Revert "Merge branch 'sbstan/bag-of-words-indexing' into 'master'"2024-01-24T12:01:05ZChad LeongRevert "Merge branch 'sbstan/bag-of-words-indexing' into 'master'"This reverts commit 820860e381079deba98e3885c33f30ff6e629ed9.
Conflicts:
provider/indexer-azure/src/main/resources/application.propertiesThis reverts commit 820860e381079deba98e3885c33f30ff6e629ed9.
Conflicts:
provider/indexer-azure/src/main/resources/application.propertieshttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/701Draft: fix spring-security-vulnerability2024-02-08T18:16:55ZVidyaDharani LokamDraft: fix spring-security-vulnerabilityVidyaDharani LokamVidyaDharani Lokamhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/700[MSCOSDU-1894] exclude unused maven-core dependency2024-01-25T17:01:10ZVidyaDharani Lokam[MSCOSDU-1894] exclude unused maven-core dependency# Change details
* reference issues: https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/127 https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/126 https://community.opengroup.org/osd...# Change details
* reference issues: https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/127 https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/126 https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/125 https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/124 https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/123
* excluded unused `maven-core` dependency to remediate vulnerability.
* upgraded `spring-security-core` to `5.8.9`
# Changes in:
* [x] GCP
* [x] Azure
* [x] AWS
* [ ] IBM
* [x] commoncodeM23 - Release 0.26VidyaDharani LokamVidyaDharani Lokamhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/699resiliency fixes on cache miss2024-01-27T14:42:28ZNeelesh Thakurresiliency fixes on cache missresiliency fixes on cache miss
- retry fetching schema from schema service if there is a cache miss, otherwise it will incorrectly index the record.
- skip populating VirtualProperties if there a cache, otherwise it throws NullPointerExc...resiliency fixes on cache miss
- retry fetching schema from schema service if there is a cache miss, otherwise it will incorrectly index the record.
- skip populating VirtualProperties if there a cache, otherwise it throws NullPointerException.
Provide audit log entry for partially successful indexed record. Currently we are populating only for successful or failed request only.M23 - Release 0.26https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/698update azure.yml ref. to disable azure deploy for indexer service2024-01-19T11:32:11Zsaketh somarajuupdate azure.yml ref. to disable azure deploy for indexer service- Indexer service's azure/m18-master branches causes issues in GLAB env. due to significant code changes from current master.
- Which in turn causes issues in indexing of newly ingested records in dev. env. effecting other services like ...- Indexer service's azure/m18-master branches causes issues in GLAB env. due to significant code changes from current master.
- Which in turn causes issues in indexing of newly ingested records in dev. env. effecting other services like dags, ddms etc.,
- This change disables azure deploy in azure/m18 master branch and it's respective child branches.
- However, one of the downside of this change is team can't test indexer service (for m18 branch) in glab env.
CC @nursheikh , @lucynliusaketh somarajusaketh somarajuhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/697Draft: additional array tests2024-01-23T12:10:48ZMark ChanceDraft: additional array testshttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/696Mchance/integration test string array2024-01-10T16:33:30ZMark ChanceMchance/integration test string arrayhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/695Cherry-pick '[MSCOSDU-1971] variablize and enable feature flags implemented' ...2024-01-09T12:04:38ZChad LeongCherry-pick '[MSCOSDU-1971] variablize and enable feature flags implemented' into release/0.25**Original MR**: !691
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !691
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/indexer-service/-/pipelines/new?ref=cherry-pick-for-691)M22 - Release 0.25David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/694Support parsing own output format for datetimes2024-03-04T12:23:17ZStanisław BienieckiSupport parsing own output format for datetimesResolves: https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/138
Currently indexer doesn't accept datetime format which formatter produces as an output. This is causing for example that you cannot simply copy ...Resolves: https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/138
Currently indexer doesn't accept datetime format which formatter produces as an output. This is causing for example that you cannot simply copy datetimes between resources.M23 - Release 0.26https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/693Fix parsing string arrays2024-02-08T19:27:30ZStanisław BienieckiFix parsing string arraysCloses https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/137
Hotfix for parsing string arrays, as previous fix in handling string introduced error in parsing string arraysCloses https://community.opengroup.org/osdu/platform/system/indexer-service/-/issues/137
Hotfix for parsing string arrays, as previous fix in handling string introduced error in parsing string arraysM22 - Release 0.25https://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/692Merge branch 'lib-version-upgrade' into 'master'2024-01-05T08:44:43ZNikhil Patilnikhil.patil5@ibm.comMerge branch 'lib-version-upgrade' into 'master'updated ibm lib version
See merge request osdu/platform/system/indexer-service!690
(cherry picked from commit 9378051f04691afabffe7ff873652fd9347ffeb5)
40924b70 updated ibm lib version
708434db updated spring security dependency
5436a...updated ibm lib version
See merge request osdu/platform/system/indexer-service!690
(cherry picked from commit 9378051f04691afabffe7ff873652fd9347ffeb5)
40924b70 updated ibm lib version
708434db updated spring security dependency
5436aa9a Merge branch 'master' of...
a70e4070 added bom for security in pomM22 - Release 0.25Nikhil Patilnikhil.patil5@ibm.comNikhil Patilnikhil.patil5@ibm.comhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/691[MSCOSDU-1971] variablize and enable feature flags implemented2024-01-09T09:22:50ZDeepa Kumari[MSCOSDU-1971] variablize and enable feature flags implementedVariablize and enable feature flags for strategy, asIngestedCoordinates, keywordLower and bagOfWordsVariablize and enable feature flags for strategy, asIngestedCoordinates, keywordLower and bagOfWordsM22 - Release 0.25Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/690updated ibm lib version2024-01-05T08:36:20ZNikhil Patilnikhil.patil5@ibm.comupdated ibm lib versionupdated ibm lib versionupdated ibm lib versionM22 - Release 0.25Nikhil Patilnikhil.patil5@ibm.comNikhil Patilnikhil.patil5@ibm.comhttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/689Updated ibm lib version2023-12-28T10:11:21ZIsha KumariUpdated ibm lib versionUpdated ibm lib versionUpdated ibm lib versionM22 - Release 0.25Isha KumariIsha Kumarihttps://community.opengroup.org/osdu/platform/system/indexer-service/-/merge_requests/688[MSCOSDU-1894] fix logback, json-smart and reactor-netty-http vulnerability2024-01-04T09:21:21ZVidyaDharani Lokam[MSCOSDU-1894] fix logback, json-smart and reactor-netty-http vulnerability* excluded `logback` dependencies and upgraded `log4j` version for Azure
* upgraded `json-smart` version to `2.5.0` to remediate vulnerability
* upgraded `reactor-netty-http` to `1.1.14` to remediate vulnerability* excluded `logback` dependencies and upgraded `log4j` version for Azure
* upgraded `json-smart` version to `2.5.0` to remediate vulnerability
* upgraded `reactor-netty-http` to `1.1.14` to remediate vulnerabilityM23 - Release 0.26VidyaDharani LokamVidyaDharani Lokam