Search merge requestshttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests2024-01-09T13:13:16Zhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/556move optimized geoquery builder to core2024-01-09T13:13:16ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.commove optimized geoquery builder to coreIssue: https://gitlab.opengroup.org/osdu/pmc/community-implementation/-/issues/20
In the Search service, there is duplicated code for building Elasticsearch queries present in CSP modules. This MR refactors Geo query building, moving it...Issue: https://gitlab.opengroup.org/osdu/pmc/community-implementation/-/issues/20
In the Search service, there is duplicated code for building Elasticsearch queries present in CSP modules. This MR refactors Geo query building, moving it to the Core, Azure version is used as a sample.
## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES/NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] Google Cloud
- [x] IBM
- [x] Common code
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Code for Geo query building duplicated (with differences) in each CSP module.
## What is the new/expected behavior?
Code for Geo query building extracted to the core, Azure optimized query builder is used.
## Have you added/updated Unit Tests and Integration Tests?
Yes
## Any other useful informationM21 - Release 0.24Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/554GONRG-7679: update policy bootstrap2023-09-11T08:30:42ZYauheni Rykhter (EPAM)GONRG-7679: update policy bootstrapM21 - Release 0.24Yauheni Rykhter (EPAM)Yauheni Rykhter (EPAM)https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/553Cherry-pick 'Enable highlighting results' into release/0.232023-09-01T15:47:55ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Enable highlighting results' into release/0.23**Original MR**: !541
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !541
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/search-service/-/pipelines/new?ref=cherry-pick-for-541)M20 - Release 0.23David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/551Cherry-pick: Merge branch 'fix-sort-test' into 'master'2023-08-29T11:36:00ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comCherry-pick: Merge branch 'fix-sort-test' into 'master'Cherry-pick from https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/550
## Type of change
- [ ] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
##...Cherry-pick from https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/550
## Type of change
- [ ] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES/NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] Google Cloud
- [ ] IBM
## Does this introduce a breaking change?
- [YES/NO]
## What is the current behavior?
## What is the new/expected behavior?
## Have you added/updated Unit Tests and Integration Tests?
## Any other useful informationM20 - Release 0.23Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/550Switch tests to common and updated sort model2023-08-29T07:50:02ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comSwitch tests to common and updated sort model
## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud prov...
## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] Common code
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
The custom model in a testing directory is not aligned with the updated model from core-common, making the sort test with filter fail.
## What is the new/expected behavior?
The common model works fine, and the request with the new filter property formed as expected.M20 - Release 0.23Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/548Merge branch 'azure/m16-master-FixVulnerability' into 'azure/m16-master'2023-08-25T17:55:46ZNathan StrelserMerge branch 'azure/m16-master-FixVulnerability' into 'azure/m16-master'## Type of change
- [ ] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES/NO]
## Does this introduce a change in the cloud p...## Type of change
- [ ] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES/NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] Google Cloud
- [ ] IBM
## Does this introduce a breaking change?
- [YES/NO]
## What is the current behavior?
## What is the new/expected behavior?
## Have you added/updated Unit Tests and Integration Tests?
## Any other useful informationNathan StrelserNathan Strelserhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/546Revert "Merge branch 'GONRG-7560-policy-update' into 'master'"2023-09-05T08:06:10ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRevert "Merge branch 'GONRG-7560-policy-update' into 'master'"## Type of change
- [ ] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES/NO]
## Does this introduce a change in the cloud p...## Type of change
- [ ] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES/NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] Google Cloud
- [ ] IBM
## Does this introduce a breaking change?
- [YES/NO]
## What is the current behavior?
## What is the new/expected behavior?
## Have you added/updated Unit Tests and Integration Tests?
## Any other useful informationM20 - Release 0.23https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/545move changes from m16 to m182023-08-23T23:22:36ZNathan Strelsermove changes from m16 to m18## Type of change
- [ ] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES/NO]
## Does this introduce a change in the cloud p...## Type of change
- [ ] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES/NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] Google Cloud
- [ ] IBM
## Does this introduce a breaking change?
- [YES/NO]
## What is the current behavior?
## What is the new/expected behavior?
## Have you added/updated Unit Tests and Integration Tests?
## Any other useful informationNathan StrelserNathan Strelserhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/544GONRG-7560: update policy in the service2023-08-24T10:19:21ZYauheni Rykhter (EPAM)GONRG-7560: update policy in the serviceM20 - Release 0.23Yauheni Rykhter (EPAM)Yauheni Rykhter (EPAM)https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/543Exclude system/metadata indices in searches with wildcard kind2023-09-04T07:33:32ZZhibin MaiExclude system/metadata indices in searches with wildcard kind## Type of change
- [ ] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud prov...## Type of change
- [ ] Bug Fix
- [x] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] Google Cloud
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
In searches with wildcard kind, only the ElasticSearch system/internal indices started with dot will be excluded.
## What is the new/expected behavior?
In searches with wildcard kind, the ElasticSearch system/internal indices started with dot and the indices with authority as "system-meta-data" will be excluded.
## Have you added/updated Unit Tests and Integration Tests?
Yes
## Any other useful information
- ADR [ADR: Exclude indices of the system/meta data from the search results unless the indices (kinds) of the system/meta data are explicitly specified in the search query](https://community.opengroup.org/osdu/platform/system/search-service/-/issues/131)
- OSDU data-definitions
- [6.1.2 Record kind](https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/Guides/Chapters/06-LifecycleProperties.md#612-record-kind)
- [Appendix D.1.3 Schema Identifier `kind` Limitations](https://community.opengroup.org/osdu/data/data-definitions/-/blob/master/Guides/Chapters/93-OSDU-Schemas.md#appendix-d13-schema-identifier-kind-limitations)
-M20 - Release 0.23Thomas Gehrmann [slb]Zhibin MaiThomas Gehrmann [slb]https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/18Added pipeline for development testing2023-08-18T22:17:16ZDaniel SchollAdded pipeline for development testingAdding an Azure Pipeline for Development Deployment.Adding an Azure Pipeline for Development Deployment.M1 - Release 0.1Daniel SchollDaniel Schollhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/19Update development-pipeline.yml2023-08-18T22:16:48ZDaniel SchollUpdate development-pipeline.ymlM1 - Release 0.1Daniel SchollDaniel Schollhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/34multi-tenancy updates for Search azure2023-08-18T22:15:49ZDuvelis Caraomulti-tenancy updates for Search azureDraft merge requests can't be merged.Draft merge requests can't be merged.M1 - Release 0.1https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/65Added Azure ADO Pipeline with Cache support2023-08-18T22:13:24ZDaniel SchollAdded Azure ADO Pipeline with Cache supportThis is an Azure only pipeline that enables ADO support for caching of Maven Libraries.This is an Azure only pipeline that enables ADO support for caching of Maven Libraries.M1 - Release 0.1Daniel SchollDaniel Schollhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/66Azure Pipeline removed reference to community settings2023-08-18T22:13:00ZDaniel SchollAzure Pipeline removed reference to community settingsAzure only pipeline for ADO fixing references to settings.Azure only pipeline for ADO fixing references to settings.M1 - Release 0.1Daniel SchollDaniel Schollhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/67Bug Fixing Azure ADO Pipeline build settings2023-08-18T22:08:54ZDaniel SchollBug Fixing Azure ADO Pipeline build settingsAzure only MR for fixing ADO customer ADO pipelines.Azure only MR for fixing ADO customer ADO pipelines.M1 - Release 0.1Daniel SchollDaniel Schollhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/542AWS Merge Dev Into Master2023-08-24T11:59:29ZGuillaume CailletAWS Merge Dev Into MasterM20 - Release 0.23Guillaume CailletGuillaume Caillethttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/541Enable highlighting results2023-08-31T06:05:10ZStanisław BienieckiEnable highlighting results## Type of change
- [ ] Bug Fix
- [X] Feature
Changes related to the ADR: https://community.opengroup.org/osdu/platform/system/search-service/-/issues/128
## Does this introduce a change in the core logic?
- [NO]
## Does this introd...## Type of change
- [ ] Bug Fix
- [X] Feature
Changes related to the ADR: https://community.opengroup.org/osdu/platform/system/search-service/-/issues/128
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [X] AWS
- [X] Azure
- [X] Google Cloud
- [X] IBM
Changes are for all provider however related code before and and after changes fully identical.
## Does this introduce a breaking change?
- [NO]
This feature had some different implementation before, but it was never enabled. Also it wasn't working correctly as it wasn't handling json paths so only top level field could be selected and there was no support for list fields.
## What is the current behavior?
Currently there is no highlighting available for the users
## What is the new/expected behavior?
Users can specify optional field `highlightedFields` which will add to each search result additional dictionary under key `highlight` with phrases from selected fields which matched the query. Example in the updated docs
## Have you added/updated Unit Tests and Integration Tests?
Unit tests have been updated and some were added
## Any other useful informationM20 - Release 0.23Stanisław BienieckiStanisław Bienieckihttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/536Fixing vulnerability2023-08-24T17:10:43ZHarsheet ShahFixing vulnerability## Type of change
- [ ] Bug Fix No
- [ ] Feature No
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES/NO] No
## Does this introduce a change in the...## Type of change
- [ ] Bug Fix No
- [ ] Feature No
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- [YES/NO] No
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS No
- [ ] Azure Yes
- [ ] Google Cloud No
- [ ] IBM No
## Does this introduce a breaking change?
- [YES/NO] No
## What is the current behavior?
## What is the new/expected behavior?
## Have you added/updated Unit Tests and Integration Tests?
## Any other useful information
Bumping version of libraries to fix vulnerabilities.Harsheet ShahHarsheet Shahhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/535Add filter to nested sort2023-08-18T22:35:31ZStanisław BienieckiAdd filter to nested sort## Type of change
- [ ] Bug Fix
- [X] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
Changes related to ADR https://community.opengroup.org/osdu/platform/system/search-service/-/issues/106
## Do...## Type of change
- [ ] Bug Fix
- [X] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
Changes related to ADR https://community.opengroup.org/osdu/platform/system/search-service/-/issues/106
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] Google Cloud
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
Currently specifying filter for nested sorting is not possible.
## What is the new/expected behavior?
Users can pass filter for nested field sorting. From official docs `A filter that the inner objects inside the nested path should match with in order for its field values to be taken into account by sorting. Common case is to repeat the query / filter inside the nested filter or query.` Filter is applied to the "first" nested level in the case of sort field being nested multiple times. Related Elastic feature docs: https://www.elastic.co/guide/en/elasticsearch/reference/7.17/sort-search-results.html#nested-sorting
## Have you added/updated Unit Tests and Integration Tests?
Unit tests have been added/updated.
## Any other useful information
parseSort method is only used in byoc-provider so that's why only this provider has changed. Others use getSortQuery which has signature unchangedM20 - Release 0.23