Search merge requestshttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests2023-08-18T22:06:48Zhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/121Nested query search2023-08-18T22:06:48ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comNested query search## Type of change
- [x] Feature
https://community.opengroup.org/osdu/platform/system/search-service/-/issues/38
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implement...## Type of change
- [x] Feature
https://community.opengroup.org/osdu/platform/system/search-service/-/issues/38
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x] IBM
- [x] Common code
## Does this introduce a breaking change?
- [NO]
## How to test:
Functionality can be tested with several requests:<br/>
With pure nested queries and their combination:
~~~
{
"kind": "osdu:wks:work-product-component--WellboreMarkerSet:1.0.0",
"query":"nested(data.Markers, (MarkerMeasuredDepth:(<15) AND MarkerName:\"North Sea Supergroup\"))"
}
~~~
~~~
{
"kind": "osdu:wks:master-data--Well:1.0.0",
"query":"nested(data.VerticalMeasurements, (VerticalMeasurement:(>15) AND VerticalReferenceID:\"Example*\")) AND nested(data.FacilityOperators, (FacilityOperatorID:\"Example*\"))",
"limit":100
}
~~~
And with combination of string queries and nested queries:
~~~
{
"kind": "osdu:wks:master-data--Well:1.0.0",
"query":"data.Source:\"Example*\" AND nested(data.VerticalMeasurements, (VerticalMeasurement:(>15) AND VerticalReferenceID:\"Example*\"))",
"limit":100
}
~~~
~~~
{
"kind": "osdu:wks:master-data--Well:1.0.0",
"query":"data.Source:\"Example*\" AND nested(data.VerticalMeasurements, (VerticalMeasurement:(>15) AND VerticalReferenceID:\"Example*\")) AND data.FacilityName:\"Example*\"",
"limit":100
}
~~~
Sort and aggregation also can be tested:
~~~
{
"kind": "osdu:wks:master-data--Well:1.0.0",
"query": "data.Source:\"Example*\" AND nested(data.VerticalMeasurements, (VerticalMeasurement:(>15) AND VerticalReferenceID:\"Example*\"))",
"limit": 100,
"sort": {
"field": [
"nested(data.FacilityOperators, TerminationDateTime, min)"
],
"order": [
"ASC"
]
},
"aggregateBy": "nested(data.VerticalMeasurements, VerticalMeasurement)"
}
~~~
## Dev Checklist:
- [x] Added Unit Tests, wherever applicable.
- [x] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M7 - Release 0.10Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/122enable sort on text fields inside the data block2023-08-18T22:06:46ZNeelesh Thakurenable sort on text fields inside the data block- enables sort on text fields (inside the data block)
- provide feedback to user if unsupported field is provided on SortQuery. Users will get error message:
```
{
"code": 400,
"reason": "Bad Request",
"message": "Sort is no...- enables sort on text fields (inside the data block)
- provide feedback to user if unsupported field is provided on SortQuery. Users will get error message:
```
{
"code": 400,
"reason": "Bad Request",
"message": "Sort is not supported for one or more of the requested fields"
}
```
- resolves issue [44](https://community.opengroup.org/osdu/platform/system/search-service/-/issues/44)
- refactor & clean field type mapping implementation and move it to core package to avoid copying same code for all providersM7 - Release 0.10https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/125Gcp timeout exception search (GONRG-2509)2023-08-18T22:06:45ZRiabokon Stanislav(EPAM)[GCP]Gcp timeout exception search (GONRG-2509)## Type of change
- [X] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] IB...## Type of change
- [X] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## What is the current behavior?
It returns 500.
## What is the new/expected behavior?
Added java.net.SocketTimeoutException during search. It will return 408 instead of 500.
## Any other useful informationM7 - Release 0.10Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/126update tutorial and swagger documentation to reflect correct max record count...2023-08-18T22:06:43ZNeelesh Thakurupdate tutorial and swagger documentation to reflect correct max record count per requestUpdate tutorial and swagger documentation to reflect correct max record count per request. It's changed to 100 --> 1000 per request some time back.Update tutorial and swagger documentation to reflect correct max record count per request. It's changed to 100 --> 1000 per request some time back.M7 - Release 0.10https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/127(GONRG-2409) gcp helm charts2023-08-18T22:06:41ZVladislav Shishko (EPAM)(GONRG-2409) gcp helm chartsM7 - Release 0.10Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/128Fix double path replacement for nested query2023-08-18T22:06:40ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comFix double path replacement for nested query# Description:
Fix for nested queries & test refactoring
# Changes include:
- [x] Bugfix (a non-breaking change that solves an issue).
# Changes in:
- [x] Common code
# Dev Checklist:
- [x] Added Unit Tests, wherever applicable.
- [x...# Description:
Fix for nested queries & test refactoring
# Changes include:
- [x] Bugfix (a non-breaking change that solves an issue).
# Changes in:
- [x] Common code
# Dev Checklist:
- [x] Added Unit Tests, wherever applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M7 - Release 0.10Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/130Add node selector2023-08-18T22:06:38ZDzmitry_Paulouski (slb)Add node selector## All Submissions:
-------------------------------------
* [YES] Have you followed our code review [guidelines](https://github.com/microsoft/code-with-engineering-playbook/blob/master/pull-requests/code-reviews/readme.md)?
* [YES] Have ...## All Submissions:
-------------------------------------
* [YES] Have you followed our code review [guidelines](https://github.com/microsoft/code-with-engineering-playbook/blob/master/pull-requests/code-reviews/readme.md)?
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [NO] I have updated the documentation accordingly.
* [NA] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [YES] I ran lint checks locally prior to submission.
## What is the current behavior?
-------------------------------------
Currently, we have only one node pool for services and system workloads.
## What is the new behavior?
-------------------------------------
We are going to move workloads to a new node pool with enabled autoscaling. Here you can find the details:
[Issue link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/167)
## Does this introduce a breaking change?
-------------------------------------
- [NO]
This change has a strong dependency on the following MR and should be merged **only** after it:
https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/324M8 - Release 0.11https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/131(GONRG-2419) Enabling publish step2023-08-18T22:06:36ZVladislav Shishko (EPAM)(GONRG-2419) Enabling publish stepM7 - Release 0.10Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/132change some info logs to debug2023-08-18T22:06:35ZAlok Joshichange some info logs to debug1. Upgrade core-lib-azure library to make use of info to debug log changes in the latest RC version
2. Incorporate debug logs into Search service. This is an effort to reduce the cost associated with storing trace (info) logs.1. Upgrade core-lib-azure library to make use of info to debug log changes in the latest RC version
2. Incorporate debug logs into Search service. This is an effort to reduce the cost associated with storing trace (info) logs.M7 - Release 0.10Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/133Minor improvements for helms [GONRG-2627]2023-08-18T22:06:33ZMikhail Piatliou (EPAM)Minor improvements for helms [GONRG-2627]M7 - Release 0.10Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/134fix bug with sort on multi text field in a request2023-08-18T22:06:32ZNeelesh Thakurfix bug with sort on multi text field in a requestCurrent implementation can only support sort on one text field at this time.
Query returns error
```
{
"kind": "osdu:csv:wellbore:2.0.0",
"sort": {
"field": [
"data.LEASE",
"data.OPERATOR"
...Current implementation can only support sort on one text field at this time.
Query returns error
```
{
"kind": "osdu:csv:wellbore:2.0.0",
"sort": {
"field": [
"data.LEASE",
"data.OPERATOR"
],
"order": [
"ASC",
"ASC"
]
}
}
```
Response returned:
```
{
"code": 400,
"reason": "Bad Request",
"message": "Sort is not supported for one or more of the requested fields"
}
```
Sort query should work with multi fieldM7 - Release 0.10https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/136Update PolicyService-Integration.md2023-08-18T22:06:30ZRiabokon Stanislav(EPAM)[GCP]Update PolicyService-Integration.mdUpdate PolicyService-Integration.mdUpdate PolicyService-Integration.mdM7 - Release 0.10Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/137Update README.md2023-08-18T22:06:29ZRiabokon Stanislav(EPAM)[GCP]Update README.mdUpdate README.md for GCPUpdate README.md for GCPM7 - Release 0.10Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/139(GONRG-2702) Update .gitlab-ci.yml2023-08-18T22:06:27ZVladislav Shishko (EPAM)(GONRG-2702) Update .gitlab-ci.ymlM7 - Release 0.10https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/140respond with 429 if server cannot process new cursor request any more2023-08-18T22:06:25ZNeelesh Thakurrespond with 429 if server cannot process new cursor request any moreElastic 7 onwards, server responds with error code 500 if there are too many scroll requests. MR convert this to 429 response.
Also log suppressed Elasticsearch exceptions to capture root cause for Elasticsearch's request failure in logs.Elastic 7 onwards, server responds with error code 500 if there are too many scroll requests. MR convert this to 429 response.
Also log suppressed Elasticsearch exceptions to capture root cause for Elasticsearch's request failure in logs.M7 - Release 0.10https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/142Changed authorize_api to v22023-08-18T22:06:24ZOleksii Tsyganov (EPAM)Changed authorize_api to v2M7 - Release 0.10Oleksii Tsyganov (EPAM)Oleksii Tsyganov (EPAM)https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/144AWS Updates2023-08-18T22:06:22ZSpencer Suttonsuttonsp@amazon.comAWS Updatescommit edbc4b49
Author: Spencer Sutton <suttonsp@amazon.com>
Date: Thu May 27 2021 10:16:11 GMT-0500 (Central Daylight Time)
"Excluding springs default security, locking down mvn central"commit edbc4b49
Author: Spencer Sutton <suttonsp@amazon.com>
Date: Thu May 27 2021 10:16:11 GMT-0500 (Central Daylight Time)
"Excluding springs default security, locking down mvn central"M7 - Release 0.10Spencer Suttonsuttonsp@amazon.comSpencer Suttonsuttonsp@amazon.comhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/147Added rollme annotation2023-08-18T22:06:21ZOleksii Tsyganov (EPAM)Added rollme annotationM8 - Release 0.11Oleksii Tsyganov (EPAM)Oleksii Tsyganov (EPAM)https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/148handle expired cursor2023-08-18T22:06:19ZNeelesh Thakurhandle expired cursorIntercept and handle expired cursor request. It should not throw 500 errors.
Addresses issue #60Intercept and handle expired cursor request. It should not throw 500 errors.
Addresses issue #60M8 - Release 0.11https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/149sort by relevancy score2023-08-18T22:06:17ZNeelesh Thakursort by relevancy scoreBy default Search service returns result in relevancy order (descending).
This MR enables reverse sort by relevancy score. This is particularly very useful feature if query response returns lot of records and users want to look at the ...By default Search service returns result in relevancy order (descending).
This MR enables reverse sort by relevancy score. This is particularly very useful feature if query response returns lot of records and users want to look at the last few pages.M8 - Release 0.11