Search merge requestshttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests2023-08-18T22:07:55Zhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/70Elastic 72023-08-18T22:07:55ZRiabokon Stanislav(EPAM)[GCP]Elastic 7ElasticSearch 7.8.1 for search serviceElasticSearch 7.8.1 for search serviceM4 - Release 0.7Dmitriy RudkoDmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/44Improve code coverage (search-azure)2023-08-18T22:13:59ZAalekh JainImprove code coverage (search-azure)## All Submissions:
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] I have updated the documentation accordingly.
* [YES] I have added tests to cover my changes.
* [YES] All new ...## All Submissions:
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] I have updated the documentation accordingly.
* [YES] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [NO] I ran lint checks locally prior to submission.
## What is the current behavior?
No Unit Tests and zero code coverage for search-azure.
## What is the new behavior?
Added Unit Tests to improve code coverage for search-azure
## Does this introduce a breaking change?
- [NO]
cc: @kibattul @polavishnu @amaverma
# Note
To exclude the Config classes from jacoco code coverage reports, add the following configuration in pom.xml for Jacoco plugin.
```xml
<configuration>
<excludes>
<exclude>org/opengroup/osdu/search/provider/azure/*/*Config*</exclude>
</excludes>
</configuration>
```M4 - Release 0.7https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/101Update libraries to fix CVE security vulnerabilities2023-08-18T22:07:15ZAlok JoshiUpdate libraries to fix CVE security vulnerabilitiesPlease refer to [this MR](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/75) for more detailsPlease refer to [this MR](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/75) for more detailsM5 - Release 0.8Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/100handle double slash in path2023-08-18T22:07:16ZNeelesh Thakurhandle double slash in pathSpring Security throws 403 when it encounters // in the path. As we have Slf4jMDCFilter as lowest precedence, it intercepts and converts to 500.
There is proper fix of this [issue ](https://github.com/spring-projects/spring-security/is...Spring Security throws 403 when it encounters // in the path. As we have Slf4jMDCFilter as lowest precedence, it intercepts and converts to 500.
There is proper fix of this [issue ](https://github.com/spring-projects/spring-security/issues/5007) when we can upgrade spring to >=2.4.
Making localized changes for Azure only as not sure how other providers are handling this at the container level.M5 - Release 0.8https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/99OSDU-GCP CI/CD Fix2023-08-18T22:07:18ZAliaksandr Ramanovich (EPAM)OSDU-GCP CI/CD FixAdd variable for redis instanceAdd variable for redis instanceM5 - Release 0.8Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/98upgrade core-common for Azure and core2023-08-18T22:07:19ZAlok Joshiupgrade core-common for Azure and corePulling the latest versions of core-common and core-lib-azure to improve logging in Azure portal. This mainly fixes the logging of inner messages when an exception occurs and also fixes the issue of 'requests' log not being logged in AI ...Pulling the latest versions of core-common and core-lib-azure to improve logging in Azure portal. This mainly fixes the logging of inner messages when an exception occurs and also fixes the issue of 'requests' log not being logged in AI due to customDimensions field being empty.M5 - Release 0.8Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/97include validation errors in response2023-08-18T22:07:21ZNeelesh Thakurinclude validation errors in responseAddresses the issue #35Addresses the issue #35M5 - Release 0.8https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/95Support ECK/Elasticsearch on EKS2023-08-18T22:07:23ZMatt WiseSupport ECK/Elasticsearch on EKScommit 11fe6d13
Author: Matt Wise <wsmatth@amazon.com>
Date: Fri Mar 19 2021 13:27:57 GMT-0500 (Central Daylight Time)
AWS update elasticsearch int test client
commit b5163a9f
Author: Matt Wise <wsmatth@amazon.com>
Date: Thu ...commit 11fe6d13
Author: Matt Wise <wsmatth@amazon.com>
Date: Fri Mar 19 2021 13:27:57 GMT-0500 (Central Daylight Time)
AWS update elasticsearch int test client
commit b5163a9f
Author: Matt Wise <wsmatth@amazon.com>
Date: Thu Mar 18 2021 14:41:32 GMT-0500 (Central Daylight Time)
update elastic clientM5 - Release 0.8Matt WiseMatt Wisehttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/94Update core common to fix headers for preflight CORS request (GONRG-2138)2023-08-18T22:07:24ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comUpdate core common to fix headers for preflight CORS request (GONRG-2138)# Description:
https://community.opengroup.org/osdu/ui/admin-ui uses preflight requests for CORS, which cannot be processed , due to lack `access-control-allow-origin` in `Access-Control-Allow-Headers` response.
Added `access-control-all...# Description:
https://community.opengroup.org/osdu/ui/admin-ui uses preflight requests for CORS, which cannot be processed , due to lack `access-control-allow-origin` in `Access-Control-Allow-Headers` response.
Added `access-control-allow-origin` to `Access-Control-Allow-Headers` for CORS preflight request.<br/>
![storage](/uploads/1ba0f428c4047866efecf93da339efcc/storage.PNG)
# How to test:
After header added, preflight requests can be processed normally <br/>
![search](/uploads/5944ce2da3f0f7bd12e519a6fa4a792e/search.PNG)
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [x] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] Common code
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M5 - Release 0.8Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/93track accurate total count if requested2023-08-18T22:07:26ZNeelesh Thakurtrack accurate total count if requestedresolves the issue #29resolves the issue #29M5 - Release 0.8https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/92index field is not being included when users are explicitly requesting via re...2023-08-18T22:07:27ZNeelesh Thakurindex field is not being included when users are explicitly requesting via returnedFieldsindex exclusion from response: index field is not being included when users are explicitly requesting via returnedFields
issue #34index exclusion from response: index field is not being included when users are explicitly requesting via returnedFields
issue #34M5 - Release 0.8https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/91ibm kind fix2023-08-18T22:07:29ZShrikant Gargibm kind fixIBM fix for kind validationIBM fix for kind validationM5 - Release 0.8Anuj GuptaAnuj Guptahttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/90(GONRG-2081)Update .gitlab-ci.yml2023-08-18T22:07:31ZVladislav Shishko (EPAM)(GONRG-2081)Update .gitlab-ci.ymlM5 - Release 0.8Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/88fix the issue with cross tenant search2023-08-18T22:07:32ZYauheni Lesnikaufix the issue with cross tenant searchFix for the issue: https://community.opengroup.org/osdu/platform/system/search-service/-/issues/28Fix for the issue: https://community.opengroup.org/osdu/platform/system/search-service/-/issues/28M5 - Release 0.8ethiraj krishnamanaiduNeelesh ThakurSherman YangAlok JoshiYauheni Lesnikauethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/87update core-common for azure and core2023-08-18T22:07:34ZAlok Joshiupdate core-common for azure and coreM5 - Release 0.8Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/86Switching the dependencies to release versions2023-08-18T22:07:35ZDavid Diederichd.diederich@opengroup.orgSwitching the dependencies to release versionsThis changes a library dependency to use the released version of the core libraries. It was previously depending on SNAPSHOT versions, which is a less stable version. More importantly, the SNAPSHOT versions are periodically purged from t...This changes a library dependency to use the released version of the core libraries. It was previously depending on SNAPSHOT versions, which is a less stable version. More importantly, the SNAPSHOT versions are periodically purged from the system to save disk space -- this happened recently. Since these libraries no longer exist on community, building becomes difficult.
This MR moves that dependency to a release version, which is better going forward and allows FOSSA to do the build and get good dependency information. I assert that there are no substantial changes between the SNAPSHOT version I moved from and the latest release version that I moved to. It's difficult to know which commit the SNAPSHOT dependency linked to, since it moved many times, but here are the differences from the last time the SNAPSHOT dependency was listed and the one commit that has the release version (0.7.0). All of these changes were from me, updating versions and references as part of the release process.
* [GCP Differences](https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/os-core-lib-gcp/-/compare/ff52818d929b7a32e491b75743285026c4c0a9b4...v0.7.0)
Separately, since I was working with FOSSA, I updated the configuration file and the corresponding NOTICE changes resulting from the new module.M5 - Release 0.8David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/85Integration Search service query api authorization with Policy service2023-08-18T22:07:37ZAlok JoshiIntegration Search service query api authorization with Policy serviceAdd support for validating authorization via Policy service. Policy service is togglable and requires setting `service.policy.enabled` configuration setting. By default this configuration is disabled and authorization works with building...Add support for validating authorization via Policy service. Policy service is togglable and requires setting `service.policy.enabled` configuration setting. By default this configuration is disabled and authorization works with building authorization filter within the elastic query
Providers must deploy policy and partition service before enabling this option.
Issue https://community.opengroup.org/osdu/platform/system/search-service/-/issues/27M5 - Release 0.8Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/84entitlements v2 cutover2023-08-18T22:07:39ZMingyang Zhuentitlements v2 cutoverM5 - Release 0.8Mingyang ZhuMingyang Zhuhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/81(GONRG-1759) Fix Security response headers issue2023-08-18T22:07:42ZIgor Filippov (EPAM)(GONRG-1759) Fix Security response headers issue**Issue**
Testing team reported "VULN-05 HSTS and CSP not implemented properly".
The report is attached to the GONRG-1637: [^API security testing report _Trajectory.pdf].
They say "The HSTS and CSP headers are not implemented. A man-i...**Issue**
Testing team reported "VULN-05 HSTS and CSP not implemented properly".
The report is attached to the GONRG-1637: [^API security testing report _Trajectory.pdf].
They say "The HSTS and CSP headers are not implemented. A man-in-the-middle attacker attempts to intercept traffic from a victim user using an invalid certificate and hopes the user will accept the bad certificate".
Reported contexts: STORAGE, DELIVERY
**Replay and analysis**
Debugged Search service API:
```
curl --location --request POST 'https://os-search-attcrcktoa-uc.a.run.app/api/search/v2/query' \
--header 'Authorization: Bearer <token>' \
--header 'data-partition-id: osdu' \
--header 'Content-Type: application/json' \
--data-raw '{
"kind": "osdu:osdu:*:0.2.0",
"query": "BIR*"
}'
```
- Noticed security headers (Strict-Transport-Security, Content-Security-Policy etc.) absence in responses
- Analyzed Search service Java code
- not found any "active" code for setting security headers on Responses
- found the inactivated class "org.opengroup.osdu.search.middleware.CorrelationIDRequestFilter" designed to set needed headers
- the class is inactivated by the commented "@Component" annotation
- CorrelationIDRequestFilter component should be reviewed, actualized and reactivated
- see the similar functionality code we have in Storage service: GONRG-1756
## Type of change
- [X] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x] IBM
## Does this introduce a breaking change?
- [NO]
## Have you added/updated Unit Tests and Integration Tests?
- [NO]M5 - Release 0.8Dmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/120GCP temp fix for policy integration2023-08-18T22:06:50ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comGCP temp fix for policy integrationM6 - Release 0.9Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]