Search merge requestshttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests2023-08-18T22:07:43Zhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/79Aws integration only2023-08-18T22:07:43ZSpencer Suttonsuttonsp@amazon.comAws integration onlyM4 - Release 0.7Matt WiseMatt Wisehttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/81(GONRG-1759) Fix Security response headers issue2023-08-18T22:07:42ZIgor Filippov (EPAM)(GONRG-1759) Fix Security response headers issue**Issue**
Testing team reported "VULN-05 HSTS and CSP not implemented properly".
The report is attached to the GONRG-1637: [^API security testing report _Trajectory.pdf].
They say "The HSTS and CSP headers are not implemented. A man-i...**Issue**
Testing team reported "VULN-05 HSTS and CSP not implemented properly".
The report is attached to the GONRG-1637: [^API security testing report _Trajectory.pdf].
They say "The HSTS and CSP headers are not implemented. A man-in-the-middle attacker attempts to intercept traffic from a victim user using an invalid certificate and hopes the user will accept the bad certificate".
Reported contexts: STORAGE, DELIVERY
**Replay and analysis**
Debugged Search service API:
```
curl --location --request POST 'https://os-search-attcrcktoa-uc.a.run.app/api/search/v2/query' \
--header 'Authorization: Bearer <token>' \
--header 'data-partition-id: osdu' \
--header 'Content-Type: application/json' \
--data-raw '{
"kind": "osdu:osdu:*:0.2.0",
"query": "BIR*"
}'
```
- Noticed security headers (Strict-Transport-Security, Content-Security-Policy etc.) absence in responses
- Analyzed Search service Java code
- not found any "active" code for setting security headers on Responses
- found the inactivated class "org.opengroup.osdu.search.middleware.CorrelationIDRequestFilter" designed to set needed headers
- the class is inactivated by the commented "@Component" annotation
- CorrelationIDRequestFilter component should be reviewed, actualized and reactivated
- see the similar functionality code we have in Storage service: GONRG-1756
## Type of change
- [X] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x] IBM
## Does this introduce a breaking change?
- [NO]
## Have you added/updated Unit Tests and Integration Tests?
- [NO]M5 - Release 0.8Dmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/82CORS Fix Update2023-08-18T22:07:40ZSpencer Suttonsuttonsp@amazon.comCORS Fix UpdateM4 - Release 0.7ethiraj krishnamanaiduDania Kodeih (Microsoft)Wladmir FrazaoJoeDmitriy RudkoMatt Wiseethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/84entitlements v2 cutover2023-08-18T22:07:39ZMingyang Zhuentitlements v2 cutoverM5 - Release 0.8Mingyang ZhuMingyang Zhuhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/85Integration Search service query api authorization with Policy service2023-08-18T22:07:37ZAlok JoshiIntegration Search service query api authorization with Policy serviceAdd support for validating authorization via Policy service. Policy service is togglable and requires setting `service.policy.enabled` configuration setting. By default this configuration is disabled and authorization works with building...Add support for validating authorization via Policy service. Policy service is togglable and requires setting `service.policy.enabled` configuration setting. By default this configuration is disabled and authorization works with building authorization filter within the elastic query
Providers must deploy policy and partition service before enabling this option.
Issue https://community.opengroup.org/osdu/platform/system/search-service/-/issues/27M5 - Release 0.8Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/86Switching the dependencies to release versions2023-08-18T22:07:35ZDavid Diederichd.diederich@opengroup.orgSwitching the dependencies to release versionsThis changes a library dependency to use the released version of the core libraries. It was previously depending on SNAPSHOT versions, which is a less stable version. More importantly, the SNAPSHOT versions are periodically purged from t...This changes a library dependency to use the released version of the core libraries. It was previously depending on SNAPSHOT versions, which is a less stable version. More importantly, the SNAPSHOT versions are periodically purged from the system to save disk space -- this happened recently. Since these libraries no longer exist on community, building becomes difficult.
This MR moves that dependency to a release version, which is better going forward and allows FOSSA to do the build and get good dependency information. I assert that there are no substantial changes between the SNAPSHOT version I moved from and the latest release version that I moved to. It's difficult to know which commit the SNAPSHOT dependency linked to, since it moved many times, but here are the differences from the last time the SNAPSHOT dependency was listed and the one commit that has the release version (0.7.0). All of these changes were from me, updating versions and references as part of the release process.
* [GCP Differences](https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/os-core-lib-gcp/-/compare/ff52818d929b7a32e491b75743285026c4c0a9b4...v0.7.0)
Separately, since I was working with FOSSA, I updated the configuration file and the corresponding NOTICE changes resulting from the new module.M5 - Release 0.8David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/87update core-common for azure and core2023-08-18T22:07:34ZAlok Joshiupdate core-common for azure and coreM5 - Release 0.8Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/88fix the issue with cross tenant search2023-08-18T22:07:32ZYauheni Lesnikaufix the issue with cross tenant searchFix for the issue: https://community.opengroup.org/osdu/platform/system/search-service/-/issues/28Fix for the issue: https://community.opengroup.org/osdu/platform/system/search-service/-/issues/28M5 - Release 0.8ethiraj krishnamanaiduNeelesh ThakurSherman YangAlok JoshiYauheni Lesnikauethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/89(GONRG-1759) - Fix post merge issue in :search-azure2021-03-22T22:59:52ZDmitriy Rudko(GONRG-1759) - Fix post merge issue in :search-azureM1 - Release 0.1Daniel SchollJasonDaniel Schollhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/90(GONRG-2081)Update .gitlab-ci.yml2023-08-18T22:07:31ZVladislav Shishko (EPAM)(GONRG-2081)Update .gitlab-ci.ymlM5 - Release 0.8Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/91ibm kind fix2023-08-18T22:07:29ZShrikant Gargibm kind fixIBM fix for kind validationIBM fix for kind validationM5 - Release 0.8Anuj GuptaAnuj Guptahttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/92index field is not being included when users are explicitly requesting via re...2023-08-18T22:07:27ZNeelesh Thakurindex field is not being included when users are explicitly requesting via returnedFieldsindex exclusion from response: index field is not being included when users are explicitly requesting via returnedFields
issue #34index exclusion from response: index field is not being included when users are explicitly requesting via returnedFields
issue #34M5 - Release 0.8https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/93track accurate total count if requested2023-08-18T22:07:26ZNeelesh Thakurtrack accurate total count if requestedresolves the issue #29resolves the issue #29M5 - Release 0.8https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/94Update core common to fix headers for preflight CORS request (GONRG-2138)2023-08-18T22:07:24ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comUpdate core common to fix headers for preflight CORS request (GONRG-2138)# Description:
https://community.opengroup.org/osdu/ui/admin-ui uses preflight requests for CORS, which cannot be processed , due to lack `access-control-allow-origin` in `Access-Control-Allow-Headers` response.
Added `access-control-all...# Description:
https://community.opengroup.org/osdu/ui/admin-ui uses preflight requests for CORS, which cannot be processed , due to lack `access-control-allow-origin` in `Access-Control-Allow-Headers` response.
Added `access-control-allow-origin` to `Access-Control-Allow-Headers` for CORS preflight request.<br/>
![storage](/uploads/1ba0f428c4047866efecf93da339efcc/storage.PNG)
# How to test:
After header added, preflight requests can be processed normally <br/>
![search](/uploads/5944ce2da3f0f7bd12e519a6fa4a792e/search.PNG)
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [x] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] Common code
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M5 - Release 0.8Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/95Support ECK/Elasticsearch on EKS2023-08-18T22:07:23ZMatt WiseSupport ECK/Elasticsearch on EKScommit 11fe6d13
Author: Matt Wise <wsmatth@amazon.com>
Date: Fri Mar 19 2021 13:27:57 GMT-0500 (Central Daylight Time)
AWS update elasticsearch int test client
commit b5163a9f
Author: Matt Wise <wsmatth@amazon.com>
Date: Thu ...commit 11fe6d13
Author: Matt Wise <wsmatth@amazon.com>
Date: Fri Mar 19 2021 13:27:57 GMT-0500 (Central Daylight Time)
AWS update elasticsearch int test client
commit b5163a9f
Author: Matt Wise <wsmatth@amazon.com>
Date: Thu Mar 18 2021 14:41:32 GMT-0500 (Central Daylight Time)
update elastic clientM5 - Release 0.8Matt WiseMatt Wisehttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/97include validation errors in response2023-08-18T22:07:21ZNeelesh Thakurinclude validation errors in responseAddresses the issue #35Addresses the issue #35M5 - Release 0.8https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/98upgrade core-common for Azure and core2023-08-18T22:07:19ZAlok Joshiupgrade core-common for Azure and corePulling the latest versions of core-common and core-lib-azure to improve logging in Azure portal. This mainly fixes the logging of inner messages when an exception occurs and also fixes the issue of 'requests' log not being logged in AI ...Pulling the latest versions of core-common and core-lib-azure to improve logging in Azure portal. This mainly fixes the logging of inner messages when an exception occurs and also fixes the issue of 'requests' log not being logged in AI due to customDimensions field being empty.M5 - Release 0.8Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/99OSDU-GCP CI/CD Fix2023-08-18T22:07:18ZAliaksandr Ramanovich (EPAM)OSDU-GCP CI/CD FixAdd variable for redis instanceAdd variable for redis instanceM5 - Release 0.8Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/100handle double slash in path2023-08-18T22:07:16ZNeelesh Thakurhandle double slash in pathSpring Security throws 403 when it encounters // in the path. As we have Slf4jMDCFilter as lowest precedence, it intercepts and converts to 500.
There is proper fix of this [issue ](https://github.com/spring-projects/spring-security/is...Spring Security throws 403 when it encounters // in the path. As we have Slf4jMDCFilter as lowest precedence, it intercepts and converts to 500.
There is proper fix of this [issue ](https://github.com/spring-projects/spring-security/issues/5007) when we can upgrade spring to >=2.4.
Making localized changes for Azure only as not sure how other providers are handling this at the container level.M5 - Release 0.8https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/101Update libraries to fix CVE security vulnerabilities2023-08-18T22:07:15ZAlok JoshiUpdate libraries to fix CVE security vulnerabilitiesPlease refer to [this MR](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/75) for more detailsPlease refer to [this MR](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/75) for more detailsM5 - Release 0.8Alok JoshiAlok Joshi