Search merge requestshttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests2023-08-18T22:13:52Zhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/51Azure SDKs Upgrade and Dependency Management2023-08-18T22:13:52ZAbhishek PatilAzure SDKs Upgrade and Dependency Management## All Submissions:
-------------------------------------
* [YES] Have you followed our code review [guidelines](https://github.com/microsoft/code-with-engineering-playbook/blob/master/pull-requests/code-reviews/readme.md)?
* [YES] Have ...## All Submissions:
-------------------------------------
* [YES] Have you followed our code review [guidelines](https://github.com/microsoft/code-with-engineering-playbook/blob/master/pull-requests/code-reviews/readme.md)?
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES] I have updated the documentation accordingly.
* [NA] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [NO] I ran lint checks locally prior to submission.
## What is the current behavior?
-------------------------------------
- Old version of Azure SDKs are used in service.
- Versions of dependencies which are common across all OSDU services are not consistent/same in all OSDU services.<!-- Please describe the current behavior that you are modifying, or link to a relevant issue. -->
Issue: https://dev.azure.com/msazure/One/_workitems/edit/8708950
## What is the new behavior?
-------------------------------------
- All Azure SDKs are upgraded to latest stable version.
- Common dependencies are inherited from core-lib-azure as managed dependencies. This will make sure that version info of those dependencies are inherited from core-lib-azure. All OSDU services using same version of core-lib-azure will automatically use same versions for all those dependencies.<!-- Please describe the behavior or changes that are being added by this PR. -->
## Does this introduce a breaking change?
-------------------------------------
- [NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Any relevant logs, error output, etc?
-------------------------------------
(If it’s long, please paste to https://ghostbin.com/ and insert the link here.)
## Other information
-------------------------------------
<!-- Any other information that is important to this PR such as screenshots of how the component looks before and after the change. -->M1 - Release 0.1Abhishek PatilAbhishek Patilhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/60switch to pod managed identities for Azure Resource access2023-08-18T22:13:39ZAliaksei Darafeyeuswitch to pod managed identities for Azure Resource accessM1 - Release 0.1ethiraj krishnamanaiduDaniel SchollPavel BachylaAlok Joshiethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/63Gcp sonar comments fix (GONRG-1364)2023-08-18T22:13:34ZAnastasiia GelmutGcp sonar comments fix (GONRG-1364)# Description:
SonarLint comments were fixed. Non-breakable refactoring.
# How to test:
mvn clean install
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking chan...# Description:
SonarLint comments were fixed. Non-breakable refactoring.
# How to test:
mvn clean install
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
# Dev Checklist:
- [x] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M1 - Release 0.1Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/64Gcp fix elastic tests (GONRG-846)2023-08-18T22:13:33ZAnastasiia GelmutGcp fix elastic tests (GONRG-846)# Description:
ElasticClientHandlerTest was fixed.
# How to test:
mav clean install
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [x] Bugfix (a non-breaking change that solves an issu...# Description:
ElasticClientHandlerTest was fixed.
# How to test:
mav clean install
# Changes include:
- [x] Refactor (a non-breaking change that improves code maintainability).
- [x] Bugfix (a non-breaking change that solves an issue).
- [ ] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
# Dev Checklist:
- [x] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M1 - Release 0.1Rostislav Dublin (EPAM)Rostislav Dublin (EPAM)https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/55Fix whitesource2023-08-18T22:13:48ZPavel BachylaFix whitesource## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- No
## Does this introduce a change in the cloud provide...## Type of change
- [x] Bug Fix
- [ ] Feature
**Please provide link to gitlab issue or ADR(Architecture Decision Record)**
## Does this introduce a change in the core logic?
- No
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [ ] GCP
- [ ] IBM
## Does this introduce a breaking change?
- No
## What is the current behavior?
N/A
## What is the new/expected behavior?
N/A
## Have you added/updated Unit Tests and Integration Tests?
N/A
## Any other useful information
Fix Major/Critical/Blocker WhiteSource vulnerabilities except those related to ElasticSearch version limitations and log4j due to incompatibilities with other librariesM1 - Release 0.1ethiraj krishnamanaiduNeelesh ThakurSherman Yangethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/68OSDU-GCP: parametrize containerization stage2023-08-18T22:07:57ZAliaksandr Ramanovich (EPAM)OSDU-GCP: parametrize containerization stageparametrize containerization stage to remove hardcoded images pathsparametrize containerization stage to remove hardcoded images pathsM1 - Release 0.1Oleksandr Kosse (EPAM)Oleksandr Kosse (EPAM)https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/71Fix response status code upon elastic search exception2023-08-18T22:07:53ZSherman YangFix response status code upon elastic search exceptionCurrently, search service returns 404 response code upon elastic search exception, which is misleading.
This MR changes the response status code to better reflect the actual status upon elastic search exception.Currently, search service returns 404 response code upon elastic search exception, which is misleading.
This MR changes the response status code to better reflect the actual status upon elastic search exception.M1 - Release 0.1https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/72Cloud-agnostic module `search-reference` for Anthos (GONRG-1423)2023-08-18T22:07:51ZRiabokon Stanislav(EPAM)[GCP]Cloud-agnostic module `search-reference` for Anthos (GONRG-1423)# Description:
Added a new cloud-agnostic module `search-reference`. This module can be used for Anthos GCP.
Database: MongoDB
Messaging Broker: RabbitMQ
Search Engine: Elasticsearch
# How to test:
Bussines cases.
# Changes include...# Description:
Added a new cloud-agnostic module `search-reference`. This module can be used for Anthos GCP.
Database: MongoDB
Messaging Broker: RabbitMQ
Search Engine: Elasticsearch
# How to test:
Bussines cases.
# Changes include:
- [ ] Refactor (a non-breaking change that improves code maintainability).
- [ ] Bugfix (a non-breaking change that solves an issue).
- [X] New feature (a non-breaking change that adds functionality).
- [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBM
# Dev Checklist:
- [x] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M1 - Release 0.1Dmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/74ibm health check changes2023-08-18T22:07:50ZAnuj Guptaibm health check changesibm health check changesibm health check changesM3 - Release 0.5Anuj GuptaAnuj Guptahttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/44Improve code coverage (search-azure)2023-08-18T22:13:59ZAalekh JainImprove code coverage (search-azure)## All Submissions:
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] I have updated the documentation accordingly.
* [YES] I have added tests to cover my changes.
* [YES] All new ...## All Submissions:
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] I have updated the documentation accordingly.
* [YES] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [NO] I ran lint checks locally prior to submission.
## What is the current behavior?
No Unit Tests and zero code coverage for search-azure.
## What is the new behavior?
Added Unit Tests to improve code coverage for search-azure
## Does this introduce a breaking change?
- [NO]
cc: @kibattul @polavishnu @amaverma
# Note
To exclude the Config classes from jacoco code coverage reports, add the following configuration in pom.xml for Jacoco plugin.
```xml
<configuration>
<excludes>
<exclude>org/opengroup/osdu/search/provider/azure/*/*Config*</exclude>
</excludes>
</configuration>
```M4 - Release 0.7https://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/70Elastic 72023-08-18T22:07:55ZRiabokon Stanislav(EPAM)[GCP]Elastic 7ElasticSearch 7.8.1 for search serviceElasticSearch 7.8.1 for search serviceM4 - Release 0.7Dmitriy RudkoDmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/77Hotfix: Updated Elastic classes in Auzre tests.2023-08-18T22:07:47ZDmitriy RudkoHotfix: Updated Elastic classes in Auzre tests.1. Updated Elastic classes in Azure tests:
`GetFieldMappingsResponse.FieldMappingMetaData` -> `GetFieldMappingsResponse.FieldMappingMetadata`
2. Ignored failing Azure tests:
```
QueryServiceImplTest.testQueryBase_useGeoShapeQueryIsFalse...1. Updated Elastic classes in Azure tests:
`GetFieldMappingsResponse.FieldMappingMetaData` -> `GetFieldMappingsResponse.FieldMappingMetadata`
2. Ignored failing Azure tests:
```
QueryServiceImplTest.testQueryBase_useGeoShapeQueryIsFalse_getByBoundingBox:183 » NullPointer
QueryServiceImplTest.testQueryBase_useGeoShapeQueryIsFalse_getByDistance:276 » NullPointer
QueryServiceImplTest.testQueryBase_useGeoShapeQueryIsFalse_getByGeoPolygon:305 » NullPointer
QueryServiceImplTest.testQueryBase_useGeoShapeQueryIsTrue_getByBoundingBox:212 » NullPointer
QueryServiceImplTest.testQueryBase_useGeoShapeQueryIsTrue_getByGeoPolygon:330 » NullPointer
QueryServiceImplTest.testQueryBase_whenSearchHitsIsEmpty:148 » NullPointer
QueryServiceImplTest.testQueryBase_whenSearchHitsIsNotEmpty:166 » NullPointer
ScrollQueryServiceImplTest.testQueryIndex_whenNoCursorInSearchQuery:197 » WrongTypeOfReturnValue
ScrollQueryServiceImplTest.testQueryIndex_whenSearchHitsIsEmpty:167 » WrongTypeOfReturnValue
ScrollQueryServiceImplTest.testQueryIndex_whenSearchHitsIsNotEmpty:128 » WrongTypeOfReturnValue
```M4 - Release 0.7Dmitriy RudkoDmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/78(GONRG-1767) GCP implement new log2023-08-18T22:07:45ZIgor Filippov (EPAM)(GONRG-1767) GCP implement new log## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] IB...## Type of change
- [ ] Bug Fix
- [X] Feature
## Does this introduce a change in the core logic?
- [NO]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [ ] AWS
- [ ] Azure
- [X] GCP
- [ ] IBM
## Does this introduce a breaking change?
- [NO]
## Have you added/updated Unit Tests and Integration Tests?
- [NO]M4 - Release 0.7Dmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/75CORS Fix, AWS Updates2023-08-18T22:07:48ZSpencer Suttonsuttonsp@amazon.comCORS Fix, AWS UpdatesM4 - Release 0.7ethiraj krishnamanaiduDania Kodeih (Microsoft)Wladmir FrazaoJoeDmitriy Rudkoethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/79Aws integration only2023-08-18T22:07:43ZSpencer Suttonsuttonsp@amazon.comAws integration onlyM4 - Release 0.7Matt WiseMatt Wisehttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/82CORS Fix Update2023-08-18T22:07:40ZSpencer Suttonsuttonsp@amazon.comCORS Fix UpdateM4 - Release 0.7ethiraj krishnamanaiduDania Kodeih (Microsoft)Wladmir FrazaoJoeDmitriy RudkoMatt Wiseethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/84entitlements v2 cutover2023-08-18T22:07:39ZMingyang Zhuentitlements v2 cutoverM5 - Release 0.8Mingyang ZhuMingyang Zhuhttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/86Switching the dependencies to release versions2023-08-18T22:07:35ZDavid Diederichd.diederich@opengroup.orgSwitching the dependencies to release versionsThis changes a library dependency to use the released version of the core libraries. It was previously depending on SNAPSHOT versions, which is a less stable version. More importantly, the SNAPSHOT versions are periodically purged from t...This changes a library dependency to use the released version of the core libraries. It was previously depending on SNAPSHOT versions, which is a less stable version. More importantly, the SNAPSHOT versions are periodically purged from the system to save disk space -- this happened recently. Since these libraries no longer exist on community, building becomes difficult.
This MR moves that dependency to a release version, which is better going forward and allows FOSSA to do the build and get good dependency information. I assert that there are no substantial changes between the SNAPSHOT version I moved from and the latest release version that I moved to. It's difficult to know which commit the SNAPSHOT dependency linked to, since it moved many times, but here are the differences from the last time the SNAPSHOT dependency was listed and the one commit that has the release version (0.7.0). All of these changes were from me, updating versions and references as part of the release process.
* [GCP Differences](https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/os-core-lib-gcp/-/compare/ff52818d929b7a32e491b75743285026c4c0a9b4...v0.7.0)
Separately, since I was working with FOSSA, I updated the configuration file and the corresponding NOTICE changes resulting from the new module.M5 - Release 0.8David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/87update core-common for azure and core2023-08-18T22:07:34ZAlok Joshiupdate core-common for azure and coreM5 - Release 0.8Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/search-service/-/merge_requests/81(GONRG-1759) Fix Security response headers issue2023-08-18T22:07:42ZIgor Filippov (EPAM)(GONRG-1759) Fix Security response headers issue**Issue**
Testing team reported "VULN-05 HSTS and CSP not implemented properly".
The report is attached to the GONRG-1637: [^API security testing report _Trajectory.pdf].
They say "The HSTS and CSP headers are not implemented. A man-i...**Issue**
Testing team reported "VULN-05 HSTS and CSP not implemented properly".
The report is attached to the GONRG-1637: [^API security testing report _Trajectory.pdf].
They say "The HSTS and CSP headers are not implemented. A man-in-the-middle attacker attempts to intercept traffic from a victim user using an invalid certificate and hopes the user will accept the bad certificate".
Reported contexts: STORAGE, DELIVERY
**Replay and analysis**
Debugged Search service API:
```
curl --location --request POST 'https://os-search-attcrcktoa-uc.a.run.app/api/search/v2/query' \
--header 'Authorization: Bearer <token>' \
--header 'data-partition-id: osdu' \
--header 'Content-Type: application/json' \
--data-raw '{
"kind": "osdu:osdu:*:0.2.0",
"query": "BIR*"
}'
```
- Noticed security headers (Strict-Transport-Security, Content-Security-Policy etc.) absence in responses
- Analyzed Search service Java code
- not found any "active" code for setting security headers on Responses
- found the inactivated class "org.opengroup.osdu.search.middleware.CorrelationIDRequestFilter" designed to set needed headers
- the class is inactivated by the commented "@Component" annotation
- CorrelationIDRequestFilter component should be reviewed, actualized and reactivated
- see the similar functionality code we have in Storage service: GONRG-1756
## Type of change
- [X] Bug Fix
- [ ] Feature
## Does this introduce a change in the core logic?
- [YES]
## Does this introduce a change in the cloud provider implementation, if so which cloud?
- [x] AWS
- [x] Azure
- [x] GCP
- [x] IBM
## Does this introduce a breaking change?
- [NO]
## Have you added/updated Unit Tests and Integration Tests?
- [NO]M5 - Release 0.8Dmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudko