OS Core Common - Spring 5 merge requestshttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests2023-08-31T11:22:00Zhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/226Allow selection of highlighted fields2023-08-31T11:22:00ZStanisław BienieckiAllow selection of highlighted fieldsFor implementation of https://community.opengroup.org/osdu/platform/system/search-service/-/issues/128For implementation of https://community.opengroup.org/osdu/platform/system/search-service/-/issues/128M20 - Release 0.23Chad LeongStanisław BienieckiChad Leonghttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/225Added check redis connection (GONRG-7597)2023-08-28T07:00:52ZYurii Ruban [EPAM / GCP]Added check redis connection (GONRG-7597)# Description:
In case of loss of connection between the service and Redis, the processing of requests to the service is delayed, and an error of the type
"Command timed out after 30 SECONDS" is received.
Added Redis connection health c...# Description:
In case of loss of connection between the service and Redis, the processing of requests to the service is delayed, and an error of the type
"Command timed out after 30 SECONDS" is received.
Added Redis connection health checks.
Issue https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/72
# How to test:
1. Choose a service with core common lib.
2. Сheck service viability.
3. Turn off the Radis.
4. Get the result of a request to the service without errors.
5. Check logs with the message "Redis connection is closed.".
# Changes include:
- [ ] Bugfix (a non-breaking change that solves an issue).
# Changes in:
- [ ] GCP
- [ ] Azure
- [ ] AWS
- [ ] IBMM20 - Release 0.23Riabokon Stanislav(EPAM)[GCP]Riabokon Stanislav(EPAM)[GCP]https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/224Adding expiry time for File DMS Security Enhancement2023-09-05T11:50:04ZDeepa KumariAdding expiry time for File DMS Security EnhancementRelated ADR: https://community.opengroup.org/osdu/platform/system/file/-/issues/78 and https://community.opengroup.org/osdu/platform/system/dataset/-/issues/57Related ADR: https://community.opengroup.org/osdu/platform/system/file/-/issues/78 and https://community.opengroup.org/osdu/platform/system/dataset/-/issues/57M20 - Release 0.23Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/223Add filter to SortQuery search model2023-08-11T20:18:39ZStanisław BienieckiAdd filter to SortQuery search model# Description:
* Introducing filter field to SortQuery to be able to implement changes for https://community.opengroup.org/osdu/platform/system/search-service/-/issues/106
Changes are backward-compatible, queries without filter are stil...# Description:
* Introducing filter field to SortQuery to be able to implement changes for https://community.opengroup.org/osdu/platform/system/search-service/-/issues/106
Changes are backward-compatible, queries without filter are still valid.
# How to test:
via Unit test
# Changes include:
* [ ] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
* [x] Common code
# Dev Checklist:
* [ ] Added Unit Tests, wherever applicable.
* [ ] Updated the Readme, if applicable.
* [x] Existing Tests pass
* [x] Verified functionality locally
* [x] Self Reviewed my code for formatting and complex business logic.M20 - Release 0.23https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/222Remove duplicates from ACLs2023-07-28T10:13:54ZAlok JoshiRemove duplicates from ACLsPlease see [issue](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/71)
By doing this change, we are making the behavior consistent with how we handle other similar scenarios. i.e. if duplicate legal...Please see [issue](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/71)
By doing this change, we are making the behavior consistent with how we handle other similar scenarios. i.e. if duplicate legaltags or tags are provided in the record create request, they are automatically ignored and record gets created with non-duplicate values. This change will make the behavior consistent with acls
This is affecting [new PATCH api](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md#records-patch-api) when a user tries to modify tags for a record that has duplicates in the acls. This issue itself needs to be addressed separately, this change is to fix having duplicates to begin with.M20 - Release 0.23Alok JoshiChad LeongAlok Joshihttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/221draft: response usability improvements2023-07-27T15:41:33ZNeelesh Thakurdraft: response usability improvementshttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/220Avoid duplicates in acls2023-07-27T16:11:10ZAlok JoshiAvoid duplicates in aclsPlease see [issue](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/71)
This is affecting [new PATCH api](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/Stor...Please see [issue](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/71)
This is affecting [new PATCH api](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/docs/tutorial/StorageService.md#records-patch-api) when a user tries to modify tags for a record that has duplicates in the acls. This issue itself needs to be addressed separately, this change is to fix having duplicates to begin with.M20 - Release 0.23Alok JoshiChad LeongAlok Joshihttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/219Update PartitionService.java, will fix invaliddata-partition-id return 500 code2023-12-07T17:23:13ZBruce JinUpdate PartitionService.java, will fix invaliddata-partition-id return 500 codeResolving issue https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/70
While making calls to OSDU services, such as `secret` and `storage` service, testers discover that if they put invalid symbols in `...Resolving issue https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/70
While making calls to OSDU services, such as `secret` and `storage` service, testers discover that if they put invalid symbols in `data_partition_id`, we will have 500 code, but with reason of Access Denied.
After investigation, we realize the partition service did not consider the situation that user put invalid URI symbols like `@#$%` in data partition
id, which will make the `normalizeStringUrl` function have Java.Lang exception in this [UrlNormalizationUtil.java](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/util/UrlNormalizationUtil.java).
```
Caused by: java.lang.IllegalArgumentException: Malformed escape pair at index 57: http://os-partition:8080/api/partition/v1/partitions/osdu%
at java.net.URI.create(URI.java:852)
at org.opengroup.osdu.core.common.util.UrlNormalizationUtil.normalizeStringUrl(UrlNormalizationUtil.java:27)
```
This will generate a 500 code in entitlement service since the service will treat this error as a general error in [SpringExceptionMapper.java](handleGeneralException), instead a 400 code.
So as an optional solution, a try-catch will handle the exception within `normalizeStringUrl` function and return a `AppException` so the service will know what it is.https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/218update error message on socket timeout2023-07-06T18:25:51ZNeelesh Thakurupdate error message on socket timeoutupdate error message on socket timeout, instead of throwing generic error message: `One or more properties could not be converted by CRS converter`update error message on socket timeout, instead of throwing generic error message: `One or more properties could not be converted by CRS converter`M19 - Release 0.22https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/217Improved error message handling for datetime conversion failure2023-06-29T17:58:58ZAlok JoshiImproved error message handling for datetime conversion failureLibrary errors out unexpectedly in a scenario when datetime conversion is trying to happen on a json object instead of a string of some date format. This is a user error that ends up in a 500 when Storage API to fetch records is called w...Library errors out unexpectedly in a scenario when datetime conversion is trying to happen on a json object instead of a string of some date format. This is a user error that ends up in a 500 when Storage API to fetch records is called with conversion enabled.
Example is added in the unit test which generates this issue.
We should handle it with a proper error message instead of failing with 500M19 - Release 0.22Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/216refactor AbstractHttpClient to use Apache Http Client and fix issues with Jav...2024-01-25T13:47:32ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comrefactor AbstractHttpClient to use Apache Http Client and fix issues with Java 17# Description:
- Required for migration to Java 17.
- Does not migrate core-common to Java 17, it will be kept with Java 8.
- Refactoring of AbstractHttpClient to switch from HttpURLConnection to Apache HTTP Client. For Patch requests s...# Description:
- Required for migration to Java 17.
- Does not migrate core-common to Java 17, it will be kept with Java 8.
- Refactoring of AbstractHttpClient to switch from HttpURLConnection to Apache HTTP Client. For Patch requests support reflection access was used for the final static fields of HttpURLConnection [(link)](https://stackoverflow.com/questions/56039341/get-declared-fields-of-java-lang-reflect-fields-in-jdk12/56043252), causing:
~~~
Caused by: java.lang.NoSuchFieldException: modifiers
at java.base/java.lang.Class.getDeclaredField(Class.java:2610)
at org.opengroup.osdu.core.common.http.AbstractHttpClient.supportPatchMethod(AbstractHttpClient.java:131)
~~~
# How to test:
via Unit test
# Changes include:
- [x] Nonbreaking change.
# Changes in:
- [x] Common code
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M19 - Release 0.22Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/215provider header information update2023-07-05T07:48:42ZShankar Pattanayakprovider header information updateIssue : https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/extern-data/home/-/issues/271Issue : https://gitlab.opengroup.org/osdu/subcommittees/ea/projects/extern-data/home/-/issues/271M19 - Release 0.22Shankar PattanayakShankar Pattanayakhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/214fix NullPointerException in PersistenceHelper2023-07-06T18:39:45ZMykyta Savchukfix NullPointerException in PersistenceHelperfix NullPointerException in PersistenceHelperfix NullPointerException in PersistenceHelperM19 - Release 0.22Mykyta SavchukMykyta Savchukhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/213Add request timeout for crs converter service2023-06-21T13:59:57ZYurii KondakovAdd request timeout for crs converter serviceThese changes are proposed to allow setting a timeout for http requests to the crs conversion service.
refactor CrsConverterFactory so that it can also create a CrsConverterService with a custom RequestConfig (custom HttpClient)
refactor...These changes are proposed to allow setting a timeout for http requests to the crs conversion service.
refactor CrsConverterFactory so that it can also create a CrsConverterService with a custom RequestConfig (custom HttpClient)
refactor CrsConverterService to use apache ClosableHttpClient that may have a custom socket timeout
Issue - https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/67
Storage MR https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/712M19 - Release 0.22Yurii KondakovYurii Kondakovhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/212Draft: Java 17 migration2024-01-22T13:58:05ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comDraft: Java 17 migration# Description:
- Migration to Java 17.
- Refactoring of AbstractHttpClient to switch from HttpURLConnection to new HttpClient. For Patch requests support reflection access was used for the final static fields of HttpURLConnection [(link...# Description:
- Migration to Java 17.
- Refactoring of AbstractHttpClient to switch from HttpURLConnection to new HttpClient. For Patch requests support reflection access was used for the final static fields of HttpURLConnection [(link)](https://stackoverflow.com/questions/56039341/get-declared-fields-of-java-lang-reflect-fields-in-jdk12/56043252), causing:
~~~
Caused by: java.lang.NoSuchFieldException: modifiers
at java.base/java.lang.Class.getDeclaredField(Class.java:2610)
at org.opengroup.osdu.core.common.http.AbstractHttpClient.supportPatchMethod(AbstractHttpClient.java:131)
~~~
- Partial test migration from Powermock to updated Mockito. Powermock uses illegal JDK reflection accesses not supported in jdk 12+ and does not get updates. PowerMockRunner in Junit was replaced with MockitoJUnitRunner.
# How to test:
via Unit test
# Changes include:
- [x] Breaking change (a change that is not backward-compatible and/or changes current functionality).
# Changes in:
- [x] Common code
# Dev Checklist:
- [ ] Added Unit Tests, wherever applicable.
- [ ] Updated the Readme, if applicable.
- [x] Existing Tests pass
- [x] Verified functionality locally
- [x] Self Reviewed my code for formatting and complex business logic.M23 - Release 0.26Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/211Draft: enable highlight2023-12-08T15:23:01ZNeelesh ThakurDraft: enable highlighthttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/210Set Content-Length in the request headers2023-06-01T09:20:14ZLarissa PereiraSet Content-Length in the request headersContext - By Default, the HttpURLConnection buffers the complete request body in memory before it is transmitted. You can't set the content length; it's done automatically. Therefore, despite setting the "Content-Length" in the HttpRequ...Context - By Default, the HttpURLConnection buffers the complete request body in memory before it is transmitted. You can't set the content length; it's done automatically. Therefore, despite setting the "Content-Length" in the HttpRequest headers we saw the following errror
_HttpResponse(body={"fault":{"faultstring":"Content-Length is missing","detail":{"errorcode":"messaging.adaptors.http.flow.LengthRequired"}}}, contentType=application/json, responseCode=411, exception=null_
Solution - We can use setFixedLengthStreamingMode(long contentLength), when the content length is known in advance.M19 - Release 0.22https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/209add list group on behalf API for entitlements service2023-05-31T10:35:16ZKelly Zhouadd list group on behalf API for entitlements serviceM18 - Release 0.21https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/208Add new constructor for Redis Cache2023-05-31T10:38:19Zharshit aggarwalAdd new constructor for Redis CacheM18 - Release 0.21harshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/207Cherry-pick 'Update FOSSA NOTICE' into release/0.202023-04-07T18:29:32ZDavid Diederichd.diederich@opengroup.orgCherry-pick 'Update FOSSA NOTICE' into release/0.20**Original MR**: !206
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporati...**Original MR**: !206
### This MR is a Cherry Pick into a Release Branch.
After the release branch is first created, any subsequent changes use this process to update the release (often resulting in a new patch tag) without incorporating all changes in the default branch.
These MRs must be approved by the PMC before they are merged, since they alter the scope of the release.
To see more details about the change itself, look at the Original MR listed above.
#### Skipped Pipeline
Normally, pipelines are not executed on the cherry pick branch/MR prior to merging.
This optimization is accepted because the code was tested when it merged into the default branch, and will be tested again in the release branch prior to tagging.
However, if anybody feels that the MR requires further scrutiny -- whether because it had conflicts in the cherry-picking, it interfaces with some drastically altered logic between the branches, or any other reason -- we can run the pipeline here prior to merging.
#### If There's Reason to Run a Pipeline
If you want to see a pipeline result before this merges, first add a comment explaining why you'd like to see the pipeline results so the PMC and others know your thinking.
Then, mark the MR as a Draft MR (using the vertical ellipsis above, choose 'Mark as Draft').
This prevents the MR from being approved & merged accidentally by a busy release coordinator who didn't see your comment.
Finally, if you are a maintainer on the project, launch a pipeline on this branch.
Since this branch is a protected branch and the MR has ~no-detached-pipeline set, all integration tests will run and there's no need for any `trusted-*` branches.
[Launch a Pipeline for this Branch](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/pipelines/new?ref=cherry-pick-for-206)M17 - Release 0.20David Diederichd.diederich@opengroup.orgChad LeongSrinivasan NarayananDavid Diederichd.diederich@opengroup.org