Lib issueshttps://community.opengroup.org/groups/osdu/platform/system/lib/-/issues2024-03-21T10:00:55Zhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/79ADR: OSDU API Versioning Strategy.2024-03-21T10:00:55ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comADR: OSDU API Versioning Strategy.# ADR: OSDU API Versioning Strategy.
## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context
- We have mature APIs solidified through time, but they are not immutable for life. From time to...# ADR: OSDU API Versioning Strategy.
## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context
- We have mature APIs solidified through time, but they are not immutable for life. From time to time, changes are suggested, for example, https://community.opengroup.org/osdu/platform/system/partition/-/issues/49
- OSDU Java-based services communicate through interfaces defined in the Core-Common library, those interfaces must be aligned with related service API, for example https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/tree/master/src/main/java/org/opengroup/osdu/core/common/storage?ref_type=heads
- This ADR mainly refers to cases when we have the necessity to change existing concepts. We are suggesting the introduction of a way to handle those changes, keeping compatibility, and minimizing configurations.
## Problem Statement
- We do not have a strategy to introduce breaking changes in APIs without disrupting existing environments or making overcomplicated configurations.
- Any change requires updates in Core-Common interfaces and in services that rely on those APIs, slowing down development time. Change should be adopted from top to bottom, from service that provides API to services that consume it.
- It's impossible to support two versions at a time, change should be implemented everywhere. Major changes are happening with the same interfaces, we're not introducing them as new entities but doing a rewrite of existing logic.
## Proposal
- Introduce API-compatible autodiscovery configurations in Core-Common. Using Spring Conditions we can introduce self-assembling configurations, enabling compatible clients in an automated manner.
- Improve /info endpoint response by providing services API version in it.
## Decision
- Add the supported API version to the /info endpoint response:
~~~
{
"groupId": "org.opengroup.osdu",
"artifactId": "partition-gc",
"version": "0.26.0-SNAPSHOT",
"api-version": "V2"
.....
}
~~~
- Introduce autodiscovery mechanism to Core-Common, client configurations that would pick implementations according to a response from /info endpoints.
![Untitled_Diagram.drawio_27_](/uploads/c76eafd5760d89c88093ee5574ed20a5/Untitled_Diagram.drawio_27_.png)
- If API changes require updates in consumer service, introduce them not as a replacement for current functionality but as an addition, relying on autoconfigures from Core-Common.
![Untitled_Diagram.drawio_28_](/uploads/bdd9d172fff18d1a2c5cf3bd28b6c933/Untitled_Diagram.drawio_28_.png)
**Pros**:
- Reduced configuration hell, no need to point out each API version manually in the environment or properties.
- Defined versioning strategy.
- Ability to introduce new API versions, keeping existing environments intact.
**Cons**:
- Bean configs that depend on HTTP requests to fetch API version could slow down initial service initialization.
Pseudocode example:
~~~
@ConditionalOn(PartitionAPICondition.class)
public class PartitionServiceV1{
V1 compatible logic...
}
@ConditionalOn(PartitionAPICondition.class)
public class PartitionServiceV2{
V2 compatible logic...
}
public class PartitionAPICondition{
getApiVersion(){
String apiVersion = partition.getApiVersion()
return apiVersion;
}
}
~~~
## Use cases
**We do not need this approach if:**
1. OSDU API change does not require changes in existing logic.
2. If a new API introduces new entities, and new endpoints, and can be handled by corresponding service without changes in their clients.
3. If new APIs are complementary to existing ones.
For example, Storage Replay API https://community.opengroup.org/osdu/platform/system/storage/-/issues/187 doesn't require changing existing Reindex API in Indexer, it adds new entities and will be handled separately.
**This approach might help us if:**
1. OSDU API change requires changes in existing services client, logic, data structure, and the way they communicate.
2. If API change affects existing entities that are widely used within service logic.
To keep backward compatibility we usually can put the required changes in place. Update the client or add a new one. The main purpose of this ADR is to simplify the maintenance of those changes. Instead of configuring each component manually, pointing out what components should be present in the operating platform, rely on auto configurations via discovery.
## Rationale
- With a versioning strategy that offers backward compatibility development process can go faster.
- Existing environments, that are bound to stable APIs wouldn't block Platform evolution.
- Defining a standard way of versioning APIs will improve Platform maintenance, and help to unify configurations that are currently a mixture of env variables, properties, etc.
## Consequences
- Adding API version to the /info endpoint.
- Changes in Core-Common, the introduction of API version self-discovery mechanism.
- Requirement to follow new versioning approach.
## Alternatives
**Similar approach but without autodiscovery:**
~~~
@ConditionalOn(PartitionAPIV1Property)
public class PartitionServiceV1{
V1 compatible logic...
}
@ConditionalOnProperty(PartitionAPIV2Property)
public class PartitionServiceV2{
V2 compatible logic...
}
public class PartitionAPICondition{
PartitionAPIV1Property = false;
PartitionAPIV2Property = true;
}
}
~~~
Pros:
- no impact on initialization time
Cons:
- It could be a configuration hell, configuring all API versions through properties
~~~
partition.api=v2
storage.api=v1
entitlements.api=v3
etc, etc.
~~~
**Keep things as they are:**
- The proposed solution might be overcomplicated, and violate the YAGNI principle, if use cases are limited, then we can define strategy per change, instead of introducing a standard.https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/os-core-lib-gcp/-/issues/7ADR: OSDU API Versioning Strategy from Service Integration Perspective.2024-02-27T08:23:54ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comADR: OSDU API Versioning Strategy from Service Integration Perspective.TBDTBDhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/78ADR: Strategy for Core-Common Migration to Spring 6 and Jakarta EE2024-03-26T11:39:59ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comADR: Strategy for Core-Common Migration to Spring 6 and Jakarta EE# ADR: Strategy for Core-Common Migration to Spring 6 and Jakarta EE
## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context
Our current challenge revolves around the compatibility of Core-...# ADR: Strategy for Core-Common Migration to Spring 6 and Jakarta EE
## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context
Our current challenge revolves around the compatibility of Core-Common with Spring 6 and Jakarta and the breaking changes caused by the Core-Common upgrade that will affect OSDU services that will keep using Spring 5 and Javax. The same issue is observed with CSP core-libs.
## Problem Statement
If we proceed with upgrading Core-Common to Spring 6 and Jakarta as is, services on Spring 5 won't be able to use the upgraded Core-Common. This also applies to CSP core-libs. Freezing the service with the old Core-Common is not a viable option.
Example of straightforward, not compatible migration: https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/261
## Decision Options
**1. Fork of Core-Common**
Create a new branch or module within the same repository to maintain a second version of Core-Common. Two versions of Core-Common will have to be maintained simultaneously.
- Pros: Less initial development effort is required. Ability to add new breaking changes to Core-Common fork.
- Cons: This approach will require cherry-picking changes, potentially increasing maintenance overhead.
**2. Byte/Source Code Modifications during Build**
Modify code during the build process using plugins.
- Pros: Possibility of a smooth upgrade, no changes would be required for services that are still using Spring 5 and Javax.
- Cons: This workaround may introduce potential build and runtime issues, and its feasibility is uncertain. Requires a lot of initial effort to develop complex build configurations.
**3. Heavy Core-Common Refactoring**
Refactored Core-Common, 2 new modules with extracted affected code, 1 compatible with Spring 5 and Javax, and another for Spring 6 and Jakarta.
- Pros: Cleaner approach, compatibility will be guaranteed, lesser chance of unexpected issues.
- Cons: A lot of effort to refactor. Breaks the current flow, auto upgrades could be blocked, it may be required to switch libraries. Code duplicates are possible, so in further development, we may have to push changes into two modules.
## Decision
TBD
## Consequences
TBDM23 - Release 0.26Chad LeongChad Leonghttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/77Increase timeout for storage service requests2024-02-01T12:51:10ZSudesh TagadpallewarIncrease timeout for storage service requestsWhen registering dataset using `/registerDataset` some users are getting 400 error. As per the Logs this request is timing out(with the error- **Unexpected error sending to URL http://storage/api/storage/v2/records METHOD PUT error java....When registering dataset using `/registerDataset` some users are getting 400 error. As per the Logs this request is timing out(with the error- **Unexpected error sending to URL http://storage/api/storage/v2/records METHOD PUT error java.net.SocketTimeoutException: Read timed out**) when it tries to upsertRecord in the Storage. We have found out that when dataset service is calling storage service and it is taking more than 5 seconds which results in a SocketTimeoutException. When creating `StorageService` instance using `StorageFactory`, new `HttpClient()` instance is used which has default timeout of 5 seconds. Instead of using new `HttpClient` instance `HttpClientHandler` instance should have been used which has 60 seconds timeout. This code is present in the core-common library. See attached image for reference![storage](/uploads/31cfbdb427cf9fc78168dc0fbc4e7f24/storage.JPG)https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/76Unset variables causing NullExceptionPointer in os-core-common AppPropertiesF...2024-01-23T05:26:54ZDeepa KumariUnset variables causing NullExceptionPointer in os-core-common AppPropertiesFeatureFlagImplThere is a scenario when an unset variable is causing the application to fail unexpectedly:
Caused by: java.lang.NullPointerException: Cannot invoke "String.equals(Object)" because the return value of "org.springframework.core.env.Envir...There is a scenario when an unset variable is causing the application to fail unexpectedly:
Caused by: java.lang.NullPointerException: Cannot invoke "String.equals(Object)" because the return value of "org.springframework.core.env.Environment.getProperty(String)" is null
at org.opengroup.osdu.core.common.feature.AppPropertiesFeatureFlagImpl.isFeatureEnabled(AppPropertiesFeatureFlagImpl.java:18)
at org.opengroup.osdu.storage.service.IngestionServiceImpl.createUpdateRecords(IngestionServiceImpl.java:119)
at org.opengroup.osdu.storage.api.RecordApi.createOrUpdateRecords(RecordApi.java:102)
The code here will need to be changed: https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/feature/AppPropertiesFeatureFlagImpl.java#L18
Its even highlighted by sonar lint.M23 - Release 0.26Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/75Empty partition id causing 500 internal server error in partition feature fla...2023-10-26T10:37:15ZAbhishek PatilEmpty partition id causing 500 internal server error in partition feature flag implementationIf `featureFlag.strategy=dataPartition` and a user makes a API call with empty data-partition-id then a 500 Internal Server Error is returned. Ideally a 4xx error should be returned in such cases where user provide incorrect input.
http...If `featureFlag.strategy=dataPartition` and a user makes a API call with empty data-partition-id then a 500 Internal Server Error is returned. Ideally a 4xx error should be returned in such cases where user provide incorrect input.
https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/feature/PartitionFeatureFlagImpl.java#L59
As per the above code, 500 internal server error is thrown because `partitionProvider.get(headers.getPartitionId())` will throw a PartitionException in case of empty data-partition-id.https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/74Absence of NotNull or NotEmpty checks for legal field in storage record causi...2023-10-05T10:22:32ZAbhishek PatilAbsence of NotNull or NotEmpty checks for legal field in storage record causing issuesThe legal field in storage record has no NotNull or NotEmpty validations for it.
https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/model/storage/Recor...The legal field in storage record has no NotNull or NotEmpty validations for it.
https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/model/storage/Record.java?ref_type=heads#L74
This is causing issue if a user try to create a storage record without passing legal in the request payload. In storage service, the legal field of record is accessed(for example [this](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/storage-core/src/main/java/org/opengroup/osdu/storage/service/IngestionServiceImpl.java?ref_type=heads#L397)) and if it's null, a null pointer exception is thrown which ultimately results in 500 Internal Server Error response.
Putting up NotNull/NotEmpty validations for legal will ensure that we throw 400 response with right message in case if legal is not passed in input request payload. Just like we have NotNull validation for acl in storage record (check [this](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/model/storage/Record.java?ref_type=heads#L69)), we can have same validation for legal too.https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/73500 Internal Server error for nonexistent data partitions2023-10-18T10:02:35ZAbhishek Patil500 Internal Server error for nonexistent data partitionsWhile making calls to OSDU services, e.g., storage service, the users are getting a 500 internal server error if they provide a nonexistent data partition id. Technically, as the input is wrong from the user side some sort of 4xx error s...While making calls to OSDU services, e.g., storage service, the users are getting a 500 internal server error if they provide a nonexistent data partition id. Technically, as the input is wrong from the user side some sort of 4xx error should have been returned.
Upon debugging I found that the issue is occurring when featureFlag.strategy=dataPartition. The following is the code from where 500 AppException is being returned if a data partition is not found.
https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/feature/PartitionFeatureFlagImpl.java?ref_type=heads#L52
Is 500 internal server error a expected response in this case as its not the service fault rather a wrong input provided by the user?M21 - Release 0.24Deepa KumariDeepa Kumarihttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/72Error getting data from Redis on lost connection2023-09-18T07:26:13ZYurii Ruban [EPAM / GCP]Error getting data from Redis on lost connectionIn case of loss of connection between the service and Redis, the processing of requests to the service is delayed and an error of type
`Command timed out after 30 SECONDS` is received.
Redis (cache service) is a productivity tool. In c...In case of loss of connection between the service and Redis, the processing of requests to the service is delayed and an error of type
`Command timed out after 30 SECONDS` is received.
Redis (cache service) is a productivity tool. In case of a lost connection, delays in processing requests increase, and, in general, the service becomes unviable.
`entitlements-v2.app: An unknown error has occurred
AppException(error=AppError(code=500, reason=Internal Server Error, message=An unknown error has occurred, errors=null, debuggingInfo=null, originalException=com.lambdaworks.redis.RedisCommandTimeoutException: Command timed out after 30 SECONDS), originalException=com.lambdaworks.redis.RedisCommandTimeoutException: Command timed out after 30 SECONDS)
...
Caused by: com.lambdaworks.redis.RedisCommandTimeoutException: Command timed out after 30 SECONDS
`
I recommend adding Redis health checks.M20 - Release 0.23Yurii Ruban [EPAM / GCP]Yurii Ruban [EPAM / GCP]https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/71Storage records can be created with duplicate ACLs2023-07-28T14:06:59ZAlok JoshiStorage records can be created with duplicate ACLsToday, we can create records with duplicate acls viewers/owners
e.g.
```
"acl": {
"viewers": [
"data.default.viewers@{{data-partition-id}}.{{domain}}",
"data.default.viewers@{{data-partition-...Today, we can create records with duplicate acls viewers/owners
e.g.
```
"acl": {
"viewers": [
"data.default.viewers@{{data-partition-id}}.{{domain}}",
"data.default.viewers@{{data-partition-id}}.{{domain}}"
],
"owners": [
"data.default.owners@{{data-partition-id}}.{{domain}}",
"data.default.owners@{{data-partition-id}}.{{domain}}"
]
}
```
Performing a getRecord for the same record will return duplicate values.
This does not happen to legaltags because it is declared as a Set, while acls are arrays. Making acls a Set can be a breaking change but we can put validation to check for duplicates.M20 - Release 0.23Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/70Invalid data-partition-id will create 500 code with no Authorize message.2023-10-10T12:16:06ZBruce JinInvalid data-partition-id will create 500 code with no Authorize message.While making calls to OSDU services, such as `secret` and `storage` service, testers discover that if they put invalid symbols in `data_partition_id`, we will have 500 code, but with reason of Access Denied.
After investigation, we reali...While making calls to OSDU services, such as `secret` and `storage` service, testers discover that if they put invalid symbols in `data_partition_id`, we will have 500 code, but with reason of Access Denied.
After investigation, we realize the partition service did not consider the situation that user put invalid URI symbols like `@#$%` in data partition
id, which will make the `normalizeStringUrl` function have Java.Lang exception in this [UrlNormalizationUtil.java](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/util/UrlNormalizationUtil.java).
```
Caused by: java.lang.IllegalArgumentException: Malformed escape pair at index 57: http://os-partition:8080/api/partition/v1/partitions/osdu%
at java.net.URI.create(URI.java:852)
at org.opengroup.osdu.core.common.util.UrlNormalizationUtil.normalizeStringUrl(UrlNormalizationUtil.java:27)
```
This will generate a 500 code in entitlement service since the service will treat this error as a general error in [SpringExceptionMapper.java](handleGeneralException), instead a 400 code.
Also in Entitlements the error message is processed within [AuthorizationServiceImpl.java](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/entitlements/AuthorizationServiceImpl.java), so it will have `"Access denied", "The user is not authorized to perform this action"` error message.
Here is a MR that will handle the 500 code produced from `java.lang.IllegalArgumentException` https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/219Bruce JinBruce Jinhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/issues/32Undelete (restore) of soft deleted blob doesn't work anymore2023-07-19T19:28:41ZAlok JoshiUndelete (restore) of soft deleted blob doesn't work anymorePlease refer to [this](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/216) original implementation of the feature.
We started seeing issues with the restore blob functionality in ...Please refer to [this](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/216) original implementation of the feature.
We started seeing issues with the restore blob functionality in our envs. We are seeing an unexpected error of type `com.fasterxml.jackson.core.JsonParseException` with every restore request. Investigation narrowed down the root cause to be [this](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/278) change, particularly the upgrade in `azure-core` version.
Steps to reproduce the issue (use Storage service APIs):
- Create a record
- Manually soft-delete only the record blob (don't touch cosmos metadata)
- Get record
The expected behavior is that Storage service attempts to restore the blob and getRecord API returns successfully. This isn't happening atm due to the mentioned issue.
None of the other later versions of `azure-core` fixes this. It seems this issue persists with any version `1.35.0` and higher
There are no observed vulnerabilities with `1.34.0`, therefore suggesting to revert back to this version.
~~[MR](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/300) for fix (abandoned)~~
After some discussion, we decided to make this [fix](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/726) exclusively in StorageM19 - Release 0.22Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/69Storage null reference exception on filtering properties2023-07-05T12:36:46ZMykyta SavchukStorage null reference exception on filtering propertiesThe error appears to relate to line 174
https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/storage/PersistenceHelper.java
a null pointer is being thro...The error appears to relate to line 174
https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/storage/PersistenceHelper.java
a null pointer is being thrown when calling
subFieldParentElement.isJsonObject()
which leads to 500 error,
so it looks like we need an extra null check on this property and to return null if it is before accessing.Mykyta SavchukMykyta Savchukhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/68Core-common post migration TODOs (Powermock, module access)2023-06-28T14:57:27ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comCore-common post migration TODOs (Powermock, module access)- Unit test refactoring is required, to remove --add-opens params from build run args.
~~~
<argLine>--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED</argLine>
~~~
Tests failed without args:
~~~
Fail...- Unit test refactoring is required, to remove --add-opens params from build run args.
~~~
<argLine>--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED</argLine>
~~~
Tests failed without args:
~~~
Failed tests:
shouldThrowExceptionWhenPropertyNotPresentInEnv(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
shouldThrowExceptionWhenPropertyInMapNull(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
shouldThrowExceptionWhenPropertyValueInMapNull(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
shouldThrowExceptionWhenPropertyNotInMap(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
Tests in error:
shouldNotThrowExceptionWhenPropertyValueInMapIsEmpty(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
shouldNotThrowExceptionWhenEnvReturnEmptyVal(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
shouldGetTypedProperties(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
getPropertyValueFromEnv(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
shouldThrowExceptionWhenEnvReturnNull(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
getPropertyValueFromMapValue(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
~~~
- Powermock dependency removal is highly recommended, Powermock not getting updates to work with Java 17, and has conflicts with Mockito:
~~~
<dependency>
<groupId>org.powermock</groupId>
<artifactId>powermock-core</artifactId>
<version>2.0.9</version>
<scope>test</scope>
</dependency>
~~~https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/67inefficient/non-performant crs-conversions causing reliability/performance is...2023-06-21T07:54:16ZYurii Kondakovinefficient/non-performant crs-conversions causing reliability/performance issues on ingestion workflowWe are seeing reliability and performance issue because of inefficient/non-performant crs-conversions on ingestion workflow. If crs conversions takes long time (on Storage /batch API), then it slows down entire ingestion workflow.
There...We are seeing reliability and performance issue because of inefficient/non-performant crs-conversions on ingestion workflow. If crs conversions takes long time (on Storage /batch API), then it slows down entire ingestion workflow.
There is a need for setting a timeout for crs-conversion requests that run for more than a certain time. For the requests to crs-conversion-service currently java.net.HttpURLConnection class is used, which is only has connectionTimeout and readTimeout properties, that don't help us to set the timeout.
It is suggested to use apache CloseableHttpClient httpClient that has socketTimeout property.
core-common MR - https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/213
storage MR - https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/712M19 - Release 0.22Yurii KondakovYurii Kondakovhttps://community.opengroup.org/osdu/platform/system/lib/cloud/aws/os-core-lib-aws/-/issues/8Improper return value prevents local mode operation2023-07-11T14:51:14ZMark ChanceImproper return value prevents local mode operationWith LOCAL_MODE true, the Search Service (and possibly others) does not start due to NullPointerException in this library.
In org.opengroup.osdu.core.aws.ssm.K8sLocalParameterProvider:93
```java
// this is for credentials, credentials...With LOCAL_MODE true, the Search Service (and possibly others) does not start due to NullPointerException in this library.
In org.opengroup.osdu.core.aws.ssm.K8sLocalParameterProvider:93
```java
// this is for credentials, credentials mounted by CSI is a json string
// if in local mode, it returns an empty HashMap, it is the responsibility of end user to getDefault
public Map<String, String> getCredentialsAsMap(String parameterKey) throws K8sParameterNotFoundException, JsonProcessingException {
if (localMode) {
return null;
}
return objectMapper.readValue(this.getParameterAsString(parameterKey), typeRef);
```
Despite the comment to the contrary, the return value causes problems for example in
org.opengroup.osdu.core.aws.entitlements.ServiceAccountJwtAwsClientImpl:50
```java
private void init() {
K8sLocalParameterProvider provider = new K8sLocalParameterProvider();
try {
client_credentials_clientid = provider.getParameterAsString("CLIENT_CREDENTIALS_ID");
client_credentials_secret = provider.getCredentialsAsMap("CLIENT_CREDENTIALS_SECRET").get("client_credentials_client_secret");
...
```
If we change "return null;" to "return new HashMap<>();" it works better.Yong ZengYong Zenghttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/66Code duplicity and misleading parameter usage in Redis Cache constructor2023-06-05T09:41:27Zharshit aggarwalCode duplicity and misleading parameter usage in Redis Cache constructorCurrently there is a lot of code duplicity in some of the RedisCache constructors which can be avoided be easily avoided by implemented a common base constructor
Also one issue is that the parameter `expTimeSeconds` which is supposed to...Currently there is a lot of code duplicity in some of the RedisCache constructors which can be avoided be easily avoided by implemented a common base constructor
Also one issue is that the parameter `expTimeSeconds` which is supposed to be the TTLs for the keys in Redis cache is also used for command execution timeout value as well which is very misleading because both values can drastically differ in values becasue TTLs can be very long duration as compared to command execution timeouts
Code reference - https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/cache/RedisCache.java#L48
The above should be cleanup or modified to take appropriate params with some default valuesM18 - Release 0.21https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/issues/30use of RequestMappingHandlerMapping in Slf4jMDCFilter2023-03-09T18:33:59ZNeelesh Thakuruse of RequestMappingHandlerMapping in Slf4jMDCFilter`Slf4jMDCFilter` is being used in quite a few services and it's invoked on each API request. It's throwing following exception:
![image](/uploads/f955b40cbc4d8652ca927fa26c00a111/image.png)
Based on traffic, we are seeing millions of s...`Slf4jMDCFilter` is being used in quite a few services and it's invoked on each API request. It's throwing following exception:
![image](/uploads/f955b40cbc4d8652ca927fa26c00a111/image.png)
Based on traffic, we are seeing millions of such exceptions are thrown across different services in several environments.https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/65Wrong encoding of special characters in URL query component2023-05-30T08:55:20ZKelly ZhouWrong encoding of special characters in URL query componentWhile consuming HttpClient from os core common library, we found that some special characters in URL query component are not properly encoded, 1 example of that is `+` (the plus symbol).
According to [RFC 2396](https://www.rfc-editor.o...While consuming HttpClient from os core common library, we found that some special characters in URL query component are not properly encoded, 1 example of that is `+` (the plus symbol).
According to [RFC 2396](https://www.rfc-editor.org/rfc/rfc2396#appendix-A), it is a reserved character in query component so it is not encoded when forming the url, however, if we encode it explicitly before constructing the URL by using library, the escape character `%` would be treated as a non-reserved character in query component which leads to double-encoding issue for `+` symbol.
As a result, if we send `+` as plain text, library forms the url as it is, if we encode `+` to `%2B` before sending it to library, it comes out as `%252B`, neither of them represents the original url when it arrives at server side.M17 - Release 0.20https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/64Spring configuration related to Feature flag should be optional2023-01-30T11:37:15ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comSpring configuration related to Feature flag should be optionalIt is possible that some OSDU services may not use Partition service, Unit service as an example. But the configuration of the recently merged code https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requ...It is possible that some OSDU services may not use Partition service, Unit service as an example. But the configuration of the recently merged code https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/189 requires PARTITION_API env variable mandatorily. It will be better to change the configuration to make it optional, instead of bringing this variable to each service, no matter if Partition is used or not.
~~~
Exception encountered during context initialization - cancelling refresh attempt:
org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'featureFlagConfig':
Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException:
Could not resolve placeholder 'PARTITION_API' in value "${PARTITION_API}"
~~~Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRiabokon Stanislav(EPAM)[GCP]Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.com