OS Core Common - Spring 5 issueshttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues2024-03-26T11:39:59Zhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/78ADR: Strategy for Core-Common Migration to Spring 6 and Jakarta EE2024-03-26T11:39:59ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comADR: Strategy for Core-Common Migration to Spring 6 and Jakarta EE# ADR: Strategy for Core-Common Migration to Spring 6 and Jakarta EE
## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context
Our current challenge revolves around the compatibility of Core-...# ADR: Strategy for Core-Common Migration to Spring 6 and Jakarta EE
## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context
Our current challenge revolves around the compatibility of Core-Common with Spring 6 and Jakarta and the breaking changes caused by the Core-Common upgrade that will affect OSDU services that will keep using Spring 5 and Javax. The same issue is observed with CSP core-libs.
## Problem Statement
If we proceed with upgrading Core-Common to Spring 6 and Jakarta as is, services on Spring 5 won't be able to use the upgraded Core-Common. This also applies to CSP core-libs. Freezing the service with the old Core-Common is not a viable option.
Example of straightforward, not compatible migration: https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/261
## Decision Options
**1. Fork of Core-Common**
Create a new branch or module within the same repository to maintain a second version of Core-Common. Two versions of Core-Common will have to be maintained simultaneously.
- Pros: Less initial development effort is required. Ability to add new breaking changes to Core-Common fork.
- Cons: This approach will require cherry-picking changes, potentially increasing maintenance overhead.
**2. Byte/Source Code Modifications during Build**
Modify code during the build process using plugins.
- Pros: Possibility of a smooth upgrade, no changes would be required for services that are still using Spring 5 and Javax.
- Cons: This workaround may introduce potential build and runtime issues, and its feasibility is uncertain. Requires a lot of initial effort to develop complex build configurations.
**3. Heavy Core-Common Refactoring**
Refactored Core-Common, 2 new modules with extracted affected code, 1 compatible with Spring 5 and Javax, and another for Spring 6 and Jakarta.
- Pros: Cleaner approach, compatibility will be guaranteed, lesser chance of unexpected issues.
- Cons: A lot of effort to refactor. Breaks the current flow, auto upgrades could be blocked, it may be required to switch libraries. Code duplicates are possible, so in further development, we may have to push changes into two modules.
## Decision
TBD
## Consequences
TBDM23 - Release 0.26Chad LeongChad Leonghttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/77Increase timeout for storage service requests2024-02-01T12:51:10ZSudesh TagadpallewarIncrease timeout for storage service requestsWhen registering dataset using `/registerDataset` some users are getting 400 error. As per the Logs this request is timing out(with the error- **Unexpected error sending to URL http://storage/api/storage/v2/records METHOD PUT error java....When registering dataset using `/registerDataset` some users are getting 400 error. As per the Logs this request is timing out(with the error- **Unexpected error sending to URL http://storage/api/storage/v2/records METHOD PUT error java.net.SocketTimeoutException: Read timed out**) when it tries to upsertRecord in the Storage. We have found out that when dataset service is calling storage service and it is taking more than 5 seconds which results in a SocketTimeoutException. When creating `StorageService` instance using `StorageFactory`, new `HttpClient()` instance is used which has default timeout of 5 seconds. Instead of using new `HttpClient` instance `HttpClientHandler` instance should have been used which has 60 seconds timeout. This code is present in the core-common library. See attached image for reference![storage](/uploads/31cfbdb427cf9fc78168dc0fbc4e7f24/storage.JPG)https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/75Empty partition id causing 500 internal server error in partition feature fla...2023-10-26T10:37:15ZAbhishek PatilEmpty partition id causing 500 internal server error in partition feature flag implementationIf `featureFlag.strategy=dataPartition` and a user makes a API call with empty data-partition-id then a 500 Internal Server Error is returned. Ideally a 4xx error should be returned in such cases where user provide incorrect input.
http...If `featureFlag.strategy=dataPartition` and a user makes a API call with empty data-partition-id then a 500 Internal Server Error is returned. Ideally a 4xx error should be returned in such cases where user provide incorrect input.
https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/feature/PartitionFeatureFlagImpl.java#L59
As per the above code, 500 internal server error is thrown because `partitionProvider.get(headers.getPartitionId())` will throw a PartitionException in case of empty data-partition-id.https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/70Invalid data-partition-id will create 500 code with no Authorize message.2023-10-10T12:16:06ZBruce JinInvalid data-partition-id will create 500 code with no Authorize message.While making calls to OSDU services, such as `secret` and `storage` service, testers discover that if they put invalid symbols in `data_partition_id`, we will have 500 code, but with reason of Access Denied.
After investigation, we reali...While making calls to OSDU services, such as `secret` and `storage` service, testers discover that if they put invalid symbols in `data_partition_id`, we will have 500 code, but with reason of Access Denied.
After investigation, we realize the partition service did not consider the situation that user put invalid URI symbols like `@#$%` in data partition
id, which will make the `normalizeStringUrl` function have Java.Lang exception in this [UrlNormalizationUtil.java](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/util/UrlNormalizationUtil.java).
```
Caused by: java.lang.IllegalArgumentException: Malformed escape pair at index 57: http://os-partition:8080/api/partition/v1/partitions/osdu%
at java.net.URI.create(URI.java:852)
at org.opengroup.osdu.core.common.util.UrlNormalizationUtil.normalizeStringUrl(UrlNormalizationUtil.java:27)
```
This will generate a 500 code in entitlement service since the service will treat this error as a general error in [SpringExceptionMapper.java](handleGeneralException), instead a 400 code.
Also in Entitlements the error message is processed within [AuthorizationServiceImpl.java](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/entitlements/AuthorizationServiceImpl.java), so it will have `"Access denied", "The user is not authorized to perform this action"` error message.
Here is a MR that will handle the 500 code produced from `java.lang.IllegalArgumentException` https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/219Bruce JinBruce Jinhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/74Absence of NotNull or NotEmpty checks for legal field in storage record causi...2023-10-05T10:22:32ZAbhishek PatilAbsence of NotNull or NotEmpty checks for legal field in storage record causing issuesThe legal field in storage record has no NotNull or NotEmpty validations for it.
https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/model/storage/Recor...The legal field in storage record has no NotNull or NotEmpty validations for it.
https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/model/storage/Record.java?ref_type=heads#L74
This is causing issue if a user try to create a storage record without passing legal in the request payload. In storage service, the legal field of record is accessed(for example [this](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/storage-core/src/main/java/org/opengroup/osdu/storage/service/IngestionServiceImpl.java?ref_type=heads#L397)) and if it's null, a null pointer exception is thrown which ultimately results in 500 Internal Server Error response.
Putting up NotNull/NotEmpty validations for legal will ensure that we throw 400 response with right message in case if legal is not passed in input request payload. Just like we have NotNull validation for acl in storage record (check [this](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/model/storage/Record.java?ref_type=heads#L69)), we can have same validation for legal too.https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/72Error getting data from Redis on lost connection2023-09-18T07:26:13ZYurii Ruban [EPAM / GCP]Error getting data from Redis on lost connectionIn case of loss of connection between the service and Redis, the processing of requests to the service is delayed and an error of type
`Command timed out after 30 SECONDS` is received.
Redis (cache service) is a productivity tool. In c...In case of loss of connection between the service and Redis, the processing of requests to the service is delayed and an error of type
`Command timed out after 30 SECONDS` is received.
Redis (cache service) is a productivity tool. In case of a lost connection, delays in processing requests increase, and, in general, the service becomes unviable.
`entitlements-v2.app: An unknown error has occurred
AppException(error=AppError(code=500, reason=Internal Server Error, message=An unknown error has occurred, errors=null, debuggingInfo=null, originalException=com.lambdaworks.redis.RedisCommandTimeoutException: Command timed out after 30 SECONDS), originalException=com.lambdaworks.redis.RedisCommandTimeoutException: Command timed out after 30 SECONDS)
...
Caused by: com.lambdaworks.redis.RedisCommandTimeoutException: Command timed out after 30 SECONDS
`
I recommend adding Redis health checks.M20 - Release 0.23Yurii Ruban [EPAM / GCP]Yurii Ruban [EPAM / GCP]https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/8WIP: Semantic release numbering (GONRG-819)2023-07-05T10:09:41ZRostislav Dublin (EPAM)WIP: Semantic release numbering (GONRG-819)## Change Type:
- [x] Feature
- [ ] Bugfix
- [ ] Refactoring
## Context and Scope
All OSDU microservices projects developed by the community. Those microservices are in the active development by different teams simultaneously, and when...## Change Type:
- [x] Feature
- [ ] Bugfix
- [ ] Refactoring
## Context and Scope
All OSDU microservices projects developed by the community. Those microservices are in the active development by different teams simultaneously, and when being merged to the community they should be versioned unambiguously and consistently. This is hard to achieve without some central point of registration and computation, and such stuff is not implemented yet.
## Decision
Improve release management for all microservices projects by implementation of proposed technical solution and an agreement of proper usage of it in OSDU context.
## Rational
We just performed an POC investigation and elaborated a suitable technical solution (scripts & Gitlab CICD template) which can be rather easily applied on the community pipelines.
## Consequences
The solution will be implemented to manage release numbering:
- based on principles of [Semantic Versioning 2.0.0](https://semver.org/),
- using technical implementation (script/pipeline template) developed as part of POC by EPAM,
- with involving repositories tags and CICD variables to the action
- by adding several specific steps to all community pipelines
- and sticking special labels (bump-major and bump-minor) to merge requests,
- and causing resulting pipelines' artifacts' names to be composed with produced release numbers inclusions
All CSPs will need to:
- follow the agreed strategy of version numbering (know, when to bump major, minor and patch parts of it);
- refactor pom.xml files (use <version>${revision}</version> substitution instead of constant version number;
- include versioning solution template to pipelines;
- instruct maintainers to set bump-major and bump-minor labels on MRsDmitriy RudkoDmitriy Rudko2020-10-27https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/19OSDU Java stack Standards2023-07-05T10:09:41ZRostislav Dublin (EPAM)OSDU Java stack Standards## Change Type:
- [x] Feature
- [ ] Bugfix
- [ ] Refactoring
## Context and Scope
Standardization has a positive impact for successful software development.
Java coding standards makes sure that all the developers working on the proje...## Change Type:
- [x] Feature
- [ ] Bugfix
- [ ] Refactoring
## Context and Scope
Standardization has a positive impact for successful software development.
Java coding standards makes sure that all the developers working on the project are following certain specified guidelines.
This is especially critical for OSDU community, where many Java teams develop independently and contribute into the common codebase.
The code should be easily understood and proper consistency maintained.
Currently, Java coding standards are not defined in the community. It leads to the following issues:
- Poor code readability
- Difficult onboarding
- Same functionality redundancy
- Libraries and frameworks versioning mess
- Testing frameworks mess
- Security Concerns
- Performance Issues
- etc.
## Decision
Define, adopt and enforce a single Java coding standard for the community. Improve code quality and maintainability.
## Rational
Achieve the following advantages:
- Enhanced Efficiency - detect the problems early or even prevent them completely, increase efficiency throughout the software process.
- Reduced risk of project failure - IT projects fails happen due to problems while developing software.
- Decreased Complexity - coding standards help to develop less complex software and thereby reduce the errors.
- Simplified Maintenance - when code is consistent it can be easily maintained, cause anyone can easily understand and modify it.
- Bug Rectification - it becomes really easy to locate and correct bugs if the source code is written in a consistent manner.
- Comprehensive Look - a clear view of how code fits in the larger application or fits into the company as a whole.
- Cost-Efficiency - a clear code gives an opportunity to reuse it. This reduces the cost along with the efforts put in the development.
## Consequences
The following definition of Java code Standard should be adopted and enforced:
[[OSDU GCP Java Coding Standards]](https://community.opengroup.org/osdu/documentation/-/wikis/Releases/R2.0/GCP/Architecture/OSDU-GCP-Java-Coding-Standards)
All CSPs will need to:
- enforce and follow the adopted Java coding Standard in their Java teams;
- start contributing only compliant new Java code;
- refactoring existing Java code where possible;Dmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/22OSDU REST API Standards2023-07-05T10:09:40ZRostislav Dublin (EPAM)OSDU REST API Standards## Change Type:
- [x] Feature
- [ ] Bugfix
- [ ] Refactoring
## Context and Scope
Standardization has a positive impact for successful software development.
REST API standards makes sure that all the developers working on the project ...## Change Type:
- [x] Feature
- [ ] Bugfix
- [ ] Refactoring
## Context and Scope
Standardization has a positive impact for successful software development.
REST API standards makes sure that all the developers working on the project are following certain specified guidelines.
This is especially critical for OSDU community, where many teams develop independently and produce dozens of common APIs.
The code should be easily understood and proper consistency maintained.
Currently, REST API standards are not defined in the community. It leads to the following issues:
- Poor code readability
- Difficult onboarding
- Same functionality redundancy
- Versioning mess
- Security Concerns
- Performance Issues
- etc.
## Decision
Define, adopt and enforce a single REST API standard for the community. Improve API quality and maintainability.
## Rational
Achieve the following advantages:
- Consistent design across an community’s APIs to ensure the smoothest possible experience for developers using OSDU platform.
- Standardization in design to make APIs feel familiar.
- Minimization the learning curve as developers;
- Design improvement making using APIs easy and intuitive.
## Consequences
The following definition of REST API Standard should be adopted and enforced:
[[OSDU REST API Standards]](https://community.opengroup.org/osdu/documentation/-/wikis/Releases/R2.0/GCP/Architecture/OSDU-GCP-REST-API-Standards)
All CSPs will need to:
- enforce and follow the adopted REST API Standard in their teams;
- start contributing only compliant new REST API;
- refactoring existing REST API where possible;Dmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/68Core-common post migration TODOs (Powermock, module access)2023-06-28T14:57:27ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comCore-common post migration TODOs (Powermock, module access)- Unit test refactoring is required, to remove --add-opens params from build run args.
~~~
<argLine>--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED</argLine>
~~~
Tests failed without args:
~~~
Fail...- Unit test refactoring is required, to remove --add-opens params from build run args.
~~~
<argLine>--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED</argLine>
~~~
Tests failed without args:
~~~
Failed tests:
shouldThrowExceptionWhenPropertyNotPresentInEnv(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
shouldThrowExceptionWhenPropertyInMapNull(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
shouldThrowExceptionWhenPropertyValueInMapNull(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
shouldThrowExceptionWhenPropertyNotInMap(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
Tests in error:
shouldNotThrowExceptionWhenPropertyValueInMapIsEmpty(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
shouldNotThrowExceptionWhenEnvReturnEmptyVal(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
shouldGetTypedProperties(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
getPropertyValueFromEnv(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
shouldThrowExceptionWhenEnvReturnNull(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
getPropertyValueFromMapValue(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
~~~
- Powermock dependency removal is highly recommended, Powermock not getting updates to work with Java 17, and has conflicts with Mockito:
~~~
<dependency>
<groupId>org.powermock</groupId>
<artifactId>powermock-core</artifactId>
<version>2.0.9</version>
<scope>test</scope>
</dependency>
~~~https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/67inefficient/non-performant crs-conversions causing reliability/performance is...2023-06-21T07:54:16ZYurii Kondakovinefficient/non-performant crs-conversions causing reliability/performance issues on ingestion workflowWe are seeing reliability and performance issue because of inefficient/non-performant crs-conversions on ingestion workflow. If crs conversions takes long time (on Storage /batch API), then it slows down entire ingestion workflow.
There...We are seeing reliability and performance issue because of inefficient/non-performant crs-conversions on ingestion workflow. If crs conversions takes long time (on Storage /batch API), then it slows down entire ingestion workflow.
There is a need for setting a timeout for crs-conversion requests that run for more than a certain time. For the requests to crs-conversion-service currently java.net.HttpURLConnection class is used, which is only has connectionTimeout and readTimeout properties, that don't help us to set the timeout.
It is suggested to use apache CloseableHttpClient httpClient that has socketTimeout property.
core-common MR - https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/213
storage MR - https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/712M19 - Release 0.22Yurii KondakovYurii Kondakovhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/39ADR: Partion Service as a TenantInfo provider2023-04-07T12:43:48ZRostislav Dublin (EPAM)ADR: Partion Service as a TenantInfo provider## Status
- [x] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context
Currently, CSPs have their own implementation of retrieving the TenantInfo data:
1. AWS. https://community.opengroup.org/osdu/platform/sy...## Status
- [x] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context
Currently, CSPs have their own implementation of retrieving the TenantInfo data:
1. AWS. https://community.opengroup.org/osdu/platform/system/lib/cloud/aws/os-core-lib-aws/-/blob/master/src/main/java/org/opengroup/osdu/core/aws/multitenancy/TenantFactory.java
2. Azure. https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/blob/master/src/main/java/org/opengroup/osdu/azure/multitenancy/TenantFactoryImpl.java
3. IBM. https://community.opengroup.org/osdu/platform/system/lib/cloud/ibm/os-core-lib-ibm/-/blob/master/src/main/java/org/opengroup/osdu/core/ibm/multitenancy/TenantFactory.java
All implementations follow a similar way to retrieve the TenantInfo from the Partition service (request partition service->map to TenantInfo class; calls to the Partition service are cached).
## Scope
Implement a common approach to retrieve the TenantInfo data and use it across all services.
## Decision
Unify an approach of retrieving TenantInfo and using it across all services:
- Implement a common tenant factory (ITenantFactory) on the core-lib level.
- Remove CSP-specific implementations of the TenantFactory.
- Some CSPs will require to restructure a code and implement missing interfaces (in fact, the code is already there):
- IBM should implement IServiceAccountJwtClient explicitly. There is already a similar method in the TenantFactory implementation.
### Outcome of the Decision (added by @rostislav.dublin on May 24):
1. A single common implementation of PartitionTenantInfoFactory in os-core-common. It just lies here and does not inject anywhere. [See it](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/73/diffs#eb73934c527564533da4c8574c831649474ee500_0_36) in the [MR 73](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/73/) ;
2. Each CSP injects the above single common implementation in its CSP library, removing its own implementation. Everyone moves at their own pace, no one rushes anyone. E.g. GCP uses AbstractFactoryBean<IPartitionFactory> [as shown here](https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/102/diffs#4ac1210ac980e2a0dcec2574ebd6eb21618e8abc_0_23).
## Rationale
This change unifies an approach of retrieving tenant information, which is used to separate tenants from each other (multitenancy). With the increased maintainability of the code, OSDU will have a unified basic data to implement multitenancy
## Consequences
Simplified CSP-specific code and code generalization (higher maintainability and reusability).https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/64Spring configuration related to Feature flag should be optional2023-01-30T11:37:15ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comSpring configuration related to Feature flag should be optionalIt is possible that some OSDU services may not use Partition service, Unit service as an example. But the configuration of the recently merged code https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requ...It is possible that some OSDU services may not use Partition service, Unit service as an example. But the configuration of the recently merged code https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/189 requires PARTITION_API env variable mandatorily. It will be better to change the configuration to make it optional, instead of bringing this variable to each service, no matter if Partition is used or not.
~~~
Exception encountered during context initialization - cancelling refresh attempt:
org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'featureFlagConfig':
Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException:
Could not resolve placeholder 'PARTITION_API' in value "${PARTITION_API}"
~~~Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRiabokon Stanislav(EPAM)[GCP]Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/29Get Group Members query parameters not working properly.2022-12-08T17:14:06ZGregGet Group Members query parameters not working properly.GET – Get Group Members. Functionality working but defects. Query parameters “limit” and “role” not working properly If query params optional,e.g., it should return all the members regardless of the role for the request URL {{osduonaws_b...GET – Get Group Members. Functionality working but defects. Query parameters “limit” and “role” not working properly If query params optional,e.g., it should return all the members regardless of the role for the request URL {{osduonaws_base_url}}/api/entitlements/v1/groups/data.testing.osduonaws-testing@opendes.shell.com/members?limit=100. Currently it only return the user with the role of OWNER If query params required, e.g., it should return error handling with the message like “required parameter xxx missing”. There is defect with either scenario. And it would make more sense to keep query params optional in this case.M1 - Release 0.1JoeRucha DeshpandeMatt WiseJoehttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/30Cursor search returns results even when version does not match.2022-12-08T17:13:50ZGregCursor search returns results even when version does not match.POST – Query With Cursor. Functionality working but defect.
For example, below request is to search "opendes:osdu:wellbore-master:0.2.0" with the cursor "2F8900678904A680D24593BC7D8BEEA5".
However, it is still returning the results wit...POST – Query With Cursor. Functionality working but defect.
For example, below request is to search "opendes:osdu:wellbore-master:0.2.0" with the cursor "2F8900678904A680D24593BC7D8BEEA5".
However, it is still returning the results with 0.2.0 even I change the version number of data, e.g. "opendes:osdu:wellbore-master:0.0.8"
{ "kind": "opendes:osdu:wellbore-master:0.2.0", "cursor": "2F8900678904A680D24593BC7D8BEEA5", "aggregateBy": "kind" }
It seems that the kind should not be required once you have the Cursor ID. Or, if it is required, the kind should need to be the same as the original query.M1 - Release 0.1Chris ZhangChris Zhanghttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/31Bean with hard-coded properties imposes policy that limits application/servic...2022-11-24T12:34:33ZSherman YangBean with hard-coded properties imposes policy that limits application/service flexibilityIn OS Core Common repo's src/main/java/org/opengroup/osdu/core/common/http/HttpConfiguration.java, a bean is automatically created with hard-coded properties that impose policy which limits service flexibility.
For example, the new ent...In OS Core Common repo's src/main/java/org/opengroup/osdu/core/common/http/HttpConfiguration.java, a bean is automatically created with hard-coded properties that impose policy which limits service flexibility.
For example, the new entitlement v2 service needs to accept case insensitive Json input but with the hard-coded ObjectMapper bean that got introduced since osdu core common version 0.0.17, the entitlement service's case insensitive json configuration is no longer available as the common library creates its own ObjectMapper which disables/overrides the service's settings and imposes case sensitivity.
Need to relax this limitation to allow application flexibility.https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/24Inconsistent error handling across services.2022-09-27T14:19:25ZGregInconsistent error handling across services.Entitlements Service & Search Service. The response body is in JSON format. The attributes of “code”, “reason”, “message” are returned along with status code, e.g., 401 Unauthorized. Legal Service. The response body is in Text format wit...Entitlements Service & Search Service. The response body is in JSON format. The attributes of “code”, “reason”, “message” are returned along with status code, e.g., 401 Unauthorized. Legal Service. The response body is in Text format with no attributes of “code”, “reason”, “message” returned.M1 - Release 0.1Chris ZhangChris Zhanghttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/25Delete LegalTag response message is not correct.2022-09-27T13:50:36ZGregDelete LegalTag response message is not correct.DEL – Delete LegalTag. Functionality working but needs improvement. Need to improve the response as it is returning 204 No Content without any message regardless of whether the legal tag exists or not. If the Legal Tag does not exist, it...DEL – Delete LegalTag. Functionality working but needs improvement. Need to improve the response as it is returning 204 No Content without any message regardless of whether the legal tag exists or not. If the Legal Tag does not exist, it is expected to return 404 Not Found with the message like “Cannot delete a LegalTag that does not exist for given name xxx”M1 - Release 0.1Chris ZhangChris Zhanghttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/49ADR : Optional Audit Logs per partition2022-05-20T16:46:09ZPreksha Beohar-SlbADR : Optional Audit Logs per partition## Status
- [X] Proposed
- [X] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
As an operator of OSDU we are seeing increased levels of costs related to logging over time. For instance in a single day we saw over 500GB of...## Status
- [X] Proposed
- [X] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
As an operator of OSDU we are seeing increased levels of costs related to logging over time. For instance in a single day we saw over 500GB of logs created per environment. This can run to costs of $X000 a day.
A large percentage of these logs can be equated to audit logs, especially for READ actions like list groups in Entitlements which can be called multiple millions of times a day.
We have deployments and partitions of varying degrees of sensitivity. For example we have deployments strictly for development purposes but even in our production systems some partitions are used solely for development by their clients.
As a client of the system I would like to be able to control which partitions have audit logs enabled and if so whether I have this turned on for write operations only i.e. create,update,delete actions or all events in the system.
This gives me the choice of whether the cost benefit of running audit logs in a particular partition is worthwhile and can give the client the control based on the sensitivity of the partition.
This has precedent in other systems e.g. in Google Cloud or Azure I have the choice for most resources I deploy whether to turn on and retain audit logs for these services even though they _could_ have sensitive information as it gives me as the client the control and choice.
## Trade-off Analysis
I could choose to have all audit logs turned on at all times. This is the current state. However this has potentially high cost implications. If the partition or deployment of OSDU does not contain sensitive information then I would like the choice to turn off audit logs.
I could make the option to turn on/off logs as part of the build configuration. However this means I need to redeploy if I want to switch logs on or off. This also prevents end users potentially ever having this control.
I could make the configuration for the entire environment rather than per partition. However a strong use case for partitions is to separate them based on data sensitivity e.g. I may have a partition for development purposes and another partition in the same deployment with production data.
## Decision
We are proposing the ability to have audit logs turned on or off by choice of the operator and/or client.
An implementation of this could be via a feature flag from the partition service which will allow this to be set at runtime and per partition.
We would like 2 separate feature flags. One which can turn the READ action audit logs on or off and another to turn the CREATE/UPDATE/DELETE/PUBLISH/JOB_RUN action audit logs on or off.
By default both of these should be turned on so it is 'secure' by default. However the operator and/or end user can then choose to turn certain ones off.Preksha Beohar-SlbPreksha Beohar-Slbhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/10Expose audit trail attributes in search results2021-10-04T22:12:09ZDebasis ChatterjeeExpose audit trail attributes in search resultsThis is very useful information and may be added to standard query result.
Username who created the record.
Date/time when record was created in OSDU.
Also expose Username who updated the record.
Date/time when update operation was per...This is very useful information and may be added to standard query result.
Username who created the record.
Date/time when record was created in OSDU.
Also expose Username who updated the record.
Date/time when update operation was performed.
@dmitry-kniazev later added the example from IBM implementation as he was showing during recent session of "Reporting and dashboarding"?JoeJoe