Lib issueshttps://community.opengroup.org/groups/osdu/platform/system/lib/-/issues2022-08-30T11:52:13Zhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/issues/200.13.0 Getting NoSuchMethodError at com.azure.storage.blob.specialized.BlobAs...2022-08-30T11:52:13ZTsvetelina Ivanova0.13.0 Getting NoSuchMethodError at com.azure.storage.blob.specialized.BlobAsyncClientBase.lambda$downloadStreamWithResponse$22When using version 0.13.0 we get exception NoSuchMethodError at com.azure.storage.blob.specialized.BlobAsyncClientBase.lambda$downloadStreamWithResponse$22 when try to read files from blob storage.
In version 0.13.0 a new version of azu...When using version 0.13.0 we get exception NoSuchMethodError at com.azure.storage.blob.specialized.BlobAsyncClientBase.lambda$downloadStreamWithResponse$22 when try to read files from blob storage.
In version 0.13.0 a new version of azure-storage-blob is introduced 12.13.0. In class BlobAsyncClientBase on line 1069 a call to FluxUtil.createRetriableDownloadFlux() is made.
In class FluxUtil method createRetriableDownloadFlux() does not exists. (This class is in library azure-core).
**Stack Trace:**
java.lang.NoSuchMethodError: com.azure.core.util.FluxUtil.createRetriableDownloadFlux(Ljava/util/function/Supplier;Ljava/util/function/BiFunction;IJ)Lreactor/core/publisher/Flux;
at com.azure.storage.blob.specialized.BlobAsyncClientBase.lambda$downloadStreamWithResponse$22(BlobAsyncClientBase.java:1069)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:249)
at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2398)
at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2194)
at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2068)
at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:192)
at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53)
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
at reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.onNext(FluxDoOnEach.java:173)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120)
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
at reactor.core.publisher.FluxDelaySubscription$DelaySubscriptionMainSubscriber.onNext(FluxDelaySubscription.java:189)
at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onNext(FluxTimeout.java:180)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:249)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:127)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:127)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onNext(FluxRetryWhen.java:174)
at reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2664)
at reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:180)
at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2400)
at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onSubscribeInner(MonoFlatMapMany.java:150)
at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:189)
at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onNext(FluxRetryWhen.java:174)
at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:165)
at reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:414)
at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:671)
at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.onStateChange(DefaultPooledConnectionProvider.java:201)
at reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnection.onStateChange(DefaultPooledConnectionProvider.java:457)
at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:637)
at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:93)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:311)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:432)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371)
at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1245)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.kqueue.AbstractKQueueStreamChannel$KQueueStreamUnsafe.readReady(AbstractKQueueStreamChannel.java:544)
at io.netty.channel.kqueue.AbstractKQueueChannel$AbstractKQueueUnsafe.readReady(AbstractKQueueChannel.java:381)
at io.netty.channel.kqueue.KQueueEventLoop.processReady(KQueueEventLoop.java:211)
at io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:289)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)M14 - Release 0.17https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/74Absence of NotNull or NotEmpty checks for legal field in storage record causi...2023-10-05T10:22:32ZAbhishek PatilAbsence of NotNull or NotEmpty checks for legal field in storage record causing issuesThe legal field in storage record has no NotNull or NotEmpty validations for it.
https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/model/storage/Recor...The legal field in storage record has no NotNull or NotEmpty validations for it.
https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/model/storage/Record.java?ref_type=heads#L74
This is causing issue if a user try to create a storage record without passing legal in the request payload. In storage service, the legal field of record is accessed(for example [this](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/storage-core/src/main/java/org/opengroup/osdu/storage/service/IngestionServiceImpl.java?ref_type=heads#L397)) and if it's null, a null pointer exception is thrown which ultimately results in 500 Internal Server Error response.
Putting up NotNull/NotEmpty validations for legal will ensure that we throw 400 response with right message in case if legal is not passed in input request payload. Just like we have NotNull validation for acl in storage record (check [this](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/model/storage/Record.java?ref_type=heads#L69)), we can have same validation for legal too.https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/1Adding support for logger name in JaxRsDpsLogger and ILogger for fine grained...2021-02-20T07:50:10ZKishore BattulaAdding support for logger name in JaxRsDpsLogger and ILogger for fine grained configuration of log messages## Context and Scope
Below are the sample logs in storage service
```
2020-06-29 19:34:19.783 INFO 5644 --- [nio-8082-exec-3] o.o.o.c.common.logging.DefaultLogWriter : storage.app: Start Web-API DELETE /records/opendes:id:1593439432354...## Context and Scope
Below are the sample logs in storage service
```
2020-06-29 19:34:19.783 INFO 5644 --- [nio-8082-exec-3] o.o.o.c.common.logging.DefaultLogWriter : storage.app: Start Web-API DELETE /records/opendes:id:1593439432354 Headers: {data-partition-id:opendes,content-type:application/json} {correlation-id=68a989a1-f1a2-4bbc-9f1c-5c734a790311, data-partition-id=opendes}
2020-06-29 19:34:22.342 INFO 5644 --- [nio-8082-exec-3] o.o.o.c.common.logging.DefaultLogWriter : {auditLog={resources=[opendes:id:1593439432354], action=DELETE, actionId=ST003, message=Record purged, user=e6ac6dfe-8198-4f9c-b2e8-b974c0f9ef5b, status=SUCCESS}} {correlation-id=68a989a1-f1a2-4bbc-9f1c-5c734a790311, data-partition-id=opendes}
2020-06-29 19:34:22.344 INFO 5644 --- [nio-8082-exec-3] o.o.o.c.common.logging.DefaultLogWriter : storage.app: Storage publishes message 68a989a1-f1a2-4bbc-9f1c-5c734a790311 {correlation-id=68a989a1-f1a2-4bbc-9f1c-5c734a790311, data-partition-id=opendes}
2020-06-29 19:34:22.670 INFO 5644 --- [nio-8082-exec-3] o.o.o.c.common.logging.DefaultLogWriter : storage.app: End Web-API DELETE /records/opendes:id:1593439432354 Headers: {correlation-id:68a989a1-f1a2-4bbc-9f1c-5c734a790311} timeTaken:2887 {correlation-id=68a989a1-f1a2-4bbc-9f1c-5c734a790311, data-partition-id=opendes}
```
If we notice above the loggerName for all the logs is `o.o.o.c.common.logging.DefaultLogWriter`. This is fully qualified name of the class in which logger instance is created by giving the class name as input.
Logging libraries like java.utils.logging, log4j2 etc., provides a way to configure log level based on logger name. As these logger names most of the times are full qualified names, this allows different configuration at package level or class level. Refer sample log4j2.xml below
```
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<appender name="console" class="org.apache.log4j.ConsoleAppender">
<param name="Target" value="System.out"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%-5p %c{1} - %m%n"/>
</layout>
</appender>
<logger name="com.example.mypackage" additivity="false">
<level value="info"/>
<appender-ref ref="console"/>
</logger>
<root>
<priority value="warn"/>
<appender-ref ref="console"/>
</root>
</log4j:configuration>
```
From the above configuration, for package `com.example.mypacage` we are logging any thing info and above where as for root we are logging warn and above.
Having loggerName will give fine grained control of log messages at different level like package or classes and different environments like dev, stage, prod.
## Decision
To support loggerName overloaded methods will be added to JaxRsDpsLogger.java and ILogger.java. ILogger.java new methods will have default implementation which will call into existing methods leaving the loggerName. This is to keep backward compatiability.
Cloud providers who what to use this additional feature will implement their own class for ILogger to support new loggerName.
## Rationale
This proposal will not affect the existing implementations and cloud providers can pick up these changes at their own pace if neededhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/49ADR : Optional Audit Logs per partition2022-05-20T16:46:09ZPreksha Beohar-SlbADR : Optional Audit Logs per partition## Status
- [X] Proposed
- [X] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
As an operator of OSDU we are seeing increased levels of costs related to logging over time. For instance in a single day we saw over 500GB of...## Status
- [X] Proposed
- [X] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
As an operator of OSDU we are seeing increased levels of costs related to logging over time. For instance in a single day we saw over 500GB of logs created per environment. This can run to costs of $X000 a day.
A large percentage of these logs can be equated to audit logs, especially for READ actions like list groups in Entitlements which can be called multiple millions of times a day.
We have deployments and partitions of varying degrees of sensitivity. For example we have deployments strictly for development purposes but even in our production systems some partitions are used solely for development by their clients.
As a client of the system I would like to be able to control which partitions have audit logs enabled and if so whether I have this turned on for write operations only i.e. create,update,delete actions or all events in the system.
This gives me the choice of whether the cost benefit of running audit logs in a particular partition is worthwhile and can give the client the control based on the sensitivity of the partition.
This has precedent in other systems e.g. in Google Cloud or Azure I have the choice for most resources I deploy whether to turn on and retain audit logs for these services even though they _could_ have sensitive information as it gives me as the client the control and choice.
## Trade-off Analysis
I could choose to have all audit logs turned on at all times. This is the current state. However this has potentially high cost implications. If the partition or deployment of OSDU does not contain sensitive information then I would like the choice to turn off audit logs.
I could make the option to turn on/off logs as part of the build configuration. However this means I need to redeploy if I want to switch logs on or off. This also prevents end users potentially ever having this control.
I could make the configuration for the entire environment rather than per partition. However a strong use case for partitions is to separate them based on data sensitivity e.g. I may have a partition for development purposes and another partition in the same deployment with production data.
## Decision
We are proposing the ability to have audit logs turned on or off by choice of the operator and/or client.
An implementation of this could be via a feature flag from the partition service which will allow this to be set at runtime and per partition.
We would like 2 separate feature flags. One which can turn the READ action audit logs on or off and another to turn the CREATE/UPDATE/DELETE/PUBLISH/JOB_RUN action audit logs on or off.
By default both of these should be turned on so it is 'secure' by default. However the operator and/or end user can then choose to turn certain ones off.Preksha Beohar-SlbPreksha Beohar-Slbhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/39ADR: Partion Service as a TenantInfo provider2023-04-07T12:43:48ZRostislav Dublin (EPAM)ADR: Partion Service as a TenantInfo provider## Status
- [x] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context
Currently, CSPs have their own implementation of retrieving the TenantInfo data:
1. AWS. https://community.opengroup.org/osdu/platform/sy...## Status
- [x] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context
Currently, CSPs have their own implementation of retrieving the TenantInfo data:
1. AWS. https://community.opengroup.org/osdu/platform/system/lib/cloud/aws/os-core-lib-aws/-/blob/master/src/main/java/org/opengroup/osdu/core/aws/multitenancy/TenantFactory.java
2. Azure. https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/blob/master/src/main/java/org/opengroup/osdu/azure/multitenancy/TenantFactoryImpl.java
3. IBM. https://community.opengroup.org/osdu/platform/system/lib/cloud/ibm/os-core-lib-ibm/-/blob/master/src/main/java/org/opengroup/osdu/core/ibm/multitenancy/TenantFactory.java
All implementations follow a similar way to retrieve the TenantInfo from the Partition service (request partition service->map to TenantInfo class; calls to the Partition service are cached).
## Scope
Implement a common approach to retrieve the TenantInfo data and use it across all services.
## Decision
Unify an approach of retrieving TenantInfo and using it across all services:
- Implement a common tenant factory (ITenantFactory) on the core-lib level.
- Remove CSP-specific implementations of the TenantFactory.
- Some CSPs will require to restructure a code and implement missing interfaces (in fact, the code is already there):
- IBM should implement IServiceAccountJwtClient explicitly. There is already a similar method in the TenantFactory implementation.
### Outcome of the Decision (added by @rostislav.dublin on May 24):
1. A single common implementation of PartitionTenantInfoFactory in os-core-common. It just lies here and does not inject anywhere. [See it](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/73/diffs#eb73934c527564533da4c8574c831649474ee500_0_36) in the [MR 73](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/73/) ;
2. Each CSP injects the above single common implementation in its CSP library, removing its own implementation. Everyone moves at their own pace, no one rushes anyone. E.g. GCP uses AbstractFactoryBean<IPartitionFactory> [as shown here](https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/102/diffs#4ac1210ac980e2a0dcec2574ebd6eb21618e8abc_0_23).
## Rationale
This change unifies an approach of retrieving tenant information, which is used to separate tenants from each other (multitenancy). With the increased maintainability of the code, OSDU will have a unified basic data to implement multitenancy
## Consequences
Simplified CSP-specific code and code generalization (higher maintainability and reusability).https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/78ADR: Strategy for Core-Common Migration to Spring 6 and Jakarta EE2024-03-26T11:39:59ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comADR: Strategy for Core-Common Migration to Spring 6 and Jakarta EE# ADR: Strategy for Core-Common Migration to Spring 6 and Jakarta EE
## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context
Our current challenge revolves around the compatibility of Core-...# ADR: Strategy for Core-Common Migration to Spring 6 and Jakarta EE
## Status
- [ ] Proposed
- [ ] Trialing
- [ ] Under review
- [ ] Approved
- [ ] Retired
## Context
Our current challenge revolves around the compatibility of Core-Common with Spring 6 and Jakarta and the breaking changes caused by the Core-Common upgrade that will affect OSDU services that will keep using Spring 5 and Javax. The same issue is observed with CSP core-libs.
## Problem Statement
If we proceed with upgrading Core-Common to Spring 6 and Jakarta as is, services on Spring 5 won't be able to use the upgraded Core-Common. This also applies to CSP core-libs. Freezing the service with the old Core-Common is not a viable option.
Example of straightforward, not compatible migration: https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/261
## Decision Options
**1. Fork of Core-Common**
Create a new branch or module within the same repository to maintain a second version of Core-Common. Two versions of Core-Common will have to be maintained simultaneously.
- Pros: Less initial development effort is required. Ability to add new breaking changes to Core-Common fork.
- Cons: This approach will require cherry-picking changes, potentially increasing maintenance overhead.
**2. Byte/Source Code Modifications during Build**
Modify code during the build process using plugins.
- Pros: Possibility of a smooth upgrade, no changes would be required for services that are still using Spring 5 and Javax.
- Cons: This workaround may introduce potential build and runtime issues, and its feasibility is uncertain. Requires a lot of initial effort to develop complex build configurations.
**3. Heavy Core-Common Refactoring**
Refactored Core-Common, 2 new modules with extracted affected code, 1 compatible with Spring 5 and Javax, and another for Spring 6 and Jakarta.
- Pros: Cleaner approach, compatibility will be guaranteed, lesser chance of unexpected issues.
- Cons: A lot of effort to refactor. Breaks the current flow, auto upgrades could be blocked, it may be required to switch libraries. Code duplicates are possible, so in further development, we may have to push changes into two modules.
## Decision
TBD
## Consequences
TBDM23 - Release 0.26Chad LeongChad Leonghttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/31Bean with hard-coded properties imposes policy that limits application/servic...2022-11-24T12:34:33ZSherman YangBean with hard-coded properties imposes policy that limits application/service flexibilityIn OS Core Common repo's src/main/java/org/opengroup/osdu/core/common/http/HttpConfiguration.java, a bean is automatically created with hard-coded properties that impose policy which limits service flexibility.
For example, the new ent...In OS Core Common repo's src/main/java/org/opengroup/osdu/core/common/http/HttpConfiguration.java, a bean is automatically created with hard-coded properties that impose policy which limits service flexibility.
For example, the new entitlement v2 service needs to accept case insensitive Json input but with the hard-coded ObjectMapper bean that got introduced since osdu core common version 0.0.17, the entitlement service's case insensitive json configuration is no longer available as the common library creates its own ObjectMapper which disables/overrides the service's settings and imposes case sensitivity.
Need to relax this limitation to allow application flexibility.https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/15Configuration Approach for Existing Microservices2020-10-27T14:41:47ZRostislav Dublin (EPAM)Configuration Approach for Existing Microservices## Change Type:
- [ ] Feature
- [ ] Bugfix
- [x] Refactoring
## Context and Scope
Currently used (legacy) microservices configuration approach is error-pron, not flexible and poorly scalable:
- The @Value annotation is commonly used. It...## Change Type:
- [ ] Feature
- [ ] Bugfix
- [x] Refactoring
## Context and Scope
Currently used (legacy) microservices configuration approach is error-pron, not flexible and poorly scalable:
- The @Value annotation is commonly used. It is redundant, hard to find and change, not type-safe (error-pron);
- Configuration properties (in .property files and @Value) are almost always mapped on capitalized names of environment variables, which reduces the set of possible means of parameter passing and possibility of hierarchic properties definition;
- The unsuitable @Component annotation is commonly used for ostensibly "configuration properties set" classes;
It does not take advantage of the modern configuration methods offered by Spring Boot.
This makes it difficult to transition to a "configuration server/clients" approach usage.
## Decision
### Extend ["Configuration Approach for New Microservices"](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/7) approach on Existing services.
### Existing services' code should be refactored:
- create @ConfigurationProperties annotated beans;
- enable properties beans in modules;
- transform all @Value annotations to @ConfigurationProperties beans fields
- create property field
- set correct datatype;
- compose property field name by properly transforming capitalized @Value's value, eg.: ```SUPER_PROPERY ==> superProperty```
- optionally: set default value;
- find that @Value usage in all classes and change it for:
- inject properties bean into the property consumer class;
- change @Value variable mentions for propertyClass.propertyGetter;
- remove @Value definition code;
- correct properties definitions in application.(properties|yaml) files:
- use the same strategy as for above mentioned "transforming capitalized @Value's value", eg.:
```SUPER_PROPERY ==> super.property```
## Rational
Current configuration approach is obsolete, error-prone, and hard for understanding maintaining.
By establishing a better configuration standard we improve, simplify and speed-up our own development process
and make it easier for all new community contributors to join and become involved faster. Plus, we wish to make our services potentially configurable with MSA "configuration server/client" model.
## Consequences
- Providers should plan refactoring their code as soon as they slots;
- No urgent activity required. Providers shouldn't sync their changes with others;
- When properties re-defined properly, nothing should break in currently defined CI/CD etcDmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/7Configuration Approach for New Microservices2020-10-28T22:24:28ZRostislav Dublin (EPAM)Configuration Approach for New Microservices## Change Type:
- [ ] Feature
- [ ] Bugfix
- [x] Refactoring
## Context and Scope
Currently used (legacy) microservices configuration approach is error-pron, not flexible and poorly scalable:
- The @Value annotation is commonly used. It...## Change Type:
- [ ] Feature
- [ ] Bugfix
- [x] Refactoring
## Context and Scope
Currently used (legacy) microservices configuration approach is error-pron, not flexible and poorly scalable:
- The @Value annotation is commonly used. It is redundant, hard to find and change, not type-safe (error-pron);
- Configuration properties (in .property files and @Value) are almost always mapped on capitalized names of environment variables, which reduces the set of possible means of parameter passing and possibility of hierarchic properties definition;
- The unsuitable @Component annotation is commonly used for ostensibly "configuration properties set" classes;
It does not take advantage of the modern configuration methods offered by Spring Boot.
This makes it difficult to transition to a "configuration server/clients" approach usage.
## Decision
### 1. Stop using obsolete/legacy approach:
- @Value annotations;
- @Component annotation for "configuration properties set" classes;
- Stop mapping application properties on environment variables names;
### 2. Start using recommended Spring Boot configuration/externalization approach:
Start following [Spring Boot 2 Externalized Configuration guide](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-external-config) tightly;
- [Type-safe @ConfigurationProperties annotated configuration properties beans](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-external-config-typesafe-configuration-properties) should be used:
- each service modules (root, core and providers') introducing at least one configuration property SHOULD have at least one such class in ```org.opengroup.osdu.<service>.<module>.config``` package, eg. ```org.opengroup.osdu.backup.provider.gcp.BackupGcpProperties```.
- More when one can be used where convenient;
- to simplify properties beans definition, Lombok @Getter and @Setter annotations should be used;
- properties beans should be [enabled with @ConfigurationPropertiesScan](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-external-config-enabling)
- properties beans should be [injected into interested classes (using constructor injection)](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-external-config-using)
- certain properties should be consumed by getters, eg. ```propertiesBeanVariable.getSomeProperty()```
## Rational
Current configuration approach is obsolete, error-prone, and hard for understanding maintaining.
By establishing a better configuration standard we improve, simplify and speed-up our own development process
and make it easier for all new community contributors to join and become involved faster. Plus, we wish to make our services potentially configurable with MSA "configuration server/client" model.
## Consequences
- Providers should plan refactoring their code as soon as they slots;
- No urgent activity required. Providers shouldn't sync their changes with others;
- When properties re-defined properly, nothing should break in currently defined CI/CD etcDmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudko2020-10-27https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/68Core-common post migration TODOs (Powermock, module access)2023-06-28T14:57:27ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comCore-common post migration TODOs (Powermock, module access)- Unit test refactoring is required, to remove --add-opens params from build run args.
~~~
<argLine>--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED</argLine>
~~~
Tests failed without args:
~~~
Fail...- Unit test refactoring is required, to remove --add-opens params from build run args.
~~~
<argLine>--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED</argLine>
~~~
Tests failed without args:
~~~
Failed tests:
shouldThrowExceptionWhenPropertyNotPresentInEnv(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
shouldThrowExceptionWhenPropertyInMapNull(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
shouldThrowExceptionWhenPropertyValueInMapNull(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
shouldThrowExceptionWhenPropertyNotInMap(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
Tests in error:
shouldNotThrowExceptionWhenPropertyValueInMapIsEmpty(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
shouldNotThrowExceptionWhenEnvReturnEmptyVal(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
shouldGetTypedProperties(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
getPropertyValueFromEnv(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
shouldThrowExceptionWhenEnvReturnNull(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
getPropertyValueFromMapValue(org.opengroup.osdu.core.common.partition.PartitionPropertyResolverTest)
~~~
- Powermock dependency removal is highly recommended, Powermock not getting updates to work with Java 17, and has conflicts with Mockito:
~~~
<dependency>
<groupId>org.powermock</groupId>
<artifactId>powermock-core</artifactId>
<version>2.0.9</version>
<scope>test</scope>
</dependency>
~~~https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/23Create LegalTag Expiration date is not human readable.2021-01-23T20:12:58ZGregCreate LegalTag Expiration date is not human readable.POST – Create LegalTag. Functionality working but defect. The attribute of "expirationDate" in payload is “2222222222222”, which is not human-readable. Need to convert epoch time to human-readable date. The schema of “expirationDate” of ...POST – Create LegalTag. Functionality working but defect. The attribute of "expirationDate" in payload is “2222222222222”, which is not human-readable. Need to convert epoch time to human-readable date. The schema of “expirationDate” of the legal tag is in the format of “yyyy-mm-dd”, e.g.,"2040-06-02".M1 - Release 0.1Chris ZhangChris Zhanghttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/30Cursor search returns results even when version does not match.2022-12-08T17:13:50ZGregCursor search returns results even when version does not match.POST – Query With Cursor. Functionality working but defect.
For example, below request is to search "opendes:osdu:wellbore-master:0.2.0" with the cursor "2F8900678904A680D24593BC7D8BEEA5".
However, it is still returning the results wit...POST – Query With Cursor. Functionality working but defect.
For example, below request is to search "opendes:osdu:wellbore-master:0.2.0" with the cursor "2F8900678904A680D24593BC7D8BEEA5".
However, it is still returning the results with 0.2.0 even I change the version number of data, e.g. "opendes:osdu:wellbore-master:0.0.8"
{ "kind": "opendes:osdu:wellbore-master:0.2.0", "cursor": "2F8900678904A680D24593BC7D8BEEA5", "aggregateBy": "kind" }
It seems that the kind should not be required once you have the Cursor ID. Or, if it is required, the kind should need to be the same as the original query.M1 - Release 0.1Chris ZhangChris Zhanghttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/25Delete LegalTag response message is not correct.2022-09-27T13:50:36ZGregDelete LegalTag response message is not correct.DEL – Delete LegalTag. Functionality working but needs improvement. Need to improve the response as it is returning 204 No Content without any message regardless of whether the legal tag exists or not. If the Legal Tag does not exist, it...DEL – Delete LegalTag. Functionality working but needs improvement. Need to improve the response as it is returning 204 No Content without any message regardless of whether the legal tag exists or not. If the Legal Tag does not exist, it is expected to return 404 Not Found with the message like “Cannot delete a LegalTag that does not exist for given name xxx”M1 - Release 0.1Chris ZhangChris Zhanghttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/issues/7EG management SDK has to be moved to GA2022-07-11T19:41:13ZKomal MakkarEG management SDK has to be moved to GAEG management is using preview [SDK](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/blob/master/pom.xml#L55), which can be risky.
Move to the GA version of it.EG management is using preview [SDK](https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/blob/master/pom.xml#L55), which can be risky.
Move to the GA version of it.harshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/75Empty partition id causing 500 internal server error in partition feature fla...2023-10-26T10:37:15ZAbhishek PatilEmpty partition id causing 500 internal server error in partition feature flag implementationIf `featureFlag.strategy=dataPartition` and a user makes a API call with empty data-partition-id then a 500 Internal Server Error is returned. Ideally a 4xx error should be returned in such cases where user provide incorrect input.
http...If `featureFlag.strategy=dataPartition` and a user makes a API call with empty data-partition-id then a 500 Internal Server Error is returned. Ideally a 4xx error should be returned in such cases where user provide incorrect input.
https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/blob/master/src/main/java/org/opengroup/osdu/core/common/feature/PartitionFeatureFlagImpl.java#L59
As per the above code, 500 internal server error is thrown because `partitionProvider.get(headers.getPartitionId())` will throw a PartitionException in case of empty data-partition-id.https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/72Error getting data from Redis on lost connection2023-09-18T07:26:13ZYurii Ruban [EPAM / GCP]Error getting data from Redis on lost connectionIn case of loss of connection between the service and Redis, the processing of requests to the service is delayed and an error of type
`Command timed out after 30 SECONDS` is received.
Redis (cache service) is a productivity tool. In c...In case of loss of connection between the service and Redis, the processing of requests to the service is delayed and an error of type
`Command timed out after 30 SECONDS` is received.
Redis (cache service) is a productivity tool. In case of a lost connection, delays in processing requests increase, and, in general, the service becomes unviable.
`entitlements-v2.app: An unknown error has occurred
AppException(error=AppError(code=500, reason=Internal Server Error, message=An unknown error has occurred, errors=null, debuggingInfo=null, originalException=com.lambdaworks.redis.RedisCommandTimeoutException: Command timed out after 30 SECONDS), originalException=com.lambdaworks.redis.RedisCommandTimeoutException: Command timed out after 30 SECONDS)
...
Caused by: com.lambdaworks.redis.RedisCommandTimeoutException: Command timed out after 30 SECONDS
`
I recommend adding Redis health checks.M20 - Release 0.23Yurii Ruban [EPAM / GCP]Yurii Ruban [EPAM / GCP]https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/10Expose audit trail attributes in search results2021-10-04T22:12:09ZDebasis ChatterjeeExpose audit trail attributes in search resultsThis is very useful information and may be added to standard query result.
Username who created the record.
Date/time when record was created in OSDU.
Also expose Username who updated the record.
Date/time when update operation was per...This is very useful information and may be added to standard query result.
Username who created the record.
Date/time when record was created in OSDU.
Also expose Username who updated the record.
Date/time when update operation was performed.
@dmitry-kniazev later added the example from IBM implementation as he was showing during recent session of "Reporting and dashboarding"?JoeJoehttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/issues/8Fix security vulnerabilities in old dependencies2022-07-11T19:47:00ZRostislav Vatolinvatolinrp@gmail.comFix security vulnerabilities in old dependenciesFix security vulnerabilities in old dependencies:
1) removing netty-bom, because netty-bom is taken from os-core-common
2) removing jackson-bom, because Jackson-bom is taken from os-core-common
3) remove javax.inject, because it is take...Fix security vulnerabilities in old dependencies:
1) removing netty-bom, because netty-bom is taken from os-core-common
2) removing jackson-bom, because Jackson-bom is taken from os-core-common
3) remove javax.inject, because it is taken from os-core-common
4) replace azure-spring-boot-metrics-starter with micrometer-registry-azure-monitor to avoid bringing extra problem dependencies
5) add explicitly version of json-smart to avoid having old version with security vulnerabilities
6) remove the unused version of azure-core-http-netty
7) remove unused dependencies msal4j, azure-core, reactor-bom from dependencyManagement section
The library is proven in projects legal and partition:
https://community.opengroup.org/osdu/platform/security-and-compliance/legal/-/merge_requests/111
https://community.opengroup.org/osdu/platform/system/partition/-/merge_requests/51https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/29Get Group Members query parameters not working properly.2022-12-08T17:14:06ZGregGet Group Members query parameters not working properly.GET – Get Group Members. Functionality working but defects. Query parameters “limit” and “role” not working properly If query params optional,e.g., it should return all the members regardless of the role for the request URL {{osduonaws_b...GET – Get Group Members. Functionality working but defects. Query parameters “limit” and “role” not working properly If query params optional,e.g., it should return all the members regardless of the role for the request URL {{osduonaws_base_url}}/api/entitlements/v1/groups/data.testing.osduonaws-testing@opendes.shell.com/members?limit=100. Currently it only return the user with the role of OWNER If query params required, e.g., it should return error handling with the message like “required parameter xxx missing”. There is defect with either scenario. And it would make more sense to keep query params optional in this case.M1 - Release 0.1JoeRucha DeshpandeMatt WiseJoehttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/issues/28Improve performance on CosmosStore pageable query method2022-09-28T15:48:40ZRafael FreireImprove performance on CosmosStore pageable query methodCurrently, CosmosStore has a queryItemsPage which interact with a BlockingIterable object twice per call. This object uses a Thread-sleep blocking way to iterate through page results causing a slow performance for heavy queries. Its impl...Currently, CosmosStore has a queryItemsPage which interact with a BlockingIterable object twice per call. This object uses a Thread-sleep blocking way to iterate through page results causing a slow performance for heavy queries. Its implementation can be improved by substituting the check then use style for a for-each approachRafael FreireRafael Freire