OS Core Common - Spring 5 issueshttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues2020-10-27T14:41:47Zhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/15Configuration Approach for Existing Microservices2020-10-27T14:41:47ZRostislav Dublin (EPAM)Configuration Approach for Existing Microservices## Change Type:
- [ ] Feature
- [ ] Bugfix
- [x] Refactoring
## Context and Scope
Currently used (legacy) microservices configuration approach is error-pron, not flexible and poorly scalable:
- The @Value annotation is commonly used. It...## Change Type:
- [ ] Feature
- [ ] Bugfix
- [x] Refactoring
## Context and Scope
Currently used (legacy) microservices configuration approach is error-pron, not flexible and poorly scalable:
- The @Value annotation is commonly used. It is redundant, hard to find and change, not type-safe (error-pron);
- Configuration properties (in .property files and @Value) are almost always mapped on capitalized names of environment variables, which reduces the set of possible means of parameter passing and possibility of hierarchic properties definition;
- The unsuitable @Component annotation is commonly used for ostensibly "configuration properties set" classes;
It does not take advantage of the modern configuration methods offered by Spring Boot.
This makes it difficult to transition to a "configuration server/clients" approach usage.
## Decision
### Extend ["Configuration Approach for New Microservices"](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/7) approach on Existing services.
### Existing services' code should be refactored:
- create @ConfigurationProperties annotated beans;
- enable properties beans in modules;
- transform all @Value annotations to @ConfigurationProperties beans fields
- create property field
- set correct datatype;
- compose property field name by properly transforming capitalized @Value's value, eg.: ```SUPER_PROPERY ==> superProperty```
- optionally: set default value;
- find that @Value usage in all classes and change it for:
- inject properties bean into the property consumer class;
- change @Value variable mentions for propertyClass.propertyGetter;
- remove @Value definition code;
- correct properties definitions in application.(properties|yaml) files:
- use the same strategy as for above mentioned "transforming capitalized @Value's value", eg.:
```SUPER_PROPERY ==> super.property```
## Rational
Current configuration approach is obsolete, error-prone, and hard for understanding maintaining.
By establishing a better configuration standard we improve, simplify and speed-up our own development process
and make it easier for all new community contributors to join and become involved faster. Plus, we wish to make our services potentially configurable with MSA "configuration server/client" model.
## Consequences
- Providers should plan refactoring their code as soon as they slots;
- No urgent activity required. Providers shouldn't sync their changes with others;
- When properties re-defined properly, nothing should break in currently defined CI/CD etcDmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/10Expose audit trail attributes in search results2021-10-04T22:12:09ZDebasis ChatterjeeExpose audit trail attributes in search resultsThis is very useful information and may be added to standard query result.
Username who created the record.
Date/time when record was created in OSDU.
Also expose Username who updated the record.
Date/time when update operation was per...This is very useful information and may be added to standard query result.
Username who created the record.
Date/time when record was created in OSDU.
Also expose Username who updated the record.
Date/time when update operation was performed.
@dmitry-kniazev later added the example from IBM implementation as he was showing during recent session of "Reporting and dashboarding"?JoeJoehttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/8WIP: Semantic release numbering (GONRG-819)2023-07-05T10:09:41ZRostislav Dublin (EPAM)WIP: Semantic release numbering (GONRG-819)## Change Type:
- [x] Feature
- [ ] Bugfix
- [ ] Refactoring
## Context and Scope
All OSDU microservices projects developed by the community. Those microservices are in the active development by different teams simultaneously, and when...## Change Type:
- [x] Feature
- [ ] Bugfix
- [ ] Refactoring
## Context and Scope
All OSDU microservices projects developed by the community. Those microservices are in the active development by different teams simultaneously, and when being merged to the community they should be versioned unambiguously and consistently. This is hard to achieve without some central point of registration and computation, and such stuff is not implemented yet.
## Decision
Improve release management for all microservices projects by implementation of proposed technical solution and an agreement of proper usage of it in OSDU context.
## Rational
We just performed an POC investigation and elaborated a suitable technical solution (scripts & Gitlab CICD template) which can be rather easily applied on the community pipelines.
## Consequences
The solution will be implemented to manage release numbering:
- based on principles of [Semantic Versioning 2.0.0](https://semver.org/),
- using technical implementation (script/pipeline template) developed as part of POC by EPAM,
- with involving repositories tags and CICD variables to the action
- by adding several specific steps to all community pipelines
- and sticking special labels (bump-major and bump-minor) to merge requests,
- and causing resulting pipelines' artifacts' names to be composed with produced release numbers inclusions
All CSPs will need to:
- follow the agreed strategy of version numbering (know, when to bump major, minor and patch parts of it);
- refactor pom.xml files (use <version>${revision}</version> substitution instead of constant version number;
- include versioning solution template to pipelines;
- instruct maintainers to set bump-major and bump-minor labels on MRsDmitriy RudkoDmitriy Rudko2020-10-27https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/7Configuration Approach for New Microservices2020-10-28T22:24:28ZRostislav Dublin (EPAM)Configuration Approach for New Microservices## Change Type:
- [ ] Feature
- [ ] Bugfix
- [x] Refactoring
## Context and Scope
Currently used (legacy) microservices configuration approach is error-pron, not flexible and poorly scalable:
- The @Value annotation is commonly used. It...## Change Type:
- [ ] Feature
- [ ] Bugfix
- [x] Refactoring
## Context and Scope
Currently used (legacy) microservices configuration approach is error-pron, not flexible and poorly scalable:
- The @Value annotation is commonly used. It is redundant, hard to find and change, not type-safe (error-pron);
- Configuration properties (in .property files and @Value) are almost always mapped on capitalized names of environment variables, which reduces the set of possible means of parameter passing and possibility of hierarchic properties definition;
- The unsuitable @Component annotation is commonly used for ostensibly "configuration properties set" classes;
It does not take advantage of the modern configuration methods offered by Spring Boot.
This makes it difficult to transition to a "configuration server/clients" approach usage.
## Decision
### 1. Stop using obsolete/legacy approach:
- @Value annotations;
- @Component annotation for "configuration properties set" classes;
- Stop mapping application properties on environment variables names;
### 2. Start using recommended Spring Boot configuration/externalization approach:
Start following [Spring Boot 2 Externalized Configuration guide](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-external-config) tightly;
- [Type-safe @ConfigurationProperties annotated configuration properties beans](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-external-config-typesafe-configuration-properties) should be used:
- each service modules (root, core and providers') introducing at least one configuration property SHOULD have at least one such class in ```org.opengroup.osdu.<service>.<module>.config``` package, eg. ```org.opengroup.osdu.backup.provider.gcp.BackupGcpProperties```.
- More when one can be used where convenient;
- to simplify properties beans definition, Lombok @Getter and @Setter annotations should be used;
- properties beans should be [enabled with @ConfigurationPropertiesScan](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-external-config-enabling)
- properties beans should be [injected into interested classes (using constructor injection)](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-external-config-using)
- certain properties should be consumed by getters, eg. ```propertiesBeanVariable.getSomeProperty()```
## Rational
Current configuration approach is obsolete, error-prone, and hard for understanding maintaining.
By establishing a better configuration standard we improve, simplify and speed-up our own development process
and make it easier for all new community contributors to join and become involved faster. Plus, we wish to make our services potentially configurable with MSA "configuration server/client" model.
## Consequences
- Providers should plan refactoring their code as soon as they slots;
- No urgent activity required. Providers shouldn't sync their changes with others;
- When properties re-defined properly, nothing should break in currently defined CI/CD etcDmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudko2020-10-27https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/5search.Config class in os-core-common does not respect Spring configuration2021-01-17T16:29:09ZDmitriy Rudkosearch.Config class in os-core-common does not respect Spring configuration## Change Type:
- [ ] Feature
- [x] Bugfix
- [x] Refactoring
## Context and Scope
Implementation of `org.opengroup.osdu.core.common.search.Config` class make it hard to use class in modules and introduce several implicit issues:
1. Spr...## Change Type:
- [ ] Feature
- [x] Bugfix
- [x] Refactoring
## Context and Scope
Implementation of `org.opengroup.osdu.core.common.search.Config` class make it hard to use class in modules and introduce several implicit issues:
1. Spring does not support ingestion of `@Values` in static fields. The fact that the class is annotated with Spring annotations (`@Component`, `@Value`) misled because class can be used only as a general Java class (`Config.QUERY_DEFAULT_LIMIT`)
2. Because Spring can't populate variables, configuration can be passed ONLY as Env variables.
3. The Class introduce implicit dependency on other Search related Variables in the code even module could not use them.
4. The Class introduce implicit dependency on GCP specific Variables for all CSPs. E.g. code of all CSPs will check presence of `GOOGLE_CLOUD_PROJECT` variable even it is GCP specific.
![Screen_Shot_2020-08-27_at_4.07.50_PM](/uploads/0e4f41a5ae23158ebebabe9803daaaff/Screen_Shot_2020-08-27_at_4.07.50_PM.png)
## Decision
- Move `org.opengroup.osdu.core.common.search.Config` class from os-core-common. Application configuration should be specified on microservice level, not in common library
## Rational
1. Dependency on Spring in common 'library' make it hard to use from hosting code
2. Dependency on `os-core-common` make application implicitly `aware` about all Spring components / variables declared in library
## Consequences
1. New `search.Config` class should be added in microservices that use the class (Search)
2. Changes in OS Core Common > QueryUtils.getResultSizeForQuery()ethiraj krishnamanaiduDania Kodeih (Microsoft)Wladmir FrazaoAlan Brazethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/4Making UrlFetchServiceImpl as singleton spring bean2021-01-17T16:28:54ZKishore BattulaMaking UrlFetchServiceImpl as singleton spring bean# Convert http clients into spring singleton components
## Context and Scope
In current OSDU core commons there are two http clients
- UrlFetchServiceImpl which is implementation of IUrlFetchService. It uses HttpClientHandler which is a ...# Convert http clients into spring singleton components
## Context and Scope
In current OSDU core commons there are two http clients
- UrlFetchServiceImpl which is implementation of IUrlFetchService. It uses HttpClientHandler which is a spring with request scope
- HttpClient which is implementation of IHttpClient
UrlFetchServiceImpl is a spring component with request scope and HttpClient is not a spring component.
## Questions
- Any specific reason for having UrlFetchServiceImpl as request scoped spring bean? Is this because of injection of JaxRsDpsLogger in HttpClientHandler?
- Any specific reason for not having HttpClient as spring bean?
## Proposal
Convert these two http clients to spring bean with singleton as scope.
- Converting the existing UrlFetchServiceImpl and HttpClientHandler to singleton scope
- Converting the HttpClient as singleton scoped spring bean. We still support the constructor way of invoking for backward compatibility.
## Rationale
I am working on logging the request/response for external clients at these clients. For this logging I want to use JaxRsDpsLogger which needs to be autowired.
# ----------------- ADR update based on discussion ---------------
Splitting this ADR into two different issues.
This ADR is about converting UrlFetchServiceImpl as singleton spring bean.ethiraj krishnamanaiduDania Kodeih (Microsoft)Wladmir FrazaoBrandt BealAlan Brazethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/1Adding support for logger name in JaxRsDpsLogger and ILogger for fine grained...2021-02-20T07:50:10ZKishore BattulaAdding support for logger name in JaxRsDpsLogger and ILogger for fine grained configuration of log messages## Context and Scope
Below are the sample logs in storage service
```
2020-06-29 19:34:19.783 INFO 5644 --- [nio-8082-exec-3] o.o.o.c.common.logging.DefaultLogWriter : storage.app: Start Web-API DELETE /records/opendes:id:1593439432354...## Context and Scope
Below are the sample logs in storage service
```
2020-06-29 19:34:19.783 INFO 5644 --- [nio-8082-exec-3] o.o.o.c.common.logging.DefaultLogWriter : storage.app: Start Web-API DELETE /records/opendes:id:1593439432354 Headers: {data-partition-id:opendes,content-type:application/json} {correlation-id=68a989a1-f1a2-4bbc-9f1c-5c734a790311, data-partition-id=opendes}
2020-06-29 19:34:22.342 INFO 5644 --- [nio-8082-exec-3] o.o.o.c.common.logging.DefaultLogWriter : {auditLog={resources=[opendes:id:1593439432354], action=DELETE, actionId=ST003, message=Record purged, user=e6ac6dfe-8198-4f9c-b2e8-b974c0f9ef5b, status=SUCCESS}} {correlation-id=68a989a1-f1a2-4bbc-9f1c-5c734a790311, data-partition-id=opendes}
2020-06-29 19:34:22.344 INFO 5644 --- [nio-8082-exec-3] o.o.o.c.common.logging.DefaultLogWriter : storage.app: Storage publishes message 68a989a1-f1a2-4bbc-9f1c-5c734a790311 {correlation-id=68a989a1-f1a2-4bbc-9f1c-5c734a790311, data-partition-id=opendes}
2020-06-29 19:34:22.670 INFO 5644 --- [nio-8082-exec-3] o.o.o.c.common.logging.DefaultLogWriter : storage.app: End Web-API DELETE /records/opendes:id:1593439432354 Headers: {correlation-id:68a989a1-f1a2-4bbc-9f1c-5c734a790311} timeTaken:2887 {correlation-id=68a989a1-f1a2-4bbc-9f1c-5c734a790311, data-partition-id=opendes}
```
If we notice above the loggerName for all the logs is `o.o.o.c.common.logging.DefaultLogWriter`. This is fully qualified name of the class in which logger instance is created by giving the class name as input.
Logging libraries like java.utils.logging, log4j2 etc., provides a way to configure log level based on logger name. As these logger names most of the times are full qualified names, this allows different configuration at package level or class level. Refer sample log4j2.xml below
```
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<appender name="console" class="org.apache.log4j.ConsoleAppender">
<param name="Target" value="System.out"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%-5p %c{1} - %m%n"/>
</layout>
</appender>
<logger name="com.example.mypackage" additivity="false">
<level value="info"/>
<appender-ref ref="console"/>
</logger>
<root>
<priority value="warn"/>
<appender-ref ref="console"/>
</root>
</log4j:configuration>
```
From the above configuration, for package `com.example.mypacage` we are logging any thing info and above where as for root we are logging warn and above.
Having loggerName will give fine grained control of log messages at different level like package or classes and different environments like dev, stage, prod.
## Decision
To support loggerName overloaded methods will be added to JaxRsDpsLogger.java and ILogger.java. ILogger.java new methods will have default implementation which will call into existing methods leaving the loggerName. This is to keep backward compatiability.
Cloud providers who what to use this additional feature will implement their own class for ILogger to support new loggerName.
## Rationale
This proposal will not affect the existing implementations and cloud providers can pick up these changes at their own pace if needed