OS Core Common - Spring 5 issueshttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues2021-02-20T07:50:10Zhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/1Adding support for logger name in JaxRsDpsLogger and ILogger for fine grained...2021-02-20T07:50:10ZKishore BattulaAdding support for logger name in JaxRsDpsLogger and ILogger for fine grained configuration of log messages## Context and Scope
Below are the sample logs in storage service
```
2020-06-29 19:34:19.783 INFO 5644 --- [nio-8082-exec-3] o.o.o.c.common.logging.DefaultLogWriter : storage.app: Start Web-API DELETE /records/opendes:id:1593439432354...## Context and Scope
Below are the sample logs in storage service
```
2020-06-29 19:34:19.783 INFO 5644 --- [nio-8082-exec-3] o.o.o.c.common.logging.DefaultLogWriter : storage.app: Start Web-API DELETE /records/opendes:id:1593439432354 Headers: {data-partition-id:opendes,content-type:application/json} {correlation-id=68a989a1-f1a2-4bbc-9f1c-5c734a790311, data-partition-id=opendes}
2020-06-29 19:34:22.342 INFO 5644 --- [nio-8082-exec-3] o.o.o.c.common.logging.DefaultLogWriter : {auditLog={resources=[opendes:id:1593439432354], action=DELETE, actionId=ST003, message=Record purged, user=e6ac6dfe-8198-4f9c-b2e8-b974c0f9ef5b, status=SUCCESS}} {correlation-id=68a989a1-f1a2-4bbc-9f1c-5c734a790311, data-partition-id=opendes}
2020-06-29 19:34:22.344 INFO 5644 --- [nio-8082-exec-3] o.o.o.c.common.logging.DefaultLogWriter : storage.app: Storage publishes message 68a989a1-f1a2-4bbc-9f1c-5c734a790311 {correlation-id=68a989a1-f1a2-4bbc-9f1c-5c734a790311, data-partition-id=opendes}
2020-06-29 19:34:22.670 INFO 5644 --- [nio-8082-exec-3] o.o.o.c.common.logging.DefaultLogWriter : storage.app: End Web-API DELETE /records/opendes:id:1593439432354 Headers: {correlation-id:68a989a1-f1a2-4bbc-9f1c-5c734a790311} timeTaken:2887 {correlation-id=68a989a1-f1a2-4bbc-9f1c-5c734a790311, data-partition-id=opendes}
```
If we notice above the loggerName for all the logs is `o.o.o.c.common.logging.DefaultLogWriter`. This is fully qualified name of the class in which logger instance is created by giving the class name as input.
Logging libraries like java.utils.logging, log4j2 etc., provides a way to configure log level based on logger name. As these logger names most of the times are full qualified names, this allows different configuration at package level or class level. Refer sample log4j2.xml below
```
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<appender name="console" class="org.apache.log4j.ConsoleAppender">
<param name="Target" value="System.out"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%-5p %c{1} - %m%n"/>
</layout>
</appender>
<logger name="com.example.mypackage" additivity="false">
<level value="info"/>
<appender-ref ref="console"/>
</logger>
<root>
<priority value="warn"/>
<appender-ref ref="console"/>
</root>
</log4j:configuration>
```
From the above configuration, for package `com.example.mypacage` we are logging any thing info and above where as for root we are logging warn and above.
Having loggerName will give fine grained control of log messages at different level like package or classes and different environments like dev, stage, prod.
## Decision
To support loggerName overloaded methods will be added to JaxRsDpsLogger.java and ILogger.java. ILogger.java new methods will have default implementation which will call into existing methods leaving the loggerName. This is to keep backward compatiability.
Cloud providers who what to use this additional feature will implement their own class for ILogger to support new loggerName.
## Rationale
This proposal will not affect the existing implementations and cloud providers can pick up these changes at their own pace if neededhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/4Making UrlFetchServiceImpl as singleton spring bean2021-01-17T16:28:54ZKishore BattulaMaking UrlFetchServiceImpl as singleton spring bean# Convert http clients into spring singleton components
## Context and Scope
In current OSDU core commons there are two http clients
- UrlFetchServiceImpl which is implementation of IUrlFetchService. It uses HttpClientHandler which is a ...# Convert http clients into spring singleton components
## Context and Scope
In current OSDU core commons there are two http clients
- UrlFetchServiceImpl which is implementation of IUrlFetchService. It uses HttpClientHandler which is a spring with request scope
- HttpClient which is implementation of IHttpClient
UrlFetchServiceImpl is a spring component with request scope and HttpClient is not a spring component.
## Questions
- Any specific reason for having UrlFetchServiceImpl as request scoped spring bean? Is this because of injection of JaxRsDpsLogger in HttpClientHandler?
- Any specific reason for not having HttpClient as spring bean?
## Proposal
Convert these two http clients to spring bean with singleton as scope.
- Converting the existing UrlFetchServiceImpl and HttpClientHandler to singleton scope
- Converting the HttpClient as singleton scoped spring bean. We still support the constructor way of invoking for backward compatibility.
## Rationale
I am working on logging the request/response for external clients at these clients. For this logging I want to use JaxRsDpsLogger which needs to be autowired.
# ----------------- ADR update based on discussion ---------------
Splitting this ADR into two different issues.
This ADR is about converting UrlFetchServiceImpl as singleton spring bean.ethiraj krishnamanaiduDania Kodeih (Microsoft)Wladmir FrazaoBrandt BealAlan Brazethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/5search.Config class in os-core-common does not respect Spring configuration2021-01-17T16:29:09ZDmitriy Rudkosearch.Config class in os-core-common does not respect Spring configuration## Change Type:
- [ ] Feature
- [x] Bugfix
- [x] Refactoring
## Context and Scope
Implementation of `org.opengroup.osdu.core.common.search.Config` class make it hard to use class in modules and introduce several implicit issues:
1. Spr...## Change Type:
- [ ] Feature
- [x] Bugfix
- [x] Refactoring
## Context and Scope
Implementation of `org.opengroup.osdu.core.common.search.Config` class make it hard to use class in modules and introduce several implicit issues:
1. Spring does not support ingestion of `@Values` in static fields. The fact that the class is annotated with Spring annotations (`@Component`, `@Value`) misled because class can be used only as a general Java class (`Config.QUERY_DEFAULT_LIMIT`)
2. Because Spring can't populate variables, configuration can be passed ONLY as Env variables.
3. The Class introduce implicit dependency on other Search related Variables in the code even module could not use them.
4. The Class introduce implicit dependency on GCP specific Variables for all CSPs. E.g. code of all CSPs will check presence of `GOOGLE_CLOUD_PROJECT` variable even it is GCP specific.
![Screen_Shot_2020-08-27_at_4.07.50_PM](/uploads/0e4f41a5ae23158ebebabe9803daaaff/Screen_Shot_2020-08-27_at_4.07.50_PM.png)
## Decision
- Move `org.opengroup.osdu.core.common.search.Config` class from os-core-common. Application configuration should be specified on microservice level, not in common library
## Rational
1. Dependency on Spring in common 'library' make it hard to use from hosting code
2. Dependency on `os-core-common` make application implicitly `aware` about all Spring components / variables declared in library
## Consequences
1. New `search.Config` class should be added in microservices that use the class (Search)
2. Changes in OS Core Common > QueryUtils.getResultSizeForQuery()ethiraj krishnamanaiduDania Kodeih (Microsoft)Wladmir FrazaoAlan Brazethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/7Configuration Approach for New Microservices2020-10-28T22:24:28ZRostislav Dublin (EPAM)Configuration Approach for New Microservices## Change Type:
- [ ] Feature
- [ ] Bugfix
- [x] Refactoring
## Context and Scope
Currently used (legacy) microservices configuration approach is error-pron, not flexible and poorly scalable:
- The @Value annotation is commonly used. It...## Change Type:
- [ ] Feature
- [ ] Bugfix
- [x] Refactoring
## Context and Scope
Currently used (legacy) microservices configuration approach is error-pron, not flexible and poorly scalable:
- The @Value annotation is commonly used. It is redundant, hard to find and change, not type-safe (error-pron);
- Configuration properties (in .property files and @Value) are almost always mapped on capitalized names of environment variables, which reduces the set of possible means of parameter passing and possibility of hierarchic properties definition;
- The unsuitable @Component annotation is commonly used for ostensibly "configuration properties set" classes;
It does not take advantage of the modern configuration methods offered by Spring Boot.
This makes it difficult to transition to a "configuration server/clients" approach usage.
## Decision
### 1. Stop using obsolete/legacy approach:
- @Value annotations;
- @Component annotation for "configuration properties set" classes;
- Stop mapping application properties on environment variables names;
### 2. Start using recommended Spring Boot configuration/externalization approach:
Start following [Spring Boot 2 Externalized Configuration guide](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-external-config) tightly;
- [Type-safe @ConfigurationProperties annotated configuration properties beans](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-external-config-typesafe-configuration-properties) should be used:
- each service modules (root, core and providers') introducing at least one configuration property SHOULD have at least one such class in ```org.opengroup.osdu.<service>.<module>.config``` package, eg. ```org.opengroup.osdu.backup.provider.gcp.BackupGcpProperties```.
- More when one can be used where convenient;
- to simplify properties beans definition, Lombok @Getter and @Setter annotations should be used;
- properties beans should be [enabled with @ConfigurationPropertiesScan](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-external-config-enabling)
- properties beans should be [injected into interested classes (using constructor injection)](https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-features.html#boot-features-external-config-using)
- certain properties should be consumed by getters, eg. ```propertiesBeanVariable.getSomeProperty()```
## Rational
Current configuration approach is obsolete, error-prone, and hard for understanding maintaining.
By establishing a better configuration standard we improve, simplify and speed-up our own development process
and make it easier for all new community contributors to join and become involved faster. Plus, we wish to make our services potentially configurable with MSA "configuration server/client" model.
## Consequences
- Providers should plan refactoring their code as soon as they slots;
- No urgent activity required. Providers shouldn't sync their changes with others;
- When properties re-defined properly, nothing should break in currently defined CI/CD etcDmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudko2020-10-27https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/8WIP: Semantic release numbering (GONRG-819)2023-07-05T10:09:41ZRostislav Dublin (EPAM)WIP: Semantic release numbering (GONRG-819)## Change Type:
- [x] Feature
- [ ] Bugfix
- [ ] Refactoring
## Context and Scope
All OSDU microservices projects developed by the community. Those microservices are in the active development by different teams simultaneously, and when...## Change Type:
- [x] Feature
- [ ] Bugfix
- [ ] Refactoring
## Context and Scope
All OSDU microservices projects developed by the community. Those microservices are in the active development by different teams simultaneously, and when being merged to the community they should be versioned unambiguously and consistently. This is hard to achieve without some central point of registration and computation, and such stuff is not implemented yet.
## Decision
Improve release management for all microservices projects by implementation of proposed technical solution and an agreement of proper usage of it in OSDU context.
## Rational
We just performed an POC investigation and elaborated a suitable technical solution (scripts & Gitlab CICD template) which can be rather easily applied on the community pipelines.
## Consequences
The solution will be implemented to manage release numbering:
- based on principles of [Semantic Versioning 2.0.0](https://semver.org/),
- using technical implementation (script/pipeline template) developed as part of POC by EPAM,
- with involving repositories tags and CICD variables to the action
- by adding several specific steps to all community pipelines
- and sticking special labels (bump-major and bump-minor) to merge requests,
- and causing resulting pipelines' artifacts' names to be composed with produced release numbers inclusions
All CSPs will need to:
- follow the agreed strategy of version numbering (know, when to bump major, minor and patch parts of it);
- refactor pom.xml files (use <version>${revision}</version> substitution instead of constant version number;
- include versioning solution template to pipelines;
- instruct maintainers to set bump-major and bump-minor labels on MRsDmitriy RudkoDmitriy Rudko2020-10-27https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/10Expose audit trail attributes in search results2021-10-04T22:12:09ZDebasis ChatterjeeExpose audit trail attributes in search resultsThis is very useful information and may be added to standard query result.
Username who created the record.
Date/time when record was created in OSDU.
Also expose Username who updated the record.
Date/time when update operation was per...This is very useful information and may be added to standard query result.
Username who created the record.
Date/time when record was created in OSDU.
Also expose Username who updated the record.
Date/time when update operation was performed.
@dmitry-kniazev later added the example from IBM implementation as he was showing during recent session of "Reporting and dashboarding"?JoeJoehttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/15Configuration Approach for Existing Microservices2020-10-27T14:41:47ZRostislav Dublin (EPAM)Configuration Approach for Existing Microservices## Change Type:
- [ ] Feature
- [ ] Bugfix
- [x] Refactoring
## Context and Scope
Currently used (legacy) microservices configuration approach is error-pron, not flexible and poorly scalable:
- The @Value annotation is commonly used. It...## Change Type:
- [ ] Feature
- [ ] Bugfix
- [x] Refactoring
## Context and Scope
Currently used (legacy) microservices configuration approach is error-pron, not flexible and poorly scalable:
- The @Value annotation is commonly used. It is redundant, hard to find and change, not type-safe (error-pron);
- Configuration properties (in .property files and @Value) are almost always mapped on capitalized names of environment variables, which reduces the set of possible means of parameter passing and possibility of hierarchic properties definition;
- The unsuitable @Component annotation is commonly used for ostensibly "configuration properties set" classes;
It does not take advantage of the modern configuration methods offered by Spring Boot.
This makes it difficult to transition to a "configuration server/clients" approach usage.
## Decision
### Extend ["Configuration Approach for New Microservices"](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/7) approach on Existing services.
### Existing services' code should be refactored:
- create @ConfigurationProperties annotated beans;
- enable properties beans in modules;
- transform all @Value annotations to @ConfigurationProperties beans fields
- create property field
- set correct datatype;
- compose property field name by properly transforming capitalized @Value's value, eg.: ```SUPER_PROPERY ==> superProperty```
- optionally: set default value;
- find that @Value usage in all classes and change it for:
- inject properties bean into the property consumer class;
- change @Value variable mentions for propertyClass.propertyGetter;
- remove @Value definition code;
- correct properties definitions in application.(properties|yaml) files:
- use the same strategy as for above mentioned "transforming capitalized @Value's value", eg.:
```SUPER_PROPERY ==> super.property```
## Rational
Current configuration approach is obsolete, error-prone, and hard for understanding maintaining.
By establishing a better configuration standard we improve, simplify and speed-up our own development process
and make it easier for all new community contributors to join and become involved faster. Plus, we wish to make our services potentially configurable with MSA "configuration server/client" model.
## Consequences
- Providers should plan refactoring their code as soon as they slots;
- No urgent activity required. Providers shouldn't sync their changes with others;
- When properties re-defined properly, nothing should break in currently defined CI/CD etcDmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/19OSDU Java stack Standards2023-07-05T10:09:41ZRostislav Dublin (EPAM)OSDU Java stack Standards## Change Type:
- [x] Feature
- [ ] Bugfix
- [ ] Refactoring
## Context and Scope
Standardization has a positive impact for successful software development.
Java coding standards makes sure that all the developers working on the proje...## Change Type:
- [x] Feature
- [ ] Bugfix
- [ ] Refactoring
## Context and Scope
Standardization has a positive impact for successful software development.
Java coding standards makes sure that all the developers working on the project are following certain specified guidelines.
This is especially critical for OSDU community, where many Java teams develop independently and contribute into the common codebase.
The code should be easily understood and proper consistency maintained.
Currently, Java coding standards are not defined in the community. It leads to the following issues:
- Poor code readability
- Difficult onboarding
- Same functionality redundancy
- Libraries and frameworks versioning mess
- Testing frameworks mess
- Security Concerns
- Performance Issues
- etc.
## Decision
Define, adopt and enforce a single Java coding standard for the community. Improve code quality and maintainability.
## Rational
Achieve the following advantages:
- Enhanced Efficiency - detect the problems early or even prevent them completely, increase efficiency throughout the software process.
- Reduced risk of project failure - IT projects fails happen due to problems while developing software.
- Decreased Complexity - coding standards help to develop less complex software and thereby reduce the errors.
- Simplified Maintenance - when code is consistent it can be easily maintained, cause anyone can easily understand and modify it.
- Bug Rectification - it becomes really easy to locate and correct bugs if the source code is written in a consistent manner.
- Comprehensive Look - a clear view of how code fits in the larger application or fits into the company as a whole.
- Cost-Efficiency - a clear code gives an opportunity to reuse it. This reduces the cost along with the efforts put in the development.
## Consequences
The following definition of Java code Standard should be adopted and enforced:
[[OSDU GCP Java Coding Standards]](https://community.opengroup.org/osdu/documentation/-/wikis/Releases/R2.0/GCP/Architecture/OSDU-GCP-Java-Coding-Standards)
All CSPs will need to:
- enforce and follow the adopted Java coding Standard in their Java teams;
- start contributing only compliant new Java code;
- refactoring existing Java code where possible;Dmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/22OSDU REST API Standards2023-07-05T10:09:40ZRostislav Dublin (EPAM)OSDU REST API Standards## Change Type:
- [x] Feature
- [ ] Bugfix
- [ ] Refactoring
## Context and Scope
Standardization has a positive impact for successful software development.
REST API standards makes sure that all the developers working on the project ...## Change Type:
- [x] Feature
- [ ] Bugfix
- [ ] Refactoring
## Context and Scope
Standardization has a positive impact for successful software development.
REST API standards makes sure that all the developers working on the project are following certain specified guidelines.
This is especially critical for OSDU community, where many teams develop independently and produce dozens of common APIs.
The code should be easily understood and proper consistency maintained.
Currently, REST API standards are not defined in the community. It leads to the following issues:
- Poor code readability
- Difficult onboarding
- Same functionality redundancy
- Versioning mess
- Security Concerns
- Performance Issues
- etc.
## Decision
Define, adopt and enforce a single REST API standard for the community. Improve API quality and maintainability.
## Rational
Achieve the following advantages:
- Consistent design across an community’s APIs to ensure the smoothest possible experience for developers using OSDU platform.
- Standardization in design to make APIs feel familiar.
- Minimization the learning curve as developers;
- Design improvement making using APIs easy and intuitive.
## Consequences
The following definition of REST API Standard should be adopted and enforced:
[[OSDU REST API Standards]](https://community.opengroup.org/osdu/documentation/-/wikis/Releases/R2.0/GCP/Architecture/OSDU-GCP-REST-API-Standards)
All CSPs will need to:
- enforce and follow the adopted REST API Standard in their teams;
- start contributing only compliant new REST API;
- refactoring existing REST API where possible;Dmitriy RudkoRostislav Dublin (EPAM)Dmitriy Rudkohttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/23Create LegalTag Expiration date is not human readable.2021-01-23T20:12:58ZGregCreate LegalTag Expiration date is not human readable.POST – Create LegalTag. Functionality working but defect. The attribute of "expirationDate" in payload is “2222222222222”, which is not human-readable. Need to convert epoch time to human-readable date. The schema of “expirationDate” of ...POST – Create LegalTag. Functionality working but defect. The attribute of "expirationDate" in payload is “2222222222222”, which is not human-readable. Need to convert epoch time to human-readable date. The schema of “expirationDate” of the legal tag is in the format of “yyyy-mm-dd”, e.g.,"2040-06-02".M1 - Release 0.1Chris ZhangChris Zhanghttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/24Inconsistent error handling across services.2022-09-27T14:19:25ZGregInconsistent error handling across services.Entitlements Service & Search Service. The response body is in JSON format. The attributes of “code”, “reason”, “message” are returned along with status code, e.g., 401 Unauthorized. Legal Service. The response body is in Text format wit...Entitlements Service & Search Service. The response body is in JSON format. The attributes of “code”, “reason”, “message” are returned along with status code, e.g., 401 Unauthorized. Legal Service. The response body is in Text format with no attributes of “code”, “reason”, “message” returned.M1 - Release 0.1Chris ZhangChris Zhanghttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/25Delete LegalTag response message is not correct.2022-09-27T13:50:36ZGregDelete LegalTag response message is not correct.DEL – Delete LegalTag. Functionality working but needs improvement. Need to improve the response as it is returning 204 No Content without any message regardless of whether the legal tag exists or not. If the Legal Tag does not exist, it...DEL – Delete LegalTag. Functionality working but needs improvement. Need to improve the response as it is returning 204 No Content without any message regardless of whether the legal tag exists or not. If the Legal Tag does not exist, it is expected to return 404 Not Found with the message like “Cannot delete a LegalTag that does not exist for given name xxx”M1 - Release 0.1Chris ZhangChris Zhanghttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/28Notification service not integrated with core services.2021-01-25T21:11:23ZGregNotification service not integrated with core services.Notification service not integrated with core services.Notification service not integrated with core services.M1 - Release 0.1Chris ZhangChris Zhanghttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/29Get Group Members query parameters not working properly.2022-12-08T17:14:06ZGregGet Group Members query parameters not working properly.GET – Get Group Members. Functionality working but defects. Query parameters “limit” and “role” not working properly If query params optional,e.g., it should return all the members regardless of the role for the request URL {{osduonaws_b...GET – Get Group Members. Functionality working but defects. Query parameters “limit” and “role” not working properly If query params optional,e.g., it should return all the members regardless of the role for the request URL {{osduonaws_base_url}}/api/entitlements/v1/groups/data.testing.osduonaws-testing@opendes.shell.com/members?limit=100. Currently it only return the user with the role of OWNER If query params required, e.g., it should return error handling with the message like “required parameter xxx missing”. There is defect with either scenario. And it would make more sense to keep query params optional in this case.M1 - Release 0.1JoeRucha DeshpandeMatt WiseJoehttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/30Cursor search returns results even when version does not match.2022-12-08T17:13:50ZGregCursor search returns results even when version does not match.POST – Query With Cursor. Functionality working but defect.
For example, below request is to search "opendes:osdu:wellbore-master:0.2.0" with the cursor "2F8900678904A680D24593BC7D8BEEA5".
However, it is still returning the results wit...POST – Query With Cursor. Functionality working but defect.
For example, below request is to search "opendes:osdu:wellbore-master:0.2.0" with the cursor "2F8900678904A680D24593BC7D8BEEA5".
However, it is still returning the results with 0.2.0 even I change the version number of data, e.g. "opendes:osdu:wellbore-master:0.0.8"
{ "kind": "opendes:osdu:wellbore-master:0.2.0", "cursor": "2F8900678904A680D24593BC7D8BEEA5", "aggregateBy": "kind" }
It seems that the kind should not be required once you have the Cursor ID. Or, if it is required, the kind should need to be the same as the original query.M1 - Release 0.1Chris ZhangChris Zhanghttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/31Bean with hard-coded properties imposes policy that limits application/servic...2022-11-24T12:34:33ZSherman YangBean with hard-coded properties imposes policy that limits application/service flexibilityIn OS Core Common repo's src/main/java/org/opengroup/osdu/core/common/http/HttpConfiguration.java, a bean is automatically created with hard-coded properties that impose policy which limits service flexibility.
For example, the new ent...In OS Core Common repo's src/main/java/org/opengroup/osdu/core/common/http/HttpConfiguration.java, a bean is automatically created with hard-coded properties that impose policy which limits service flexibility.
For example, the new entitlement v2 service needs to accept case insensitive Json input but with the hard-coded ObjectMapper bean that got introduced since osdu core common version 0.0.17, the entitlement service's case insensitive json configuration is no longer available as the common library creates its own ObjectMapper which disables/overrides the service's settings and imposes case sensitivity.
Need to relax this limitation to allow application flexibility.https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/39ADR: Partion Service as a TenantInfo provider2023-04-07T12:43:48ZRostislav Dublin (EPAM)ADR: Partion Service as a TenantInfo provider## Status
- [x] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context
Currently, CSPs have their own implementation of retrieving the TenantInfo data:
1. AWS. https://community.opengroup.org/osdu/platform/sy...## Status
- [x] Proposed
- [x] Trialing
- [x] Under review
- [x] Approved
- [ ] Retired
## Context
Currently, CSPs have their own implementation of retrieving the TenantInfo data:
1. AWS. https://community.opengroup.org/osdu/platform/system/lib/cloud/aws/os-core-lib-aws/-/blob/master/src/main/java/org/opengroup/osdu/core/aws/multitenancy/TenantFactory.java
2. Azure. https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/blob/master/src/main/java/org/opengroup/osdu/azure/multitenancy/TenantFactoryImpl.java
3. IBM. https://community.opengroup.org/osdu/platform/system/lib/cloud/ibm/os-core-lib-ibm/-/blob/master/src/main/java/org/opengroup/osdu/core/ibm/multitenancy/TenantFactory.java
All implementations follow a similar way to retrieve the TenantInfo from the Partition service (request partition service->map to TenantInfo class; calls to the Partition service are cached).
## Scope
Implement a common approach to retrieve the TenantInfo data and use it across all services.
## Decision
Unify an approach of retrieving TenantInfo and using it across all services:
- Implement a common tenant factory (ITenantFactory) on the core-lib level.
- Remove CSP-specific implementations of the TenantFactory.
- Some CSPs will require to restructure a code and implement missing interfaces (in fact, the code is already there):
- IBM should implement IServiceAccountJwtClient explicitly. There is already a similar method in the TenantFactory implementation.
### Outcome of the Decision (added by @rostislav.dublin on May 24):
1. A single common implementation of PartitionTenantInfoFactory in os-core-common. It just lies here and does not inject anywhere. [See it](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/73/diffs#eb73934c527564533da4c8574c831649474ee500_0_36) in the [MR 73](https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/73/) ;
2. Each CSP injects the above single common implementation in its CSP library, removing its own implementation. Everyone moves at their own pace, no one rushes anyone. E.g. GCP uses AbstractFactoryBean<IPartitionFactory> [as shown here](https://community.opengroup.org/osdu/platform/system/schema-service/-/merge_requests/102/diffs#4ac1210ac980e2a0dcec2574ebd6eb21618e8abc_0_23).
## Rationale
This change unifies an approach of retrieving tenant information, which is used to separate tenants from each other (multitenancy). With the increased maintainability of the code, OSDU will have a unified basic data to implement multitenancy
## Consequences
Simplified CSP-specific code and code generalization (higher maintainability and reusability).https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/49ADR : Optional Audit Logs per partition2022-05-20T16:46:09ZPreksha Beohar-SlbADR : Optional Audit Logs per partition## Status
- [X] Proposed
- [X] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
As an operator of OSDU we are seeing increased levels of costs related to logging over time. For instance in a single day we saw over 500GB of...## Status
- [X] Proposed
- [X] Under review
- [x] Approved
- [ ] Retired
## Context & Scope
As an operator of OSDU we are seeing increased levels of costs related to logging over time. For instance in a single day we saw over 500GB of logs created per environment. This can run to costs of $X000 a day.
A large percentage of these logs can be equated to audit logs, especially for READ actions like list groups in Entitlements which can be called multiple millions of times a day.
We have deployments and partitions of varying degrees of sensitivity. For example we have deployments strictly for development purposes but even in our production systems some partitions are used solely for development by their clients.
As a client of the system I would like to be able to control which partitions have audit logs enabled and if so whether I have this turned on for write operations only i.e. create,update,delete actions or all events in the system.
This gives me the choice of whether the cost benefit of running audit logs in a particular partition is worthwhile and can give the client the control based on the sensitivity of the partition.
This has precedent in other systems e.g. in Google Cloud or Azure I have the choice for most resources I deploy whether to turn on and retain audit logs for these services even though they _could_ have sensitive information as it gives me as the client the control and choice.
## Trade-off Analysis
I could choose to have all audit logs turned on at all times. This is the current state. However this has potentially high cost implications. If the partition or deployment of OSDU does not contain sensitive information then I would like the choice to turn off audit logs.
I could make the option to turn on/off logs as part of the build configuration. However this means I need to redeploy if I want to switch logs on or off. This also prevents end users potentially ever having this control.
I could make the configuration for the entire environment rather than per partition. However a strong use case for partitions is to separate them based on data sensitivity e.g. I may have a partition for development purposes and another partition in the same deployment with production data.
## Decision
We are proposing the ability to have audit logs turned on or off by choice of the operator and/or client.
An implementation of this could be via a feature flag from the partition service which will allow this to be set at runtime and per partition.
We would like 2 separate feature flags. One which can turn the READ action audit logs on or off and another to turn the CREATE/UPDATE/DELETE/PUBLISH/JOB_RUN action audit logs on or off.
By default both of these should be turned on so it is 'secure' by default. However the operator and/or end user can then choose to turn certain ones off.Preksha Beohar-SlbPreksha Beohar-Slbhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/64Spring configuration related to Feature flag should be optional2023-01-30T11:37:15ZRustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comSpring configuration related to Feature flag should be optionalIt is possible that some OSDU services may not use Partition service, Unit service as an example. But the configuration of the recently merged code https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requ...It is possible that some OSDU services may not use Partition service, Unit service as an example. But the configuration of the recently merged code https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/189 requires PARTITION_API env variable mandatorily. It will be better to change the configuration to make it optional, instead of bringing this variable to each service, no matter if Partition is used or not.
~~~
Exception encountered during context initialization - cancelling refresh attempt:
org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'featureFlagConfig':
Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException:
Could not resolve placeholder 'PARTITION_API' in value "${PARTITION_API}"
~~~Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comRiabokon Stanislav(EPAM)[GCP]Rustam Lotsmanenko (EPAM)rustam_lotsmanenko@epam.comhttps://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/issues/67inefficient/non-performant crs-conversions causing reliability/performance is...2023-06-21T07:54:16ZYurii Kondakovinefficient/non-performant crs-conversions causing reliability/performance issues on ingestion workflowWe are seeing reliability and performance issue because of inefficient/non-performant crs-conversions on ingestion workflow. If crs conversions takes long time (on Storage /batch API), then it slows down entire ingestion workflow.
There...We are seeing reliability and performance issue because of inefficient/non-performant crs-conversions on ingestion workflow. If crs conversions takes long time (on Storage /batch API), then it slows down entire ingestion workflow.
There is a need for setting a timeout for crs-conversion requests that run for more than a certain time. For the requests to crs-conversion-service currently java.net.HttpURLConnection class is used, which is only has connectionTimeout and readTimeout properties, that don't help us to set the timeout.
It is suggested to use apache CloseableHttpClient httpClient that has socketTimeout property.
core-common MR - https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/213
storage MR - https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/712M19 - Release 0.22Yurii KondakovYurii Kondakov