OS Core Lib Azure merge requestshttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests2021-08-28T08:56:29Zhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/141Updating OSDU dependencies to latest release2021-08-28T08:56:29ZDavid Diederichd.diederich@opengroup.orgUpdating OSDU dependencies to latest releaseM8 - Release 0.11David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/137Modified the code to add Content-Type and Content-Disposition headers in sign...2023-08-18T12:44:14Zsachin GuptaModified the code to add Content-Type and Content-Disposition headers in signed URL generation logicIssue #https://community.opengroup.org/osdu/platform/system/file/-/issues/35Issue #https://community.opengroup.org/osdu/platform/system/file/-/issues/35M9 - Release 0.12Paresh BehedeParesh Behedehttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/143Publisher Facade2023-08-18T12:44:07ZNikhil Singh[MicroSoft]Publisher Facade## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] Does the MR contain pipeline/ helm chart related changes?
* [NA] I have u...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [NO] Does the MR contain pipeline/ helm chart related changes?
* [NA] I have updated the documentation accordingly.
* [YES] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [NA] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
<!-- Please describe the current behavior that you are modifying, 'or' link to a relevant issue.
Feel free to add references to any design documents you might have shared with the team or any
related MR that you are building on top of. -->
This MR introduces a publisher facade which can be used across services in order to publish messages to message brokers(Service Bus/ Event Grid).It will help to manage and update at a single source instead of each service doing it individually.The pub sub configuration can be used to configure publishing for Event grid and Service bus.
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
- All existing and new TC's passed.
## Does this introduce a breaking change?
-------------------------------------
- [NO]M9 - Release 0.12Nikhil Singh[MicroSoft]Nikhil Singh[MicroSoft]https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/145Quality Improvements Publisher Facade2023-08-18T12:44:05ZNikhil Singh[MicroSoft]Quality Improvements Publisher Facade**INTRODUCTION**
This MR inculdes following quality improvements items:
1. Usage of builder patterns for amps and json objects
1. Removal of conditional beans as some services (ex schema) can have both EG and SB disabled.
1. Unit Tests u...**INTRODUCTION**
This MR inculdes following quality improvements items:
1. Usage of builder patterns for amps and json objects
1. Removal of conditional beans as some services (ex schema) can have both EG and SB disabled.
1. Unit Tests updatedM9 - Release 0.12Nikhil Singh[MicroSoft]Nikhil Singh[MicroSoft]https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/142Circuitbreaker configuration being configurable through application.properties2023-08-18T12:44:09ZRonak SakhujaCircuitbreaker configuration being configurable through application.properties## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] I have updated the documentation accordingly.
* [YES] I have added tests...## All Submissions:
-------------------------------------
* [YES] I have added an explanation of what changes in this merge do and why we should include it?
* [YES] I have updated the documentation accordingly.
* [YES] I have added tests to cover my changes.
* [YES] All new and existing tests passed.
* [YES] My code follows the code style of this project.
* [YES] I ran lint checks locally prior to submission.
## What is the issue or story related to the change?
-------------------------------------
Circuit Breaker can be made configurable through adding a CircuitBreakerConfig class to the registry. We are adding few boolean flags to the configuration to set those properties in circuitbreaker.
azure.circuitbreaker.enable = false;
azure.circuitbreaker.defaultCircuitBreaker = false;
azure.circuitbreaker.slidingWindowSize = 50;
azure.circuitbreaker.minimumNumberOfCalls = 50;
azure.circuitbreaker.failureRate = 50;
azure.circuitbreaker.slidingWindowType = "TIME_BASED";
azure.circuitbreaker.permittedCallsInHalfOpenState = 10;
Details about each is in Readme.
We are using postConstruct in setCBR method, to run that after initializing all values of configuration.
High level design:
Issue: <!-- Link any __GitLab__ workitem(s) to this pull request. -->
<!-- Please add implementation details of current set of changes and how the code changes are
doing what they are expected to do. Are there any complex loops or designated code blocks that
should be elaborated? Is there some contextual knowledge that the reviewer should be aware of? -->
Change details:
## Test coverage:
------------------
<!-- Mention unit test coverage of changes. -->
All tests are passing and changes are covered.
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
No
## Pending items
----------------
<!-- Are there changes that you'll introduce in upcoming MRs and hence did not add in this one? Next steps of your
feature can also be mentioned here. -->
## Reviewer request
-------------------
- Please provide an ETA when you plan to review this MR. Write a comment to decline or provide an ETA.
- Block the MR if you feel there is less testing or no details in the MR
- Please cover the following aspects in the MR
-- Coding design: _\<Reviewer1>_
-- Backward Compatibility: _\<Reviewer2>_
-- Feature Logic: _\<Logic design\>_
-- _\<Any other context mention here>_
OR
-- _\<Component 1>_: _\<Reviewer1>_
-- _\<CosmosDB>_: _\<Reviewer2>_
-- _\<ServiceBus>_ _\<Reviewer3>_
-- _\<Mention any other component and owner>_
## Other information
-------------------------------------
<!-- Any other information that is important to this MR such as screenshots of how the component looks before and after the change. -->M9 - Release 0.12https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/147Remove unused dependencies2023-08-18T12:44:04ZRonak SakhujaRemove unused dependenciesM9 - Release 0.12https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/148Update NOTICE2023-08-18T12:44:02ZAbhishek PatilUpdate NOTICEM9 - Release 0.12Abhishek PatilAbhishek Patilhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/149Added @Configuration2023-08-18T12:44:00ZRonak SakhujaAdded @ConfigurationSpring doesn't create bean if @Configuration is not present. Hence adding @Configuration to config classSpring doesn't create bean if @Configuration is not present. Hence adding @Configuration to config classM9 - Release 0.12https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/140Added LegalFactoryAzure and LegalAPIConfigBean2023-08-18T12:44:10ZMuskan SrivastavaAdded LegalFactoryAzure and LegalAPIConfigBean## All Submissions:
-------------------------------------
* I have added an explanation of what changes in this merge do and why we should include it? - YES
* Does the MR contain pipeline/ helm chart related changes?- NO
* I have updated...## All Submissions:
-------------------------------------
* I have added an explanation of what changes in this merge do and why we should include it? - YES
* Does the MR contain pipeline/ helm chart related changes?- NO
* I have updated the documentation accordingly. - NO
* I have added tests to cover my changes. -NA
* All new and existing tests passed. -YES
* My code follows the code style of this project. -YES
* I ran lint checks locally prior to submission. -YES
## What is the issue or story related to the change?
-------------------------------------
This change is a step forward to enable the services to configure following variables for service-to-service communication:
- Circuit Breaker
- Retry count for service unavailable strategy
- Connect timeout
- Connection Request Timeout
- Socket Timeout
<br>
All the above features are handled by HttpClientAzure class.
This MR essentially enables use of HttpClientAzure for legal service.
These configurations can be used for legal service now.
<br>
High level design: This MR adds 2 classes in total. <br>
- LegalAPIConfigAzure : creates a bean of type LegalAPIConfig
- LegalFactoryAzure : uses newly created HttpClientAzure internally so that configurable retires, CB and timeouts can be used for Legal Service.
## Does this introduce a breaking change?
-------------------------------------
- NO
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Tests and Results
To test the CB and timeouts, I used the scenario where storage service calls legal service. Simulations were triggered by Firefly Flight
1. Circuit Breaker: To test this legal service was killed. Result is as follows![Screenshot__29_](/uploads/b3398d97c9b06a574209eeddf28d3231/Screenshot__29_.png)
2. Timeouts: To test this Socket Timeouts was configured to be 40 sec for any calls made to legal service. Latency was induced to legal service between the range of 45 sec and 50 sec.
![Screenshot__30_](/uploads/66d4191433187731e39be420c98fea5a/Screenshot__30_.png)M9 - Release 0.12Muskan SrivastavaMuskan Srivastavahttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/150too many cosmosdb request exception handler2023-08-18T12:43:59ZNeelesh Thakurtoo many cosmosdb request exception handlerM9 - Release 0.12https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/152fix fossa NOTICE to get latest rc package2023-08-18T12:43:57ZNeelesh Thakurfix fossa NOTICE to get latest rc packageM9 - Release 0.12https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/154Update common.properties2023-08-18T12:43:53ZSmitha ManjunathUpdate common.propertiesM9 - Release 0.12Smitha ManjunathSmitha Manjunathhttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/155throw explicit 503 when server is busy to process BlobStorage request so clie...2023-08-18T12:43:51ZNeelesh Thakurthrow explicit 503 when server is busy to process BlobStorage request so client can retryWe see Storage API starts to responds with 500 when system is under heavy ingestion for very few requests. On closer look, we found that BlobStorage is throttled and underlaying error message is following:
AppException(error=AppError(co...We see Storage API starts to responds with 500 when system is under heavy ingestion for very few requests. On closer look, we found that BlobStorage is throttled and underlaying error message is following:
AppException(error=AppError(code=500, reason=Failed to read specified blob, message=Status code 503, "<?xml version="1.0" encoding="utf-8"?><Error><Code>ServerBusy</Code><Message>The server is busy._RequestId:82895e12-701e-0066-26d3-b568b7000000_Time:2021-09-30T08:13:49.5116925Z</Message></Error>", errors=null, debuggingInfo=null, originalException=com.azure.storage.blob.models.BlobStorageException: Status code 503, "<?xml version="1.0" encoding="utf-8"?><Error><Code>ServerBusy</Code><Message>The server is busy.
RequestId:82895e12-701e-0066-26d3-b568b7000000
Time:2021-09-30T08:13:49.5116925Z</Message></Error>"), originalException=com.azure.storage.blob.models.BlobStorageException: Status code 503, "<?xml version="1.0" encoding="utf-8"?><Error><Code>ServerBusy</Code><Message>The server is busy.
RequestId:82895e12-701e-0066-26d3-b568b7000000
Time:2021-09-30T08:13:49.5116925Z</Message></Error>")
![image](/uploads/2ac4650821f1a3ae252b65f4a5958d3c/image.png)
With this change, client can retry based of response code.M9 - Release 0.12https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/153Added code to disable ai appender based on jvm argument2023-08-18T12:43:55ZVineeth Guna [Microsoft]Added code to disable ai appender based on jvm argumentM9 - Release 0.12https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/156Update core common version2023-08-18T12:43:50ZNikhil Singh[MicroSoft]Update core common version**Introduction**
This MR upgrades the os core-common version for azure provider to verify the recent security fixes made in the below mentioned MR are working properly.
**Issue Link**
https://community.opengroup.org/osdu/platform/syst...**Introduction**
This MR upgrades the os core-common version for azure provider to verify the recent security fixes made in the below mentioned MR are working properly.
**Issue Link**
https://community.opengroup.org/osdu/platform/system/lib/core/os-core-common/-/merge_requests/119M9 - Release 0.12Nikhil Singh[MicroSoft]Nikhil Singh[MicroSoft]https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/157Update OSDU dependencies2021-10-21T16:10:34ZDavid Diederichd.diederich@opengroup.orgUpdate OSDU dependenciesM9 - Release 0.12David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/158Make HttpClientAzure conditionally primary2023-08-18T12:43:48ZNikhil Singh[MicroSoft]Make HttpClientAzure conditionally primary** Introduction **
This MR makes the bean HttpClientAzure Conditional so that worker services like Indexer Queue,WKS,Notification,Storage can use the non request scope HttpClient while working with message bus.** Introduction **
This MR makes the bean HttpClientAzure Conditional so that worker services like Indexer Queue,WKS,Notification,Storage can use the non request scope HttpClient while working with message bus.M9 - Release 0.12Nikhil Singh[MicroSoft]Nikhil Singh[MicroSoft]https://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/159Cherry-pick branch 'users/nikhil/fixHttpClientRS' into release/0.122022-12-22T18:58:01ZDavid Diederichd.diederich@opengroup.orgCherry-pick branch 'users/nikhil/fixHttpClientRS' into release/0.12Make HttpClientAzure conditionally primary
See original merge request osdu/platform/system/lib/cloud/azure/os-core-lib-azure!158Make HttpClientAzure conditionally primary
See original merge request osdu/platform/system/lib/cloud/azure/os-core-lib-azure!158M9 - Release 0.12David Diederichd.diederich@opengroup.orgDavid Diederichd.diederich@opengroup.orghttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/160Catch request too large exception2023-08-18T12:43:46ZAlok JoshiCatch request too large exceptionPlease refer to https://community.opengroup.org/osdu/platform/system/storage/-/issues/97 for detailsPlease refer to https://community.opengroup.org/osdu/platform/system/storage/-/issues/97 for detailsM10 - Release 0.13Alok JoshiAlok Joshihttps://community.opengroup.org/osdu/platform/system/lib/cloud/azure/os-core-lib-azure/-/merge_requests/161Update notice2023-08-18T12:43:45ZAlok JoshiUpdate noticeFix notice fileFix notice fileM10 - Release 0.13Alok JoshiAlok Joshi