infra-azure-provisioning issueshttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues2023-02-02T14:28:12Zhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/218Enable pod security policies for AKS cluster2023-02-02T14:28:12ZBharathi SelvarajEnable pod security policies for AKS cluster**Azure Policy built-in definitions for Azure Kubernetes Service**
For new OSDU deployment on Azure, make sure the following security policies are enabled for AKS
[Built-in policy definitions for Azure Kubernetes Service - Azure Kuberne...**Azure Policy built-in definitions for Azure Kubernetes Service**
For new OSDU deployment on Azure, make sure the following security policies are enabled for AKS
[Built-in policy definitions for Azure Kubernetes Service - Azure Kubernetes Service | Microsoft Docs](https://docs.microsoft.com/en-us/azure/aks/policy-reference)
- [x] Enable AKS Policies for osdu-azure namespace
- [x] Enable AKS policies for istio-system namespace
- [x] Enable AKS policies for airflow2 namespace
- [x] Move csi secrets to kube-system as recommended by Microsoft docs [Docs](https://docs.microsoft.com/en-us/azure-stack/aks-hci/secrets-store-csi-driver#install-the-secrets-store-csi-driver)
- [x] Fix non compliant policies - Allow Authorized IP Ranges, Enable private clusters.
- [x] Documentation on how to enable the feature and fix compliance issues.M11 - Release 0.14Krishna Nikhil VedurumudiKrishna Nikhil Vedurumudihttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/216Vulnerability Periodic Scans and Processes2022-03-03T16:04:00ZKrishna Nikhil VedurumudiVulnerability Periodic Scans and ProcessesCreate a process where the team can pro-actively catch vulnerabilities such that no surprises come when a build is needed. A periodic job that would run nightly to scan the containers.
Include all the containers that are used in the O...Create a process where the team can pro-actively catch vulnerabilities such that no surprises come when a build is needed. A periodic job that would run nightly to scan the containers.
Include all the containers that are used in the OSDU system - airflow, java, python under the same process.
Create templates in CI-CD project such that the pipelines can be re-used.
Send email notification to a email group in case of pipeline errors such that team is alerted on the new vulnerability that is detected.
Make sure the build fails if there are any vulnerabilities are detected.
## Tasks:
- [ ] Create a distribution list for pipeline failure notifications.
- [x] Create re-usable pipeline that periodically scans build images. (nightly?)
- [x] Send email to the above DL only in case of failures.
- [x] Make sure the build fails if vulnerabilities are detected.
- [x] Include all the containers that are used in the OSDU system - airflow, java, python, terraform under the same process.M10 - Release 0.13Arturo Hernandez [EPAM]Arturo Hernandez [EPAM]https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/212Azure Infrastructure manual deployment Central Resources ERROR2021-11-23T07:46:49ZSergey ZemskovAzure Infrastructure manual deployment Central Resources ERROR`common_prepare.sh`script has been executed successfully. But I get the error on deployment Central Recourses step.
Error ocured on step `terraform plan -var-file custom.tfvars`
```
Error: expected "object_id" to be a valid UUID, got
...`common_prepare.sh`script has been executed successfully. But I get the error on deployment Central Recourses step.
Error ocured on step `terraform plan -var-file custom.tfvars`
```
Error: expected "object_id" to be a valid UUID, got
on ../../../modules/providers/azure/keyvault-policy/main.tf line 15, in resource "azurerm_key_vault_access_policy" "keyvault":
15: resource "azurerm_key_vault_access_policy" "keyvault" {
```
UPD:
File with params `.envrc` was not completely filled after running `common_prepare.sh` script, so object_id couldn't be foundhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/207Preshipping provisioning for Azure2022-07-04T13:23:28ZMANISH KUMARPreshipping provisioning for Azure**Release name**: `M12`
The following steps must be completed OSDU Azure release.
For more information, visit our release documentation [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisio...**Release name**: `M12`
The following steps must be completed OSDU Azure release.
For more information, visit our release documentation [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/CHANGELOG.md).
## Steps:
**Infra Board closure**
- [ ] Mark all issues closed which have been completed in the release. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/boards). Note that creating an issue ticket as "Close Release - <version>" helps as a mark up for the release
**Deploy terraform scripts**
- [x] Central Resources
- [x] Service Resources
- [x] Data Partitions.
**Create service release images**
- [x] Upload release service images to ACR.
- [x] Create and Update service charts
- [x] Upload service charts to ACR.
**Create Data seeding**
- [ ] Update documentation for following seeding data, Config, Manifest DAG, CSV Parser DAG, Schema, Entitlements, Policy, ZGY DAG and VDS DAG.
- [ ] Create and Upload versioned image for following seeding data, Config, Manifest DAG, CSV Parser DAG, Schema, Entitlements, Policy, ZGY DAG and VDS DAG.
- [ ] Update scripts for data seeding.
**Upload Data prior to service deployment**
- [x] Upload Config Data
- [x] Upload Manifest Ingest DAG
**Service deployment (Install Charts to ACR)**
- [x] Partition Service
- [x] Security Services
- [x] Core Services (Can be installed in parallel)
- [x] Reference Services (Grouped under helm charts repo)
- [x] Ingest Services
- [x] Seismic Services
- [x] Wellbore Services
**Upload Data post to service deployment**
- [x] Upload Entitlements Data
- [x] Upload Schema
- [ ] Upload Policies
- [x] Upload CSV Parser DAG
- [x] Upload SEGY to ZGY DAG Conversion
- [x] Upload SEGY to VDS DAG Conversion
**Load community provided datasets post to service deployment**
- [ ] TNO data
- [ ] Volve data
**Validation**
Test services and dags using REST scripts[Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/rest).
- [ ] Test services
- [ ] Test Manifest DAG
- [ ] Test CSV DAG
- [ ] Test ZGY DAG
- [ ] Test VDS DAG
**Add Smoke Tests for services for M12 using Postman**
Preshipping community require test collections for validation which are placed here [Link](https://community.opengroup.org/osdu/platform/pre-shipping/-/tree/main/R3-M9/Azure-M9).
Need to enhance these test at par with our probe tests and any new development since M8 release.
Internal Probe Tests location for reference [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/source/probe-test-client)
Platform Validation Tests for reference [Link](https://community.opengroup.org/osdu/platform/testing)M12 - Release 0.15Madhur Tanwani [Microsoft]Bharathi SelvarajPrashanth KKrishnan GanesanVivek KalraMadhur Tanwani [Microsoft]https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/205Istio version upgrade2021-11-18T10:44:30ZDzmitry_Paulouski (slb)Istio version upgradeWe are using 1.6.7 version in OSDU, it's a little outdated and soon won't be supported by the latest deployable AKS version.
**Upgrade details**
* Current Version - 1.6.7
* New Version - 1.11.3
Steps to upgrade:
Run `osdu-istio` pipe...We are using 1.6.7 version in OSDU, it's a little outdated and soon won't be supported by the latest deployable AKS version.
**Upgrade details**
* Current Version - 1.6.7
* New Version - 1.11.3
Steps to upgrade:
Run `osdu-istio` pipeline to add in Gitops repo the updated manifests of Istio Operator, which takes care of upgrade.
https://dev.azure.com/ms-slb-cobuild/ms-slb-collab/_workitems/edit/665/Dzmitry_Paulouski (slb)Dzmitry_Paulouski (slb)https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/202Upgrade Airflow Lib with release version2021-11-12T03:43:34Zharshit aggarwalUpgrade Airflow Lib with release versionWith this MR we are adding a dev version of a airflow python package which should be replaced with release version during release
https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_r...With this MR we are adding a dev version of a airflow python package which should be replaced with release version during release
https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/485harshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/197Enable BYOAD by adding feature flag for ad application in central resources2023-08-16T10:40:37ZVivek OjhaEnable BYOAD by adding feature flag for ad application in central resourcesVivek OjhaVivek Ojhahttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/196Close Release 0.112023-08-16T10:40:37ZMANISH KUMARClose Release 0.11Vivek OjhaVivek Ojhahttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/193Onboard Seismic Dms File Metadata Service2021-08-30T17:16:12ZVladimir MoiseevOnboard Seismic Dms File Metadata Service**Service name**: `Seismic Dms File Metadata Service`
The following steps must be completed for a service to onboard with OSDU on Azure. Additionally, please add the `Service Onboarding` tag to this issue when it is created.
For more i...**Service name**: `Seismic Dms File Metadata Service`
The following steps must be completed for a service to onboard with OSDU on Azure. Additionally, please add the `Service Onboarding` tag to this issue when it is created.
For more information, visit our service onboarding documentation [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-onboarding.md).
## Steps:
**Infrastructure and Initial Requirements**
- [x] Add any additional Azure cloud infrastructure (Cosmos containers, Storage containers, fileshares, etc.) to the Terraform template. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/infra/templates/osdu-r3-mvp). Note that if the infrastructure is a part of the data-partition template, you may need to add secrets to the keyvault that are partition specific; if doing so, update the createPartition REST request to include the keys that you have added so they are accessible in service code. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/rest/partition.http#L48)
- [x] Create an ingress point for the service. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/appgw-ingress.yaml)
- [x] Add any test data that is required for the service integration tests. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/test_data)
- [x] Update `upload-data.py` to upload any new test data files you created. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/test_data/upload-data.py).
- [x] Update the integration tester with any entitlements required to test the service. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/test_data/user_info_1.json)
- [x] Add in any new secrets that the service needs to run. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/kv-secrets.yaml)
- [x] Create environment variable script to generate .yaml files to be used with Intellij [EnvFile](https://plugins.jetbrains.com/plugin/7861-envfile) plugin and .envrc files to be used with [direnv](https://direnv.net/). [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/variables)
**Gitlab Code and Documentation**
- [x] Complete the service code such that it passes all integration tests locally. There is some documentation on starting off implementing an Azure provider. [Link](./gitlab-service-readme-template.md)
- [x] Create helm charts for service. The charts for each service are located in the `devops/azure` directory. You can look at charts from other services as a model. The charts will be nearly identical except for the different environment variables, values, etc each service needs to run. [Link](./gitlab-service-guide.md)
- [x] Implement Istio for the service if this has not already been done. Here is an example MR that shows what steps are required. [Link](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/64)
- [x] Create an Istio auth policy in the `devops/azure/chart/templates` directory. Here is an example of an Istio auth policy that is generic and can be used by other services. [Link](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/devops/azure/chart/templates/azure-istio-auth-policy.yaml)
- [x] Add any variables that are required for the service integration tests to the Azure CI-CD file. [Link](https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/blob/master/cloud-providers/azure.yml)
- [x] Verify that the README for the Azure provider correctly and clearly describes how to run and test the service. There is a README template to help. [Link](./gitlab-service-readme-template.md)
- [x] Push any changes and verify that the Gitlab pipeline is passing in master.
**Development and Demo Azure Devops Pipelines**
- [ ] Create development ADO pipeline at `devops/azure/development-pipeline.yml` in the service repo.
- [ ] Verify development pipeline passes in ADO.
- [ ] Create Demo ADO pipeline at `devops/azure/pipeline.yml` in the service repo.
- [ ] Verify demo pipeline is passing in ADO.
**User Documentation**
- [ ] Add the service to the mirror pipeline instructions. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/code-mirroring.md)
- [ ] Add the service to the manual deployment instructions. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/charts)
- [ ] Add any required variables to the already existing variable group instructions for automated deployment. You should know if any variables need to be added to existing variable groups from creating the development and demo pipelines. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [ ] Add a variable group `Azure Service Release - $SERVICE_NAME` to the documentation. You should know what values to set for this variable group from creating the development and demo pipelines. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [ ] Add a step for creating the service pipeline at the bottom of the service-automation page. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [ ] Create a rest script with sample calls to the service for users. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/rest)Vladimir MoiseevVladimir Moiseevhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/192Release 0.10.02021-08-09T16:03:14ZMANISH KUMARRelease 0.10.0**Release name**: `0.10.0`
The following steps must be completed OSDU Azure release.
For more information, visit our release documentation [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provi...**Release name**: `0.10.0`
The following steps must be completed OSDU Azure release.
For more information, visit our release documentation [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/CHANGELOG.md).
## Steps:
**Infra Board closure**
- [ ] Mark all issues closed which have been completed in the release. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/boards). Note that creating an issue ticket as "Close Release - <version>" helps as a mark up for the release
**Deploy terraform scripts**
- [ ] Central Resources
- [ ] Service Resources
- [ ] Data Paritions.
**Create service release images**
- [ ] Upload release service images to ACR.
- [ ] Create and Update service charts
- [ ] Upload service charts to ACR.
**Create Data seeding**
- [ ] Update documentation for following seeding data, Config, Manifest DAG, CSV Parser DAG, Schema, Entitlements, Policy, ZGY DAG and VDS DAG.
- [ ] Create and Upload versioned image for following seeding data, Config, Manifest DAG, CSV Parser DAG, Schema, Entitlements, Policy, ZGY DAG and VDS DAG.
- [ ] Update scripts for data seeding.
**Upload Data prior to service deployment**
- [ ] Upload Config Data
- [ ] Upload Manifest Ingest DAG
**Service deployment**
- [ ] Partition Service
- [ ] Security Services
- [ ] Core Services
- [ ] Reference Services
- [ ] Ingest Services
- [ ] Seismic Services
- [ ] Wellbore Services
**Upload Data post to service deployment**
- [ ] Upload Entitlements Data
- [ ] Upload Schema
- [ ] Upload Policies
- [ ] Upload CSV Parser DAG
- [ ] Upload SEGY to ZGY DAG Conversion
- [ ] Upload SEGY to VDS DAG Conversion
**Validation**
Test services and dags using REST scripts[Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/rest).
- [ ] Test services
- [ ] Test Manifest DAG
- [ ] Test CSV DAG
- [ ] Test ZGY DAG
- [ ] Test VDS DAGM7 - Release 0.10.0 - removeMANISH KUMARMANISH KUMARhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/190Close Release 0.10.02021-07-28T05:59:15ZMANISH KUMARClose Release 0.10.0M7 - Release 0.10.0 - removeMANISH KUMARVivek OjhaMANISH KUMARhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/187Use latest secure TLS version for Storage accounts2021-06-29T08:33:55ZVasyl Leskiv [SLB]Use latest secure TLS version for Storage accountsMotivation:
* Use most recent secure TLS version in storage account module by default.
* Some client subscriptions have configured Azure policies that prevent OSDU terraform deployment with TLS version 1.0 (it is [non explicitly set by ...Motivation:
* Use most recent secure TLS version in storage account module by default.
* Some client subscriptions have configured Azure policies that prevent OSDU terraform deployment with TLS version 1.0 (it is [non explicitly set by default](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_account#min_tls_version)) and is non-secure outdated version.Vasyl Leskiv [SLB]Vasyl Leskiv [SLB]https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/186Adding Airflow Multipartition Partition Changes2022-08-23T10:47:33Zharshit aggarwalAdding Airflow Multipartition Partition ChangesFor enabling multi partitioning support for Airflow following infrastructure changes are required
**New AKS Cluster needs to be created in dp resource group**
- Same configuration as what we have in service resources group
- Autoscal...For enabling multi partitioning support for Airflow following infrastructure changes are required
**New AKS Cluster needs to be created in dp resource group**
- Same configuration as what we have in service resources group
- Autoscaling needs to be enabled for AKS cluster
- The virtual network used by node pool should accommodate for atleast 2500 ip addresses
- AKS should have access to node resource group in which node pools exist
- AKS Access to Create and Remove VM's in Node Resource Group
- AKS should have access to only pull images from central resources ACR as well as ACR created in data partition
- AKS should have access to the data partition specific pod identity
**New managed identity needs to be created in dp resource group**
- Need read access to keyvault which is present in dp resource group
- Need access to fileshares/ blob storage for the storage account used by other osdu services
- Need access to storage queue to read and process
**New Postgresql server needs to be created in dp resource group**
- Same configuration steps as what we have in service resources group
- Only difference is any secrets related to postgres needs to be stored in data partition.
**Use existing storage account used by other osdu services**
- Create fileshares and directories internally similar to service resource group
- Create storage container similar to service resource group
- Create storage queue similar to service resource group
- Adding storage account secrets in dp keyvault
**Create event grid subscription to push logs to log analytics**
**New container registry needs to be created in dp resource group**
**New keyvault needs to be created in dp resource group**
**New redis cluster needs to be created in dp resource group**
- Same configuration steps as what we have in service resource group
- Only difference is any secrets related to redis needs to be stored in data partition.
**New log analytics workspace needs to be created in data partition to store task logs**
**Kubernetes changes needed**
- Install KEDA helm chart version 2.1.0
- Install Cert manager helm chart
- Install Kvsecrets helm chart
- Install aad-pod-identity helm chart
- Create OSDU namespace with istio injection enabled
Create airflow specific secrets and store it in dp specific keyvault
**AKS, Postgres, Redis, Virtual network diagnostics**
**New keyvault to be created in central resources which will have app insights key which is shared across all data partitions**
- The pod identity in data partition should have get access to this keyvault.
**Create NSG for aks subnet in data partition AKS cluster**
- Whitelist sr aks egress ip in this NSG
**All the resources created should be feature flagged**https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/185[Feature] Airflow Monitoring Dashboards.2021-07-28T06:23:16ZMayank Saggar [Microsoft][Feature] Airflow Monitoring Dashboards.For Monitoring of Airflow and it's services, three dashboards, one for airflow infra, one for airflow service and one for airflow dags will be deployed as a part of Monitoring resources. The infra and service dashboards would be viewable...For Monitoring of Airflow and it's services, three dashboards, one for airflow infra, one for airflow service and one for airflow dags will be deployed as a part of Monitoring resources. The infra and service dashboards would be viewable at data partition level, whereas the dags dashboard would be viewable at data-partition and dag level.M7 - Release 0.10Mayank Saggar [Microsoft]Mayank Saggar [Microsoft]https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/181[Breaking Change] Zonal Redundancy for Airflow2022-08-23T10:47:30ZAbhishek Chowdhry[Breaking Change] Zonal Redundancy for AirflowPoint Airflow to use the newly created Zone Redundant Redis Instance for future purposes. This will break the existing Airflow Runs and they will need to be triggered again.
**Consuming this Change**:
This change will break the existin...Point Airflow to use the newly created Zone Redundant Redis Instance for future purposes. This will break the existing Airflow Runs and they will need to be triggered again.
**Consuming this Change**:
This change will break the existing Airflow Runs. If they can be retriggered without losing any data, just retrigger the Airflow Runs once this change is merged.
If retriggering end to end runs is not possible due to any reason and we don't want to lose the existing runs, there are 2 suggested methods:
1) Drain the entire Queue by do not sending any new requests to Airflow. Once the queue is drained, take the changes for pointing to the new queue(new redis instance) and resume the traffic to Airflow.
2) Stop sending any new Requests to Airflow and take the changes for pointing to the new Queue(new Redis instance). Now requeue all the tasks from the old queue into the new queue. Resume the traffic to Airlfow.
Prefer the first option to the second one as the second option has a big overhead of requeuing and may still result in data loss.M7 - Release 0.10.0 - removeAbhishek ChowdhryAbhishek Chowdhryhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/180Zonal Redundancy for Redis2022-08-23T10:47:29ZAbhishek ChowdhryZonal Redundancy for RedisEnable Zone Redundancy for Redis by creating a new Redis Premium Instance with zone Redundancy Enabled
## Acceptance Criteria
* [X] Infra changes to add new Redis
* [X] Premerge pipeline
* [ ] Changes for Glab/dev/demo
* [ ] Changes fo...Enable Zone Redundancy for Redis by creating a new Redis Premium Instance with zone Redundancy Enabled
## Acceptance Criteria
* [X] Infra changes to add new Redis
* [X] Premerge pipeline
* [ ] Changes for Glab/dev/demo
* [ ] Changes for ManualM7 - Release 0.10.0 - removeAbhishek ChowdhryAbhishek Chowdhryhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/179Zonal redundancy for AKS and App Gateway2021-07-27T07:50:44ZVivek OjhaZonal redundancy for AKS and App GatewayEnable zonal redundancy for AKS and App GatewayEnable zonal redundancy for AKS and App GatewayM7 - Release 0.10.0 - removeVivek OjhaVivek Ojhahttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/178Pipeline Variables for Service and Data Partition RG not present2021-06-21T11:22:43ZAbhishek ChowdhryPipeline Variables for Service and Data Partition RG not presentPipeline Variables for Service and Data Partition RG are not present in Infrastructure Pipelines Variables Groups(present for central RG). Add those variables in all the infra pipeline variable groups and also update templates in infra-a...Pipeline Variables for Service and Data Partition RG are not present in Infrastructure Pipelines Variables Groups(present for central RG). Add those variables in all the infra pipeline variable groups and also update templates in infra-azure-repo to propagate these variables into env variables. Also make corresponding changes to Manual/CI-CD deployment docs and necessary files.
![MicrosoftTeams-image__1_](/uploads/f1282eb51abf8fa9d7e04188130aa391/MicrosoftTeams-image__1_.png)Abhishek ChowdhryAbhishek Chowdhryhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/177Feature Change - Support auto delete of files on file-staging-area2021-07-12T08:04:51ZKrishna Nikhil VedurumudiFeature Change - Support auto delete of files on file-staging-area## Why is this change needed?
The *file-staging-area* Blob Container is used by File Service for Generating Signed URLs for Upload operations. Once the file is uploaded, the client uses File Service's metadata API which will internally ...## Why is this change needed?
The *file-staging-area* Blob Container is used by File Service for Generating Signed URLs for Upload operations. Once the file is uploaded, the client uses File Service's metadata API which will internally copy the blob contents from the *file-staging-area* to *file-persistent-area* and update the blob's metadata to Storage Service.
This results in lot of temporary files left over in the Staging area. Removing the files periodically will better the COGS.
## Proposal
Use life-cycle management policies to recycle old files. The number of days after which the data should be deleted must be configurable.
The Auto delete of Files should also be an Opt-In feature.
## Acceptance Criteria
* [x] Design Feature to ensure can be implemented with a non breaking change.
* [x] Update Storage Module
* [x] Ensure all Module Unit Tests Pass
* [x] Ensure all Template Unit Tests and Integration Tests Pass. Make sure the change will be applicable only to File-Staging-Area.
* [x] Update all required documentationNidhi JainNidhi Jainhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/176Update airflow resource requests and limits2023-08-16T10:40:37ZAalekh JainUpdate airflow resource requests and limitsThe ITs for glab environment are failing because of timeout. This is due to the unavailability of enough resources to execute dag runs in case the number of dags in the airflow increases. Since this slows down the dag runs execution lead...The ITs for glab environment are failing because of timeout. This is due to the unavailability of enough resources to execute dag runs in case the number of dags in the airflow increases. Since this slows down the dag runs execution leading to timeout, hence, we need to request for more resources for the following -
1. Airflow - WebUI
2. Airflow - Worker
3. Airflow - Scheduler
Link to the MR: !331
cc: @vineethguna @kibattulAalekh JainAalekh Jain