infra-azure-provisioning issueshttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues2022-03-24T13:55:41Zhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/209Move to ARM for IaaC from Terraform2022-03-24T13:55:41ZMANISH KUMARMove to ARM for IaaC from Terraform**Topic**: `Move to ARM for IaaC from Terraform`
Being a part of Azure, ARM is the right choice for writing Infrastructure code.
**Tasks**
- [ ] Rewrite current OSDU Azure Infra code in ARM
- [ ] Adhere to feature flags in OSDU Azure
-...**Topic**: `Move to ARM for IaaC from Terraform`
Being a part of Azure, ARM is the right choice for writing Infrastructure code.
**Tasks**
- [ ] Rewrite current OSDU Azure Infra code in ARM
- [ ] Adhere to feature flags in OSDU Azure
- [ ] Automate scripts for deploying service charts
- [ ] Update documentation for creating an env
- [ ] Update release process
- [ ] Create tooling / documentation that helps in migrating users from terraform based Infrastructure to ARM based infrastructure.
- [ ] Unit Tests
**TODO**
- [ ] Reuse parameters files to deploy ARM templates (similar to custom.tfvars)M11 - Release 0.14Madhur Tanwani [Microsoft]Bharathi SelvarajPrashanth KKrishnan GanesanMadhur Tanwani [Microsoft]https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/208Platform Validation Env2021-12-27T11:53:08ZMANISH KUMARPlatform Validation Env**Env**: `PLATFORM VALIDATION / QA`
Preshipping community requires maintenance of QA env where test suite [runs](https://community.opengroup.org/osdu/platform/testing).
Community guidelines for [env](https://community.opengroup.org/osd...**Env**: `PLATFORM VALIDATION / QA`
Preshipping community requires maintenance of QA env where test suite [runs](https://community.opengroup.org/osdu/platform/testing).
Community guidelines for [env](https://community.opengroup.org/osdu/governance/project-management-committee/-/wikis/Environments)
## Tasks:
- [ ] Refresh token for the env and provide support to the community
- [ ] Maintain the env, keeping it updated with latest changes in Glab
- [ ] Test cases should have a pass rate of more than 99%Madhur Tanwani [Microsoft]Bharathi SelvarajPrashanth KKrishnan GanesanMadhur Tanwani [Microsoft]https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/188OSDU Azure infra setup2021-11-18T15:13:03ZArvind BhojOSDU Azure infra setupCompleted OSDU Azure infra setup but Unit test for service_resources failing with the following error:
unit.go:143: Plan unexpectedly had 79 resources instead of 92
Followed the steps in the following link and all the tearrforms comple...Completed OSDU Azure infra setup but Unit test for service_resources failing with the following error:
unit.go:143: Plan unexpectedly had 79 resources instead of 92
Followed the steps in the following link and all the tearrforms completed successfully:
[](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/infra/templates/osdu-r3-mvp)https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/151Event Grid Role Assignment for Webhook.2021-06-23T09:27:44ZKomal MakkarEvent Grid Role Assignment for Webhook.
<!--- Provide a general summary of the issue in the Title above -->
# Priority: High
## Expected Behavior
<!--- Tell us what should happen -->
Register Service should be able to create a subscriber on Event Grid Topic.
## Current Beh...
<!--- Provide a general summary of the issue in the Title above -->
# Priority: High
## Expected Behavior
<!--- Tell us what should happen -->
Register Service should be able to create a subscriber on Event Grid Topic.
## Current Behavior
<!--- Tell us what happens instead of the expected behavior -->
EG errors out during the creation of subscribers is blocked.
## Possible Solution
<!--- Not obligatory, but suggest a fix/reason for the bug, -->
https://docs.microsoft.com/en-us/azure/event-grid/secure-webhook-delivery
## Steps to Reproduce
<!--- Provide a link to a live example or an unambiguous set of steps to -->
<!--- reproduce this bug. Include code to reproduce, if relevant -->
Run ITs on Notification or register. Examine the logs.
## Impact
<!--- How has this issue affected you? What are you trying to accomplish? -->
<!--- Providing context helps us come up with a solution that is most useful in the real world -->
#### Environments impacted
All the environments that we have been impacted.
#### Release impacted
OSDU M5 tagging is waiting for this fix.
#### Services impacted
Register, Notification
#### Scenario impacted
Creation of a new subscriber to a topic.
<!--- Provide a general summary of the issue in the Title above -->
## Detailed Description
<!--- Provide a detailed description of the change or addition you are proposing -->
Due to security issues, EG updated the API behavior and hence all the environments got impacted. For more details, refer [this](https://docs.microsoft.com/en-us/azure/event-grid/secure-webhook-delivery)Komal MakkarKomal Makkarhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/142AAD Graph api going to be retired in 2022 and this is a pre-requisite for inf...2021-05-04T09:29:59ZNitin-slbAAD Graph api going to be retired in 2022 and this is a pre-requisite for infrastructure deployment.Terraform code to setup the infrastructure for Core services has a pre-requisite on AAD Graph api, which are going to be retired in 2022. Terraform code should be updated to use Microsoft Graph Apis instead or find an alternate approach ...Terraform code to setup the infrastructure for Core services has a pre-requisite on AAD Graph api, which are going to be retired in 2022. Terraform code should be updated to use Microsoft Graph Apis instead or find an alternate approach to satisfy the dependency for infra deployment.https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/139Issue - Cosmos DB Limitations || Shared throughputs2021-06-23T09:30:15ZKrishna Nikhil VedurumudiIssue - Cosmos DB Limitations || Shared throughputsCosmos DB's shared throughput cannot be stretched beyond 25 collections. Quoting Azure documentation ->
> Azure Cosmos DB accounts using shared database throughput are now limited to 25 collections per database. This will allow for bett...Cosmos DB's shared throughput cannot be stretched beyond 25 collections. Quoting Azure documentation ->
> Azure Cosmos DB accounts using shared database throughput are now limited to 25 collections per database. This will allow for better throughput sharing across collections. Create additional databases with shared throughput and add more collections or add collections to the same database with dedicated throughput.
In OSDU we are currently at 25 collections. So, we have already hit the limitation.
This would require us to re-organize our collections to multiple databases so that they can scale in a better way.Krishna Nikhil VedurumudiKrishna Nikhil Vedurumudihttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/138Feature Change - Add tags create and modified timestamps as tags to azure res...2021-06-23T09:22:25ZKrishna Nikhil VedurumudiFeature Change - Add tags create and modified timestamps as tags to azure resourcesFor an azure resource, the activity logs are available only for 90 days.
Post that there is no way to find out the "Creation Date" for the resource.
Sometimes creation date matters w.r.t availability of features. For eg: All the Cosmo...For an azure resource, the activity logs are available only for 90 days.
Post that there is no way to find out the "Creation Date" for the resource.
Sometimes creation date matters w.r.t availability of features. For eg: All the Cosmos DB instances that have created post a certain date have default partition key size more than 2k bytes. Prior to that the default value was 100 bytes.
Quoting Azure support team
> Unless you add a tag with the creation date, the only way to get a creation date is to use Azure Activity Log to search for the creation operation of the resource. These logs are only saved for 90 days, so if the resource was created more than 90 days ago, there is no way to find the creation date.Krishna Nikhil VedurumudiKrishna Nikhil Vedurumudihttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/132Upgrade AGIC to 1.4.0 to support Health Probe annotation2021-06-23T04:46:15ZDaniel SchollUpgrade AGIC to 1.4.0 to support Health Probe annotationApplication Gateway Ingress Controller has updated to version 1.4.0 which enables [Health Probe Annotations](https://azure.github.io/application-gateway-kubernetes-ingress/annotations/#health-probe-status-codes).
The feature adds ingres...Application Gateway Ingress Controller has updated to version 1.4.0 which enables [Health Probe Annotations](https://azure.github.io/application-gateway-kubernetes-ingress/annotations/#health-probe-status-codes).
The feature adds ingress annotations to customize health probes.Ankit Sharma [Microsoft]Ankit Sharma [Microsoft]https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/117Feature Change - Enable Pod to Pod transport security2021-06-14T04:26:42ZDaniel SchollFeature Change - Enable Pod to Pod transport securityCurrent implementation terminates and ssl offloads transport security at the Load Balancer. Transport Security should exist all the way to the Kubernetes Pod and between Pods.
OSDU Security - Universal Encryption - B2
Acceptance Crite...Current implementation terminates and ssl offloads transport security at the Load Balancer. Transport Security should exist all the way to the Kubernetes Pod and between Pods.
OSDU Security - Universal Encryption - B2
Acceptance Criteria
---
1. A design decision should be made on the best way to handle this feature.
2. Infrastructure/Helm automation should automatically configure and enable this feature.
3. Service Helm charts should be changed to move to httpshttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/93Ingestion Framework Onboarding Service2021-06-14T04:26:41ZSumra ZafarIngestion Framework Onboarding Service**Service name**: `Ingestion Framework Service`
The following steps must be completed for a service to onboard with OSDU on Azure. Additionally, please add the `Service Onboarding` tag to this issue when it is created.
For more informa...**Service name**: `Ingestion Framework Service`
The following steps must be completed for a service to onboard with OSDU on Azure. Additionally, please add the `Service Onboarding` tag to this issue when it is created.
For more information, visit our service onboarding documentation [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-onboarding.md).
## Steps:
**Infrastructure and Initial Requirements**
- [x] Add any additional Azure cloud infrastructure (Cosmos containers, Storage containers, fileshares, etc.) to the Terraform template. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/infra/templates/osdu-r3-mvp). Note that if the infrastructure is a part of the data-partition template, you may need to add secrets to the keyvault that are partition specific; if doing so, update the createPartition REST request to include the keys that you have added so they are accessible in service code. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/rest/partition.http#L48)
- [x] Create an ingress point for the service. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/appgw-ingress.yaml)
- [x] Add any test data that is required for the service integration tests. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/test_data)
- [x] Update `upload-data.py` to upload any new test data files you created. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/test_data/upload-data.py).
- [x] Update the integration tester with any entitlements required to test the service. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/test_data/user_info_1.json)
- [x] Add in any new secrets that the service needs to run. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/kv-secrets.yaml)
- [x] Create environment variable script to generate .yaml files to be used with Intellij [EnvFile](https://plugins.jetbrains.com/plugin/7861-envfile) plugin and .envrc files to be used with [direnv](https://direnv.net/). [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/variables)
**Gitlab Code and Documentation**
- [x] Complete the service code such that it passes all integration tests locally. There is some documentation on starting off implementing an Azure provider. [Link](./gitlab-service-readme-template.md)
- [x] Create helm charts for service. The charts for each service are located in the `devops/azure` directory. You can look at charts from other services as a model. The charts will be nearly identical except for the different environment variables, values, etc each service needs to run. [Link](./gitlab-service-guide.md)
- [x] Implement Istio for the service if this has not already been done. Here is an example MR that shows what steps are required. [Link](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/64)
- [x] Create an Istio auth policy in the `devops/azure/chart/templates` directory. Here is an example of an Istio auth policy that is generic and can be used by other services. [Link](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/devops/azure/chart/templates/azure-istio-auth-policy.yaml)
- [x] Add any variables that are required for the service integration tests to the Azure CI-CD file. [Link](https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/blob/master/cloud-providers/azure.yml)
- [ ] Verify that the README for the Azure provider correctly and clearly describes how to run and test the service. There is a README template to help. [Link](./gitlab-service-readme-template.md)
- [x] Push any changes and verify that the Gitlab pipeline is passing in master.
**Development and Demo Azure Devops Pipelines**
- [ ] Create development ADO pipeline at `devops/azure/development-pipeline.yml` in the service repo.
- [ ] Verify development pipeline passes in ADO.
- [ ] Create Demo ADO pipeline at `devops/azure/pipeline.yml` in the service repo.
- [ ] Verify demo pipeline is passing in ADO.
**User Documentation**
- [ ] Add the service to the mirror pipeline instructions. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/code-mirroring.md)
- [ ] Add the service to the manual deployment instructions. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/charts)
- [ ] Add any required variables to the already existing variable group instructions for automated deployment. You should know if any variables need to be added to existing variable groups from creating the development and demo pipelines. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [ ] Add a variable group `Azure Service Release - $SERVICE_NAME` to the documentation. You should know what values to set for this variable group from creating the development and demo pipelines. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [ ] Add a step for creating the service pipeline at the bottom of the service-automation page. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [ ] Create a rest script with sample calls to the service for users. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/rest)M6 - Release 0.9 - removeSumra ZafarSumra Zafarhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/69Feature change - Helm values override support - Support of overriding helm co...2021-06-23T09:18:00ZKiran VeerapaneniFeature change - Helm values override support - Support of overriding helm config in airflow deployment**Why is this change needed**
Customers requested to provide a way to override default helm-config.yaml that is checkedin into the repository. Airflow is deployed in AKS cluster using helm charts. For airflow helm charts configuration l...**Why is this change needed**
Customers requested to provide a way to override default helm-config.yaml that is checkedin into the repository. Airflow is deployed in AKS cluster using helm charts. For airflow helm charts configuration like database to connect to, redis, replicas etc., are provided using helm-config.yaml. This helm config is checkedin into master repository which is basic and tuned for gitlab and dev environment. Using the override feature customer can provide a override configuration file which will override default configuration provided in the repository. This is optional and customer can deploy the existing templates without any change.
**Current Behavior**
Currently customers are changing the helm-config.yaml file directly once forked from infrastructure repository. This is causing conflicts when the helm-config.yaml file is updated in the infrastructure repository and manual intervention is needed to resolve this.
**Expected Behavior**
Provide a way for customers to provide a override file without worrying about handling merge conflicts.
**Design proposal**
ADO pipelines are modified to take extra values files "helm-config-override.yaml" which is always empty in the infrastructure repository. ADO pipelines are modified to take two input value files "helm-config.yaml" and "helm-config-override.yaml". After this change customers who want to deploy airflow can give overriding values through "helm-config-override.yaml". As this file is always empty there won't be any conflicts.
**Acceptance Criteria**
- Design Feature to ensure can be implemented with a non breaking change.
- Update all required documentationKishore BattulaKishore Battulahttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/23Need to configure App Gateway to return required security response headers2021-06-23T04:32:08ZSherman YangNeed to configure App Gateway to return required security response headersCurrently, the mandatory HTTP response headers are not returned when a path is not found or JWT header is missing or invalid in a request. Need to return the following mandatory headers from App Gateway:
"X-XSS-Protection"
"X-Content-Typ...Currently, the mandatory HTTP response headers are not returned when a path is not found or JWT header is missing or invalid in a request. Need to return the following mandatory headers from App Gateway:
"X-XSS-Protection"
"X-Content-Type-Options"
"X-Frame-Options"
"Cache-Control
"Expires"
"Strict-Transport-Security"
This feature may be required or custom rules can be added.Vivek OjhaVivek Ojhahttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/14Remove unnecessary secrets in kv-secrets.yaml that relate to a specific data ...2022-08-23T10:47:31ZDaniel SchollRemove unnecessary secrets in kv-secrets.yaml that relate to a specific data partition.Originally the kv-secrets.yaml file was holding secrets that were used for services to connect to Cosmos and Storage. Now that Data Partition Service has been fully implemented and all charts migrated to use Data partition these secrets...Originally the kv-secrets.yaml file was holding secrets that were used for services to connect to Cosmos and Storage. Now that Data Partition Service has been fully implemented and all charts migrated to use Data partition these secrets should be removed so that the dependency of a datapartition called opendes is not required.
```yaml
- secretName: storage
type: Opaque
data:
- objectName: "opendes-storage"
key: storage-account
- objectName: "opendes-storage-key"
key: storage-key
- secretName: cosmos
type: Opaque
data:
- objectName: "opendes-cosmos-endpoint"
key: cosmos-endpoint
- objectName: "opendes-cosmos-connection"
key: cosmos-connection
- objectName: "opendes-cosmos-primary-key"
key: cosmos-primary-key
- secretName: servicebus
type: Opaque
data:
- objectName: "opendes-sb-namespace"
key: servicebus-namespace
- objectName: "opendes-sb-connection"
key: servicebus-connection
```Kishore BattulaVibhuti Sharma [Microsoft]Kishore Battula