infra-azure-provisioning issueshttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues2021-06-14T04:26:38Zhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/28Some sensitive values shown in TF output.2021-06-14T04:26:38ZDzmitry_Paulouski (slb)Some sensitive values shown in TF output.The following sensitive values shows in TF output: "primary_master_key", "connection_strings", "redis_primary_access_key".
It's a good idea to hide these values in output by adding "sensitive = true" flag.The following sensitive values shows in TF output: "primary_master_key", "connection_strings", "redis_primary_access_key".
It's a good idea to hide these values in output by adding "sensitive = true" flag.Sprint 10/25 - 10/31Daniel SchollDaniel Schollhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/18Disable TLS1.0, 1.1 on weak cyphers2021-06-14T04:26:37ZDzmitry_Paulouski (slb)Disable TLS1.0, 1.1 on weak cyphersDisable old TLS versions and weak cyphers according to www.ssllabs.comDisable old TLS versions and weak cyphers according to www.ssllabs.comSprint 10/25 - 10/31Daniel SchollDzmitry_Paulouski (slb)Daniel Schollhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/17Ignore_changes block in app_gw resource doesn't work for ssl_certificate2021-06-14T04:26:37ZDzmitry Paulouski (slb)Ignore_changes block in app_gw resource doesn't work for ssl_certificateLooks like it’s the same error as in this [issue](https://github.com/terraform-providers/terraform-provider-azurerm/issues/6330).
It was fixed in 2.33.0 version of “azurerm” provider, we are using 2.29. We’ve tested the pipeline with th...Looks like it’s the same error as in this [issue](https://github.com/terraform-providers/terraform-provider-azurerm/issues/6330).
It was fixed in 2.33.0 version of “azurerm” provider, we are using 2.29. We’ve tested the pipeline with the recommended version and all works well. @danielscholl please take a look.
![error](/uploads/0168eab2e2552ee79868333beaddd1cb/error.jpg)Sprint 10/25 - 10/31https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/13Assign Redis namespace number for services2021-06-14T04:26:36Zashley kelhamAssign Redis namespace number for servicesMultiple services are using the same Redis instance deployed.
There is possibility for conflict between services with keys between different services. This could cause data leakge, corruption etc.
To prevent this we want different serv...Multiple services are using the same Redis instance deployed.
There is possibility for conflict between services with keys between different services. This could cause data leakge, corruption etc.
To prevent this we want different services to use different Redis namespaces (no. 0-15).
The infrastructure can assign the Redis database number for individual services to use so each service can just pull the assigned number.
This keeps a centralized view on the separation, lets us see when we have hit capacity (15) and means the service has less work to do to make sure they maintain a separation with one another.Sprint 10/25 - 10/31Daniel SchollDaniel Schollhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/7Secrets in secret store not getting updated2021-06-14T04:26:36ZKiran VeerapaneniSecrets in secret store not getting updatedSecrets in secret store is not getting updated, After the secret is created, Not able to perform add, update and delete keys to the secret.Secrets in secret store is not getting updated, After the secret is created, Not able to perform add, update and delete keys to the secret.Sprint 10/25 - 10/31https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/6Update Documentation and Scripts For Deploying Infrastructure and Services2021-06-14T04:26:36ZJasonUpdate Documentation and Scripts For Deploying Infrastructure and ServicesThe documentation and deployment scripts need to be updated to provide detailed instructions and assistance on how to deploy OSDU on Azure so that customers and developers can better understand the infrastructure and how they can stand u...The documentation and deployment scripts need to be updated to provide detailed instructions and assistance on how to deploy OSDU on Azure so that customers and developers can better understand the infrastructure and how they can stand up an environment for themselves:
- [x] Update common resources deployment [script](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/infra/templates/osdu-r3-mvp/common_prepare.sh) and verify that it correctly provisions required resources for deployment.
- [x] Update [documentation](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/pipeline-setup.md) to deploy infrastructure using pipelines.
- [x] Update [documentation](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/infra/templates/osdu-r3-mvp/README.md) to deploy infrastructure manually.
- [x] Update [documentation](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/README.md) to deploy services manually.
- [x] Update [documentation](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/pipeline-setup.md#deploy-osdu-services) to deploy services using pipelines.
---
__Acceptance Criteria__
- [x] Manually verify that documentation is all clear and concise.
- [x] Deploy OSDU infrastructure and services manually by following instructions.
- [x] Deploy OSDU infrastructure and service using pipelines by following instructions.
---Sprint 10/25 - 10/31JasonJasonhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/42Default Infrastructure Sizing is to small to support more then just functiona...2021-06-14T04:26:38ZDaniel SchollDefault Infrastructure Sizing is to small to support more then just functionality testingThe default sizing of the infrastructure and service replicas needs to be increased slightly to support a more common usage pattern for a default OSDU usage.
__Minimum Supported Storage Rate__
- Concurrency: 10 threads.
- Batch Size: ...The default sizing of the infrastructure and service replicas needs to be increased slightly to support a more common usage pattern for a default OSDU usage.
__Minimum Supported Storage Rate__
- Concurrency: 10 threads.
- Batch Size: 150.
- Records: 50,000.
- Time: ~ 8:00 minutes.
### Default Configuration *New
__AKS Cluster__
- Node Count: 5
- Server Size: Standard_E4s_v3. (4x32)
- Subnet Size: /23
- AKS Version: 1.18.8
__Cosmos DB__
- Throughput Auto Scale Shared RU’s: 12000
#### OSDU Service Replicas
__Partition:__
- Replicas: 2
- Requests CPU: 100m
- Limits CPU: 300m
__Entitlements:__
- Replicas: 2
- Requests CPU: 100m
- Limits CPU: 300m
__Legal:__
- Replicas: 2
- Requests CPU: 100m
- Limits CPU: 300m
__Storage:__
- Replicas: 10
- Requests CPU: 100m
- Limits CPU: 800m
__Indexer-Queue:__
- pollingInterval: 30
- cooldownPeriod: 60
- minReplicaCount: 1
- maxReplicaCount: 10
__Indexer:__
- Requests CPU: 100m
- Limits CPU: 500m
- HPA:
- minReplicas: 2
- maxReplicas: 5
- cpu:averageUtilization: 75
__Ingress Annotations__
- appgw.ingress.kubernetes.io/request-timeout: "300"
- appgw.ingress.kubernetes.io/connection-draining: "true"
- appgw.ingress.kubernetes.io/connection-draining-timeout: "30"Sprint 11/1 - 11/7Daniel SchollDaniel Schollhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/41Istio Auth - All services need to open up /actuator/health for liveness probes2021-06-14T04:26:38ZDaniel SchollIstio Auth - All services need to open up /actuator/health for liveness probesEach service has a health probe but RBAC doesn't allow it currently to be available. Istio Auth Policies need to be updated to allow access to this URL endpoint.Each service has a health probe but RBAC doesn't allow it currently to be available. Istio Auth Policies need to be updated to allow access to this URL endpoint.Sprint 11/1 - 11/7Daniel SchollDaniel Schollhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/40Onboard File Service with Infrastructure2022-08-23T10:47:32ZJasonOnboard File Service with Infrastructure**Service name**: `File`
The following steps must be completed for a service to onboard with OSDU on Azure. Additionally, please add the `Service Onboarding` tag to this issue when it is created.
For more information, visit our service...**Service name**: `File`
The following steps must be completed for a service to onboard with OSDU on Azure. Additionally, please add the `Service Onboarding` tag to this issue when it is created.
For more information, visit our service onboarding documentation [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-onboarding.md).
## Steps:
**Infrastructure and Initial Requirements**
- [x] Add any additional Azure cloud infrastructure (Cosmos containers, Storage containers, fileshares, etc.) to the Terraform template. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/infra/templates/osdu-r3-mvp). Note that if the infrastructure is a part of the data-partition template, you may need to add secrets to the keyvault that are partition specific; if doing so, update the createPartition REST request to include the keys that you have added so they are accessible in service code. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/rest/partition.http#L48)
- [x] Create an ingress point for the service. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/appgw-ingress.yaml)
- [x] Add any test data that is required for the service integration tests. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/test_data)
- [x] Update `upload-data.py` to upload any new test data files you created. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/test_data/upload-data.py).
- [x] Update the integration tester with any entitlements required to test the service. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/test_data/user_info_1.json)
- [x] Add in any new secrets that the service needs to run. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/kv-secrets.yaml)
- [x] Create environment variable script to generate .yaml files to be used with Intellij [EnvFile](https://plugins.jetbrains.com/plugin/7861-envfile) plugin and .envrc files to be used with [direnv](https://direnv.net/). [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/variables)
**Gitlab Code and Documentation**
- [x] Complete the service code such that it passes all integration tests locally. There is some documentation on starting off implementing an Azure provider. [Link](./gitlab-service-readme-template.md)
- [x] Create helm charts for service. The charts for each service are located in the `devops/azure` directory. You can look at charts from other services as a model. The charts will be nearly identical except for the different environment variables, values, etc each service needs to run. [Link](./gitlab-service-guide.md)
- [x] Implement Istio for the service if this has not already been done. Here is an example MR that shows what steps are required. [Link](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/64)
- [x] Create an Istio auth policy in the `devops/azure/chart/templates` directory. Here is an example of an Istio auth policy that is generic and can be used by other services. [Link](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/devops/azure/chart/templates/azure-istio-auth-policy.yaml)
- [x] Add any variables that are required for the service integration tests to the Azure CI-CD file. [Link](https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/blob/master/cloud-providers/azure.yml)
- [x] Verify that the README for the Azure provider correctly and clearly describes how to run and test the service. There is a README template to help. [Link](./gitlab-service-readme-template.md)
- [x] Push any changes and verify that the Gitlab pipeline is passing in master.
**Development and Demo Azure Devops Pipelines**
- [x] Create development ADO pipeline at `devops/azure/development-pipeline.yml` in the service repo.
- [x] Verify development pipeline passes in ADO.
- [x] Create Demo ADO pipeline at `devops/azure/pipeline.yml` in the service repo.
- [x] Verify demo pipeline is passing in ADO.
**User Documentation**
- [x] Add the service to the mirror pipeline instructions. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/code-mirroring.md)
- [x] Add the service to the manual deployment instructions. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/charts)
- [x] Add any required variables to the already existing variable group instructions for automated deployment. You should know if any variables need to be added to existing variable groups from creating the development and demo pipelines. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [x] Add a variable group `Azure Service Release - $SERVICE_NAME` to the documentation. You should know what values to set for this variable group from creating the development and demo pipelines. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [x] Add a step for creating the service pipeline at the bottom of the service-automation page. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)Sprint 11/1 - 11/7https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/39Bug - Developer Environment Variables don't support Redis Cache Database2021-06-14T04:26:38ZDaniel SchollBug - Developer Environment Variables don't support Redis Cache DatabaseRedis Cache Database Numbers are assigned and consumed by a [config map](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/redis-map.yaml).
When a...Redis Cache Database Numbers are assigned and consumed by a [config map](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/redis-map.yaml).
When a developer is using the system those values are not being translated into Environment Variable settings.Sprint 11/1 - 11/7Daniel SchollDaniel Schollhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/34Tasks sharing in workflow -- Infrastructure requirements (storage container)2021-06-14T04:26:38ZAalekh JainTasks sharing in workflow -- Infrastructure requirements (storage container)Currently there is no container to share the data between the tasks in workflow. Hence a new container is required for the same.
storage container name: `workflow-tasks-sharing`
__Why is this change needed?__
To store the data that is...Currently there is no container to share the data between the tasks in workflow. Hence a new container is required for the same.
storage container name: `workflow-tasks-sharing`
__Why is this change needed?__
To store the data that is shared between the tasks in workflow.
__Current behavior__
No container available to store such data.
__Expected behavior__
A new storage container will be created and we will be able to store the data that will be shared across tasks in the workflow.Sprint 11/1 - 11/7https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/32Tasks sharing in workflow -- Infrastructure requirements (cosmos collection)2021-06-14T04:26:38ZAalekh JainTasks sharing in workflow -- Infrastructure requirements (cosmos collection)We need to provide signed tokens for blobs present in the storage container (#34) depending on the `workflowid` and `runid` (which will be used to share data between tasks in workflow). In order to keep a track of this, we need to store ...We need to provide signed tokens for blobs present in the storage container (#34) depending on the `workflowid` and `runid` (which will be used to share data between tasks in workflow). In order to keep a track of this, we need to store the metadata which comprises of what all workflowids and runids have been used to generate these signed tokens along with some other information such as "createdBy", "createdAt", "state" etc. This cosmos collection will store the metadata.
collection-name: `WorkflowTasksSharingInfo`
__Why is this change needed?__
To store the metadata such as "workflowid", "runid", "state" etc. in Cosmos collection for the storage container for which the SAS tokens will be generated. The usage of this collection is not limited to storing the information relted to SAS token generation and might get extended later.
__Current behavior__
No such collection available to store information related to workflow tasks.
__Expected behavior__
A new cosmos collection is created and we will be able to store the metadata related to workflow tasks in this cosmos collection.Sprint 11/1 - 11/7https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/30Add roles and namespace to airflow chart to create pods using pod operator2021-06-14T04:26:38ZKiran VeerapaneniAdd roles and namespace to airflow chart to create pods using pod operatorCreate new namespace airflow.
Add roles to the service account such that kubernetes pod operator can create pods in this namespaceCreate new namespace airflow.
Add roles to the service account such that kubernetes pod operator can create pods in this namespaceSprint 11/1 - 11/7https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/21Onboard Delivery Service with Infrastructure2022-08-23T10:47:31ZJasonOnboard Delivery Service with Infrastructure**Service name**: `INSERT SERVICE NAME HERE`
The following steps must be completed for a service to onboard with OSDU on Azure. Additionally, please add the `Service Onboarding` tag to this issue when it is created.
For more informatio...**Service name**: `INSERT SERVICE NAME HERE`
The following steps must be completed for a service to onboard with OSDU on Azure. Additionally, please add the `Service Onboarding` tag to this issue when it is created.
For more information, visit our service onboarding documentation [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-onboarding.md).
## Steps:
**Infrastructure and Initial Requirements**
- [x] Add any additional Azure cloud infrastructure (Cosmos containers, Storage containers, fileshares, etc.) to the Terraform template. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/infra/templates/osdu-r3-mvp). Note that if the infrastructure is a part of the data-partition template, you may need to add secrets to the keyvault that are partition specific; if doing so, update the createPartition REST request to include the keys that you have added so they are accessible in service code. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/rest/partition.http#L48)
- [x] Create an ingress point for the service. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/appgw-ingress.yaml)
- [x] Add any test data that is required for the service integration tests. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/test_data)
- [x] Update `upload-data.py` to upload any new test data files you created. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/test_data/upload-data.py).
- [x] Update the integration tester with any entitlements required to test the service. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/test_data/user_info_1.json)
- [x] Add in any new secrets that the service needs to run. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/kv-secrets.yaml)
- [x] Create environment variable script to generate .yaml files to be used with Intellij [EnvFile](https://plugins.jetbrains.com/plugin/7861-envfile) plugin and .envrc files to be used with [direnv](https://direnv.net/). [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/variables)
**Gitlab Code and Documentation**
- [x] Complete the service code such that it passes all integration tests locally. There is some documentation on starting off implementing an Azure provider. [Link](./gitlab-service-readme-template.md)
- [x] Create helm charts for service. The charts for each service are located in the `devops/azure` directory. You can look at charts from other services as a model. The charts will be nearly identical except for the different environment variables, values, etc each service needs to run. [Link](./gitlab-service-guide.md)
- [x] Implement Istio for the service if this has not already been done. Here is an example MR that shows what steps are required. [Link](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/64)
- [x] Create an Istio auth policy in the `devops/azure/chart/templates` directory. Here is an example of an Istio auth policy that is generic and can be used by other services. [Link](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/devops/azure/chart/templates/azure-istio-auth-policy.yaml)
- [x] Add any variables that are required for the service integration tests to the Azure CI-CD file. [Link](https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/blob/master/cloud-providers/azure.yml)
- [x] Verify that the README for the Azure provider correctly and clearly describes how to run and test the service. There is a README template to help. [Link](./gitlab-service-readme-template.md)
- [x] Push any changes and verify that the Gitlab pipeline is passing in master.
**Development and Demo Azure Devops Pipelines**
- [x] Create development ADO pipeline at `devops/azure/development-pipeline.yml` in the service repo.
- [x] Verify development pipeline passes in ADO.
- [x] Create Demo ADO pipeline at `devops/azure/pipeline.yml` in the service repo.
- [x] Verify demo pipeline is passing in ADO.
**User Documentation**
- [x] Add the service to the mirror pipeline instructions. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/code-mirroring.md)
- [x] Add the service to the manual deployment instructions. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/charts)
- [x] Add any required variables to the already existing variable group instructions for automated deployment. You should know if any variables need to be added to existing variable groups from creating the development and demo pipelines. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [x] Add a variable group `Azure Service Release - $SERVICE_NAME` to the documentation. You should know what values to set for this variable group from creating the development and demo pipelines. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [x] Add a step for creating the service pipeline at the bottom of the service-automation page. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)Sprint 11/1 - 11/7JasonJasonhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/37Wellbore DMS -- Chart Requirements2021-06-14T04:26:38ZDaniel SchollWellbore DMS -- Chart RequirementsThe following chart requirements have been identified as necessary to onboard.
- [x] Public Ingress. Path: /api/os-wellbore-ddms/*
Decisions by the development teams working on Wellbore around namespace isolation have been made to de...The following chart requirements have been identified as necessary to onboard.
- [x] Public Ingress. Path: /api/os-wellbore-ddms/*
Decisions by the development teams working on Wellbore around namespace isolation have been made to defer isolation to a later date and attempt to leverage the OSDU namespace.
Acceptance Criteria
---
This is a phased rollout for the service. Manual verification is all that is required for completion.Sprint 11/8 - 11/14Daniel SchollDaniel Scholl2020-11-07https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/33Wellbore DMS -- Infrastructure Requirements2021-06-14T04:26:38ZDaniel SchollWellbore DMS -- Infrastructure Requirements
The following infrastructure requirements have been identified as necessary to onboard.
- [x] Data Partition Storage Container. Name: wdms-osdu
Acceptance Criteria
---
This is a phased rollout for the service. Manual verification o...
The following infrastructure requirements have been identified as necessary to onboard.
- [x] Data Partition Storage Container. Name: wdms-osdu
Acceptance Criteria
---
This is a phased rollout for the service. Manual verification of container and path is all that is required for completion.Sprint 11/8 - 11/14Daniel SchollDaniel Scholl2020-11-07https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/70Unit Service Onboarding Fix doc2021-06-14T04:26:40ZFabien BosquetUnit Service Onboarding Fix docThis is a follow-up on Issue #55 (Unit Service Onboarding)
The git repository adress and the ADO library name for this service are incorrects.
Since I am unable to create a new branch to propose a new pull request, I have attached a pat...This is a follow-up on Issue #55 (Unit Service Onboarding)
The git repository adress and the ADO library name for this service are incorrects.
Since I am unable to create a new branch to propose a new pull request, I have attached a patch [0001-Fix-unit-service-doc.patch](/uploads/be9fe3e6d91dbe0f6df7b524f9082a71/0001-Fix-unit-service-doc.patch)Decemberhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/57New Service Bus topic subscribing to Event Grid Topic for WKS2021-06-14T04:26:39ZKomal MakkarNew Service Bus topic subscribing to Event Grid Topic for WKS## Type
<!-- Please choose the type of ticket. -->
- [x] Feature Request
- [ ] Bug Report
## Priority
- [x] High
- [ ] Medium
- [ ] Low
------------------------
------------------------
## Feature Request
__Why is this change nee...## Type
<!-- Please choose the type of ticket. -->
- [x] Feature Request
- [ ] Bug Report
## Priority
- [x] High
- [ ] Medium
- [ ] Low
------------------------
------------------------
## Feature Request
__Why is this change needed?__
WKS service consumes the storage record changed notification via Service Bus R2. When we move to R3 infrastructure, we will have to make WKS consume from Service Bus R3. This change will have to make sure for all the environments, no notifications are lost. All notifications from SB R3 and SB R2 should be consumed by WKS, in all environments. To prevent that, we can make WKS consume from Service Bus R3 before it gets in production.
__Current behavior__
Storage publishes to Service Bus R2 and WKS has a subscriber listening to the notifications.
__Expected behavior__
Introduction of **Service Bus R3** in the following fashion
![image](/uploads/d53b66dbb49637d2cff12d6b7564cfb8/image.png)
--------------------------
--------------------------
## Other information
<!-- Any other information that is important to this PR such as screenshots of how the component looks before and after the change. -->
The above can be broken down into the following tasks.
1. Service Bus topic will be created.
2. Service Bus topic will subscribe to Event Grid topic.
3. Service bus topic will have a subscriber which WKS will listen to.
To be discussed:
1. TTL for messages, DLQ, and other characteristics of the Service bus Topic.
2. Any special permissions/roles to be granted.
```DecemberKomal MakkarKomal Makkarhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/56CRS Catalog Service Onboarding2022-08-23T10:47:29ZNicholas KarskyCRS Catalog Service Onboarding**Service name**: `CRS Catalog`
The following steps must be completed for a service to onboard with OSDU on Azure. Additionally, please add the `Service Onboarding` tag to this issue when it is created.
For more information, visit our ...**Service name**: `CRS Catalog`
The following steps must be completed for a service to onboard with OSDU on Azure. Additionally, please add the `Service Onboarding` tag to this issue when it is created.
For more information, visit our service onboarding documentation [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-onboarding.md).
## Steps:
**Infrastructure and Initial Requirements**
- [x] Add any additional Azure cloud infrastructure (Cosmos containers, Storage containers, fileshares, etc.) to the Terraform template. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/infra/templates/osdu-r3-mvp). Note that if the infrastructure is a part of the data-partition template, you may need to add secrets to the keyvault that are partition specific; if doing so, update the createPartition REST request to include the keys that you have added so they are accessible in service code. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/rest/partition.http#L48)
- [x] Create an ingress point for the service. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/appgw-ingress.yaml)
- [x] Add any test data that is required for the service integration tests. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/test_data)
- [x] Update `upload-data.py` to upload any new test data files you created. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/test_data/upload-data.py).
- [x] Update the integration tester with any entitlements required to test the service. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/test_data/user_info_1.json)
- [x] Add in any new secrets that the service needs to run. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/kv-secrets.yaml)
- [x] Create environment variable script to generate .yaml files to be used with Intellij [EnvFile](https://plugins.jetbrains.com/plugin/7861-envfile) plugin and .envrc files to be used with [direnv](https://direnv.net/). [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/variables)
**Gitlab Code and Documentation**
- [x] Complete the service code such that it passes all integration tests locally. There is some documentation on starting off implementing an Azure provider. [Link](./gitlab-service-readme-template.md)
- [x] Create helm charts for service. The charts for each service are located in the `devops/azure` directory. You can look at charts from other services as a model. The charts will be nearly identical except for the different environment variables, values, etc each service needs to run. [Link](./gitlab-service-guide.md)
- [x] Implement Istio for the service if this has not already been done. Here is an example MR that shows what steps are required. [Link](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/64)
- [x] Create an Istio auth policy in the `devops/azure/chart/templates` directory. Here is an example of an Istio auth policy that is generic and can be used by other services. [Link](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/devops/azure/chart/templates/azure-istio-auth-policy.yaml)
- [x] Add any variables that are required for the service integration tests to the Azure CI-CD file. [Link](https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/blob/master/cloud-providers/azure.yml)
- [x] Verify that the README for the Azure provider correctly and clearly describes how to run and test the service. There is a README template to help. [Link](./gitlab-service-readme-template.md)
- [x] Push any changes and verify that the Gitlab pipeline is passing in master.
**Development and Demo Azure Devops Pipelines**
- [x] Create development ADO pipeline at `devops/azure/development-pipeline.yml` in the service repo.
- [x] Verify development pipeline passes in ADO.
- [x] Create Demo ADO pipeline at `devops/azure/pipeline.yml` in the service repo.
- [x] Verify demo pipeline is passing in ADO.
**User Documentation**
- [x] Add the service to the mirror pipeline instructions. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/code-mirroring.md)
- [x] Add the service to the manual deployment instructions. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/charts)
- [x] Add any required variables to the already existing variable group instructions for automated deployment. You should know if any variables need to be added to existing variable groups from creating the development and demo pipelines. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [x] Add a variable group `Azure Service Release - $SERVICE_NAME` to the documentation. You should know what values to set for this variable group from creating the development and demo pipelines. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [x] Add a step for creating the service pipeline at the bottom of the service-automation page. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [x] Create a rest script with sample calls to the service for users. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/rest)
## Setup:
1. Create an empty repo `crs-catalog-service`
2. Add a variable into `Mirror Variables`
> ADO_ORGANIZATION and ADO_PROJECT should be your actual names.
| Variable | Value |
|----------|-------|
| CRS_CATALOG_REPO | `https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/crs-catalog-service` |
3. Edit the Mirror Pipeline and add the task
```
- task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1
displayName: 'crs-catalog'
inputs:
sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service.git'
destinationGitRepositoryUri: '$(CRS_CATALOG_REPO)'
destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN)
```
4. Run the Mirror Pipeline
5. Create a Variable Group `Azure Service Release - crs catalog` with the variables:
| Variable | Value |
|----------|-------|
| MAVEN_DEPLOY_POM_FILE_PATH | `drop/provider/crs-catalog-azure/crs-catalog-aks` |
6. Create a new pipeline using the `crs-catalog-service` repo and the `/devops/azure/pipeline.yml` file of that repo.
7. Upload the [crs_catalog_v2.json](https://community.opengroup.org/osdu/platform/system/reference/crs-catalog-service/-/blob/master/data/crs_catalog_v2.json) file located in the Project data folder to the fileshare `crs` of the storage account in the service resources.
8. Execute the PipelineDecemberNicholas KarskyNicholas Karsky2020-12-19https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/55Unit Service Onboarding2022-08-23T10:47:31ZNicholas KarskyUnit Service Onboarding**Service name**: `Unit`
The following steps must be completed for a service to onboard with OSDU on Azure. Additionally, please add the `Service Onboarding` tag to this issue when it is created.
For more information, visit our service...**Service name**: `Unit`
The following steps must be completed for a service to onboard with OSDU on Azure. Additionally, please add the `Service Onboarding` tag to this issue when it is created.
For more information, visit our service onboarding documentation [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-onboarding.md).
## Steps:
**Infrastructure and Initial Requirements**
- [x] Add any additional Azure cloud infrastructure (Cosmos containers, Storage containers, fileshares, etc.) to the Terraform template. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/infra/templates/osdu-r3-mvp). Note that if the infrastructure is a part of the data-partition template, you may need to add secrets to the keyvault that are partition specific; if doing so, update the createPartition REST request to include the keys that you have added so they are accessible in service code. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/rest/partition.http#L48)
- [x] Create an ingress point for the service. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/appgw-ingress.yaml)
- [x] Add any test data that is required for the service integration tests. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/test_data)
- [x] Update `upload-data.py` to upload any new test data files you created. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/test_data/upload-data.py).
- [x] Update the integration tester with any entitlements required to test the service. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/test_data/user_info_1.json)
- [x] Add in any new secrets that the service needs to run. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/kv-secrets.yaml)
- [x] Create environment variable script to generate .yaml files to be used with Intellij [EnvFile](https://plugins.jetbrains.com/plugin/7861-envfile) plugin and .envrc files to be used with [direnv](https://direnv.net/). [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/variables)
**Gitlab Code and Documentation**
- [x] Complete the service code such that it passes all integration tests locally. There is some documentation on starting off implementing an Azure provider. [Link](./gitlab-service-readme-template.md)
- [x] Create helm charts for service. The charts for each service are located in the `devops/azure` directory. You can look at charts from other services as a model. The charts will be nearly identical except for the different environment variables, values, etc each service needs to run. [Link](./gitlab-service-guide.md)
- [x] Implement Istio for the service if this has not already been done. Here is an example MR that shows what steps are required. [Link](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/64)
- [x] Create an Istio auth policy in the `devops/azure/chart/templates` directory. Here is an example of an Istio auth policy that is generic and can be used by other services. [Link](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/devops/azure/chart/templates/azure-istio-auth-policy.yaml)
- [x] Add any variables that are required for the service integration tests to the Azure CI-CD file. [Link](https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/blob/master/cloud-providers/azure.yml)
- [x] Verify that the README for the Azure provider correctly and clearly describes how to run and test the service. There is a README template to help. [Link](./gitlab-service-readme-template.md)
- [x] Push any changes and verify that the Gitlab pipeline is passing in master.
**Development and Demo Azure Devops Pipelines**
- [x] Create development ADO pipeline at `devops/azure/development-pipeline.yml` in the service repo.
- [x] Verify development pipeline passes in ADO.
- [x] Create Demo ADO pipeline at `devops/azure/pipeline.yml` in the service repo.
- [x] Verify demo pipeline is passing in ADO.
**User Documentation**
- [x] Add the service to the mirror pipeline instructions. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/code-mirroring.md)
- [x] Add the service to the manual deployment instructions. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/charts)
- [x] Add any required variables to the already existing variable group instructions for automated deployment. You should know if any variables need to be added to existing variable groups from creating the development and demo pipelines. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [x] Add a variable group `Azure Service Release - $SERVICE_NAME` to the documentation. You should know what values to set for this variable group from creating the development and demo pipelines. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [x] Add a step for creating the service pipeline at the bottom of the service-automation page. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [x] Create a rest script with sample calls to the service for users. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/rest)
## Setup:
1. Create an empty repo `unit-service`
2. Add a variable into `Mirror Variables`
> ADO_ORGANIZATION and ADO_PROJECT should be your actual names.
| Variable | Value |
|----------|-------|
| UNIT_REPO | `https://dev.azure.com/${ADO_ORGANIZATION}/$ADO_PROJECT/_git/unit-service` |
3. Edit the Mirror Pipeline and add the task
```
- task: swellaby.mirror-git-repository.mirror-git-repository-vsts-task.mirror-git-repository-vsts-task@1
displayName: 'unit'
inputs:
sourceGitRepositoryUri: 'https://community.opengroup.org/osdu/platform/system/reference/unit-service.git'
destinationGitRepositoryUri: '$(UNIT_REPO)'
destinationGitRepositoryPersonalAccessToken: $(ACCESS_TOKEN)
```
4. Run the Mirror Pipeline
5. Create a Variable Group `Azure Service Release - unit` with the variables:
| Variable | Value |
|----------|-------|
| MAVEN_DEPLOY_POM_FILE_PATH | `drop/provider/unit-azure/unit-aks` |
6. Create a Pipeline `service-unit` against the Repo `unit-service` for file `/devops/azure/pipeline.yml`
7. Upload the [unit_catalog_v2.json](https://community.opengroup.org/osdu/platform/system/reference/unit-service/-/blob/master/data/unit_catalog_v2.json) file located in the Project data folder to the fileshare `unit` of the storage account in the service resources.
8. Execute the PipelineDecemberNicholas KarskyNicholas Karsky2020-12-19