infra-azure-provisioning issueshttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues2021-06-23T04:42:36Zhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/136Enhance entitlements service with redis2021-06-23T04:42:36ZRostislav Vatolinvatolinrp@gmail.comEnhance entitlements service with redisEntitlements service needs redis for caching mechanism. Redis will store group names, to which a user belongs.Entitlements service needs redis for caching mechanism. Redis will store group names, to which a user belongs.https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/134Airflow task logging not printing correlation-id2021-03-08T07:23:01ZKishore BattulaAirflow task logging not printing correlation-idAirflow task logs have correlation-id as None. Ideally this should be the correlation-id with which the workflow service run API is triggered.Airflow task logs have correlation-id as None. Ideally this should be the correlation-id with which the workflow service run API is triggered.Kishore BattulaKishore Battulahttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/133Update path in AppGateway Ingress Controller for File service2021-03-05T10:26:41ZVibhuti Sharma [Microsoft]Update path in AppGateway Ingress Controller for File serviceFile service has undergone a change in its URL versioning, as discussed in this [issue](https://community.opengroup.org/osdu/platform/system/file/-/issues/21) - . The context path for file-azure has been updated from api/file/v2 to api/f...File service has undergone a change in its URL versioning, as discussed in this [issue](https://community.opengroup.org/osdu/platform/system/file/-/issues/21) - . The context path for file-azure has been updated from api/file/v2 to api/file. This change needs to be mirrored in the path in AppGate Ingress Controller as well, otherwise it will not be able to pick up the correct path for APIs like {FILE_HOST}/api/file/swagger-ui.html.Vibhuti Sharma [Microsoft]Vibhuti Sharma [Microsoft]https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/132Upgrade AGIC to 1.4.0 to support Health Probe annotation2021-06-23T04:46:15ZDaniel SchollUpgrade AGIC to 1.4.0 to support Health Probe annotationApplication Gateway Ingress Controller has updated to version 1.4.0 which enables [Health Probe Annotations](https://azure.github.io/application-gateway-kubernetes-ingress/annotations/#health-probe-status-codes).
The feature adds ingres...Application Gateway Ingress Controller has updated to version 1.4.0 which enables [Health Probe Annotations](https://azure.github.io/application-gateway-kubernetes-ingress/annotations/#health-probe-status-codes).
The feature adds ingress annotations to customize health probes.Ankit Sharma [Microsoft]Ankit Sharma [Microsoft]https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/131Update test data for new database structure for Entitlements V2 service2021-07-09T02:33:31ZRostislav Vatolinvatolinrp@gmail.comUpdate test data for new database structure for Entitlements V2 serviceEntitlements V2 service uses graph database. The structure of this database used to be like this: parent node has an edge pointing to child node. To enhance performance during lookup operations (finding all parents of a child node) a dec...Entitlements V2 service uses graph database. The structure of this database used to be like this: parent node has an edge pointing to child node. To enhance performance during lookup operations (finding all parents of a child node) a decision was made to add an additional edge from a child node to parent node. A change in test data is required to make integration tests pass. Also, a change in Entitlements service is required.https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/122Feature Change - Triage the data in Service Accounts and Migrate on the basis...2021-06-23T04:50:54ZKomal MakkarFeature Change - Triage the data in Service Accounts and Migrate on the basis of design/back up needs.TBD.TBD.https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/120Release - Drop Release 0.52021-06-14T04:26:42ZDaniel SchollRelease - Drop Release 0.5[Helm Charts](https://community.opengroup.org/osdu/platform/deployment-and-operations/helm-charts-azure) need to be updated to support the M3 release (0.5).[Helm Charts](https://community.opengroup.org/osdu/platform/deployment-and-operations/helm-charts-azure) need to be updated to support the M3 release (0.5).Daniel SchollDaniel Schollhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/119Bug - Add Airflow Python Package `python-keycloak`2021-06-14T04:26:42ZDaniel SchollBug - Add Airflow Python Package `python-keycloak`Ingestion DAGS pushed an MR that requires airflow pip install of python-keycloak and now Azure Implementation of Airflow is broken.
Acceptance Criteria
---
1. Change Airflow Chart for Pipeline Support
2. Change Airflow Release ChartIngestion DAGS pushed an MR that requires airflow pip install of python-keycloak and now Azure Implementation of Airflow is broken.
Acceptance Criteria
---
1. Change Airflow Chart for Pipeline Support
2. Change Airflow Release ChartDaniel SchollDaniel Schollhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/118Feature Change - Enable AKS host-based encryption2022-08-23T11:17:12ZDaniel SchollFeature Change - Enable AKS host-based encryptionCurrently the AKS nodes aren't configured to use host-based encryption which needs to be enabled to support encryption at rest security requirements.
OSDU Security - Universal Encryption - A2
> Host-based encryption on Azure Kubernetes...Currently the AKS nodes aren't configured to use host-based encryption which needs to be enabled to support encryption at rest security requirements.
OSDU Security - Universal Encryption - A2
> Host-based encryption on Azure Kubernetes Service is in Preview. [Link](https://docs.microsoft.com/en-us/azure/aks/enable-host-encryption#:~:text=With%20host%2Dbased%20encryption%2C%20the,encrypted%20to%20the%20Storage%20service.&text=The%20cache%20of%20OS%20and,type%20set%20on%20those%20disks.)
Acceptance Criteria
---
1. Infrastructure Automation should automatically configure this feature.
2. Unit and Integration Tests Should passM7 - Release 0.10.0 - removeVivek OjhaVivek Ojhahttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/117Feature Change - Enable Pod to Pod transport security2021-06-14T04:26:42ZDaniel SchollFeature Change - Enable Pod to Pod transport securityCurrent implementation terminates and ssl offloads transport security at the Load Balancer. Transport Security should exist all the way to the Kubernetes Pod and between Pods.
OSDU Security - Universal Encryption - B2
Acceptance Crite...Current implementation terminates and ssl offloads transport security at the Load Balancer. Transport Security should exist all the way to the Kubernetes Pod and between Pods.
OSDU Security - Universal Encryption - B2
Acceptance Criteria
---
1. A design decision should be made on the best way to handle this feature.
2. Infrastructure/Helm automation should automatically configure and enable this feature.
3. Service Helm charts should be changed to move to httpshttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/116Feature Change - Base Containers should allow for container hardening and os ...2021-06-14T04:26:42ZDaniel SchollFeature Change - Base Containers should allow for container hardening and os patchingCurrent java service containers are built using a default `openjdk:8-jdk-alpine` base image. This should be an image that be maintained and allow for identified security hardening items with the ability to then patch OS in case of found...Current java service containers are built using a default `openjdk:8-jdk-alpine` base image. This should be an image that be maintained and allow for identified security hardening items with the ability to then patch OS in case of found vulnerabilities.
OSDU Security - Container Security - Item E2
Acceptance Criteria
---
1. A design decision should be made on the best way to handle this feature.
2. Necessary base container code should be created.
3. Pipelines should be leveraged to create base containers for services to use.Daniel SchollMANISH KUMARDaniel Schollhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/112Ops Procedures: Disable Health Check Endpoint Logs2021-06-23T04:49:09ZJasonOps Procedures: Disable Health Check Endpoint LogsWe are currently logging each call to that Kubernetes makes to our health check endpoints. This generates a log of logs that don't have very much significance. This causes two issues:
1. Our logs are hard to parse through, since between...We are currently logging each call to that Kubernetes makes to our health check endpoints. This generates a log of logs that don't have very much significance. This causes two issues:
1. Our logs are hard to parse through, since between every request, there are many health check requests.
1. The storage of all of these unnecessary logs costs money.
We should investigate how we can disable generating these unnecessary logs for health checks.https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/111Onboard - CSV DAGS2023-10-19T09:40:41Zethiraj krishnamanaiduOnboard - CSV DAGSCSV DAGS for Ingestion are being onboarded. These are not JAVA services so the onboarding flow is not being used. But this issue is to be used for MR tracking on those activities.
- Add Airflow Environment VariablesCSV DAGS for Ingestion are being onboarded. These are not JAVA services so the onboarding flow is not being used. But this issue is to be used for MR tracking on those activities.
- Add Airflow Environment Variablesethiraj krishnamanaiduDaniel SchollSwapnilethiraj krishnamanaiduhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/110Bug: Recipe mentions upload of file that is not available locally2021-06-23T09:16:58ZJan MortensenBug: Recipe mentions upload of file that is not available locally**Description**
The following part of Manual Installation; [configuration-data](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/configuration-data.md) mentions several ...**Description**
The following part of Manual Installation; [configuration-data](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/configuration-data.md) mentions several files that needs uploading to the blob for Airflow Storage. Challenge is that these files are not available locally as they are part of the service repositories that previously was cloned locally.
**Workaround**
Download the needed files manually, but take care to review and evaluate the versioning needed for the specific installation you are running.https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/109Entitlement Role Add - New storage role `Storage.creator`2023-10-19T09:40:41ZMayank Saggar [Microsoft]Entitlement Role Add - New storage role `Storage.creator`A new entitlement role is required to be added to support for Ingestion Services calls against the storage API.
The role should be added to userinfo
Storage role required to access ingestion service API. Currently Storage.creator is no...A new entitlement role is required to be added to support for Ingestion Services calls against the storage API.
The role should be added to userinfo
Storage role required to access ingestion service API. Currently Storage.creator is not a member of entitlements list of user_info_1.jsonMayank Saggar [Microsoft]Mayank Saggar [Microsoft]https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/108Bug fix - Not able to change the max number of VMs in AKS node pool2021-06-14T04:26:42ZHema Vishnu Pola [Microsoft]Bug fix - Not able to change the max number of VMs in AKS node pool## Bug Details
Currently the AKS module supports for changing the maximum number of node pool VM count. But that is not reflected in usage at service resources AKS creation.
### Design Question
Do we need to create the same set of varia...## Bug Details
Currently the AKS module supports for changing the maximum number of node pool VM count. But that is not reflected in usage at service resources AKS creation.
### Design Question
Do we need to create the same set of variables required for the module at places where we use, or is there going to be a way to bring forward the variables needed for modules?
Or the only way is create same set of variables needed for the modules at the point where we are using and pass them?
_PS: This is soon going to be an issue as perf environment is maxing out already on VM count._https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/107External Data Service Onboarding2022-08-23T10:47:30ZGarrett EdmondsonExternal Data Service Onboarding**Service name**: `External Data Service`
[External Data Service Repo](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/tree/master)
[External Data Service CI/CD bran...**Service name**: `External Data Service`
[External Data Service Repo](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/tree/master)
[External Data Service CI/CD branch](https://community.opengroup.org/osdu/platform/data-flow/ingestion/external-data-sources/core-external-data-workflow/-/tree/CI-CD)
**Infrastructure and Initial Requirements**
- [ ] Add any additional Azure cloud infrastructure (Cosmos containers, Storage containers, fileshares, etc.) to the Terraform template. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/infra/templates/osdu-r3-mvp). Note that if the infrastructure is a part of the data-partition template, you may need to add secrets to the keyvault that are partition specific; if doing so, update the createPartition REST request to include the keys that you have added so they are accessible in service code. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/rest/partition.http#L48)
- [ ] Create an ingress point for the service. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/appgw-ingress.yaml)
- [ ] Add any test data that is required for the service integration tests. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/test_data)
- [ ] Update `upload-data.py` to upload any new test data files you created. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/test_data/upload-data.py).
- [ ] Update the integration tester with any entitlements required to test the service. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/test_data/user_info_1.json)
- [ ] Add in any new secrets that the service needs to run. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/kv-secrets.yaml)
- [ ] Create environment variable script to generate .yaml files to be used with Intellij [EnvFile](https://plugins.jetbrains.com/plugin/7861-envfile) plugin and .envrc files to be used with [direnv](https://direnv.net/). [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/variables)
**Gitlab Code and Documentation**
- [ ] Complete the service code such that it passes all integration tests locally. There is some documentation on starting off implementing an Azure provider. [Link](./gitlab-service-readme-template.md)
- [ ] Create helm charts for service. The charts for each service are located in the `devops/azure` directory. You can look at charts from other services as a model. The charts will be nearly identical except for the different environment variables, values, etc each service needs to run. [Link](./gitlab-service-guide.md)
- [ ] Implement Istio for the service if this has not already been done. Here is an example MR that shows what steps are required. [Link](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/64)
- [ ] Create an Istio auth policy in the `devops/azure/chart/templates` directory. Here is an example of an Istio auth policy that is generic and can be used by other services. [Link](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/devops/azure/chart/templates/azure-istio-auth-policy.yaml)
- [ ] Add any variables that are required for the service integration tests to the Azure CI-CD file. [Link](https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/blob/master/cloud-providers/azure.yml)
- [ ] Verify that the README for the Azure provider correctly and clearly describes how to run and test the service. There is a README template to help. [Link](./gitlab-service-readme-template.md)
- [ ] Push any changes and verify that the Gitlab pipeline is passing in master.
**Development and Demo Azure Devops Pipelines**
- [ ] Create development ADO pipeline at `devops/azure/development-pipeline.yml` in the service repo.
- [ ] Verify development pipeline passes in ADO.
- [ ] Create Demo ADO pipeline at `devops/azure/pipeline.yml` in the service repo.
- [ ] Verify demo pipeline is passing in ADO.
**User Documentation**
- [ ] Add the service to the mirror pipeline instructions. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/code-mirroring.md)
- [ ] Add the service to the manual deployment instructions. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/charts)
- [ ] Add any required variables to the already existing variable group instructions for automated deployment. You should know if any variables need to be added to existing variable groups from creating the development and demo pipelines. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [ ] Add a variable group `Azure Service Release - $SERVICE_NAME` to the documentation. You should know what values to set for this variable group from creating the development and demo pipelines. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [ ] Add a step for creating the service pipeline at the bottom of the service-automation page. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [ ] Create a rest script with sample calls to the service for users. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/rest)Krishna Nikhil VedurumudiVivek OjhaKrishna Nikhil Vedurumudihttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/106Arch Change - Data Partition - Ingestion Workflow Database and Storage new co...2023-10-19T09:40:41ZVineeth Guna [Microsoft]Arch Change - Data Partition - Ingestion Workflow Database and Storage new collections and filesharesNeed to create new collections for R3 ingestion workflow, as there is a change in partition key semantics
Collections to be created
- WorkflowV2 - (Partition key - /partitionKey)
- WorkflowRunV2 - (Partition key - /partitionKey)
- Workf...Need to create new collections for R3 ingestion workflow, as there is a change in partition key semantics
Collections to be created
- WorkflowV2 - (Partition key - /partitionKey)
- WorkflowRunV2 - (Partition key - /partitionKey)
- WorkflowCustomOperatorV2 - (Partition key - /partitionKey)
File Share folders to be created
- plugins/hooks
- plugins/sensors
All the above collections should support large size partition keysVineeth Guna [Microsoft]Vineeth Guna [Microsoft]https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/103Bug - Airflow UI needs to be available for manual management of dags.2023-10-19T09:40:41ZDaniel SchollBug - Airflow UI needs to be available for manual management of dags.The decision has been made that until a better solution is available that the Airflow UI needs to be exposed and available for manual management of DAGS.The decision has been made that until a better solution is available that the Airflow UI needs to be exposed and available for manual management of DAGS.Daniel SchollDaniel Schollhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/102Bug - Users with a lot of groups receive a 400 Bad Request when making API ca...2023-10-19T09:40:41ZDaniel SchollBug - Users with a lot of groups receive a 400 Bad Request when making API calls.**Problem**: We are currently experiencing an issue with some environments where their decoded tokens become very large because we are including security groups as a claim in the token. There is a claim included in the tokens called "gro...**Problem**: We are currently experiencing an issue with some environments where their decoded tokens become very large because we are including security groups as a claim in the token. There is a claim included in the tokens called "groups" that is an array of the IDs of AAD security groups that the user is a member of. When a user is a member of many groups, Istio has trouble processing the token and the users receive a "400 Bad Request" error.
**Solution**: We can set `group_membership_claims` to `None` for the AAD application in Terraform. This changes the application manifest so that the tokens generated for that application no longer include groups for which the user is a member.
Consequence on terraform plan in central resources:
```
# module.ad_application.azuread_application.main will be updated in-place
~ resource "azuread_application" "main" {
~ group_membership_claims = "SecurityGroup" -> "None"
id = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
name = "osdu-mvp-crxxxx-xxxx-app"
# (13 unchanged attributes hidden)
# (1 unchanged block hidden)
}
```
**Testing**: I generated a token for myself in my personal environment and saw it included a "groups" claim with two groups. I then applied this change to my environment and generated a new token. This new token had no "groups" claim. This same approach has also been used by customers by them manually changing the AAD application manifest and it has worked at eliminating the problem of token bloat.
References
- https://registry.terraform.io/providers/hashicorp/azuread/latest/docs/resources/application#group_membership_claims
- https://docs.microsoft.com/en-us/azure/active-directory/hybrid/how-to-connect-fed-group-claims#configure-the-azure-ad-application-registration-for-group-attributesJasonJason