infra-azure-provisioning issueshttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues2022-01-25T08:04:48Zhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/217Azure Ad Application URIIdentifiers new restrictions added causing deployment...2022-01-25T08:04:48ZVivek OjhaAzure Ad Application URIIdentifiers new restrictions added causing deployment failureCreating OSDU azure instance central resources giving following error
Error: graphrbac.ApplicationsClient#Create: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code...Creating OSDU azure instance central resources giving following error
Error: graphrbac.ApplicationsClient#Create: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="Unknown" Message="Unknown service error" Details=[{"odata.error":{"code":"Request_BadRequest","date":"2021-12-15T14:49:28","message":{"lang":"en","value":"Values of identifierUris property must use a verified domain of the organization or its subdomain: 'http://osdu-mvp-cr022-0bsd-app'"},"requestId":"84ba8e6f-224b-4b88-9a0c-587a52afc283","values":[{"item":"PropertyName","value":"identifierUris"},{"item":"PropertyErrorCode","value":"HostNameNotOnVerifiedDomain"},{"item":"HostName","value":"http://osdu-mvp-cr022-0bsd-app"}]}}]
on ../../../modules/providers/azure/ad-application/main.tf line 20, in resource "azuread_application" "main":
20: resource "azuread_application" "main" {https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/215Required secrets for postgresql in keyvault2021-12-10T06:24:37ZAalekh JainRequired secrets for postgresql in keyvaultWorkflow ingestion service needs to connect to the postgresql database (that is primarily used by airflow). This is required in order to implement the feature where we have to query the postgresql dataset.
As of now, there's no clear wa...Workflow ingestion service needs to connect to the postgresql database (that is primarily used by airflow). This is required in order to implement the feature where we have to query the postgresql dataset.
As of now, there's no clear way to obtain the hostname and username (for db) that will allow us to connect to the postgreql for running the custom queries.
These changes are added as part of the following MR in workflow ingestion service -
https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-workflow/-/merge_requests/199
The corresponding MR (in infra azure provisioning) that adds these changes is - !549
cc: @kibattulhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/214Error Azure Infra setup on step "Deploy Monitoring Resources"2021-11-23T08:53:37ZSergey ZemskovError Azure Infra setup on step "Deploy Monitoring Resources"I have successfully completed all the required steps before:
- `common_prepare.sh` script has executed without errors and warnings
- `.envrc` file contain all necessary parameters
I get this error while execute deployment `terraform app...I have successfully completed all the required steps before:
- `common_prepare.sh` script has executed without errors and warnings
- `.envrc` file contain all necessary parameters
I get this error while execute deployment `terraform apply -var-file custom.tfvars`:
```
Error: Error creating or updating Scheduled Query Rule "airflow-import-errors-alert-osdu-mvp-mrdemo-e5sm" (resource group "osdu-mvp-mrdemo-e5sm-rg"): insights.ScheduledQueryRulesClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="BadRequest" Message="Scope 'osdu-mvp-crdemo-dz5-ai' does not exists"
on main.tf line 239, in resource "azurerm_monitor_scheduled_query_rules_alert" "alerts":
239: resource "azurerm_monitor_scheduled_query_rules_alert" "alerts" {
```https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/213Ariflow 2.0 Performance Improvements2021-11-17T17:08:56ZKishore BattulaAriflow 2.0 Performance Improvements**Topic:**: `Airflow 2.0 Performance Improvements`
**Tasks**
- [ ] Airflow to support 10000 parallel DAG runs at any point in time
- [ ] Airflow autoscaling shouldn't disrupt running workflows as part of scale in.
- [ ] Airflow to suppo...**Topic:**: `Airflow 2.0 Performance Improvements`
**Tasks**
- [ ] Airflow to support 10000 parallel DAG runs at any point in time
- [ ] Airflow autoscaling shouldn't disrupt running workflows as part of scale in.
- [ ] Airflow to support 8M queuing capacity
- [ ] Documentation with necessary configuration to run above mentioned performancehttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/212Azure Infrastructure manual deployment Central Resources ERROR2021-11-23T07:46:49ZSergey ZemskovAzure Infrastructure manual deployment Central Resources ERROR`common_prepare.sh`script has been executed successfully. But I get the error on deployment Central Recourses step.
Error ocured on step `terraform plan -var-file custom.tfvars`
```
Error: expected "object_id" to be a valid UUID, got
...`common_prepare.sh`script has been executed successfully. But I get the error on deployment Central Recourses step.
Error ocured on step `terraform plan -var-file custom.tfvars`
```
Error: expected "object_id" to be a valid UUID, got
on ../../../modules/providers/azure/keyvault-policy/main.tf line 15, in resource "azurerm_key_vault_access_policy" "keyvault":
15: resource "azurerm_key_vault_access_policy" "keyvault" {
```
UPD:
File with params `.envrc` was not completely filled after running `common_prepare.sh` script, so object_id couldn't be foundhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/211Airflow Pipelines - Include airflow docker image build as part of pipelines2021-11-12T03:54:14Zharshit aggarwalAirflow Pipelines - Include airflow docker image build as part of pipelinesGoing forward we plan to install Osdu python packages as part of airflow docker image itself, rather than via extraPipPackages during airflow charts deployment. As of now we are manually building the airflow image, we should automate thi...Going forward we plan to install Osdu python packages as part of airflow docker image itself, rather than via extraPipPackages during airflow charts deployment. As of now we are manually building the airflow image, we should automate this process by including a step in pipelines itself to build the airflow image along with relevant versions of python packages and publish the same to a container registry
The changes to remove python packages from extraPipPackages are part of this [MR](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/514).
The image hardcoded in the values file **msosdu.azurecr.io/airflow-docker-image:v0.10** already contains the latest release/0.12 python packages installedhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/210Milestone and Release Management for Azure2021-11-11T12:59:46ZKrishnan GanesanMilestone and Release Management for AzureRelated to #207.
**Release name**: `M10`
The following steps must be completed OSDU Azure release.
For more information, visit our release documentation [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/in...Related to #207.
**Release name**: `M10`
The following steps must be completed OSDU Azure release.
For more information, visit our release documentation [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/CHANGELOG.md).
## Steps:
**Infra Board closure**
- [ ] Mark all issues closed which have been completed in the release. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/boards). Note that creating an issue ticket as "Close Release - <version>" helps as a mark up for the release
**Deploy terraform scripts**
- [ ] Central Resources
- [ ] Service Resources
- [ ] Data Paritions.
**Create service release images**
- [ ] Upload release service images to ACR.
- [ ] Create and Update service charts
- [ ] Upload service charts to ACR.
**Create Data seeding**
- [ ] Update documentation for following seeding data, Config, Manifest DAG, CSV Parser DAG, Schema, Entitlements, Policy, ZGY DAG and VDS DAG.
- [ ] Create and Upload versioned image for following seeding data, Config, Manifest DAG, CSV Parser DAG, Schema, Entitlements, Policy, ZGY DAG and VDS DAG.
- [ ] Update scripts for data seeding.
**Upload Data prior to service deployment**
- [ ] Upload Config Data
- [ ] Upload Manifest Ingest DAG
**Service deployment**
- [ ] Partition Service
- [ ] Security Services
- [ ] Core Services
- [ ] Reference Services
- [ ] Ingest Services
- [ ] Seismic Services
- [ ] Wellbore Services
**Upload Data post to service deployment**
- [ ] Upload Entitlements Data
- [ ] Upload Schema
- [ ] Upload Policies
- [ ] Upload CSV Parser DAG
- [ ] Upload SEGY to ZGY DAG Conversion
- [ ] Upload SEGY to VDS DAG Conversion
**Load community provided datasets post to service deployment**
- [ ] TNO data
- [ ] Volve data
**Validation**
Test services and dags using REST scripts[Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/rest).
- [ ] Test services
- [ ] Test Manifest DAG
- [ ] Test CSV DAG
- [ ] Test ZGY DAG
- [ ] Test VDS DAG
**Add Smoke Tests for services for M10 using Postman**
Community require test collections for validation which are placed here [Link](https://community.opengroup.org/osdu/platform/pre-shipping/-/tree/main/R3-M9/Azure-M9).
Need to enhance these test at par with our probe tests and any new development since M8 release.
Internal Probe Tests location for reference [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/source/probe-test-client)
Platform Validation Tests for reference [Link](https://community.opengroup.org/osdu/platform/testing)https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/208Platform Validation Env2021-12-27T11:53:08ZMANISH KUMARPlatform Validation Env**Env**: `PLATFORM VALIDATION / QA`
Preshipping community requires maintenance of QA env where test suite [runs](https://community.opengroup.org/osdu/platform/testing).
Community guidelines for [env](https://community.opengroup.org/osd...**Env**: `PLATFORM VALIDATION / QA`
Preshipping community requires maintenance of QA env where test suite [runs](https://community.opengroup.org/osdu/platform/testing).
Community guidelines for [env](https://community.opengroup.org/osdu/governance/project-management-committee/-/wikis/Environments)
## Tasks:
- [ ] Refresh token for the env and provide support to the community
- [ ] Maintain the env, keeping it updated with latest changes in Glab
- [ ] Test cases should have a pass rate of more than 99%Madhur Tanwani [Microsoft]Bharathi SelvarajPrashanth KKrishnan GanesanMadhur Tanwani [Microsoft]https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/206Azure provisioning requires a trusted domain for central_resources2021-10-28T13:19:37ZChris SmithAzure provisioning requires a trusted domain for central_resourcesWhen running the `central_resources` terraform scripts for Azure, the build fails with the following error:
```
Value of identifierUris property must use a verified domain of the organization or subdomain: 'http://osdu-mvp-crdemo-iesk-a...When running the `central_resources` terraform scripts for Azure, the build fails with the following error:
```
Value of identifierUris property must use a verified domain of the organization or subdomain: 'http://osdu-mvp-crdemo-iesk-app'"
{"item": "PropertyErrorCode", "value": "HostNameNotOnVerifiedDomain"}
{"item": "HostName", "value": "http://osdu-mvp-crdemo-iesk-app"}
with module.ad_application_azuread_application.main[0].
on ../../../modules/provides/azure/ad-application/main.tf line 20, in resource "azuread_application" "main":
20: resource azuread_application" "main" {
```
Microsoft [enforced this change on Oct 15, 2021](https://docs.microsoft.com/en-us/azure/active-directory/develop/reference-breaking-changes#appid-uri-in-single-tenant-applications-will-require-use-of-default-scheme-or-verified-domains).
We worked some updates based on the `release/0.11` tag and will push a branch PR shortly for review.https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/205Istio version upgrade2021-11-18T10:44:30ZDzmitry_Paulouski (slb)Istio version upgradeWe are using 1.6.7 version in OSDU, it's a little outdated and soon won't be supported by the latest deployable AKS version.
**Upgrade details**
* Current Version - 1.6.7
* New Version - 1.11.3
Steps to upgrade:
Run `osdu-istio` pipe...We are using 1.6.7 version in OSDU, it's a little outdated and soon won't be supported by the latest deployable AKS version.
**Upgrade details**
* Current Version - 1.6.7
* New Version - 1.11.3
Steps to upgrade:
Run `osdu-istio` pipeline to add in Gitops repo the updated manifests of Istio Operator, which takes care of upgrade.
https://dev.azure.com/ms-slb-cobuild/ms-slb-collab/_workitems/edit/665/Dzmitry_Paulouski (slb)Dzmitry_Paulouski (slb)https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/203infrastructure-data-partition Issue2021-09-29T09:54:14ZPrajakta Kumbharinfrastructure-data-partition IssueHi Team,
# Create and Deploy the Pipeline
az pipelines create \
--name 'infrastructure-data-partition' \
--repository infra-azure-provisioning \
--branch master \
--repository-type tfsgit \
--yaml-path /devops/pipelines/in...Hi Team,
# Create and Deploy the Pipeline
az pipelines create \
--name 'infrastructure-data-partition' \
--repository infra-azure-provisioning \
--branch master \
--repository-type tfsgit \
--yaml-path /devops/pipelines/infrastructure-data-partition.yml \
-ojson
While executing infrastructure-data-partition Pipeline in build stage, under Run unit tests tasks getting below error. can someone please help here
Error: Unable to find remote state.![infrastructure-data-partition_pipeline_issue](/uploads/186ce6a87e1224291b5b984cb1a5e32a/infrastructure-data-partition_pipeline_issue.png)https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/202Upgrade Airflow Lib with release version2021-11-12T03:43:34Zharshit aggarwalUpgrade Airflow Lib with release versionWith this MR we are adding a dev version of a airflow python package which should be replaced with release version during release
https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_r...With this MR we are adding a dev version of a airflow python package which should be replaced with release version during release
https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/485harshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/201Concurrent runs dags dashboard not reflecting correct information on graph2021-09-06T10:21:59ZBhakti ThakkarConcurrent runs dags dashboard not reflecting correct information on graph4 dags running concurrent. Atached is the screenshot of running dags concurrently.
![image](/uploads/6af01530b1f3ac0a482a764a7b65ca3a/image.png)
Graph shows number of concurrent dags as 0
![image](/uploads/93c26eabf7da46ed68a6530fe5b79...4 dags running concurrent. Atached is the screenshot of running dags concurrently.
![image](/uploads/6af01530b1f3ac0a482a764a7b65ca3a/image.png)
Graph shows number of concurrent dags as 0
![image](/uploads/93c26eabf7da46ed68a6530fe5b793bf/image.png)https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/200[Airflow Alert] Alert not created for worker host count2021-09-03T16:57:54ZBhakti Thakkar[Airflow Alert] Alert not created for worker host countEven when worker host count goes to 0, no alert is received.
Attached is the graph showing host count going to 0.
![image](/uploads/f647bea76bca42b45db484d8b6463e75/image.png)
Pods screenshot
![image](/uploads/042361ca94c6402b4ef6666...Even when worker host count goes to 0, no alert is received.
Attached is the graph showing host count going to 0.
![image](/uploads/f647bea76bca42b45db484d8b6463e75/image.png)
Pods screenshot
![image](/uploads/042361ca94c6402b4ef66663a65eb5a8/image.png)https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/199[Airflow Alert] Alert not created for scheduler host count2021-09-03T16:54:36ZBhakti Thakkar[Airflow Alert] Alert not created for scheduler host countEven when scheduler host count goes to 0, no alert is received.
Attached is the graph showing host count going to 0.
![image](/uploads/b80264a8eafbebe53700ca909b3ca17a/image.png)Even when scheduler host count goes to 0, no alert is received.
Attached is the graph showing host count going to 0.
![image](/uploads/b80264a8eafbebe53700ca909b3ca17a/image.png)https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/198[Airflow Alert] Web host count alert triggered even when host count is above ...2021-09-06T05:34:10ZBhakti Thakkar[Airflow Alert] Web host count alert triggered even when host count is above threshold value1. Attached is the graph which shows host count is 3.
Threshold value is 2.
![image](/uploads/e76a1b0815d3e12d7112219529c6b54e/image.png)
Screenshot for aks
![image](/uploads/792944ddebec49578526a4c48ba2997d/image.png)1. Attached is the graph which shows host count is 3.
Threshold value is 2.
![image](/uploads/e76a1b0815d3e12d7112219529c6b54e/image.png)
Screenshot for aks
![image](/uploads/792944ddebec49578526a4c48ba2997d/image.png)https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/197Enable BYOAD by adding feature flag for ad application in central resources2023-08-16T10:40:37ZVivek OjhaEnable BYOAD by adding feature flag for ad application in central resourcesVivek OjhaVivek Ojhahttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/196Close Release 0.112023-08-16T10:40:37ZMANISH KUMARClose Release 0.11Vivek OjhaVivek Ojhahttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/195terraform init failed with L Error: Unreadable module directory Unable to ev...2021-08-26T22:28:04ZAsraful Chowdhuryterraform init failed with L Error: Unreadable module directory Unable to evaluate directory symlink: lstat ../../../modules: no such file or directoryBelow command is failing as : (**Terraform v0.14.4**)
` terraform init -backend-config "storage_account_name=${TF_VAR_remote_state_account}" -backend-config "container_name=${TF_VAR_remote_state_container}"`
Initializing modules...
- a...Below command is failing as : (**Terraform v0.14.4**)
` terraform init -backend-config "storage_account_name=${TF_VAR_remote_state_account}" -backend-config "container_name=${TF_VAR_remote_state_container}"`
Initializing modules...
- ad_application in
- app_insights in
- container_registry in
- graph_account in
- keyvault in
- keyvault_policy in
- log_analytics in
- service_principal in
- storage_account in
```
Error: Unreadable module directory
Unable to evaluate directory symlink: lstat ../../../modules: no such file or
directory
Any thoughs anout this ?
```https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/194Update airflow with latest OSDU python SDK2021-07-30T07:54:11ZKishore BattulaUpdate airflow with latest OSDU python SDKUpdate airflow with latest version of OSDU python SDKUpdate airflow with latest version of OSDU python SDKM7 - Release 0.10.0 - remove