infra-azure-provisioning merge requestshttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests2024-03-26T22:05:21Zhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/933Adding ingestion queue2024-03-26T22:05:21ZJose Jimenez RochaAdding ingestion queue## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them? Adding queue deployment to be used by ingestion workflow to get past scale pro...## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them? Adding queue deployment to be used by ingestion workflow to get past scale problems with Airflow
* [YES] I have updated the documentation accordingly.
* [YES] My code follows the code style of this project.
## Current Behavior or Linked Issues
-------------------------------------
Adding an optional queue deployment on the data storage account. By default the queue is not deployed. The ingestion workflow will use the queue to prevent problems with Airflow scalability
## Does this introduce a breaking change?
-------------------------------------
- NOM23 - Release 0.26Jose Jimenez RochaJose Jimenez Rochahttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/932Replace WorkflowTasksSharingInfo collection with ReplayStatus2024-03-08T06:38:15Zsaketh somarajuReplace WorkflowTasksSharingInfo collection with ReplayStatus## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES] I have updated the documentation accordingly.
* [YES] My code follows ...## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES] I have updated the documentation accordingly.
* [YES] My code follows the code style of this project.
## Current Behavior or Linked Issues
-------------------------------------
Related to https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/931
## Does this introduce a breaking change?
-------------------------------------
- [NO]
## Other information
-------------------------------------
- There is a [requirement](https://community.opengroup.org/osdu/platform/system/storage/-/issues/186) to add a new cosmos collection `ReplayStatus` to database `osdu-db`.
- After adding the new collection, number of collections in database `osdu-db` becomes # 26. We have [limitation](https://learn.microsoft.com/en-us/azure/cosmos-db/set-throughput#set-throughput-on-a-database-and-a-container) of 25 collections per database for shared throughput.
- Right now data partition's `terraform apply` to create `ReplayStatus` collection is failing with the above reason.
`{\\\"Errors\\\":[\\\"Collection create forbidden as collection count in database offer has exceeded 25.\\\"]}`
- Refer https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/931
- This MR proposes to **delete obsolete collection** `WorkflowTasksSharingInfo` and replace it with `ReplayStatus`
- Please be noted that for brownfield environments to apply this change, one must delete the resource lock for cosmos db account.M23 - Release 0.26Akshat Joshisaketh somarajuAkshat Joshihttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/931Introducing Replay Db and Replay topic2024-03-05T10:59:18ZAkshat JoshiIntroducing Replay Db and Replay topic## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES] I have updated the documentation accordingly.
* [YES] My code follows ...## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES] I have updated the documentation accordingly.
* [YES] My code follows the code style of this project.
## Current Behavior or Linked Issues
-------------------------------------
<!-- Please describe the current behavior that you are modifying, or link to a relevant issue. -->
This change introduce the following new infra for azure as part the [Replay ADR ](https://community.opengroup.org/osdu/platform/system/storage/-/issues/186)- <br>
- replay topic i.e. - replayrecordtopic <br>
- replay db i.e. - ReplayIndexStatus <br>
## Does this introduce a breaking change?
-------------------------------------
- NO
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Other information
-------------------------------------
- Nohttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/930Merge Azure changes back to master2024-02-26T20:09:21ZNathan StrelserMerge Azure changes back to master## All Submissions:
-------------------------------------
* [YES/NO] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] My c...## All Submissions:
-------------------------------------
* [YES/NO] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] My code follows the code style of this project.
## Current Behavior or Linked Issues
-------------------------------------
These changes bring changes made to Azure milestone branches back to master. these include vulnerability fixes. These changes have been made by a handful of people, and I have listed contacts for each change in "Other Information" for comments about changes in those folders.
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Other information
-------------------------------------
For questions about specific changes, please ping the following people:
ddms-smoke tests: @sumrazafar
initialization agent: @arysinghM23 - Release 0.26Nathan StrelserNathan Strelserhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/929Draft: Update docs2024-02-26T03:51:53Zsaketh somarajuDraft: Update docs## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES] I have updated the documentation accordingly.
* [YES] My code follows ...## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES] I have updated the documentation accordingly.
* [YES] My code follows the code style of this project.
## Does this introduce a breaking change?
-------------------------------------
- [NO]
## Other information
-------------------------------------
- Documentation UpdatesM23 - Release 0.26saketh somarajusaketh somarajuhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/928Added the env parameter in infrastructure-data-partition.yml to set different...2024-02-28T12:04:46ZAnastasiia DolotAdded the env parameter in infrastructure-data-partition.yml to set different sets of partitions for each env## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [NO] I have updated the documentation accordingly.
* [YES] My code follows t...## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [NO] I have updated the documentation accordingly.
* [YES] My code follows the code style of this project.
## Current Behavior or Linked Issues
-------------------------------------
For now, we can't have different sets of Data Partitions for each environment. These changes add such functionality.
## Does this introduce a breaking change?
-------------------------------------
- [NO]M23 - Release 0.26Anastasiia DolotAnastasiia Dolothttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/927Use the "implicit dependency" approach as a best practice. In this case...2024-02-13T14:28:36ZAnastasiia DolotUse the "implicit dependency" approach as a best practice. In this case...## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [NO] I have updated the documentation accordingly.
* [YES] My code follows t...## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [NO] I have updated the documentation accordingly.
* [YES] My code follows the code style of this project.
## Current Behavior or Linked Issues
-------------------------------------
Connected MR https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/910/diffs
Issue:
Terraform data sources need to be read during terraform apply. The data source azurerm_cosmosdb_sql_role_definition can only be read after the cosmos DB resource is created. If missing the dependency, terraform will fail due to timeout reading azurerm_cosmosdb_sql_role_definition data source. Error: Sql Role Definition: (Name "00000000-0000-0000-0000-000000000002" / Database Account Name "db account name" / Resource Group "rg name") was not found
Use the "implicit dependency" approach as a best practice. Implicit dependency is faster and relies on the property account_name in this case. if we refer value to the module output - this will be "Implicit dependency" and we don't need to specify "depends_on":
account_name = module.graph_account.account_name
## Does this introduce a breaking change?
-------------------------------------
- [NO]
## Other information
-------------------------------------
This issue only happens on the first time the cosmos DB account is provisioned.M23 - Release 0.26Anastasiia DolotAnastasiia Dolothttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/926Parametrize network location2024-02-08T15:01:10ZAnastasiia DolotParametrize network location## All Submissions:
* \[YES\] Have you added an explanation of what your changes do and why you'd like us to include them?
* \[NO\] I have updated the documentation accordingly.
* \[YES\] My code follows the code style of this project.
...## All Submissions:
* \[YES\] Have you added an explanation of what your changes do and why you'd like us to include them?
* \[NO\] I have updated the documentation accordingly.
* \[YES\] My code follows the code style of this project.
## Current Behavior or Linked Issues
---
Added ability to specify location for network module
## Does this introduce a breaking change?
---
* \[NO\]
## Additional information
Old MR https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/911M23 - Release 0.26Anastasiia DolotAnastasiia Dolothttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/925add configurations to support admin ui2024-02-12T16:22:25Zsaketh somarajuadd configurations to support admin ui## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES] I have updated the documentation accordingly.
* [YES] My code follows ...## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES] I have updated the documentation accordingly.
* [YES] My code follows the code style of this project.
## Current Behavior or Linked Issues
-------------------------------------
<!-- Please describe the current behavior that you are modifying, or link to a relevant issue. -->
## Does this introduce a breaking change?
-------------------------------------
- [NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Other information
-------------------------------------
<!-- Any other information that is important to this PR such as screenshots of how the component looks before and after the change. -->
- Add `"User.ReadBasic.All","Application.Read.All"` for the azure ad application to support admin ui configuration
- Ref: https://community.opengroup.org/osdu/ui/admin-ui-group/admin-ui-totalenergies/admin-ui-totalenergies/-/blob/main/OSDUApp/src/environments/environment.ts?ref_type=heads#L15
- Configure redirect URI for Single Page ApplicationM23 - Release 0.26saketh somarajusaketh somarajuhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/924Update version of release branch to common: 0.25.1-SNAPSHOT2024-02-05T09:26:14ZDavyd DerkachUpdate version of release branch to common: 0.25.1-SNAPSHOT## All Submissions:
-------------------------------------
* [YES/NO] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] My c...## All Submissions:
-------------------------------------
* [YES/NO] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] My code follows the code style of this project.
## Current Behavior or Linked Issues
-------------------------------------
<!-- Please describe the current behavior that you are modifying, or link to a relevant issue. -->
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Other information
-------------------------------------
<!-- Any other information that is important to this PR such as screenshots of how the component looks before and after the change. -->https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/923Remove outdated istio svc data - use fixed var val2024-02-12T17:02:32ZArturo Hernandez [EPAM]Remove outdated istio svc data - use fixed var val## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES] I have updated the documentation accordingly.
* [YES] My code follows ...## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES] I have updated the documentation accordingly.
* [YES] My code follows the code style of this project.
## Current Behavior or Linked Issues
-------------------------------------
Related to [slack infra comment](https://opensdu.slack.com/archives/C01JM5YLDV2/p1706546267471869)
Basically the get data seems to be outdated and causing issues when using the terraform code in brownfield environments.
## Does this introduce a breaking change?
-------------------------------------
- [NO]
However, customers will need to be aware that the value should be changed to match the ip address if this was assigned already dynamically by the Azure CNI controller, if this is already set, there shouldn't be any issue.
I tempt to believe in azure environments this is already set, therefore, wouldn't expect any issue.
## Other information
-------------------------------------
Related to [slack infra comment](https://opensdu.slack.com/archives/C01JM5YLDV2/p1706546267471869)M23 - Release 0.26Arturo Hernandez [EPAM]Arturo Hernandez [EPAM]https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/922#327 Terraform upgrade 1.6 and minor network fixes2024-02-01T20:12:43ZArturo Hernandez [EPAM]#327 Terraform upgrade 1.6 and minor network fixes## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them? In changelog, there were no issues to report when this change was tested.
* [Y...## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them? In changelog, there were no issues to report when this change was tested.
* [YES] I have updated the documentation accordingly.
* [YES] My code follows the code style of this project.
## Current Behavior or Linked Issues
-------------------------------------
Reated to #327 - Upgrade terraform version for compliance.
## Does this introduce a breaking change?
-------------------------------------
- [NO]
- Tests were made on EPAM azure internal environment, the upgrade it is transparent and smooth. No issues/changes to report.
- Also covering missing backend/CIDR's for allow networks in backend components such as cosmos and service bus - No breaking changes, just filling gaps on allowed networks vars.
## Other information
-------------------------------------
Related to #327
Closes #327M23 - Release 0.26Arturo Hernandez [EPAM]Arturo Hernandez [EPAM]https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/921Draft: Analytics RAFS Azure Infrastructure2024-01-18T15:33:17ZArturo Hernandez [EPAM]Draft: Analytics RAFS Azure Infrastructure## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them? It is also included in osdu/platform/domain-data-mgmt-services/rock-and-fluid-...## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them? It is also included in osdu/platform/domain-data-mgmt-services/rock-and-fluid-sample/rafs-ddms-services#28
* [YES] I have updated the documentation accordingly.
* [YES] My code follows the code style of this project.
## Current Behavior or Linked Issues
-------------------------------------
* osdu/platform/domain-data-mgmt-services/rock-and-fluid-sample/rafs-ddms-services#28
Implemented feature flag to enable analytics infrastructure for RAFS-DDMS, subject to be changed in the near future by next ADR's, this is phase1 of the analytics for RAFS.
## Does this introduce a breaking change?
-------------------------------------
- [NO]
**Introduces new feature for RAFS** It can be enabled as it is disabled by default, shouldn't affect current infrastructure.
## Other information
-------------------------------------
* Related to osdu/platform/domain-data-mgmt-services/rock-and-fluid-sample/rafs-ddms-services#28
* Queue for RAFS worker
* MongoDB for RAFS worker
* Fixed auth ip ranges for some modulesM23 - Release 0.26Arturo Hernandez [EPAM]Arturo Hernandez [EPAM]https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/920Adding eds_dag postman collection2024-02-16T07:29:14ZNaga Aneesh MylavarapuAdding eds_dag postman collection## All Submissions:
-------------------------------------
* [YES/NO] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] My c...## All Submissions:
-------------------------------------
* [YES/NO] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES/NO] I have updated the documentation accordingly.
* [YES/NO/NA] My code follows the code style of this project.
## Current Behavior or Linked Issues
-------------------------------------
<!-- Please describe the current behavior that you are modifying, or link to a relevant issue. -->
Adding a new postman collection to test the functioning of EDS DAGs
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO]
No
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Other information
-------------------------------------
<!-- Any other information that is important to this PR such as screenshots of how the component looks before and after the change. -->Naga Aneesh MylavarapuNaga Aneesh Mylavarapuhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/919Refactoring: Add centralized timeout configuration set by default2023-12-22T15:41:31ZVasyl Leskiv [SLB]Refactoring: Add centralized timeout configuration set by default## Infrastructure Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [NO] I have updated the documentation accordingly.
* [NA] I have ...## Infrastructure Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [NO] I have updated the documentation accordingly.
* [NA] I have added tests to cover my changes.
* [NA] All new and existing tests passed.
* [NA] I have formatted the terraform code. _(`terraform fmt -recursive && go fmt ./...`)_
## Current Behavior or Linked Issues
-------------------------------------
The same timeout value is hardcoded in multiple pipeline files. Added this value as default which can be optionally overrided when required.
## Does this introduce a breaking change?
-------------------------------------
- [NO]
## MR Guildelines
- [ ] Paste TF Plan for the MR.
- [ ] Pre-Merge pipeline should be run before merging. (Azure team)
- [ ] Does the module exists for new resource.
- [ ] Is there a new variable added in the MR. (Don’t use library variables and use locals)
## Other information
-------------------------------------M23 - Release 0.26Vasyl Leskiv [SLB]Vasyl Leskiv [SLB]https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/918Added TF target option2023-12-21T14:57:40ZVasyl Leskiv [SLB]Added TF target option## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES] I have updated the documentation accordingly.
* [YES] My code follows ...## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES] I have updated the documentation accordingly.
* [YES] My code follows the code style of this project.
## Current Behavior or Linked Issues
-------------------------------------
## Does this introduce a breaking change?
-------------------------------------
- [NO]Vasyl Leskiv [SLB]Vasyl Leskiv [SLB]https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/917#325 Upgrade AKS to v1.27 and docs2024-01-10T00:17:14ZArturo Hernandez [EPAM]#325 Upgrade AKS to v1.27 and docs## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES] I have updated the documentation accordingly.
* [YES] My code follows ...## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES] I have updated the documentation accordingly.
* [YES] My code follows the code style of this project.
## Current Behavior or Linked Issues
-------------------------------------
* AKS `1.25.x` will EOS next year January, we are already behind schedule if we want to need to check whether will be any issues with recent AKS versions such as `1.27`
* Upgrading 2 major versions as k8s it is being deprecated fast, `1.27` it is scheduled to EOS on July 2024.
* KEDA needs to be upgraded to avoid warns about upgrade.
## Does this introduce a breaking change?
-------------------------------------
- [NO]
* Provided instructions on how to avoid downtime, if the proper surge for nodes it is configured, there should not be any downtime, moreover, AKS it is more stable and faster than ever, it took me about ~30m to upgrade our internal cluster by using azure web ui, not terraform.
* By using terraform it might take more than 1hr in my experience, however, the upgrade should go smoothly with current Azure-osdu configuration.
## Other information
-------------------------------------
Closes #325M23 - Release 0.26Arturo Hernandez [EPAM]Arturo Hernandez [EPAM]https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/916Remove go package files from milestone branches2023-12-12T15:00:24Zharshit aggarwalRemove go package files from milestone branchesRemove go package files from milestone branches as those are required for terraformRemove go package files from milestone branches as those are required for terraformharshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/915Adding env variable placeholders which can be provided as env-vars Flags for ...2023-11-24T08:42:12ZMayank Saggar [Microsoft]Adding env variable placeholders which can be provided as env-vars Flags for running## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [NA] I have updated the documentation accordingly.
* [NA] My code follows th...## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [NA] I have updated the documentation accordingly.
* [NA] My code follows the code style of this project.
## Current Behavior or Linked Issues
-------------------------------------
Added default place holder values to environment file for running tests using CLI
## Does this introduce a breaking change?
-------------------------------------
- [YES/NO]
<!-- If this introduces a breaking change, please describe the impact and migration path for existing applications below. -->
## Other information
-------------------------------------
<!-- Any other information that is important to this PR such as screenshots of how the component looks before and after the change. -->Mayank Saggar [Microsoft]Mayank Saggar [Microsoft]https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/914Draft: feat: add blob inventory sdms2023-12-09T13:11:05ZDaniel PerezDraft: feat: add blob inventory sdms## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES] I have updated the documentation accordingly.
* [YES] My code follows ...## All Submissions:
-------------------------------------
* [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
* [YES] I have updated the documentation accordingly.
* [YES] My code follows the code style of this project.
## Current Behavior or Linked Issues
-------------------------------------
This enables blob inventory to be added in a container named inventory.
This is needed in sdms to calculate size of blobs per dataset, subproject, and containers.
## Does this introduce a breaking change?
-------------------------------------
- [NO]
## Other information
-------------------------------------
After change a new container is going to be created and, under storage account in blob inventory there is going to exist a new rule
![image](/uploads/a91e86ccefa6ebabedcac86598006d41/image.png)M23 - Release 0.26Daniel PerezDaniel Perez