... | ... | @@ -2,13 +2,14 @@ _This section holds Azure specific operating procedure for the open source versi |
|
|
|
|
|
The OSDU Gitlab deployment on Azure is delivered as an open-source contribution from Microsoft and is intended to be used by customers, partners and ISVs to accelerate their delivery of energy solutions on Azure. As this is an open source contribution, it is delivered without an explicit SLA.
|
|
|
|
|
|
To support commercial quality SLAs around OSDU we will work with partners, SIs and ISVs as well as customer to create bespoke deployments and solutions based on this contributions.
|
|
|
To support commercial quality SLAs around OSDU we will work with partners, SIs and ISVs as well as customer to create bespoke deployments and solutions based on this contribution.
|
|
|
|
|
|
These procedures documented below serve as guidance for customers or ISVs looking to deploy OSDU on their own.
|
|
|
|
|
|
## Integrating with Identity Providers
|
|
|
OSDU services on Azure run in AKS and use Istio to orchestrate identity validation. The Azure Istio configuration is provided as a [template](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-istio/templates/authentication.yaml) which can be edited by OSDU providers on Azure or customers and configure to attach to their preferred identity providers.
|
|
|
If you would like to customize your identity provider you can:
|
|
|
If you would like to customize your identity provider you can either
|
|
|
|
|
|
- Use the template as-is and simply set the variable ` .Values.global.azure.tenant ` to point to your own Azure AD tenant.
|
|
|
- Edit the the template and change the `issuer` and the `jwksUri` in the `jwtRules:` section of the template, then use the [helm charts](https://community.opengroup.org/osdu/platform/deployment-and-operations/helm-charts-azure/-/blob/master/osdu-istio/README.md) to deploy your configuration.
|
|
|
|
... | ... | @@ -28,10 +29,10 @@ Logging and Log searching in Azure is documented [here](https://community.opengr |
|
|
The team is currently working on a series of monitoring dashboards and playbooks which will be provided soon (R3 timeframe) -Daniel to provide by end of Feb.
|
|
|
|
|
|
## Managing OSDU Configuration
|
|
|
For the OSS Version of OSDU R3 on Azure, configuration is managed through a series of scripts that are available out of the box or that may need to be edited to include individual customer data.
|
|
|
For the OSS Version of OSDU R3 on Azure, the configuration is managed through a series of scripts that are available out of the box but that may need to be edited to include individual customer data.
|
|
|
|
|
|
### Loading Users and Entitlements
|
|
|
Users can be loaded into the entitlement service using the entitlement service API, along with their individual access and authorization. As this is customer-specific, scripts will need to be developed by each customer.
|
|
|
User identities can be loaded into the entitlement service using the entitlement service API, along with their individual access and authorization. As this is customer-specific, scripts will need to be developed for each customer.
|
|
|
The OSDU user management project is working on a user interface that interacts with the entitlement service. We will look to deploy this tool into the Azure OSDU instance when it's available.
|
|
|
|
|
|
This [postman collection](https://community.opengroup.org/osdu/platform/testing/-/blob/master/Postman%20Collection/14_CICD_Setup_EntitlementAPI/Entitlement%20API%20CI-CD%20v1.1.postman_collection.json) provides examples for how to connect to the entitlements service to add users.
|
... | ... | @@ -40,36 +41,36 @@ This [postman collection](https://community.opengroup.org/osdu/platform/testing/ |
|
|
Many of the core services require configuration data to be loaded up-front, this data is available and maintained in various gitlab repositories. These instructions will guide you on how to load [Unit, CRS Catalog, and CRS data](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/configuration-data.md).
|
|
|
|
|
|
### Bootstrapping Schemas
|
|
|
The data definition team has completed the schemas for the R3 scope on January 29th. We now have a stable version of data schemas that can be loaded at deployment time that forms the basis of the R3 data validation, and is used by the initial ingestion validation tests. The schemas are located [here](https://community.opengroup.org/osdu/platform/system/schema-service/-/tree/master/deployments/shared-schemas)
|
|
|
The data definitions team has completed the schemas for the R3 scope on January 29th. We now have a stable version of data schemas that can be loaded at deployment time and that form the basis of the R3 data validation. These schemas will be used by the initial ingestion validation tests. The schemas are located [here](https://community.opengroup.org/osdu/platform/system/schema-service/-/tree/master/deployments/shared-schemas)
|
|
|
Schemas are loaded into the Azure OSDU instance at deployment time through the deployment scripts. It is important that schemas are loaded in order as defined in [load script](https://community.opengroup.org/osdu/platform/system/schema-service/-/blob/master/deployments/shared-schemas/osdu/load_sequence.1.0.0.json).
|
|
|
|
|
|
### Bring-Your-Own Schemas
|
|
|
To load your own schemas, both Microsoft and GCP/EPAM are currently working on scripts to load schemas into OSDU
|
|
|
To load your own schemas, both Microsoft and GCP/EPAM are currently working on scripts to load schemas into OSDU.
|
|
|
|
|
|
### Managing multiple data partitions
|
|
|
The Microsoft version of OSDU enables support for multiple data partition separated by physical boundaries. While the MVP instance provided in Gitlab enables one data partition by default, [this section](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/infra/templates/osdu-r3-mvp/data_partition) walks you through the process of adding a new partition to your terraform deployment.
|
|
|
The Microsoft version of OSDU enables support for multiple data partitions separated by cloud "physical" boundaries. While the MVP instance provided in Gitlab enables a single data partition by default, [this section](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/infra/templates/osdu-r3-mvp/data_partition) walks you through the process of adding a new partition to your terraform deployment.
|
|
|
|
|
|
### Managing multiple environments
|
|
|
Multiple OSDU environments can be supported in Azure as multiple OSDU deployments using the same TF scripts. There are two ways to deploy into multiple environments:
|
|
|
1. Automatically, by using Gitlab mirroring into your person ADO environment, and using ADO deployment pipelines
|
|
|
2. Manually, by using a modified version of the following [helm charts](https://community.opengroup.org/osdu/platform/deployment-and-operations/helm-charts-azure)
|
|
|
1. Automatically, by using Gitlab mirroring into your individual ADO environment, and using ADO deployment pipelines. The pipelines can then pull from the master branch or a feature branch or a fork.
|
|
|
2. Manually, by using a modified version of the following [helm charts](https://community.opengroup.org/osdu/platform/deployment-and-operations/helm-charts-azure).
|
|
|
|
|
|
### Middleware Configuration
|
|
|
To configure OSDU correctly you must also be aware of infrastructure deployment, such as Airflow and its dependencies, including Redis cache. This [readme](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/charts) provides a walkthrough of setting up the right configuration for your environment.
|
|
|
To configure OSDU correctly you must also be aware of the deployment and configuration of middleware components, such as Airflow and its dependencies, including Redis cache. This [readme](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/charts) provides a walkthrough for setting up the right configuration for your environment.
|
|
|
|
|
|
### Environment Specific Configuration Files
|
|
|
The Azure deployment provides configuration templates to enable customers to deploy into their own Azure subscriptions or a managed subscription. A configuration file is provided [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/infra/templates/osdu-r3-mvp/central_resources#azure-osdu-mvc-central-resources-configuration) to enable configuration of central resources.
|
|
|
The Azure deployment provides configuration templates to enable customers to deploy into their own Azure subscriptions or a managed Azure subscription provided for them by an ISV or SI. A configuration file is provided [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/infra/templates/osdu-r3-mvp/central_resources#azure-osdu-mvc-central-resources-configuration) to enable configuration of central resources.
|
|
|
|
|
|
|
|
|
## Deployment and Release Management
|
|
|
|
|
|
### Environment Provisioning
|
|
|
- [Deployment to Azure](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning) is done with [Terraform](https://www.terraform.io/), which deploys a single instance of OSDU with a fixed number of data partitions at the start. Future improvements will include the ability to dynamically add partitions to the platform. Support for multi-region will be provided post-R3.
|
|
|
- CI/CD pipelines exist for each service which deploy the services into an environment attached to GitLab and where the integration tests execute for each code merge to master or when a "trusted-branch" is created.
|
|
|
- CI/CD pipelines exist for each service which can deploy the service into an environment attached to GitLab and where the integration tests execute for each code merged to master or when a "trusted-branch" is created.
|
|
|
- Microsoft has configured mirroring to move code from the master branch into a dev copy and this is used to deploy services as well as template changes on merges to master.
|
|
|
- Helm charts ("manual") cherry picking is used to deploy code into the QA or "pre-shipping" environment.
|
|
|
- User administration/ tenant setup, done at tenant level (not scripted), not included in the platform, will be done by customer
|
|
|
- Workflow/application access will leverage user management tools, either OSDU, scripts, or commercial software
|
|
|
- User administration/ tenant setup, done at tenant level (not scripted), not included in the platform, will be done by customer.
|
|
|
- Workflow/application access will leverage user management tools, either OSDU, scripts, or commercial software.
|
|
|
|
|
|
|
|
|
### Continuous delivery pipelines
|
... | ... | |