This section holds Azure specific operating procedure for the open source version of OSDU
The OSDU Gitlab deployment on Azure is delivered as an open-source contribution from Microsoft and is intended to be used by customers, partners and ISVs to accelerate their delivery of energy solutions on Azure. As this is an open source contribution, it is delivered without an explicit SLA.
To support commercial quality SLAs around OSDU we will work with partners, SIs and ISVs as well as customer to create bespoke deployments and solutions based on this contribution.
These procedures documented below serve as guidance for customers or ISVs looking to deploy OSDU on their own.
Integrating with Identity Providers
OSDU services on Azure run in AKS and use Istio to orchestrate identity validation. The Azure Istio configuration is provided as a template which can be edited by OSDU providers on Azure or customers and configure to attach to their preferred identity providers. If you would like to customize your identity provider you can either
- Use the template as-is and simply set the variable
.Values.global.azure.tenantto point to your own Azure AD tenant.
- Edit the the template and change the
jwtRules:section of the template, then use the helm charts to deploy your configuration.
Backup and Restore for OSDU Data
Backups in OSDU are handled at the persistence layer of individual data containers such as databases, storage accounts and other persistent data stores. To backup and restore to a consistent state in an eventually consistent system, there are some considerations that need to be thought through. The current document describes the approach for H1 in OSDU R3. Considering the system is still evolving, H1 functionality is all that is expected in the Mercury timeline.
Logging & Monitoring for OSDU services and infrastructure
Logging and Log searching in Azure is documented here
The team is currently working on a series of monitoring dashboards and playbooks. Examples and recommendations on how to build your own dashboards can be found here
Managing OSDU Configuration
For the OSS Version of OSDU R3 on Azure, the configuration is managed through a series of scripts that are available out of the box but that may need to be edited to include individual customer data.
Loading Users and Entitlements
User identities can be loaded into the entitlement service using the entitlement service API, along with their individual access and authorization. As this is customer-specific, scripts will need to be developed for each customer. The OSDU user management project is working on a user interface that interacts with the entitlement service. We will look to deploy this tool into the Azure OSDU instance when it's available.
This postman collection provides examples for how to connect to the entitlements service to add users.
Loading Unit, CRS Catalog and CRS Conversion data
Many of the core services require configuration data to be loaded up-front, this data is available and maintained in various gitlab repositories. These instructions will guide you on how to load Unit, CRS Catalog, and CRS data.
The data definitions team has completed the schemas for the R3 scope on January 29th. We now have a stable version of data schemas that can be loaded at deployment time and that form the basis of the R3 data validation. These schemas will be used by the initial ingestion validation tests. The schemas are located here Schemas are loaded into the Azure OSDU instance at deployment time through the deployment scripts. It is important that schemas are loaded in order as defined in load script.
To load your own schemas, both Microsoft and GCP/EPAM are currently working on scripts to load schemas into OSDU.
Managing multiple data partitions
The Microsoft version of OSDU enables support for multiple data partitions separated by cloud "physical" boundaries. While the MVP instance provided in Gitlab enables a single data partition by default, this section walks you through the process of adding a new partition to your terraform deployment.
Managing multiple environments
Multiple OSDU environments can be supported in Azure as multiple OSDU deployments using the same TF scripts. There are two ways to deploy into multiple environments:
- Automatically, by using Gitlab mirroring into your individual ADO environment, and using ADO deployment pipelines. The pipelines can then pull from the master branch or a feature branch or a fork.
- Manually, by using a modified version of the following helm charts.
To configure OSDU correctly you must also be aware of the deployment and configuration of middleware components, such as Airflow and its dependencies, including Redis cache. This readme provides a walkthrough for setting up the right configuration for your environment.
Environment Specific Configuration Files
The Azure deployment provides configuration templates to enable customers to deploy into their own Azure subscriptions or a managed Azure subscription provided for them by an ISV or SI. A configuration file is provided here to enable configuration of central resources.
Deployment and Release Management
- Deployment to Azure is done with Terraform, which deploys a single instance of OSDU with a fixed number of data partitions at the start. Future improvements will include the ability to dynamically add partitions to the platform. Support for multi-region will be provided post-R3.
- CI/CD pipelines exist for each service which can deploy the service into an environment attached to GitLab and where the integration tests execute for each code merged to master or when a "trusted-branch" is created.
- Microsoft has configured mirroring to move code from the master branch into a dev copy and this is used to deploy services as well as template changes on merges to master.
- Helm charts ("manual") cherry picking is used to deploy code into the QA or "pre-shipping" environment.
- User administration/ tenant setup, done at tenant level (not scripted), not included in the platform, will be done by customer.
- Workflow/application access will leverage user management tools, either OSDU, scripts, or commercial software.
Continuous delivery pipelines
- The infrastructure automation configuration section needs to be edited to connect to your own Azure subscription and credentials.
- The service automation templates hold the environment variables and configuration for service deployments into OSDU on Azure.
- OSDU code, including Azure code, can be deployed directly from Gitlab. Each cloud provider publishes their custom pipeline to the CI/CD repository. For Azure, we currently deploy into the staging environment to validate code merges and run tests in alignment with the agreed CI/CD standards for OSDU, however, for private deployments, we recommend deploying from a private ADO environment.
The following table maps a release type to the activities incurred within a release per the OSDU. This is for guidance for OSDU contributors and here as a reference. For more in-depth details on release management from an OSDU perspective consult this.
|Release Activities||Release Type|
A Major Release represents a full scale functional change and includes..
• External Validation, pen testing, etc..
• Full scale user tests
• Data structure changes
• Security validation
• Upgrade path needs to be managed, 1 at a time
• Deployment is not required to be backwards compatible, so data migration needs to be included in the major release planning
• Introduces major functionality (ex, Upstream Production, New Energy Data Domains)
•May be linked to an updated OSDU standard
Milestone release is branches and has a quality check from OSDU
• Run through validation scenarios as defined by OSDU QA, ex, probes, search stories, test cases, workflow validation
• Incremental functionality or new data type
• Follows standard PMC procedures (ADR, backlog grooming, etc..)
Bug Fixes and Minor Changes
• Integration tests only checks
• Bug fixes, minor enhancements
• patch is an insider ring update, dev update
• Blue/Green strategy supported
Insider Rings or Forks as Canaries
This can start as an incubation but then will need to be managed through the regular release management processes, such as Major/Minor
Follow the link for more details on infrastructure, services, and overall platform release technical details.
- Management of change
- Deprecation strategy
- Microsoft will use existing field and customer facing resources to communicate changes and releases