infra-azure-provisioning issueshttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues2022-08-23T11:19:13Zhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/174Issue with Manifest ingestion DAG runs2022-08-23T11:19:13Zpreeti singh[Microsoft]Issue with Manifest ingestion DAG runsA new [MR ](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/50) was merged due to which this error started coming.
Steps to reporduce :
Try to trigger the DAG for Manifest ingestion.
The...A new [MR ](https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/50) was merged due to which this error started coming.
Steps to reporduce :
Try to trigger the DAG for Manifest ingestion.
The trigger won't be successful .
In Validate manifest schema task, Following error was seen in airflow logs :
[2021-05-21 07:47:06,894] {taskinstance.py:1150} ERROR - 'date-time'
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
result = task_copy.execute(context=context)
File "/opt/airflow/plugins/operators/validate_manifest_schema.py", line 80, in execute
_ = schema_validator.validate_common_schema(manifest_data)
File "/opt/airflow/dags/libs/validation/validate_schema.py", line 354, in validate_common_schema
self._validate_against_schema(schema_without_refs, manifest)
File "/opt/airflow/dags/libs/validation/validate_schema.py", line 313, in _validate_against_schema
formats=("date-time", "time", "date")preeti singh[Microsoft]preeti singh[Microsoft]https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/170BUG: Schema service update elastic for any new schema add2022-08-23T11:19:15ZMANISH KUMARBUG: Schema service update elastic for any new schema addSchema service has a bug that it doesn't update indices for new schema insertion which is being fixed by this issue.
The changes are merged using [MR](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-p...Schema service has a bug that it doesn't update indices for new schema insertion which is being fixed by this issue.
The changes are merged using [MR](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/merge_requests/268).M6 - Release 0.9 - removeAbhishek Kumar (SLB)Abhishek Kumar (SLB)https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/169Container hardening for Java based services2022-08-23T11:19:11ZMANISH KUMARContainer hardening for Java based servicesHardened container for Java based servicesHardened container for Java based servicesM6 - Release 0.9 - removeMANISH KUMARMANISH KUMARhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/168Close Release - Drop Release 0.92022-08-23T11:19:18ZMANISH KUMARClose Release - Drop Release 0.9Release 0.9 (Milestone M6) has dropped and we need to close out the Release.
#### Chart `partition-services`
- [x] Partition Service
#### Chart `security-services`
- [x] Entitlements V1 Service
- [x] Entitlements V2 Service
- [x] Legal...Release 0.9 (Milestone M6) has dropped and we need to close out the Release.
#### Chart `partition-services`
- [x] Partition Service
#### Chart `security-services`
- [x] Entitlements V1 Service
- [x] Entitlements V2 Service
- [x] Legal Service
- [x] Policy Service
#### Chart `core-services`
- [x] Storage Service
- [x] Indexer Queue
- [x] Indexer Service
- [x] Search Service
- [x] File Service
- [x] Register Service
- [x] Notification Service
- [x] Indexer Queue
#### Chart `reference-services`
- [x] Unit Service
- [x] CRS Catalog
- [x] CRS Conversion
#### Chart `ingest-services`
- [x] WKS
- [x] Workflow
#### Chart `seismic-services`
- [x] Seismic DDMS
#### Chart `wellbore-services`
- [x] Wellbore DDMSM6 - Release 0.9 - removehttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/158Add a new airflow variable for search service cursor url2022-08-23T11:19:22ZVineeth Guna [Microsoft]Add a new airflow variable for search service cursor urlWe need to add a new airflow variable AIRFLOW_VAR_CORE__SERVICE__SEARCH_WITH_CURSOR__URL pointing to OSDU search service query cursor API, as this is consumed in manifest ingestion
For reference check this MR - https://community.opengro...We need to add a new airflow variable AIRFLOW_VAR_CORE__SERVICE__SEARCH_WITH_CURSOR__URL pointing to OSDU search service query cursor API, as this is consumed in manifest ingestion
For reference check this MR - https://community.opengroup.org/osdu/platform/data-flow/ingestion/ingestion-dags/-/merge_requests/47 which is mergedhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/156Resource requests/limits for service containers2022-08-23T11:19:14ZAbhishek PatilResource requests/limits for service containers**Context**
Each service pod running in `osdu` namespace should have resource (i.e., cpu and memory) requests/limits defined. It is a best practice guidance as it helps kubernetes to schedule the pods according to their resource require...**Context**
Each service pod running in `osdu` namespace should have resource (i.e., cpu and memory) requests/limits defined. It is a best practice guidance as it helps kubernetes to schedule the pods according to their resource requirements and aid in better resource management.
**Issue**
In a multi micro-services system, like OSDU, it is difficult to incorporate resource limits across all micro-services. Services can miss out to have resource limits or can either specify too small or large resource limits. This can cause some services to use more resources than their fare quota and some services to starve for resources.
**Solution**
Default requests/limits for compute resources can be set at namespace level. This will ensure that all container created in that namespace will pick those default values for compute resources if they does not specify there own values. Services (or containers/pods) which needs different settings for compute resources can override the default namespace level settings by specifying requests/limits values in their deployment.yaml.
**High Level Design**
**Introduction**
This document explains how the default resource (i.e., CPU and memory) requests and limits can be set at a namespace level. Any container created in a namespace, that has default resource requests and limits set, will be assigned those default values if that container does not specify its own values.
**LimitRange**
Kubernetes’s LimitRange object can be used to specify default cpu/memory requests and limits for a namespace. Below is a sample configuration file for a LimitRange object.
```
apiVersion: v1
kind: LimitRange
metadata:
name: resource-limit-range
namespace: osdu
spec:
limits:
- default:
memory: 2Gi
cpu: 1
defaultRequest:
memory: 2Gi
cpu: 0.5
type: Container
```
Above configuration will set default cpu and memory requests as 0.5 and 2Gi respectively. Also, it will set the default cpu and memory limits to 1 and 2Gi respectively. These default values will be applicable for osdu namespace.
Any container created in osdu namespace will get these default settings for cpu and memory if that container doesn’t specify its own cpu/memory requests and limits. A container can override the default namespace level resource requests and limits by specifying its own values for the same.
**Motivation**
It is a best practice to set resource (i.e., CPU/memory) requests and limits for every pod in AKS cluster. But in a multi micro-services system, it is difficult to incorporate resource limits across all micro-services. Services can miss out to have resource limits or can either specify too small or large resource limits. This can cause some services to use more resources than their fare quota and some services to starve for resources.
Thus, specifying resource requests and limits at namespace level will guarantee the default share of resources to services (or pods) which don’t explicitly define resource limits. Services (or pods) which are fine-tuned with different set of requirements can specify their own requests and limits values which will override the namespace level settings.
**Namespace limits in OSDU**
![image](/uploads/b76af2d992fda8d5a3a63e0ad8c82993/image.png)
Above figure describes the architecture of OSDU infra with namespace limits applied. All the OSDU services are deployed in osdu namespace. This osdu namespace will be configured with default settings for resource requests and limits by using LimitRange.
Services will pick up these default settings (e.g., Service1 and Service2). Services which have different requirements for resources can explicitly mention requests and limits values in their deployment.yaml (e.g., Service3). The values specified in deployment.yaml will override the namespace settings.
**Potential Blockers**
The default resource requests and limits set in a LimitRange object will be applicable for all the containers created in the specified namespace. If a service’s pod contains multiple containers, then all those containers will have the default settings.
Different containers within a pod can have different requirements for resources. Therefore, finding a common default values for resource requests and limits among different types of containers may be a difficult task.
NOTE: As of now only two types of containers are present in every pod in osdu namespace. One is istio-proxy and other is the service’s container. Resource requests and limits values for every istio-proxy container can be configured through common configuration file. Whereas default for service containers can be configured through LimitRange.
**References**
https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/
https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/
https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/cpu-default-namespace/M6 - Release 0.9 - removeAbhishek PatilAbhishek Patilhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/150Feature Change - Add alert rule for CPU usage of OSDU services2022-08-23T11:19:19ZVibhuti Sharma [Microsoft]Feature Change - Add alert rule for CPU usage of OSDU servicesWe need to add an alert rule which will alert when the CPU usage of any OSDU service is above a threshold value. Currently there are no alert rules configured in the infrastructure.
**Acceptance Criteria**
1) ADR to document the decisio...We need to add an alert rule which will alert when the CPU usage of any OSDU service is above a threshold value. Currently there are no alert rules configured in the infrastructure.
**Acceptance Criteria**
1) ADR to document the decision to add alert rules in the appropriate resource group.
2) Necessary code to add alert rule.
3) Create or update required documentationM6 - Release 0.9 - removeVibhuti Sharma [Microsoft]Vibhuti Sharma [Microsoft]https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/105Offboard Delivery Service2022-08-23T11:18:45ZJasonOffboard Delivery ServiceDelivery service no longer exists independently as its functionality will be merged into file service. We need to remove references to delivery service from our documentation since it no longer exists.Delivery service no longer exists independently as its functionality will be merged into file service. We need to remove references to delivery service from our documentation since it no longer exists.M6 - Release 0.9 - removeMANISH KUMARMANISH KUMARhttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/95Policy Service Onboarding2022-08-23T11:19:14ZHrvoje MarkovicPolicy Service Onboarding**Service name**: `Policy`
The following steps must be completed for a service to onboard with OSDU on Azure. Additionally, please add the `Service Onboarding` tag to this issue when it is created.
For more information, visit our servi...**Service name**: `Policy`
The following steps must be completed for a service to onboard with OSDU on Azure. Additionally, please add the `Service Onboarding` tag to this issue when it is created.
For more information, visit our service onboarding documentation [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-onboarding.md).
## Steps:
**Infrastructure and Initial Requirements**
- [x] Add any additional Azure cloud infrastructure (Cosmos containers, Storage containers, fileshares, etc.) to the Terraform template. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/infra/templates/osdu-r3-mvp). Note that if the infrastructure is a part of the data-partition template, you may need to add secrets to the keyvault that are partition specific; if doing so, update the createPartition REST request to include the keys that you have added so they are accessible in service code. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/rest/partition.http#L48)
- [x] Create an ingress point for the service. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/appgw-ingress.yaml)
- [x] Add any test data that is required for the service integration tests. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/test_data)
- [ ] Update `upload-data.py` to upload any new test data files you created. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/test_data/upload-data.py).
- [ ] Update the integration tester with any entitlements required to test the service. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/test_data/user_info_1.json)
- [x] Add in any new secrets that the service needs to run. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/kv-secrets.yaml)
- [ ] Create environment variable script to generate .yaml files to be used with Intellij [EnvFile](https://plugins.jetbrains.com/plugin/7861-envfile) plugin and .envrc files to be used with [direnv](https://direnv.net/). [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/variables)
**Gitlab Code and Documentation**
- [x] Complete the service code such that it passes all integration tests locally. There is some documentation on starting off implementing an Azure provider. [Link](./gitlab-service-readme-template.md)
- [x] Create helm charts for service. The charts for each service are located in the `devops/azure` directory. You can look at charts from other services as a model. The charts will be nearly identical except for the different environment variables, values, etc each service needs to run. [Link](./gitlab-service-guide.md)
- [x] Implement Istio for the service if this has not already been done. Here is an example MR that shows what steps are required. [Link](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/64)
- [x] Create an Istio auth policy in the `devops/azure/chart/templates` directory. Here is an example of an Istio auth policy that is generic and can be used by other services. [Link](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/devops/azure/chart/templates/azure-istio-auth-policy.yaml)
- [x] Add any variables that are required for the service integration tests to the Azure CI-CD file. [Link](https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/blob/master/cloud-providers/azure.yml)
- [x] Verify that the README for the Azure provider correctly and clearly describes how to run and test the service. There is a README template to help. [Link](./gitlab-service-readme-template.md)
- [x] Push any changes and verify that the Gitlab pipeline is passing in master.
**Development and Demo Azure Devops Pipelines**
- [x] Create development ADO pipeline at `devops/azure/development-pipeline.yml` in the service repo.
- [x] Verify development pipeline passes in ADO.
- [ ] Create Demo ADO pipeline at `devops/azure/pipeline.yml` in the service repo.
- [ ] Verify demo pipeline is passing in ADO.
**User Documentation**
- [x] Add the service to the mirror pipeline instructions. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/code-mirroring.md)
- [x] Add the service to the manual deployment instructions. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/charts)
- [x] Add any required variables to the already existing variable group instructions for automated deployment. You should know if any variables need to be added to existing variable groups from creating the development and demo pipelines. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [ ] Add a variable group `Azure Service Release - $SERVICE_NAME` to the documentation. You should know what values to set for this variable group from creating the development and demo pipelines. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [x] Add a step for creating the service pipeline at the bottom of the service-automation page. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [x] Create a rest script with sample calls to the service for users. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/rest)M6 - Release 0.9 - removehttps://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/issues/36Onboard Wellbore DMS Services2022-08-23T11:18:46ZDaniel SchollOnboard Wellbore DMS Services**Service name**: `Wellbore DMS`
The following steps must be completed for a service to onboard with OSDU on Azure. Additionally, please add the `Service Onboarding` tag to this issue when it is created.
For more information, visit our...**Service name**: `Wellbore DMS`
The following steps must be completed for a service to onboard with OSDU on Azure. Additionally, please add the `Service Onboarding` tag to this issue when it is created.
For more information, visit our service onboarding documentation [here](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-onboarding.md).
## Steps:
**Infrastructure and Initial Requirements**
- [x] Add any additional Azure cloud infrastructure (Cosmos containers, Storage containers, fileshares, etc.) to the Terraform template. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/infra/templates/osdu-r3-mvp). Note that if the infrastructure is a part of the data-partition template, you may need to add secrets to the keyvault that are partition specific; if doing so, update the createPartition REST request to include the keys that you have added so they are accessible in service code. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/rest/partition.http#L48)
- [x] Create an ingress point for the service. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/appgw-ingress.yaml)
- [x] Add any test data that is required for the service integration tests. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/test_data)
- [x] Update `upload-data.py` to upload any new test data files you created. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/test_data/upload-data.py).
- [x] Update the integration tester with any entitlements required to test the service. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/tools/test_data/user_info_1.json)
- [x] Add in any new secrets that the service needs to run. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/charts/osdu-common/templates/kv-secrets.yaml)
- [x] Create environment variable script to generate .yaml files to be used with Intellij [EnvFile](https://plugins.jetbrains.com/plugin/7861-envfile) plugin and .envrc files to be used with [direnv](https://direnv.net/). [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/variables)
**Gitlab Code and Documentation**
- [x] Complete the service code such that it passes all integration tests locally. There is some documentation on starting off implementing an Azure provider. [Link](./gitlab-service-readme-template.md)
- [x] Create helm charts for service. The charts for each service are located in the `devops/azure` directory. You can look at charts from other services as a model. The charts will be nearly identical except for the different environment variables, values, etc each service needs to run. [Link](./gitlab-service-guide.md)
- [x] Implement Istio for the service if this has not already been done. Here is an example MR that shows what steps are required. [Link](https://community.opengroup.org/osdu/platform/system/storage/-/merge_requests/64)
- [x] Create an Istio auth policy in the `devops/azure/chart/templates` directory. Here is an example of an Istio auth policy that is generic and can be used by other services. [Link](https://community.opengroup.org/osdu/platform/system/storage/-/blob/master/devops/azure/chart/templates/azure-istio-auth-policy.yaml)
- [x] Add any variables that are required for the service integration tests to the Azure CI-CD file. [Link](https://community.opengroup.org/osdu/platform/ci-cd-pipelines/-/blob/master/cloud-providers/azure.yml)
- [x] Verify that the README for the Azure provider correctly and clearly describes how to run and test the service. There is a README template to help. [Link](./gitlab-service-readme-template.md)
- [x] Push any changes and verify that the Gitlab pipeline is passing in master.
**Development and Demo Azure Devops Pipelines**
- [x] Create development ADO pipeline at `devops/azure/development-pipeline.yml` in the service repo.
- [ ] Verify development pipeline passes in ADO.
- [x] Create Demo ADO pipeline at `devops/azure/pipeline.yml` in the service repo.
- [ ] Verify demo pipeline is passing in ADO.
**User Documentation**
- [x] Add the service to the mirror pipeline instructions. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/code-mirroring.md)
- [x] Add the service to the manual deployment instructions. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/charts)
- [x] Add any required variables to the already existing variable group instructions for automated deployment. You should know if any variables need to be added to existing variable groups from creating the development and demo pipelines. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [x] Add a variable group `Azure Service Release - $SERVICE_NAME` to the documentation. You should know what values to set for this variable group from creating the development and demo pipelines. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [x] Add a step for creating the service pipeline at the bottom of the service-automation page. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/blob/master/docs/service-automation.md#create-osdu-service-libraries)
- [x] Create a rest script with sample calls to the service for users. [Link](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-azure-provisioning/-/tree/master/tools/rest)
## Setup:
Wellbore DDS requires some Indexation Schemas to be created on each DataPartition for the service to be operationnal on the corresponding DataPartition.
Please refer to documentation on the following [Wiki](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services/-/blob/master/schema/indexation/README.md)M6 - Release 0.9 - removeFrancois VinyesLuc YriarteVincent RondotFrancois Vinyes