Commit 76953f39 authored by Sumra Zafar's avatar Sumra Zafar
Browse files

Merge branch 'w-azure-pipelines' into 'master'

Azure Pipelines

See merge request !40
parents 0318d3c5 df20c6ea
Pipeline #31746 failed with stages
in 10 minutes and 42 seconds
......@@ -18,7 +18,11 @@ variables:
OSDU_GCP_APPLICATION_NAME: wellbore-ddms
OSDU_GCP_VENDOR: gcp
AZURE_SERVICE: wellbore-ddms
AZURE_DOCKER_SUBDIR: build/Dockerfile
AZURE_TEST_SUBDIR: tests/integration
AZURE_TEST_TYPE: python
include:
- project: "osdu/platform/ci-cd-pipelines"
file: "standard-setup.yml"
......@@ -37,6 +41,8 @@ include:
- project: 'osdu/platform/ci-cd-pipelines'
file: 'cloud-providers/ibm-wellbore.yml'
- local: "/devops/azure/azure-wellbore.yml"
# --------------------------------------------------------------------------------
containerize:
......
......@@ -50,7 +50,6 @@ BSD-2-Clause
The following software have components provided under the terms of this license:
- mock (from https://github.com/testing-cabal/mock)
- packaging (from https://github.com/pypa/packaging)
- ply (from http://www.dabeaz.com/ply/)
- pyasn1 (from http://sourceforge.net/projects/pyasn1/)
- pyasn1-modules (from http://sourceforge.net/projects/pyasn1/)
......@@ -64,6 +63,7 @@ The following software have components provided under the terms of this license:
- click (from http://github.com/mitsuhiko/click)
- cryptography (from https://github.com/pyca/cryptography)
- decorator (from https://github.com/micheles/decorator)
- grpcio (from http://www.grpc.io)
- hiredis (from https://github.com/redis/hiredis-py)
- httpcore (from https://github.com/encode/httpcore)
- httpx (from https://github.com/encode/httpx)
......@@ -89,13 +89,6 @@ The following software have components provided under the terms of this license:
- starlette (from https://github.com/encode/starlette)
- uvicorn (from https://github.com/tomchristie/uvicorn)
========================================================================
CC0-1.0
========================================================================
The following software have components provided under the terms of this license:
- coverage (from https://coverage.readthedocs.io)
========================================================================
CNRI-Python
========================================================================
......@@ -110,6 +103,7 @@ GPL-2.0-only
The following software have components provided under the terms of this license:
- coverage (from https://coverage.readthedocs.io)
- grpcio (from http://www.grpc.io)
========================================================================
GPL-3.0-only
......@@ -117,6 +111,7 @@ GPL-3.0-only
The following software have components provided under the terms of this license:
- coverage (from https://coverage.readthedocs.io)
- grpcio (from http://www.grpc.io)
- numpy (from http://www.numpy.org)
- pyparsing (from http://pyparsing.wikispaces.com/)
- rfc3986 (from https://rfc3986.readthedocs.org)
......@@ -133,8 +128,16 @@ ISC
========================================================================
The following software have components provided under the terms of this license:
- grpcio (from http://www.grpc.io)
- requests-oauthlib (from https://github.com/requests/requests-oauthlib)
========================================================================
Info-ZIP
========================================================================
The following software have components provided under the terms of this license:
- grpcio (from http://www.grpc.io)
========================================================================
JSON
========================================================================
......@@ -186,11 +189,11 @@ The following software have components provided under the terms of this license:
- cffi (from http://cffi.readthedocs.org)
- coverage (from https://coverage.readthedocs.io)
- fastapi (from https://github.com/tiangolo/fastapi)
- grpcio (from http://www.grpc.io)
- h11 (from https://github.com/python-hyper/h11)
- iniconfig (from http://github.com/RonnyPfannschmidt/iniconfig)
- jmespath (from https://github.com/jmespath/jmespath.py)
- jsonschema (from http://github.com/Julian/jsonschema)
- mockito (from https://github.com/kaste/mockito-python)
- msal (from https://github.com/AzureAD/microsoft-authentication-library-for-python)
- msal-extensions (from https://pypi.org/project/msal-extensions/0.1.3/)
- msrest (from https://github.com/Azure/msrest-for-python)
......@@ -223,14 +226,19 @@ The following software have components provided under the terms of this license:
- certifi (from http://certifi.io/)
========================================================================
OpenSSL
========================================================================
The following software have components provided under the terms of this license:
- grpcio (from http://www.grpc.io)
========================================================================
Python-2.0
========================================================================
The following software have components provided under the terms of this license:
- async-timeout (from https://github.com/aio-libs/async_timeout/)
- coverage (from https://coverage.readthedocs.io)
- cryptography (from https://github.com/pyca/cryptography)
- google-auth (from https://github.com/GoogleCloudPlatform/google-auth-library-python)
- portalocker (from https://github.com/WoLpH/portalocker)
- python-dateutil (from https://dateutil.readthedocs.org)
......@@ -240,6 +248,13 @@ The following software have components provided under the terms of this license:
- typing-extensions (from https://github.com/python/typing)
- urllib3 (from https://urllib3.readthedocs.io/)
========================================================================
Unlicense
========================================================================
The following software have components provided under the terms of this license:
- grpcio (from http://www.grpc.io)
========================================================================
WTFPL
========================================================================
......@@ -254,13 +269,20 @@ The following software have components provided under the terms of this license:
- pytz (from http://pythonhosted.org/pytz)
========================================================================
Zlib
========================================================================
The following software have components provided under the terms of this license:
- grpcio (from http://www.grpc.io)
========================================================================
public-domain
========================================================================
The following software have components provided under the terms of this license:
- botocore (from https://github.com/boto/botocore)
- coverage (from https://coverage.readthedocs.io)
- grpcio (from http://www.grpc.io)
- py (from http://pylib.readthedocs.org/)
- pytz (from http://pythonhosted.org/pytz)
......
# EXPECTED PIPELINE INHERITED GROUP VARIABLES
# --------------------------------------------------------------------------------
# AZURE (Protected Branch)
# AZURE_APP_ID (Protected Branch)
# AZURE_APP_ID_OTHER (Protected Branch)
# AZURE_APP_OID_OTHER (Protected Branch)
# AZURE_BASE (Protected Branch)
# AZURE_BASENAME_21 (Protected Branch)
# AZURE_DNS_NAME (Protected Branch)
# AZURE_ELASTIC_HOST (Protected Branch)
# AZURE_ELASTIC_PASSWORD (Protected Branch/Masked Variable)
# AZURE_INVALID_JWT (Protected Branch)
# AZURE_NO_ACCESS_ID (Protected Branch)
# AZURE_NO_ACCESS_SECRET (Protected Branch/Masked Variable)
# AZURE_PRINCIPAL_ID (Protected Branch)
# AZURE_PRINCIPAL_SECRET (Protected Branch/Masked Variable)
# AZURE_REGISTRY (Protected Branch)
# AZURE_SERVICEBUS_KEY (Protected Branch/Masked Variable)
# AZURE_STORAGE_KEY (Protected Branch/Masked Variable)
# AZURE_SUBSCRIPTION_ID (Protected Branch)
# AZURE_SUBSCRIPTION_NAME (Protected Branch)
# AZURE_TENANT_ID (Protected Branch)
# EXPECTED PIPELINE VARIABLES
# --------------------------------------------------------------------------------
# AZURE_TEST_SUBDIR
.azure_variables:
variables:
LOG_LEVEL: INFO
# Common Section
ENTITLEMENT_URL: https://${AZURE_DNS_NAME}/entitlements/v1/
ENTITLEMENT_V2_URL: https://${AZURE_DNS_NAME}/api/entitlements/v2/
LEGAL_URL: https://${AZURE_DNS_NAME}/api/legal/v1/
STORAGE_URL: https://${AZURE_DNS_NAME}/api/storage/v2/
SEARCH_URL: https://${AZURE_DNS_NAME}/api/search/v2/
INDEXER_URL: https://${AZURE_DNS_NAME}/api/indexer/v2/
DELIVERY_URL: https://${AZURE_DNS_NAME}/api/delivery/v2/
FILE_URL: https://${AZURE_DNS_NAME}/api/file/v2/
WORKFLOW_URL: https://${AZURE_DNS_NAME}/api/workflow/v1/
AZURE_AD_TENANT_ID: $AZURE_TENANT_ID
INTEGRATION_TESTER: $AZURE_PRINCIPAL_ID
AZURE_TESTER_SERVICEPRINCIPAL_SECRET: $AZURE_PRINCIPAL_SECRET
AZURE_AD_APP_RESOURCE_ID: $AZURE_APP_ID
AZURE_STORAGE_ACCOUNT: ${AZURE_BASE}data
MY_TENANT: opendes
SHARED_TENANT: opendes
DOMAIN: contoso.com
ELASTIC_HOST: $AZURE_ELASTIC_HOST
ELASTIC_PORT: 9243
ELASTIC_USER_NAME: elastic
ELASTIC_PASSWORD: $AZURE_ELASTIC_PASSWORD
VENDOR: azure
HOST: https://${AZURE_DNS_NAME}
ACL_OWNERS: data.test1
ACL_VIEWERS: data.test1
WELLBORE_URL: https://${AZURE_DNS_NAME}/api/os-wellbore-ddms
LEGAL_TAG: "opendes-public-usa-dataset-7643990"
DATA_PARTITION_ID: opendes
# JOBS
# --------------------------------------------------------------------------------
azure_containerize:
tags: ["osdu-medium"]
image: danielscholl/azure-build-image
stage: containerize
needs: ["compile-and-unit-test"]
variables:
SHA_IMAGE: ${CI_PROJECT_NAME}-${CI_COMMIT_REF_SLUG}:${CI_COMMIT_SHA}
LATEST_IMAGE: ${CI_PROJECT_NAME}-${CI_COMMIT_REF_SLUG}:latest
before_script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- az --version
- az login --service-principal -u $AZURE_PRINCIPAL_ID -p $AZURE_PRINCIPAL_SECRET --tenant $AZURE_TENANT_ID
script:
- echo "Azure Deployment for Wellbore DMS"
# Dockerfile used from AZURE_DOCKER_SUBDIR
- echo "Startup docker build is $AZURE_DOCKER_SUBDIR"
- docker build -f $AZURE_DOCKER_SUBDIR -t $CI_REGISTRY_IMAGE/$SHA_IMAGE .
# Gitlab Container Registry
- docker push ${CI_REGISTRY_IMAGE}/$SHA_IMAGE
- docker tag $CI_REGISTRY_IMAGE/$SHA_IMAGE $CI_REGISTRY_IMAGE/$LATEST_IMAGE
- docker push ${CI_REGISTRY_IMAGE}/$LATEST_IMAGE
# Azure Container Registry
- az acr login -n $AZURE_REGISTRY
- docker tag $CI_REGISTRY_IMAGE/$SHA_IMAGE ${AZURE_REGISTRY}.azurecr.io/$SHA_IMAGE
- docker push ${AZURE_REGISTRY}.azurecr.io/$SHA_IMAGE
- docker tag $CI_REGISTRY_IMAGE/$SHA_IMAGE ${AZURE_REGISTRY}.azurecr.io/$LATEST_IMAGE
- docker push ${AZURE_REGISTRY}.azurecr.io/$LATEST_IMAGE
only:
variables:
- $AZURE == 'true'
azure_deploy:
image: danielscholl/azure-build-image
tags: ["osdu-medium"]
stage: deploy
needs: ["azure_containerize"]
variables:
BRANCH: ${CI_PROJECT_NAME}-${CI_COMMIT_REF_SLUG}
TAG: $CI_COMMIT_SHA
extends:
- .azure_variables
before_script:
- az login --service-principal -u $AZURE_PRINCIPAL_ID -p $AZURE_PRINCIPAL_SECRET --tenant $AZURE_TENANT_ID
- az aks get-credentials -g $AZURE_UNIQUE-rg -n $AZURE_UNIQUE-aks
script:
# Install Service
- helm upgrade -i osdu-gitlab-$CI_PROJECT_NAME devops/azure/chart --set image.repository=${AZURE_REGISTRY}.azurecr.io --set image.branch=$BRANCH --set image.tag=$TAG
# Increasing to 900s as rolling updates are happening and each service is expected to have minimum 2 containers.
- kubectl rollout status deployment.v1.apps/osdu-gitlab-$CI_PROJECT_NAME -n osdu --timeout=900s
- pod=$(kubectl get pod -n osdu|grep $CI_PROJECT_NAME |tail -1 |awk '{print $1}')
- status=$(kubectl wait -n osdu --for=condition=Ready pod/$pod --timeout=300s)
- if [[ "$status" != *"met"* ]]; then echo "POD didn't start correctly" ; exit 1 ; fi
only:
variables:
- $AZURE == 'true'
except:
variables:
- $AZURE_SKIP_DEPLOY == 'true'
azure_test_py:
image: python:3.8
stage: integration
needs: ["azure_deploy"]
allow_failure: true
extends:
- .azure_variables
script:
- pip install virtualenv
- virtualenv venv
- source venv/bin/activate
- pip install --upgrade pip
- pip install wheel pytest pytest-cov
- pip install -r requirements.txt
- pip install -r requirements_dev.txt
- svctoken=$(python devops/scripts/azure_jwt_client.py)
- cd $AZURE_TEST_SUBDIR
- python ./gen_postman_env.py --token ${svctoken} --base_url ${WELLBORE_URL} --cloud_provider ${VENDOR} --acl_domain ${DOMAIN} --legal_tag ${LEGAL_TAG} --data_partition ${DATA_PARTITION_ID}
- pytest ./functional --environment="./generated/postman_environment.json" --insecure --timeout-request=15000 --filter-tag=basic
only:
variables:
- $AZURE == 'true' && $AZURE_SKIP_DEPLOY != 'true' && $AZURE_TEST_TYPE == 'python'
except:
variables:
- $AZURE_SKIP_TEST == 'true'
......@@ -13,9 +13,9 @@
# limitations under the License.
apiVersion: v2
name: wdms
name: wellbore-domain-services
description: OSDU Wellbore DDMS Service
appVersion: "latest"
# A chart can be either an 'application' or a 'library' chart.
#
# Application charts are a collection of templates that can be packaged into versioned archives
......
{{/*
Common Annotations
*/}}
{{- define "os-wellbore-ddms.commonAnnotations" -}}
build-number: {{ .Values.annotations.buildNumber | quote }}
build-origin: {{ .Values.annotations.buildOrigin | quote }}
commit-branch: {{ .Values.annotations.commitBranch | quote }}
commit-id: {{ .Values.annotations.commitId | quote }}
{{- end}}
{{/*
Common Labels
*/}}
......
......@@ -11,30 +11,27 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
annotations:
{{ include "os-wellbore-ddms.commonAnnotations" . | indent 4}}
labels:
{{ include "os-wellbore-ddms.commonLabels" . | indent 4}}
name: {{ .Values.authorizationPolicy.name }}{{ include "os-wellbore-ddms.name-suffix" . }}
{{ include "os-wellbore-ddms.namespace" . | indent 2}}
name: wellbore-jwt-authz
namespace: osdu
spec:
action: DENY
rules:
- from:
- source:
notRequestPrincipals:
- '*'
to:
- operation:
notPaths:
- {{ include "os-wellbore-ddms.prefix" . }}/
- {{ include "os-wellbore-ddms.prefix" . }}/ddms/v2/about
- {{ include "os-wellbore-ddms.prefix" . }}/docs
- {{ include "os-wellbore-ddms.prefix" . }}/openapi.json
selector:
matchLabels:
{{ include "os-wellbore-ddms.commonLabels" . | indent 6}}
app: osdu-gitlab-wellbore-domain-services
action: DENY
rules:
- from:
- source:
notRequestPrincipals: ["*"]
to:
- operation:
notPaths: ["/","*/index.html",
"*/v2/api-docs",
"*/swagger","*/swagger-resources","*/swagger-ui.html",
"*/actuator/health", "*/health",
"*/configuration/ui","*/configuration/security",
"/api/os-wellbore-ddms/swagger-resources/*",
"/api/os-wellbore-ddms/*",
"/api/os-wellbore-ddms/webjars/*"]
......@@ -25,8 +25,6 @@ data:
AZ_LOGGER_LEVEL: {{ .Values.configMap.data.loggerLevel }}
kind: ConfigMap
metadata:
annotations:
{{ include "os-wellbore-ddms.commonAnnotations" . | indent 4}}
labels:
{{ include "os-wellbore-ddms.commonLabels" . | indent 4}}
name: {{ .Values.configMap.name }}{{ $nameSuffix }}
......
......@@ -16,11 +16,9 @@
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
{{ include "os-wellbore-ddms.commonAnnotations" . | indent 4}}
labels:
{{ include "os-wellbore-ddms.commonLabels" . | indent 4}}
name: {{ .Values.deployment.name }}{{ $nameSuffix }}
name: {{ .Values.deployment.name }}
{{ include "os-wellbore-ddms.namespace" . | indent 2}}
spec:
replicas: {{ .Values.replicaCount }}
......@@ -29,8 +27,6 @@ spec:
{{ include "os-wellbore-ddms.commonLabels" . | indent 6}}
template:
metadata:
annotations:
{{ include "os-wellbore-ddms.commonAnnotations" . | indent 8}}
labels:
aadpodidbinding: "{{ .Values.labels.aadpodidbinding }}"
{{ include "os-wellbore-ddms.commonLabels" . | indent 8}}
......@@ -51,7 +47,7 @@ spec:
secretProviderClass: "azure-keyvault"
containers:
- name: {{ .Values.deployment.name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
image: {{ .Values.image.repository }}/{{ .Values.image.branch }}:{{ .Values.image.tag | default .Chart.AppVersion }}
ports:
- containerPort: 8080
# This preStop hook has been added as a temporary workaround to minimize downtime during deployments until this limitation is addressed at the AGIC level
......
......@@ -13,12 +13,9 @@
# limitations under the License.
{{- if .Values.ingress.enabled -}}
{{$nameSuffix := include "os-wellbore-ddms.name-suffix" .}}
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
{{ include "os-wellbore-ddms.commonAnnotations" . | indent 4}}
appgw.ingress.kubernetes.io/ssl-redirect: "true"
appgw.ingress.kubernetes.io/connection-draining: "true"
appgw.ingress.kubernetes.io/connection-draining-timeout: "30"
......@@ -30,14 +27,14 @@ metadata:
labels:
{{ include "os-wellbore-ddms.commonLabels" . | indent 4}}
name: {{ .Values.deployment.name }}{{ $nameSuffix }}
name: {{ .Values.deployment.name }}
{{ include "os-wellbore-ddms.namespace" . | indent 2}}
spec:
rules:
- http:
paths:
- backend:
serviceName: {{ .Values.deployment.name }}{{ $nameSuffix }}
serviceName: {{ .Values.deployment.name }}
servicePort: 80
path: {{ include "os-wellbore-ddms.prefix" . }}/*
{{ if .Values.ingress.hosts.host }}
......
......@@ -15,11 +15,9 @@
apiVersion: v1
kind: Service
metadata:
annotations:
{{ include "os-wellbore-ddms.commonAnnotations" . | indent 4}}
labels:
{{ include "os-wellbore-ddms.commonLabels" . | indent 4}}
name: {{ .Values.deployment.name }}{{ include "os-wellbore-ddms.name-suffix" . }}
name: wellbore-domain-services
{{ include "os-wellbore-ddms.namespace" . | indent 2}}
spec:
ports:
......
......@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Default values for os-wellbore-ddms.
# Default values for osdu-gitlab-wellbore-domain-services.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
......@@ -22,20 +22,15 @@ deployment:
secretName: central-logging
osduSvcProperties: osdu-svc-properties
keyvaultUrlPropertyName: ENV_KEYVAULT
name: os-wellbore-ddms
name: osdu-gitlab-wellbore-domain-services
replicaCount: 2
annotations:
buildNumber: #{Build.BuildNumber}#
buildOrigin: AzureDevops build/#{Build.DefinitionName}#
commitBranch: #{Build.SourceBranch}#
commitId: #{Build.SourceVersion}#
replicaCount: 1
image:
repository: #{CONTAINER_REGISTRY_NAME}#.azurecr.io/#{app}#-#{env}#
# Overrides the image tag whose default is the chart appVersion.
tag: #{Build.SourceVersion}#
repository: community.opengroup.org:5555/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services
branch: trusted-w-azure-pipelines
tag: latest
tempDeployment:
enabled: false
......@@ -43,7 +38,7 @@ tempDeployment:
labels:
aadpodidbinding: osdu-identity
env: #{env}#
env: glab
configMap:
data:
......
import os
import msal
import logging
import sys
logging.basicConfig(level=os.environ.get("LOG_LEVEL", "INFO"))
def get_id_token():
tenant_id = os.getenv('AZURE_TENANT_ID')
resource_id = os.getenv('AZURE_AD_APP_RESOURCE_ID')
client_id = os.getenv('INTEGRATION_TESTER')
client_secret = os.getenv('AZURE_TESTER_SERVICEPRINCIPAL_SECRET')
authority_host_uri = 'https://login.microsoftonline.com'
authority_uri = authority_host_uri + '/' + tenant_id
scopes = [resource_id + '/.default']
try:
app = msal.ConfidentialClientApplication(client_id=client_id, authority=authority_uri, client_credential=client_secret)
result = app.acquire_token_for_client(scopes=scopes)
print(result.get('access_token'))
return result.get('access_token')
except Exception as e:
print(e)
def get_invalid_token():
'''
This is dummy jwt
{
"sub": "dummy@dummy.com",
"iss": "dummy@dummy.com",
"aud": "dummy.dummy.com",
"iat": 1556137273,
"exp": 1556223673,
"provider": "dummy.com",
"client": "dummy.com",
"userid": "dummytester.com",
"email": "dummytester.com",
"authz": "",
"lastname": "dummy",
"firstname": "dummy",
"country": "",
"company": "",
"jobtitle": "",
"subid": "dummyid",
"idp": "dummy",
"hd": "dummy.com",
"desid": "dummyid",
"contact_email": "dummy@dummy.com"
}
'''
return "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJkdW1teUBkdW1teS5jb20iLCJpc3MiOiJkdW1teUBkdW1teS5jb20iLCJhdWQiOiJkdW1teS5kdW1teS5jb20iLCJpYXQiOjE1NTYxMzcyNzMsImV4cCI6MTU1NjIzMDk3OSwicHJvdmlkZXIiOiJkdW1teS5jb20iLCJjbGllbnQiOiJkdW1teS5jb20iLCJ1c2VyaWQiOiJkdW1teXRlc3Rlci5jb20iLCJlbWFpbCI6ImR1bW15dGVzdGVyLmNvbSIsImF1dGh6IjoiIiwibGFzdG5hbWUiOiJkdW1teSIsImZpcnN0bmFtZSI6ImR1bW15IiwiY291bnRyeSI6IiIsImNvbXBhbnkiOiIiLCJqb2J0aXRsZSI6IiIsInN1YmlkIjoiZHVtbXlpZCIsImlkcCI6ImR1bW15IiwiaGQiOiJkdW1teS5jb20iLCJkZXNpZCI6ImR1bW15aWQiLCJjb250YWN0X2VtYWlsIjoiZHVtbXlAZHVtbXkuY29tIiwianRpIjoiNGEyMWYyYzItZjU5Yy00NWZhLTk0MTAtNDNkNDdhMTg4ODgwIn0.nkiyKtfXXxAlC60iDjXuB2EAGDfZiVglP-CyU1T4etc"
if __name__ == '__main__':
get_id_token()
\ No newline at end of file
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment