Skip to content
GitLab
Explore
Sign in
Register
Primary navigation
Search or go to…
Project
W
WITSML Parser DAG
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Iterations
Wiki
Requirements
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Locked files
Build
Pipelines
Jobs
Pipeline schedules
Test cases
Artifacts
Deploy
Releases
Package Registry
Container Registry
Model registry
Operate
Environments
Terraform modules
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Code review analytics
Issue analytics
Insights
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
OSDU Software
OSDU Data Platform
Data Flow
Data Ingestion
Energistics DAGs - OSDU Integration
WITSML Parser DAG
Commits
deac66ee
Commit
deac66ee
authored
1 year ago
by
saketh somaraju [EPAM]
Browse files
Options
Downloads
Patches
Plain Diff
standardize azure related dag implementation
parent
25c78db8
No related branches found
No related tags found
1 merge request
!154
standardize azure related dag implementation
Changes
3
Hide whitespace changes
Inline
Side-by-side
Showing
3 changed files
.gitlab-ci.yml
+11
-7
11 additions, 7 deletions
.gitlab-ci.yml
deployments/scripts/azure/prepare_dags.sh
+3
-0
3 additions, 0 deletions
deployments/scripts/azure/prepare_dags.sh
deployments/scripts/azure/setup/render_dag_file.py
+2
-2
2 additions, 2 deletions
deployments/scripts/azure/setup/render_dag_file.py
with
16 additions
and
9 deletions
.gitlab-ci.yml
+
11
−
7
View file @
deac66ee
...
...
@@ -27,6 +27,8 @@ workflow:
variables
:
OSDU_AZURE_POSTFIX
:
osdu-azure
OSDU_AZURE_BASE_IMAGE_TAG
:
$CI_REGISTRY_IMAGE/$OSDU_AZURE_POSTFIX:$CI_COMMIT_SHORT_SHA
AZURE_WITSML_IMAGE_NAME
:
${CI_PROJECT_NAME}:${CI_COMMIT_SHA}
AZURE_PUBLISH_PARSER_IMAGE
:
"
true"
WORKDIR
:
energistics
SRCDIR
:
energistics/src
...
...
@@ -77,6 +79,7 @@ stages:
-
bootstrap
-
end_to_end_postman_tests
-
deploy_preship
-
publish
.azure_prepare_dag_env
:
variables
:
...
...
@@ -100,18 +103,15 @@ build_parser_azure:
tags
:
[
"
osdu-medium"
]
image
:
danielscholl/azure-build-image
variables
:
SHA_IMAGE
:
${CI_PROJECT_NAME}
-${CI_COMMIT_REF_SLUG}
:${CI_COMMIT_SHA}
SHA_IMAGE
:
${CI_PROJECT_NAME}:${CI_COMMIT_SHA}
LATEST_IMAGE
:
${CI_PROJECT_NAME}-${CI_COMMIT_REF_SLUG}:latest
RELEASE_IMAGE
:
release-${CI_COMMIT_TAG}:
${CI_PROJECT_NAME}
-
${CI_COMMIT_
TAG
}
RELEASE_IMAGE
:
${CI_PROJECT_NAME}
:
${CI_COMMIT_
SHA
}
before_script
:
-
az login --service-principal -u $AZURE_PRINCIPAL_ID -p $AZURE_PRINCIPAL_SECRET --tenant $AZURE_TENANT_ID
-
az acr login -n $AZURE_REGISTRY
script
:
-
|
if [ "$CI_COMMIT_TAG" != "" ]; then
docker build --target azure -t ${AZURE_REGISTRY}.azurecr.io/$RELEASE_IMAGE --rm -f ./build/providers/azure/Dockerfile .
docker push ${AZURE_REGISTRY}.azurecr.io/$RELEASE_IMAGE
elif [ "$CI_COMMIT_REF_NAME" = "master" ]; then
if [ "$CI_COMMIT_REF_NAME" = "master" ]; then
docker build --target azure -t ${AZURE_REGISTRY}.azurecr.io/$LATEST_IMAGE --rm -f ./build/providers/azure/Dockerfile .
docker push ${AZURE_REGISTRY}.azurecr.io/$LATEST_IMAGE
fi
...
...
@@ -134,6 +134,10 @@ azure_build_dag:
-
.azure_variables
-
.azure_prepare_dag_env
before_script
:
-
|
if [ ! -z "$CI_COMMIT_TAG" ]; then
AZURE_WITSML_IMAGE_NAME="${CI_PROJECT_NAME}:${CI_COMMIT_SHA}"
fi
-
|
# Generating environment file to be passed while running the docker container
cat > .env << EOF
...
...
@@ -153,7 +157,7 @@ azure_build_dag:
DAG_IMAGE=${DAG_IMAGE}
SHARED_TENANT=$SHARED_TENANT
AZURE_DNS_NAME=$AZURE_DNS_NAME
WITSML_IMAGE_NAME=${AZURE_
REGISTRY}.azurecr.io/${CI_PROJECT_NAME}-${CI_COMMIT_REF_SLUG}:${CI_COMMIT_SHA
}
WITSML_IMAGE_NAME=${AZURE_
WITSML_IMAGE_NAME
}
K8S_NAMESPACE=$AZURE_K8S_NAMESPACE
AZURE_DAG_DIRECTORY=$AZURE_DAG_DIRECTORY
DATA_PARTITION=$DATA_PARTITION
...
...
This diff is collapsed.
Click to expand it.
deployments/scripts/azure/prepare_dags.sh
+
3
−
0
View file @
deac66ee
...
...
@@ -20,6 +20,9 @@ echo "WITSML_IMAGE_NAME: $WITSML_IMAGE_NAME"
echo
"K8S_NAMESPACE:
$K8S_NAMESPACE
"
echo
"DATA_PARTITION:
$DATA_PARTITION
"
echo
"OSDU_AZURE_BASE_URL:
$OSDU_AZURE_BASE_URL
"
echo
"CI_PROJECT_NAME:
$CI_PROJECT_NAME
"
echo
"CI_COMMIT_SHA:
$CI_COMMIT_SHA
"
pip
install
-r
"/home/scripts/requirements.txt"
...
...
This diff is collapsed.
Click to expand it.
deployments/scripts/azure/setup/render_dag_file.py
+
2
−
2
View file @
deac66ee
...
...
@@ -11,7 +11,7 @@ class DAGFileRenderer:
def
__init__
(
self
,
file_path
:
str
):
self
.
file_path
=
file_path
self
.
dag_name
=
DAG_NAME
self
.
docker_image
=
os
.
getenv
(
"
WITSML_IMAGE
_NAME
"
)
self
.
ci_project_name
=
os
.
environ
.
get
(
"
CI_PROJECT
_NAME
"
)
self
.
k8s_namespace
=
os
.
getenv
(
"
K8S_NAMESPACE
"
)
def
_prepare_env_vars
(
self
)
->
dict
:
...
...
@@ -52,7 +52,7 @@ class DAGFileRenderer:
params
=
{
"
DAG_NAME
"
:
self
.
dag_name
,
"
DOCKER_IMAGE
"
:
self
.
docker_image
,
"
DOCKER_IMAGE
"
:
"
{{ var.value.dag_image_acr }}/witsml-parser:
"
+
"
{{ var.value.
"
+
self
.
ci_project_name
.
replace
(
"
-
"
,
"
_
"
)
+
"
_image_tag }}
"
,
"
K8S_NAMESPACE
"
:
self
.
k8s_namespace
,
"
K8S_POD_KWARGS
"
:
self
.
_prepare_operator_kwargs
(),
"
ENV_VARS
"
:
self
.
_prepare_env_vars
(),
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment