service-data.md 8.16 KB
Newer Older
1
2
3
4
5
6
7
8
9
10
11
12
# Load Service Data

## Service Schema Loading

Schema Service has standard shared schemas that have to be loaded.


```bash
# Setup Variables
UNIQUE="<your_osdu_unique>"         # ie: demo
AZURE_DNS_NAME="<your_osdu_fqdn>"   # ie: osdu-$UNIQUE.contoso.com
DATA_PARTITION="<your_partition>"   # ie:opendes
13
ACR_REGISTRY="<repository>"         # ie: msosdu.azurecr.io
Vivek Ojha's avatar
Vivek Ojha committed
14
TAG="<app_version>"                 # ie: 0.10.0
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

# This logs your local Azure CLI in using the configured service principal.
az login --service-principal -u $ARM_CLIENT_ID -p $ARM_CLIENT_SECRET --tenant $ARM_TENANT_ID

GROUP=$(az group list --query "[?contains(name, 'cr${UNIQUE}')].name" -otsv)
ENV_VAULT=$(az keyvault list --resource-group $GROUP --query [].name -otsv)

cat > .env << EOF
DATA_PARTITION=$DATA_PARTITION
AZURE_TENANT_ID=$ARM_TENANT_ID
AZURE_DNS_NAME=$AZURE_DNS_NAME
AZURE_AD_APP_RESOURCE_ID=$(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/aad-client-id --query value -otsv)
AZURE_CLIENT_ID=$(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/app-dev-sp-username --query value -otsv)
AZURE_CLIENT_SECRET=$(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/app-dev-sp-password --query value -otsv)
EOF

# Execute container to load the schema's
32
docker run --env-file .env $ACR_REGISTRY/schema-load:$TAG
33
```
MANISH KUMAR's avatar
MANISH KUMAR committed
34
35
36
37
38
39
40
41
42
43
44
## Policy Data Loading

Policy Service has standard shared policies that have to be loaded.


```bash
# Setup Variables
UNIQUE="<your_osdu_unique>"         # ie: demo
AZURE_DNS_NAME="<your_osdu_fqdn>"   # ie: osdu-$UNIQUE.contoso.com
DATA_PARTITION="<your_partition>"   # ie:opendes
ACR_REGISTRY="<repository>"         # ie: msosdu.azurecr.io
Vivek Ojha's avatar
Vivek Ojha committed
45
TAG="<app_version>"                 # ie: 0.10.0
MANISH KUMAR's avatar
MANISH KUMAR committed
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

# This logs your local Azure CLI in using the configured service principal.
az login --service-principal -u $ARM_CLIENT_ID -p $ARM_CLIENT_SECRET --tenant $ARM_TENANT_ID

GROUP=$(az group list --query "[?contains(name, 'cr${UNIQUE}')].name" -otsv)
ENV_VAULT=$(az keyvault list --resource-group $GROUP --query [].name -otsv)

cat > .env << EOF
DATA_PARTITION=$DATA_PARTITION
AZURE_TENANT_ID=$ARM_TENANT_ID
AZURE_DNS_NAME=$AZURE_DNS_NAME
AZURE_AD_APP_RESOURCE_ID=$(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/aad-client-id --query value -otsv)
AZURE_CLIENT_ID=$(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/app-dev-sp-username --query value -otsv)
AZURE_CLIENT_SECRET=$(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/app-dev-sp-password --query value -otsv)
EOF

# Execute container to load the policies
docker run --env-file .env $ACR_REGISTRY/policy-data:$TAG
```
Daniel Scholl's avatar
Daniel Scholl committed
65
66
67
68
69
70
71
72
73
74
75
76

## CSV Parser DAG Loading

The CSV Ingestion is an Airflow DAG that has to be loaded.  The images are created as part of the [csv-parser](https://community.opengroup.org/osdu/platform/data-flow/ingestion/csv-parser/csv-parser) project and are tied to a release.


```bash
# Setup Variables
UNIQUE="<your_osdu_unique>"         # ie: demo
DNS_HOST="<your_osdu_fqdn>"         # ie: osdu-$UNIQUE.contoso.com
DATA_PARTITION="<your_partition>"   # ie:opendes
ACR_REGISTRY="<repository>"         # ie: msosdu.azurecr.io
Vivek Ojha's avatar
Vivek Ojha committed
77
TAG="<app_version>"                 # ie: 0.10.0
Daniel Scholl's avatar
Daniel Scholl committed
78
79
80
81
82
83
84
85
86

# This logs your local Azure CLI in using the configured service principal.
az login --service-principal -u $ARM_CLIENT_ID -p $ARM_CLIENT_SECRET --tenant $ARM_TENANT_ID

GROUP=$(az group list --query "[?contains(name, 'cr${UNIQUE}')].name" -otsv)
ENV_VAULT=$(az keyvault list --resource-group $GROUP --query [].name -otsv)

cat > .env << EOF
CSV_PARSER_IMAGE=${ACR_REGISTRY}/csv-parser:${TAG}
87
DATA_PARTITION=$DATA_PARTITION
Daniel Scholl's avatar
Daniel Scholl committed
88
89
90
91
92
93
94
95
96
AZURE_TENANT_ID=$ARM_TENANT_ID
AZURE_DNS_NAME=$DNS_HOST
AZURE_AD_APP_RESOURCE_ID=$(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/aad-client-id --query value -otsv)
AZURE_CLIENT_ID=$(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/app-dev-sp-username --query value -otsv)
AZURE_CLIENT_SECRET=$(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/app-dev-sp-password --query value -otsv)
EOF

docker run -it --env-file .env $ACR_REGISTRY/csv-parser-dag:$TAG
```
97
98
99
100
101
102
103
104
105

## SEGY to ZGY DAG Conversion - DAG Loading

For the SEGY to ZGY Conversion to happen, the conversion DAG needs to be loaded. The images are created as part of the [segy-to-zgy-conversion] (https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion.git) project and are tied to a release. [Here](community.opengroup.org:5555/osdu/platform/data-flow/ingestion/segy-to-zgy-conversion:latest) is the baseline image for SEGY.

Reference: [Open ZGY](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-zgy)

```bash
# Setup Variables
Sumra Zafar's avatar
Sumra Zafar committed
106
107
108
109
110
111
112
UNIQUE="<your_osdu_unique>"                    # ie: demo
AZURE_DNS_NAME="<your_osdu_fqdn>"              # ie: osdu-$UNIQUE.contoso.com
DATA_PARTITION="<your_partition>"              # ie:opendes
ACR_REGISTRY="<your_acr_fqdn>"                 # ie: myacr.azurecr.io
TAG="<app_version>"                            # ie: 0.10.0
DAG_TASK_IMAGE="segy-to-zgy-conversion-dag"    # i.e. name for the image in ACR
AZURE_TENANT_ID="<azure tenant>"
113
114
115
116
117
118
119
120

# This logs your local Azure CLI in using the configured service principal.
az login --service-principal -u $ARM_CLIENT_ID -p $ARM_CLIENT_SECRET --tenant $ARM_TENANT_ID

GROUP=$(az group list --query "[?contains(name, 'cr${UNIQUE}')].name" -otsv)
ENV_VAULT=$(az keyvault list --resource-group $GROUP --query [].name -otsv)

cat > .env << EOF
Sumra Zafar's avatar
Sumra Zafar committed
121
DAG_TASK_IMAGE=${ACR_REGISTRY}/segy-to-zgy-conversion:$TAG
122
SHARED_TENANT=$DATA_PARTITION
123
AZURE_DNS_NAME=$AZURE_DNS_NAME
Sumra Zafar's avatar
Sumra Zafar committed
124
AZURE_TENANT_ID=$AZURE_TENANT_ID
125
126
127
AZURE_AD_APP_RESOURCE_ID=$(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/aad-client-id --query value -otsv)
AZURE_CLIENT_ID=$(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/app-dev-sp-username --query value -otsv)
AZURE_CLIENT_SECRET=$(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/app-dev-sp-password --query value -otsv)
128
129
EOF

130
docker run -it --env-file .env $ACR_REGISTRY/segy-to-zgy-conversion-dag:$TAG
131
132
```

133
134


135
136
137
138
139
140
141
142
143
144
145
146
## SEGY to VDS DAG Conversion - DAG Loading

For the SEGY to VDS Conversion to happen, the conversion DAG needs to be loaded. The images are created as part of the [segy-to-vds-conversion] (https://community.opengroup.org/osdu/platform/data-flow/ingestion/segy-to-vds-conversion.git) project and are tied to a release. [Here](community.opengroup.org:5555/osdu/platform/domain-data-mgmt-services/seismic/open-vds/openvds-ingestion:latest) is the baseline image for SEGY.

Reference: [Open VDS](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/seismic/open-vds)

```bash
# Setup Variables
UNIQUE="<your_osdu_unique>"         # ie: demo
AZURE_DNS_NAME="<your_osdu_fqdn>"   # ie: osdu-$UNIQUE.contoso.com
DATA_PARTITION="<your_partition>"   # ie:opendes
ACR_REGISTRY="<your_acr_fqdn>"      # ie: myacr.azurecr.io
Vivek Ojha's avatar
Vivek Ojha committed
147
148
DAG_NAME="vds_dag"
TAG="latest"                        # For now the latest tag should be used for the image places in the Open VDS Project. For example, 0.10.0
Sumra Zafar's avatar
Sumra Zafar committed
149
AZURE_TENANT_ID="<azure tenant>"
150
151
152
153
154
155
156
157

# This logs your local Azure CLI in using the configured service principal.
az login --service-principal -u $ARM_CLIENT_ID -p $ARM_CLIENT_SECRET --tenant $ARM_TENANT_ID

GROUP=$(az group list --query "[?contains(name, 'cr${UNIQUE}')].name" -otsv)
ENV_VAULT=$(az keyvault list --resource-group $GROUP --query [].name -otsv)

cat > .env << EOF
Sumra Zafar's avatar
Sumra Zafar committed
158
DAG_TASK_IMAGE=${ACR_REGISTRY}/$DAG_NAME:$TAG
Sumra Zafar's avatar
Sumra Zafar committed
159
SHARED_TENANT=$DATA_PARTITION
160
161
AZURE_TENANT_ID=$AZURE_TENANT_ID
AZURE_DNS_NAME=$AZURE_DNS_NAME
Sumra Zafar's avatar
Sumra Zafar committed
162
163
164
AZURE_AD_APP_RESOURCE_ID=$(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/aad-client-id --query value -otsv)
AZURE_CLIENT_ID=$(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/app-dev-sp-username --query value -otsv)
AZURE_CLIENT_SECRET=$(az keyvault secret show --id https://${ENV_VAULT}.vault.azure.net/secrets/app-dev-sp-password --query value -otsv)
165
166
EOF

Vivek Ojha's avatar
Vivek Ojha committed
167
#(cd ../../.. && docker build -f deployments/scripts/azure/Dockerfile -t $ACR_REGISTRY/$DAG_NAME:$TAG .)
Sumra Zafar's avatar
Sumra Zafar committed
168
docker run -it --env-file .env $ACR_REGISTRY/segy-to-vds-conversion-dag:$TAG
169
```