Commit 99e6a215 authored by Kishore Battula's avatar Kishore Battula
Browse files

Merge branch 'tenant-level-mappings-enhancements' into 'master'

Tenant level Mappings Enhancement

See merge request !22
parents 35f73d7c 057e5b92
Pipeline #18691 passed with stages
in 14 minutes and 50 seconds
......@@ -3,9 +3,6 @@ variables:
AZURE_BUILD_SUBDIR: provider/wks-azure
AZURE_TEST_SUBDIR: testing/wks-test-core
AZURE_DEPLOYMENTS_SUBDIR: deployments/scripts/azure
STORAGE_CONTAINER: osdu-wks-mappings
AZURE_STORAGE_ACCOUNT_NAME: ${AZURE_BASE}data
AZURE_STORAGE_ACCOUNT_KEY: ${AZURE_STORAGE_KEY}
include:
- project: 'osdu/platform/ci-cd-pipelines'
......@@ -28,4 +25,4 @@ include:
ref: master
file: "cloud-providers/azure.yml"
- local: "/devops/azure/bootstrap.yaml"
- local: "/devops/azure/bootstrap.yml"
......@@ -350,7 +350,7 @@ The following software have components provided under the terms of this license:
- jakarta.xml.bind-api (from )
========================================================================
CC-BY-3.0
CC-BY-2.5
========================================================================
The following software have components provided under the terms of this license:
......@@ -379,7 +379,6 @@ CDDL-1.0
The following software have components provided under the terms of this license:
- JavaBeans Activation Framework API jar (from )
- JavaBeans(TM) Activation Framework (from http://java.sun.com/javase/technologies/desktop/javabeans/jaf/index.jsp)
- JavaMail API (from )
- javax.annotation-api (from http://jcp.org/en/jsr/detail?id=250)
......@@ -392,14 +391,21 @@ The following software have components provided under the terms of this license:
- Java Architecture For XML Binding (from )
- JavaBeans Activation Framework (from )
- JavaBeans(TM) Activation Framework (from http://java.sun.com/javase/technologies/desktop/javabeans/jaf/index.jsp)
- JavaBeans(TM) Activation Framework (from http://java.sun.com/javase/technologies/desktop/javabeans/jaf/index.jsp)
- tomcat-embed-core (from http://tomcat.apache.org/)
========================================================================
EPL-1.0
CPL-1.0
========================================================================
The following software have components provided under the terms of this license:
- JUnit (from http://junit.org)
========================================================================
EPL-1.0
========================================================================
The following software have components provided under the terms of this license:
- JUnit Jupiter (Aggregator) (from https://junit.org/junit5/)
- JUnit Jupiter (Aggregator) (from https://junit.org/junit5/)
- Logback Classic Module (from )
......@@ -499,7 +505,6 @@ LGPL-2.1-or-later
========================================================================
The following software have components provided under the terms of this license:
- Java Native Access (from https://github.com/java-native-access/jna)
- Java Native Access Platform (from https://github.com/java-native-access/jna)
- SnakeYAML (from http://www.snakeyaml.org)
......@@ -568,6 +573,7 @@ The following software have components provided under the terms of this license:
- Spongy Castle (from http://rtyley.github.io/spongycastle/)
- Spring Data for Azure Cosmos DB SQL API (from https://github.com/Azure/azure-sdk-for-java/tree/master/sdk/cosmos/azure-spring-data-cosmos)
- adal4j (from https://github.com/AzureAD/azure-activedirectory-library-for-java)
- azure-cosmos (from https://github.com/Azure/azure-sdk-for-python)
- azure-storage-blob (from https://github.com/Azure/azure-storage-python)
- micrometer-core (from https://github.com/micrometer-metrics/micrometer)
- mockito-junit-jupiter (from https://github.com/mockito/mockito)
......@@ -604,7 +610,6 @@ Public-Domain
========================================================================
The following software have components provided under the terms of this license:
- AOP alliance (from http://aopalliance.sourceforge.net)
- HdrHistogram (from http://hdrhistogram.github.io/HdrHistogram/)
- LatencyUtils (from http://latencyutils.github.io/LatencyUtils/)
- Spongy Castle (from http://rtyley.github.io/spongycastle/)
......@@ -628,6 +633,7 @@ public-domain
========================================================================
The following software have components provided under the terms of this license:
- AOP alliance (from http://aopalliance.sourceforge.net)
- Apache Groovy (from http://groovy-lang.org)
- Asynchronous Http Client (from )
- Guava: Google Core Libraries for Java (from https://github.com/google/guava.git)
......@@ -641,6 +647,7 @@ The following software have components provided under the terms of this license:
- Microsoft Azure client library for Blob Storage (from https://github.com/Azure/azure-sdk-for-java)
- Project Lombok (from https://projectlombok.org)
- Spring Web (from https://github.com/spring-projects/spring-framework)
- StAX API (from http://stax.codehaus.org/)
- msal4j (from https://github.com/AzureAD/microsoft-authentication-library-for-java)
- reactive-streams (from http://www.reactive-streams.org/)
......@@ -651,6 +658,7 @@ The following software have components provided under the terms of this license:
- Byte Buddy (without dependencies) (from )
- JAXB OSGi bundle for GlassFish V3 (from http://jaxb.java.net/)
- JUnit (from http://junit.org)
- JUnit Jupiter (Aggregator) (from https://junit.org/junit5/)
- JUnit Jupiter (Aggregator) (from https://junit.org/junit5/)
- JavaBeans Activation Framework API jar (from )
......
{
"id": "1a3c1f16-25c7-44cf-9fbd-7d9cbc90805a",
"mappingId": "1a3c1f16-25c7-44cf-9fbd-7d9cbc90805a",
"mappingVersion": "1.0.0",
"mappingAuthority": "osdu",
"mappingKind": "osdu:wks:well:1.0.0",
"mappingSchemaMajorVersion": "1",
"mappingSchemaMinorVersion": "0",
"mappingSchemaPatchVersion": "0",
"sourceSchemaAuthority": "opendes",
"sourceSchemaSource": "oga",
"sourceEntityType": "well",
"sourceSchemaMajorVersion": "1",
"sourceSchemaKind": "opendes:oga:well:1",
"targetSchemaAuthority": "slb",
"targetSchemaSource": "wks",
"targetEntityType": "well",
"targetSchemaMajorVersion": "4",
"createdBy": "testUser",
"fileName": "1a3c1f16-25c7-44cf-9fbd-7d9cbc90805a.json",
"createdOnEpoch": "1606039291",
"mappingScope": "INTERNAL"
}
\ No newline at end of file
{
"id": "4b312ef2-447d-4898-9d54-2a401975bd0b",
"mappingId": "4b312ef2-447d-4898-9d54-2a401975bd0b",
"mappingVersion": "1.0.0",
"mappingAuthority": "osdu",
"mappingSchemaMajorVersion": "1",
"mappingSchemaMinorVersion": "0",
"mappingSchemaPatchVersion": "0",
"mappingKind": "osdu:wks:wellbore:1.0.0",
"sourceSchemaAuthority": "opendes",
"sourceSchemaSource": "ihs",
"sourceEntityType": "log",
"sourceSchemaMajorVersion": "1",
"sourceSchemaKind": "opendes:ihs:log:1",
"targetSchemaAuthority": "slb",
"targetSchemaSource": "wks",
"targetEntityType": "log",
"targetSchemaMajorVersion": "2",
"createdBy": "testUser",
"fileName": "4b312ef2-447d-4898-9d54-2a401975bd0b.json",
"createdOnEpoch": "1606039291",
"mappingScope": "INTERNAL"
}
\ No newline at end of file
{
"id": "aa0ee7f6-2c0b-474a-962c-3e837a3b7de3",
"mappingId": "aa0ee7f6-2c0b-474a-962c-3e837a3b7de3",
"mappingVersion": "1.0.0",
"mappingAuthority": "osdu",
"mappingKind": "osdu:wks:well:1.0.0",
"mappingSchemaMajorVersion": "1",
"mappingSchemaMinorVersion": "0",
"mappingSchemaPatchVersion": "0",
"sourceSchemaAuthority": "opendes",
"sourceSchemaSource": "oga",
"sourceEntityType": "well",
"sourceSchemaMajorVersion": "1",
"sourceSchemaKind": "opendes:oga:well:1",
"targetSchemaAuthority": "abc",
"targetSchemaSource": "wks",
"targetEntityType": "well",
"targetSchemaMajorVersion": "1",
"createdBy": "testUser",
"fileName": "aa0ee7f6-2c0b-474a-962c-3e837a3b7de3.json",
"createdOnEpoch": "1606039291",
"mappingScope": "INTERNAL"
}
\ No newline at end of file
{
"id": "d61a36bc-7b7b-4e7c-8a9c-33e7315bc076",
"mappingId": "d61a36bc-7b7b-4e7c-8a9c-33e7315bc076",
"mappingVersion": "1.0.0",
"mappingAuthority": "osdu",
"mappingSchemaMajorVersion": "1",
"mappingSchemaMinorVersion": "0",
"mappingSchemaPatchVersion": "0",
"mappingKind": "osdu:wks:log:1.0.0",
"sourceSchemaAuthority": "opendes",
"sourceSchemaSource": "ihs",
"sourceEntityType": "log",
"sourceSchemaMajorVersion": "1",
"sourceSchemaKind": "opendes:ihs:log:1",
"targetSchemaAuthority": "abc",
"targetSchemaSource": "wks",
"targetEntityType": "log",
"targetSchemaMajorVersion": "3",
"createdBy": "testUser",
"fileName": "d61a36bc-7b7b-4e7c-8a9c-33e7315bc076.json",
"createdOnEpoch": "1606039291",
"mappingScope": "INTERNAL"
}
\ No newline at end of file
{
"id": "dda587cd-770b-4104-bae3-3f834c3a8aec",
"mappingId": "dda587cd-770b-4104-bae3-3f834c3a8aec",
"mappingVersion": "1.0.0",
"mappingAuthority": "osdu",
"mappingKind": "osdu:wks:well:1.0.0",
"mappingSchemaMajorVersion": "1",
"mappingSchemaMinorVersion": "0",
"mappingSchemaPatchVersion": "0",
"sourceSchemaAuthority": "opendes",
"sourceSchemaSource": "oga",
"sourceEntityType": "well",
"sourceSchemaMajorVersion": "1",
"sourceSchemaKind": "opendes:oga:well:1",
"targetSchemaAuthority": "slb",
"targetSchemaSource": "wks",
"targetEntityType": "well",
"targetSchemaMajorVersion": "5",
"createdBy": "testUser",
"fileName": "dda587cd-770b-4104-bae3-3f834c3a8aec.json",
"createdOnEpoch": "1606039291",
"mappingScope": "INTERNAL"
}
\ No newline at end of file
{
"id": "e2678b5f-5917-4bb1-8b30-2006efcb720d",
"mappingId": "e2678b5f-5917-4bb1-8b30-2006efcb720d",
"mappingVersion": "1.0.0",
"mappingAuthority": "osdu",
"mappingSchemaMajorVersion": "1",
"mappingSchemaMinorVersion": "0",
"mappingSchemaPatchVersion": "0",
"mappingKind": "osdu:wks:wellbore:1.0.0",
"sourceSchemaAuthority": "opendes",
"sourceSchemaSource": "at",
"sourceEntityType": "wellbore",
"sourceSchemaMajorVersion": "1",
"sourceSchemaKind": "opendes:at:wellbore:1",
"targetSchemaAuthority": "slb",
"targetSchemaSource": "wks",
"targetEntityType": "wellbore",
"targetSchemaMajorVersion": "1",
"createdBy": "testUser",
"fileName": "e2678b5f-5917-4bb1-8b30-2006efcb720d.json",
"createdOnEpoch": "1606039291",
"mappingScope": "INTERNAL"
}
\ No newline at end of file
{
"targetSchemaKind": "slb:wks:well:4.0.0",
"mappingId": "1a3c1f16-25c7-44cf-9fbd-7d9cbc90805a",
"attributeMappings": [
{
"rawAttributeName": "acl",
"wksAttributeName": "acl"
},
{
"rawAttributeName": "ancestry",
"wksAttributeName": "ancestry"
},
{
"rawAttributeName": "id",
"wksAttributeName": "id"
},
{
"rawAttributeName": "kind",
"wksAttributeName": "kind"
},
{
"rawAttributeName": "legal",
"wksAttributeName": "legal"
},
{
"rawAttributeName": "meta",
"wksAttributeName": "meta"
},
{
"rawAttributeName": "version",
"wksAttributeName": "version"
},
{
"rawAttributeName": "data.Spud date",
"wksAttributeName": "data.spudDate"
},
{
"rawAttributeName": "data.UWI",
"wksAttributeName": "data.uwi"
},
{
"rawAttributeName": "data.dlLatLongWGS84.latitude",
"wksAttributeName": "data.dlWGS84.latitude"
},
{
"rawAttributeName": "data.dlLatLongWGS84.longitude",
"wksAttributeName": "data.dlWGS84.longitude"
}
]
}
{
"wksSchemaKind": "opendes:wks:wellbore:1.0.0",
"targetSchemaKind": "slb:wks:log:2.0.0",
"mappingId": "4b312ef2-447d-4898-9d54-2a401975bd0b",
"attributeMappings": [
{
"rawAttributeName": "acl",
......
{
"targetSchemaKind": "abc:wks:well:1.0.0",
"mappingId": "aa0ee7f6-2c0b-474a-962c-3e837a3b7de3",
"attributeMappings": [
{
"rawAttributeName": "acl",
"wksAttributeName": "acl"
},
{
"rawAttributeName": "ancestry",
"wksAttributeName": "ancestry"
},
{
"rawAttributeName": "id",
"wksAttributeName": "id"
},
{
"rawAttributeName": "kind",
"wksAttributeName": "kind"
},
{
"rawAttributeName": "legal",
"wksAttributeName": "legal"
},
{
"rawAttributeName": "meta",
"wksAttributeName": "meta"
},
{
"rawAttributeName": "version",
"wksAttributeName": "version"
},
{
"rawAttributeName": "data.Spud date",
"wksAttributeName": "data.spudDate"
},
{
"rawAttributeName": "data.UWI",
"wksAttributeName": "data.uwi"
},
{
"rawAttributeName": "data.dlLatLongWGS84.latitude",
"wksAttributeName": "data.dlWGS84.latitude"
},
{
"rawAttributeName": "data.dlLatLongWGS84.longitude",
"wksAttributeName": "data.dlWGS84.longitude"
}
]
}
{
"wksSchemaKind": "opendes:wks:wellbore:1.0.0",
"targetSchemaKind": "abc:wks:log:3.0.0",
"mappingId": "d61a36bc-7b7b-4e7c-8a9c-33e7315bc076",
"attributeMappings": [
{
"rawAttributeName": "acl",
......
{
"targetSchemaKind": "slb:wks:well:5.0.0",
"mappingId": "dda587cd-770b-4104-bae3-3f834c3a8aec",
"attributeMappings": [
{
"rawAttributeName": "acl",
"wksAttributeName": "acl"
},
{
"rawAttributeName": "ancestry",
"wksAttributeName": "ancestry"
},
{
"rawAttributeName": "id",
"wksAttributeName": "id"
},
{
"rawAttributeName": "kind",
"wksAttributeName": "kind"
},
{
"rawAttributeName": "legal",
"wksAttributeName": "legal"
},
{
"rawAttributeName": "meta",
"wksAttributeName": "meta"
},
{
"rawAttributeName": "version",
"wksAttributeName": "version"
},
{
"rawAttributeName": "data.Spud date",
"wksAttributeName": "data.spudDate"
},
{
"rawAttributeName": "data.UWI",
"wksAttributeName": "data.uwi"
},
{
"rawAttributeName": "data.dlLatLongWGS84.latitude",
"wksAttributeName": "data.dlWGS84.latitude"
},
{
"rawAttributeName": "data.dlLatLongWGS84.longitude",
"wksAttributeName": "data.dlWGS84.longitude"
}
]
}
{
"targetSchemaKind": "slb:wks:wellbore:1.0.0",
"mappingId": "e2678b5f-5917-4bb1-8b30-2006efcb720d",
"attributeMappings": [
{
"rawAttributeName": "acl",
"wksAttributeName": "acl"
},
{
"rawAttributeName": "ancestry",
"wksAttributeName": "ancestry"
},
{
"rawAttributeName": "id",
"wksAttributeName": "id"
},
{
"rawAttributeName": "kind",
"wksAttributeName": "kind"
},
{
"rawAttributeName": "legal",
"wksAttributeName": "legal"
},
{
"rawAttributeName": "meta",
"wksAttributeName": "meta"
},
{
"rawAttributeName": "version",
"wksAttributeName": "version"
},
{
"rawAttributeName": "data.Spud date",
"wksAttributeName": "data.spudDate"
},
{
"rawAttributeName": "data.UWI",
"wksAttributeName": "data.uwi"
},
{
"rawAttributeName": "data.dlLatLongWGS84.latitude",
"wksAttributeName": "data.dlWGS84.latitude"
},
{
"rawAttributeName": "data.dlLatLongWGS84.longitude",
"wksAttributeName": "data.dlWGS84.longitude"
}
]
}
import json
import os
from os import listdir
from os.path import isfile, join
from azure.cosmos import CosmosClient
local_folder = os.path.join(os.path.dirname(__file__), "../../mapping_info_records/opendes")
cosmos_key = os.environ.get('AZURE_COSMOS_KEY')
cosmos_url = os.environ.get('AZURE_COSMOS_URL')
cosmos_database = 'osdu-db'
cosmos_container = 'MappingInfo'
client = CosmosClient(cosmos_url, cosmos_key)
database = client.get_database_client(cosmos_database)
container = database.get_container_client(cosmos_container)
def upload_records():
files = [f for f in listdir(local_folder) if isfile(join(local_folder, f))]
for file in files:
local_file = join(local_folder, file)
with open(local_file) as f:
data = json.load(f)
container.upsert_item(data)
if __name__=="__main__":
upload_records()
......@@ -6,9 +6,9 @@ from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
storage_account_name = os.environ.get('AZURE_STORAGE_ACCOUNT_NAME')
storage_account_key = os.environ.get('AZURE_STORAGE_ACCOUNT_KEY')
storage_account_url = "https://{}.blob.core.windows.net".format(storage_account_name)
container_name = os.environ.get('STORAGE_CONTAINER')
container_name = os.environ.get('AZURE_STORAGE_CONTAINER')
blob_service_client = BlobServiceClient(account_url = storage_account_url,credential = storage_account_key)
local_folder = os.path.join(os.path.dirname(__file__), "../../mappings")
local_folder = os.path.join(os.path.dirname(__file__), "../../mappings/opendes")
def upload_files():
files = [f for f in listdir(local_folder) if isfile(join(local_folder, f))]
......
azure.storage.blob
\ No newline at end of file
azure.storage.blob
azure.cosmos
\ No newline at end of file
.bootstrap_variables:
variables:
AZURE_STORAGE_CONTAINER: osdu-wks-mappings
AZURE_COSMOS_KEY: $AZURE_COSMOS_PRIMARY_KEY
AZURE_COSMOS_URL: $AZURE_COSMOS_CONNECTION
AZURE_STORAGE_ACCOUNT_NAME: ${AZURE_BASE}data
AZURE_STORAGE_ACCOUNT_KEY: ${AZURE_STORAGE_KEY}
azure_bootstrap:
stage: bootstrap
image: python:3.8
needs: ["azure_deploy"]
extends:
- .bootstrap_variables
script:
- pip install -r $AZURE_DEPLOYMENTS_SUBDIR/requirements.txt
- python $AZURE_DEPLOYMENTS_SUBDIR/deploy_mappings.py
- python $AZURE_DEPLOYMENTS_SUBDIR/deploy_mapping_info_records.py
only:
variables:
- $AZURE == 'true'
......
......@@ -94,6 +94,8 @@ spec:
value: "http://storage/api/storage/v2"
- name: search_service_endpoint
value: "http://search-service/api/search/v2"
- name: schema_service_endpoint
value: "http://schema-service/api/schema-service/v1"
- name: partition_service_endpoint
value: "http://partition/api/partition/v1"
- name: default_tenant
......@@ -103,4 +105,4 @@ spec:
- name: executor_n_threads
value: "8"
- name: max_lock_renew_duration_seconds
value: "120"
value: "600"
......@@ -34,8 +34,11 @@ Definitions for Azure Environment variables used
| `AZURE_TENANT_ID` | `********` | AD tenant to authenticate users from | yes | keyvault secret: `$KEYVAULT_URI/secrets/app-dev-sp-tenant-id` |
| `AZURE_CLIENT_SECRET` | `********` | Secret for `$AZURE_CLIENT_ID` | yes | keyvault secret: `$KEYVAULT_URI/secrets/app-dev-sp-password` |
| `search_service_endpoint` | ex `https://search-service/api/search/v2` | Search service endpoint | no | - |
| `storage_service_endpoint` | ex `https://storage-service/api/storage/v2` | Storage service endpoint | no | - |
| `schmea_service_endpoint` | ex `https://schema-service/api/schema/v1` | Schema service endpoint | no | - |
| `partition_service_endpoint` | ex `http://partition/api/partition/v1` | Partition service endpoint | no | - |
| `default_tenant` | ex `opendes` | Default tenant that hosts mapping files | no | output of infrastructure deployment |
>>>>>>> master
### Build and run the application
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment