Commit 95213e06 authored by Daniel Scholl's avatar Daniel Scholl
Browse files

Merge branch 'master' into schema_load

* master:
  Update version of default branch to 0.9.0-SNAPSHOT
  Deploy shared schema changes incorporated
  update aws bootstrap
  Dont print bearer token
  Accept bootstrap values from environment variables
  Accept bootstrap values from environment variables
  bootstrap resources built from DD c0ea3096980ca92023df6b1886c4e804bdaef105
  update documentation
  accept schema authority as argument, default osdu; try POST first, on failure try PUT (only supported for DEVELOPMENT); tolerate "Schema Id is already present" errors.
  introduce SCHEMA_AUTHORITY; python 3.9
  Making spring app name and container name consistent
  updating the notice file
  upgrading the core-lib-azure version
  Azure cutting over to entitlements v2
  Adding schema-service tutorial
  Updating FOSSA configuration and the resulting NOTICE file
  Switching the dependencies to release versions
  Update version of default branch to 0.8.0-SNAPSHOT
parents 7605c303 d9fdadee
......@@ -5,7 +5,7 @@ version: 2
cli:
server: https://app.fossa.com
fetcher: custom
project: OS Schema
project: Schema
analyze:
modules:
- name: os-schema
......@@ -16,15 +16,35 @@ analyze:
type: mvn
target: schema-core/pom.xml
path: .
- name: os-schema-aws
type: mvn
target: provider/schema-aws/pom.xml
path: .
- name: os-schema-gcp
type: mvn
target: provider/schema-gcp/pom.xml
path: .
- name: os-schema-azure
- name: os-schema-ibm
type: mvn
target: provider/schema-azure/pom.xml
target: provider/schema-ibm/pom.xml
path: .
- name: os-schema-aws
- name: os-schema-azure
type: mvn
target: provider/schema-aws/pom.xml
target: provider/schema-azure/pom.xml
path: .
- name: aws
type: pip
target: deployments/scripts/aws
path: deployments/scripts/aws
- name: azure
type: pip
target: deployments/scripts/azure
path: deployments/scripts/azure
- name: google
type: pip
target: deployments/scripts/google
path: deployments/scripts/google
- name: scripts
type: pip
target: deployments/scripts
path: deployments/scripts
......@@ -21,6 +21,8 @@ Apache-2.0
========================================================================
The following software have components provided under the terms of this license:
- AMQP 1.0 JMS Spring Boot AutoConfiguration (from https://repo1.maven.org/maven2/org/amqphub/spring/amqp-10-jms-spring-boot-autoconfigure)
- AMQP 1.0 JMS Spring Boot Starter (from https://repo1.maven.org/maven2/org/amqphub/spring/amqp-10-jms-spring-boot-starter)
- ASM Core (from )
- ASM based accessors helper used by json-smart (from )
- AWS Event Stream (from https://github.com/awslabs/aws-eventstream-java)
......@@ -225,9 +227,11 @@ The following software have components provided under the terms of this license:
- AWS SDK for Java - Models (from https://aws.amazon.com/sdkforjava)
- Adapter: RxJava (from )
- Apache Commons Codec (from http://commons.apache.org/proper/commons-codec/)
- Apache Commons Collections (from http://commons.apache.org/proper/commons-collections/)
- Apache Commons Lang (from http://commons.apache.org/proper/commons-lang/)
- Apache Commons Logging (from http://commons.apache.org/proper/commons-logging/)
- Apache Commons Logging (from http://commons.apache.org/proper/commons-logging/)
- Apache Geronimo JMS Spec 2.0 (from http://geronimo.apache.org/maven/${siteId}/${version})
- Apache HttpAsyncClient (from http://hc.apache.org/httpcomponents-asyncclient)
- Apache HttpClient (from http://hc.apache.org/httpcomponents-client)
- Apache HttpClient Cache (from http://hc.apache.org/httpcomponents-client)
......@@ -250,6 +254,7 @@ The following software have components provided under the terms of this license:
- ClassMate (from http://github.com/cowtowncoder/java-classmate)
- Cloud Storage JSON API v1-rev58-1.21.0 (from )
- CloudWatch Metrics for AWS Java SDK (from https://aws.amazon.com/sdkforjava)
- Commons IO (from http://commons.apache.org/io/)
- Converter: Jackson (from )
- Elastic JNA Distribution (from https://github.com/java-native-access/jna)
- Elasticsearch: 5.0.0-alpha5 (from https://github.com/elastic/elasticsearch)
......@@ -276,6 +281,10 @@ The following software have components provided under the terms of this license:
- HPPC Collections (from http://labs.carrotsearch.com)
- Hibernate Validator Engine (from )
- Hibernate Validator Engine (from )
- IBM COS Java SDK for Amazon S3 (from https://github.com/ibm/ibm-cos-sdk-java)
- IBM COS Java SDK for COS KMS (from https://github.com/ibm/ibm-cos-sdk-java)
- IBM COS SDK For Java (from https://github.com/ibm/ibm-cos-sdk-java)
- IBM COS SDK for Java - Core (from https://github.com/ibm/ibm-cos-sdk-java)
- Identity and Access Management (IAM) API v1-rev247-1.23.0 (from )
- IntelliJ IDEA Annotations (from http://www.jetbrains.org)
- J2ObjC Annotations (from https://github.com/google/j2objc/)
......@@ -310,6 +319,7 @@ The following software have components provided under the terms of this license:
- Java Libraries for Amazon Simple WorkFlow (from https://aws.amazon.com/sdkforjava)
- Java Native Access (from https://github.com/java-native-access/jna)
- Java Native Access Platform (from https://github.com/java-native-access/jna)
- Java UUID Generator (from http://wiki.fasterxml.com/JugHome)
- Javassist (from http://www.javassist.org/)
- Joda-Time (from http://www.joda.org/joda-time/)
- Json Path (from https://github.com/jayway/JsonPath)
......@@ -343,6 +353,7 @@ The following software have components provided under the terms of this license:
- Microsoft Application Insights Java SDK Spring Boot starter (from https://github.com/Microsoft/ApplicationInsights-Java)
- Microsoft Application Insights Java SDK Web Module (from https://github.com/Microsoft/ApplicationInsights-Java)
- Microsoft Application Insights Log4j 2 Appender (from https://github.com/Microsoft/ApplicationInsights-Java)
- Microsoft Azure Java Core Library (from https://github.com/Azure/azure-sdk-for-java)
- Microsoft Azure Netty HTTP Client Library (from https://github.com/Azure/azure-sdk-for-java)
- Microsoft Azure SDK for SQL API of Azure Cosmos DB Service (from https://github.com/Azure/azure-sdk-for-java)
- Mockito (from http://www.mockito.org)
......@@ -374,6 +385,7 @@ The following software have components provided under the terms of this license:
- Nimbus Content Type (from https://bitbucket.org/connect2id/nimbus-content-type)
- Nimbus JOSE+JWT (from https://bitbucket.org/connect2id/nimbus-jose-jwt)
- Nimbus JOSE+JWT (from https://bitbucket.org/connect2id/nimbus-jose-jwt)
- Nimbus JOSE+JWT (from https://bitbucket.org/connect2id/nimbus-jose-jwt)
- Nimbus LangTag (from https://bitbucket.org/connect2id/nimbus-language-tags)
- Non-Blocking Reactive Foundation for the JVM (from https://github.com/reactor/reactor)
- OAuth 2.0 SDK with OpenID Connect extensions (from https://bitbucket.org/connect2id/oauth-2.0-sdk-with-openid-connect-extensions)
......@@ -383,6 +395,7 @@ The following software have components provided under the terms of this license:
- OkHttp Logging Interceptor (from )
- OkHttp URLConnection (from )
- Okio (from )
- Okio (from )
- OpenCensus (from https://github.com/census-instrumentation/opencensus-java)
- OpenCensus (from https://github.com/census-instrumentation/opencensus-java)
- OpenCensus (from https://github.com/census-instrumentation/opencensus-java)
......@@ -390,6 +403,7 @@ The following software have components provided under the terms of this license:
- OpenCensus (from https://github.com/census-instrumentation/opencensus-java)
- PWDB :: Database (from https://repo1.maven.org/maven2/org/linguafranca/pwdb/database)
- Protocol Buffer extensions to the Google HTTP Client Library for Java. (from )
- QpidJMS Client (from )
- Reactive Streams Netty driver (from https://github.com/reactor/reactor-netty)
- Retrofit (from )
- Simple XML (from http://simple.sourceforge.net)
......@@ -419,6 +433,8 @@ The following software have components provided under the terms of this license:
- Spring Core (from https://github.com/spring-projects/spring-framework)
- Spring Data Core (from )
- Spring Expression Language (SpEL) (from https://github.com/spring-projects/spring-framework)
- Spring JMS (from https://github.com/spring-projects/spring-framework)
- Spring Messaging (from https://github.com/spring-projects/spring-framework)
- Spring Plugin - Core (from )
- Spring Plugin - Metadata Extension (from )
- Spring TestContext Framework (from https://github.com/spring-projects/spring-framework)
......@@ -433,6 +449,8 @@ The following software have components provided under the terms of this license:
- aalto-xml (from )
- aggs-matrix-stats (from https://github.com/elastic/elasticsearch)
- aws-ssm-java-caching-client (from https://github.com/awslabs/aws-ssm-java-caching-client)
- boto3 (from https://github.com/boto/boto3)
- botocore (from https://github.com/boto/botocore)
- cli (from https://github.com/elastic/elasticsearch)
- com.google.api.grpc:grpc-google-cloud-pubsub-v1 (from https://github.com/googleapis/googleapis)
- com.google.api.grpc:proto-google-cloud-logging-v2 (from https://github.com/googleapis/googleapis)
......@@ -443,6 +461,7 @@ The following software have components provided under the terms of this license:
- datastore-v1-proto-client (from )
- elasticsearch-core (from https://github.com/elastic/elasticsearch)
- error-prone annotations (from )
- google-auth (from https://github.com/GoogleCloudPlatform/google-auth-library-python)
- io.grpc:grpc-alts (from https://github.com/grpc/grpc-java)
- io.grpc:grpc-api (from https://github.com/grpc/grpc-java)
- io.grpc:grpc-auth (from https://github.com/grpc/grpc-java)
......@@ -455,8 +474,11 @@ The following software have components provided under the terms of this license:
- io.grpc:grpc-protobuf-lite (from https://github.com/grpc/grpc-java)
- io.grpc:grpc-stub (from https://github.com/grpc/grpc-java)
- ion-java (from https://github.com/amznlabs/ion-java/)
- ion-java (from https://github.com/amznlabs/ion-java/)
- jackson-databind (from http://github.com/FasterXML/jackson)
- jackson-databind (from http://github.com/FasterXML/jackson)
- java-cloudant (from https://cloudant.com)
- java-cloudant (from https://cloudant.com)
- javatuples (from http://www.javatuples.org)
- javax.inject (from http://code.google.com/p/atinject/)
- jose4j (from https://bitbucket.org/b_c/jose4j/)
......@@ -473,12 +495,16 @@ The following software have components provided under the terms of this license:
- proto-google-cloud-datastore-v1 (from https://github.com/googleapis/api-client-staging)
- proton-j (from )
- rank-eval (from https://github.com/elastic/elasticsearch)
- requests (from http://python-requests.org)
- rest (from https://github.com/elastic/elasticsearch)
- rest-high-level (from https://github.com/elastic/elasticsearch)
- rxjava (from https://github.com/ReactiveX/RxJava)
- secure-sm (from https://github.com/elastic/elasticsearch)
- spring-security-config (from http://spring.io/spring-security)
- spring-security-core (from http://spring.io/spring-security)
- spring-security-oauth2-core (from http://spring.io/spring-security)
- spring-security-oauth2-jose (from http://spring.io/spring-security)
- spring-security-oauth2-resource-server (from http://spring.io/spring-security)
- spring-security-web (from http://spring.io/spring-security)
- springfox-core (from https://github.com/springfox/springfox)
- springfox-schema (from https://github.com/springfox/springfox)
......@@ -548,6 +574,7 @@ The following software have components provided under the terms of this license:
- Spring Core (from https://github.com/spring-projects/spring-framework)
- Stax2 API (from http://github.com/FasterXML/stax2-api)
- ThreeTen backport (from https://www.threeten.org/threetenbp)
- google (from http://breakingcode.wordpress.com/)
- jakarta.xml.bind-api (from )
========================================================================
......@@ -727,6 +754,7 @@ The following software have components provided under the terms of this license:
- Animal Sniffer Annotations (from )
- Azure AD Spring Security Integration Spring Boot Starter (from https://github.com/Microsoft/azure-spring-boot)
- Azure Java Client Authentication Library for AutoRest (from https://github.com/Azure/autorest-clientruntime-for-java)
- Azure Java Client Runtime for ARM (from https://github.com/Azure/autorest-clientruntime-for-java)
- Azure Java Client Runtime for AutoRest (from https://github.com/Azure/autorest-clientruntime-for-java)
- Azure Metrics Spring Boot Starter (from https://github.com/Microsoft/azure-spring-boot)
- Azure Spring Boot AutoConfigure (from https://github.com/Microsoft/azure-spring-boot)
......@@ -757,10 +785,13 @@ The following software have components provided under the terms of this license:
- Microsoft Azure Java Core Library (from https://github.com/Azure/azure-sdk-for-java)
- Microsoft Azure Netty HTTP Client Library (from https://github.com/Azure/azure-sdk-for-java)
- Microsoft Azure SDK annotations (from https://github.com/Microsoft/java-api-annotations)
- Microsoft Azure SDK for EventGrid Management (from https://github.com/Azure/azure-sdk-for-java)
- Microsoft Azure SDK for SQL API of Azure Cosmos DB Service (from https://github.com/Azure/azure-sdk-for-java)
- Microsoft Azure SDK for Service Bus (from https://github.com/Azure/azure-sdk-for-java)
- Microsoft Azure SDK for eventgrid (from https://github.com/Azure/azure-sdk-for-java)
- Microsoft Azure client library for Blob Storage (from https://github.com/Azure/azure-sdk-for-java)
- Microsoft Azure client library for Identity (from https://github.com/Azure/azure-sdk-for-java)
- Microsoft Azure client library for KeyVault Keys (from https://github.com/Azure/azure-sdk-for-java)
- Microsoft Azure client library for KeyVault Secrets (from https://github.com/Azure/azure-sdk-for-java)
- Microsoft Azure common module for Storage (from https://github.com/Azure/azure-sdk-for-java)
- Microsoft Azure internal Avro module for Storage (from https://github.com/Azure/azure-sdk-for-java)
......@@ -772,13 +803,18 @@ The following software have components provided under the terms of this license:
- Netty/Common (from )
- Project Lombok (from https://projectlombok.org)
- Project Lombok (from https://projectlombok.org)
- PyJWT (from http://github.com/jpadilla/pyjwt)
- SLF4J API Module (from http://www.slf4j.org)
- Spongy Castle (from http://rtyley.github.io/spongycastle/)
- Spring Data for Azure Cosmos DB SQL API (from https://github.com/Azure/azure-sdk-for-java/tree/master/sdk/cosmos/azure-spring-data-cosmos)
- adal4j (from https://github.com/AzureAD/azure-activedirectory-library-for-java)
- azure-documentdb (from https://azure.microsoft.com/en-us/services/cosmos-db/)
- botocore (from https://github.com/boto/botocore)
- documentdb-bulkexecutor (from http://azure.microsoft.com/en-us/services/documentdb/)
- micrometer-core (from https://github.com/micrometer-metrics/micrometer)
- mockito-inline (from https://github.com/mockito/mockito)
- mockito-junit-jupiter (from https://github.com/mockito/mockito)
- msal (from https://github.com/AzureAD/microsoft-authentication-library-for-python)
- msal4j (from https://github.com/AzureAD/microsoft-authentication-library-for-java)
- msal4j-persistence-extension (from https://github.com/AzureAD/microsoft-authentication-extensions-for-java)
- spring-security-core (from http://spring.io/spring-security)
......@@ -815,6 +851,13 @@ The following software have components provided under the terms of this license:
- LatencyUtils (from http://latencyutils.github.io/LatencyUtils/)
- Spongy Castle (from http://rtyley.github.io/spongycastle/)
========================================================================
Python-2.0
========================================================================
The following software have components provided under the terms of this license:
- google-auth (from https://github.com/GoogleCloudPlatform/google-auth-library-python)
========================================================================
SPL-1.0
========================================================================
......@@ -845,12 +888,15 @@ The following software have components provided under the terms of this license:
- Joda-Time (from http://www.joda.org/joda-time/)
- LatencyUtils (from http://latencyutils.github.io/LatencyUtils/)
- Microsoft Application Insights Java SDK Core (from https://github.com/Microsoft/ApplicationInsights-Java)
- Microsoft Azure SDK for EventGrid Management (from https://github.com/Azure/azure-sdk-for-java)
- Microsoft Azure SDK for SQL API of Azure Cosmos DB Service (from https://github.com/Azure/azure-sdk-for-java)
- Microsoft Azure client library for Blob Storage (from https://github.com/Azure/azure-sdk-for-java)
- Project Lombok (from https://projectlombok.org)
- Project Lombok (from https://projectlombok.org)
- Spring Web (from https://github.com/spring-projects/spring-framework)
- StAX API (from http://stax.codehaus.org/)
- azure-documentdb (from https://azure.microsoft.com/en-us/services/cosmos-db/)
- botocore (from https://github.com/boto/botocore)
- msal4j (from https://github.com/AzureAD/microsoft-authentication-library-for-java)
- reactive-streams (from http://www.reactive-streams.org/)
......
......@@ -4,93 +4,134 @@ import time
import requests
import argparse
from Utility import Utility, RunEnv
from typing import Tuple
class DeploySharedSchemas:
SCHEMA_AUTHORITY_TO_REPLACE = '{{schema-authority}}'
SCHEMA_EXISTS = 'Schema Id is already present'
PUBLISHED_SCHEMA_ERROR = ['Only schema in development stage can be updated',
'Only schema in developement stage can be updated']
ALREADY_PUBLISHED = 'AlreadyPublished'
SUCCESS = 'Success'
def __init__(self):
parser = argparse.ArgumentParser(
description="Given a path to an load sequence file, load/update the schemas "
"listed in the load sequence file.")
parser.add_argument('-l', type=str,
help='The path to the load sequence file, e.g. load_sequence.?.?.?',
default=None)
parser.add_argument('-u', help='The complete URL to the Schema Service.',
default=None)
arguments = parser.parse_args()
if arguments.l is not None:
RunEnv.LOAD_SEQUENCE = arguments.l
if arguments.u is not None:
RunEnv.SCHEMA_SERVICE_URL = arguments.u
if RunEnv.SCHEMA_SERVICE_URL is None:
exit('The schema service URL is not specified')
self.url = RunEnv.SCHEMA_SERVICE_URL
self.schema_registered = None
self.token = RunEnv.BEARER_TOKEN
self.schema_info_registered = None
self.headers = {
'data-partition-id': RunEnv.DATA_PARTITION,
'Content-Type': 'application/json',
'AppKey': RunEnv.APP_KEY,
'Authorization': RunEnv.BEARER_TOKEN
}
print('Current data-partition-id: {}'.format(RunEnv.DATA_PARTITION))
ok, error_mess = RunEnv().is_ok()
if not ok:
exit('Error: environment setting incomplete: {}'.format(error_mess))
def create_schema(self):
messages = list()
tenant = RunEnv.DATA_PARTITION
deployments = Utility.path_to_deployments()
start = time.time()
headers = {
'data-partition-id': tenant,
'Content-Type': 'application/json',
'AppKey': RunEnv.APP_KEY,
'Authorization': self.token
}
bootstrap_options = json.loads(RunEnv.BOOTSTRAP_OPTIONS)
for option in bootstrap_options:
try:
schema_path = option['folder']
schema_authority = option['authority']
load_sequence = option['load-sequence']
except KeyError as e:
exit('Key missing in bootstrap-options::{}'.format(str(e)))
sequence = Utility.load_json(os.path.join(deployments, RunEnv.SCHEMAS_FOLDER, schema_path, load_sequence))
for item in sequence:
self.schema_registered = None
schema_file = os.path.join(deployments, item['relativePath'])
schema = open(schema_file, 'r').read()
schema = schema.replace(self.SCHEMA_AUTHORITY_TO_REPLACE, schema_authority)
kind = self.__kind_from_schema_info(schema)
self.__register_one(kind, schema, messages)
sequence = Utility.load_json(os.path.join(deployments, *RunEnv.OSDU, RunEnv.LOAD_SEQUENCE))
for item in sequence:
self.schema_registered = None
schema_file = os.path.join(deployments, item['relativePath'])
kind = item['kind']
schema = open(schema_file, 'r').read()
# attempt to load this kind
exists = self.__does_kind_exist(kind, headers)
method = 'POST'
if exists:
if schema_file.endswith('.dev.json'):
method = 'PUT'
response = requests.request(method, self.url, headers=headers, data=schema)
else:
message = 'Error: The published kind {} cannot be updated; it already exists.'.format(kind)
print(message)
messages.append(message)
response = None
elapsed = time.time() - start
print('This update took {:.2f} seconds.'.format(elapsed))
if len(messages) != 0:
print('Following schemas failed:')
print('\n'.join(messages))
exit(1)
else:
response = requests.request(method, self.url, headers=headers, data=schema)
print('All {} schemas registered, updated or '
'left unchanged because of status PUBLISHED.'.format(str(len(sequence))))
if response is not None:
print('Success: kind {} submitted with method {} schema.'.format(kind, method))
code = response.status_code
if code not in range(200,300):
# response_message = json.loads(response.text)
# if response_message.get('error'):
messages.append(
'Error with kind {}: Message: {}'.format(kind, response.text))
@staticmethod
def __kind_from_schema_info(schema_as_str: str) -> str:
kind = 'Error'
try:
schema = json.loads(schema_as_str)
si = schema.get('schemaInfo', dict()).get('schemaIdentity', dict())
authority = si.get('authority', '')
source = si.get('source', '')
entity = si.get('entityType', '')
major = str(si.get('schemaVersionMajor', 0))
minor = str(si.get('schemaVersionMinor', 0))
patch = str(si.get('schemaVersionPatch', 0))
kind = '{}:{}:{}:{}.{}.{}'.format(authority, source, entity, major, minor, patch)
except Exception as e:
exit('Invalid JSON in payload {}'.format(str(e)))
return kind
elapsed = time.time() - start
print('This update took {:.2f} seconds.'.format(elapsed))
if len(messages) != 0:
print('Following schemas failed:')
print('\n'.join(messages))
exit(1)
else:
print('All {} schemas registered or updated.'.format(str(len(sequence))))
def __register_one(self, kind, schema, messages):
method = 'POST'
try_it = 'Try {} for id: {}'
print(try_it.format(method, kind))
response = requests.request(method, self.url, headers=self.headers, data=schema)
is_error, message, method = self.__evaluate_response(response)
if method == 'PUT': # try again
print(try_it.format(method, kind))
response = requests.request(method, self.url, headers=self.headers, data=schema)
is_error, message, method = self.__evaluate_response(response)
if is_error:
message = 'Error with kind {}: Message: {}'.format(kind, message)
print(message)
messages.append(message)
elif method == self.SUCCESS:
print('The kind {} was registered successfully.'.format(kind))
elif method == self.ALREADY_PUBLISHED:
print('The kind {} was already registered with status PUBLISHED '
'and was not updated.'.format(kind))
def __does_kind_exist(self, kind, headers):
url = '{}/{}'.format(self.url, kind)
response = requests.request("GET", url, headers=headers)
if response.status_code in range(200,300):
self.schema_registered = json.loads(response.text, encoding='utf-8')
return response.status_code in range(200,300)
def __evaluate_response(self, response: requests.Response) -> Tuple[bool, str, str]:
code = response.status_code
message = ''
method = 'Give up'
error = code not in range(200, 300)
if error:
# further test:
try:
js_err = json.loads(response.text)
message = js_err.get('error', dict()).get('message', '')
if message == self.SCHEMA_EXISTS:
method = 'PUT' # try PUT, it might have been DEVELOPMENT, than we can overwrite
elif message in self.PUBLISHED_SCHEMA_ERROR: # already PUBLISHED, no bootstrap required
method = self.ALREADY_PUBLISHED
error = False # this is not considered an error
# everything else is an error
except Exception as e:
message = str(e)
else:
method = self.SUCCESS
return error, message, method
if __name__ == '__main__':
......
......@@ -13,8 +13,10 @@ class RunEnv(object):
SCHEMA_SERVICE_URL = None
STORAGE_SERVICE_URL = None
DATA_PARTITION = os.environ.get('DATA_PARTITION')
OSDU = ['shared-schemas', 'osdu']
LOAD_SEQUENCE = 'load_sequence.0.2.0.json'
SCHEMA_AUTHORITY = os.environ.get('SCHEMA_AUTHORITY')
SCHEMAS_FOLDER = 'shared-schemas'
DEFAULT_BOOTSTRAP_OPTIONS = '[{"authority": "osdu", "folder": "osdu", "load-sequence": "load_sequence.1.0.0.json"}]'
BOOTSTRAP_OPTIONS = os.environ.get('BOOTSTRAP_OPTIONS', DEFAULT_BOOTSTRAP_OPTIONS)
def __init__(self):
"""Empty constructor"""
......@@ -118,9 +120,12 @@ class Utility(object):
@staticmethod
def load_json(path):
"""Load a JSON file"""
with open(path, "r", encoding='utf-8') as text_file:
j_obj = json.load(text_file)
return j_obj
try:
with open(path, "r", encoding='utf-8') as text_file:
j_obj = json.load(text_file)
return j_obj
except FileNotFoundError as e:
exit("Given File path not found::{}".format(str(e)))
@staticmethod
def save_json(schema, path, sort_keys=False):
......@@ -152,7 +157,7 @@ class StorageService(object):
url = '{}/schemas/{}'.format(self.url, urllib.parse.quote_plus(kind))
response = requests.request("GET", url, headers=self.headers)
if response.status_code in range(200, 300):
schema = json.loads(response.text, encoding='utf-8')
schema = json.loads(response.text)
else:
messages.append('Error: Storage Service GET schema {}, {}'.format(response.status_code, response.text))
return response.status_code in range(200, 300), schema
......@@ -167,7 +172,7 @@ class StorageService(object):
url += '&cursor={}'.format(cursor)
response = requests.request("GET", url, headers=self.headers)
if response.status_code in range(200, 300):
rs = json.loads(response.text, encoding='utf-8')
rs = json.loads(response.text)
cursor = rs['cursor']
carry_on = len(rs['results']) > 0
for kind in rs['results']:
......@@ -188,7 +193,7 @@ class SchemaService(object):
url = '{}/{}'.format(self.url, kind)
response = requests.request("GET", url, headers=self.headers)
if response.status_code in range(200,300):
schema = json.loads(response.text, encoding='utf-8')
schema = json.loads(response.text)
return response.status_code in range(200,300), schema
def post_or_put_schema(self, kind: str, schema: dict, schema_status: str, messages: list):
......@@ -242,11 +247,9 @@ class SchemaService(object):
response = requests.request("GET", url, headers=self.headers)
schema_infos = list()
if response.status_code in range(200,300):
r = json.loads(response.text, encoding='utf-8')
r = json.loads(response.text)
schema_infos = r['schemaInfos']
for info in schema_infos:
if self.__match(info['schemaIdentity'], si):
return response.status_code in range(200,300), info
return response.status_code in range(200,300), None
return response.status_code in range(200,300), None
\ No newline at end of file
......@@ -21,7 +21,7 @@ if [ -z "$BEARER_TOKEN" ];
then BEARER_TOKEN=`python3 $AWS_DEPLOYMENTS_SUBDIR/Token.py`;
export BEARER_TOKEN=$BEARER_TOKEN
fi
echo Bearer Token: $BEARER_TOKEN
export APP_KEY=""
export DATA_PARTITION=common
python3 $DEPLOYMENTS_BASE_DIR/scripts/DeploySharedSchemas.py -l load_sequence.1.0.0.json -u $AWS_SCHEMA_SERVICE_URL
python3 $DEPLOYMENTS_BASE_DIR/scripts/DeploySharedSchemas.py -u $AWS_SCHEMA_SERVICE_URL
......@@ -5,5 +5,5 @@ export AZURE_SCHEMA_URL=https://$AZURE_DNS_NAME/api/schema-service/v1/schema
BEARER_TOKEN=`python $AZURE_DEPLOYMENTS_SUBDIR/Token.py`
export BEARER_TOKEN=$BEARER_TOKEN
python $AZURE_DEPLOYMENTS_SCRIPTS_SUBDIR/DeploySharedSchemas.py -l load_sequence.1.0.0.json -u $AZURE_SCHEMA_URL
python $AZURE_DEPLOYMENTS_SCRIPTS_SUBDIR/DeploySharedSchemas.py -u $AZURE_SCHEMA_URL
......@@ -19,13 +19,13 @@ POST/PUT requests:
{
"schemaInfo": {
"schemaIdentity": {
"authority": "osdu",
"authority": "{{schema-authority}}",
"source": "wks",
"entity": "work-product-component.WellLog",
"entityType": "work-product-component--WellLog",
"schemaVersionMajor": 1,
"schemaVersionMinor": 0,
"schemaVersionPatch": 0,
"id": "osdu:wks:work-product-component.WellLog:1.0.0"
"id": "{{schema-authority}}:wks:work-product-component--WellLog:1.0.0"
},
"createdBy": "OSDU Data Definition Group",
"scope": "SHARED",
......@@ -51,19 +51,21 @@ done via the [DeploySharedSchemas.py](../scripts/DeploySharedSchemas.py):
```shell script
python deployments\scripts\DeploySharedSchemas.py -h
usage: DeploySharedSchemas.py [-h] [-l L] [-u U]
usage: DeploySharedSchemas.py [-h] [-a A] [-l L] [-u U]
Given a path to an load sequence file, load/update the schemas listed in the
load sequence file.
optional arguments:
-h, --help show this help message and exit
-a A The schema authority or partition-id to replace (default via
Env)
-l L The path to the load sequence file, e.g. load_sequence.?.?.?
-u U The complete URL to the Schema Service.
example:
python deployments\scripts\DeploySharedSchemas.py -l load_sequence.1.0.0.json -u https://opengroup.test.org/api/schema-service/v1/schema
python deployments\scripts\DeploySharedSchemas.py -l load_sequence.1.0.0.json -a osdu -u https://opengroup.test.org/api/schema-service/v1/schema
```
......
{
"schemaInfo": {
"schemaIdentity": {
"authority": "opendes",
"authority": "{{schema-authority}}",
"source": "wks",
"entityType": "AbstractAccessControlList",
"schemaVersionMajor": 1,
"schemaVersionMinor": 0,
"schemaVersionPatch": 0,
"id": "opendes:wks:AbstractAccessControlList:1.0.0"
"id": "{{schema-authority}}:wks:AbstractAccessControlList:1.0.0"
},
"createdBy": "OSDU Data Definition Group",
"scope": "SHARED",
......
{
"schemaInfo": {
"schemaIdentity": {
"authority": "opendes",
"authority": "{{schema-authority}}",
"source": "wks",
"entityType": "AbstractAliasNames",
"schemaVersionMajor": 1,
"schemaVersionMinor": 0,
"schemaVersionPatch": 0,
"id": "opendes:wks:AbstractAliasNames:1.0.0"
"id": "{{schema-authority}}:wks:AbstractAliasNames:1.0.0"
},
"createdBy": "OSDU Data Definition Group",
"scope": "SHARED",
......