Commit 6d6f70e3 authored by Rustam Lotsmanenko (EPAM)'s avatar Rustam Lotsmanenko (EPAM) Committed by Riabokon Stanislav(EPAM)[GCP]
Browse files

Ecs impl changes

commit 5d462e69
Author: Rucha deshpande <deshruch@amazon.com>
Date: Thu Aug 13 2020 12:16:03 GMT-0500 (Central Daylight Time)

    Adding appropriate copyright statements

commit 28bc590c
Author: Rucha deshpande <deshruch@amazon.com>
Date: Fri Aug 07 2020 17:23:15 GMT-0500 (Central Daylight Time)

    core-lib-aws to 0.3.7

commit a50622fe
Author: Rucha deshpande <deshruch@amazon.com>
Date: Fri Aug 07 2020 16:05:24 GMT-0500 (Central Daylight Time)

    updated buildspec.yaml for coretto

commit 2c47c8ff
Author: Rucha deshpande <deshruch@amazon.com>
Date: Fri Aug 07 2020 15:49:48 GMT-0500 (Central Daylight Time)

    entitlements-on-ecs update
parent 939d9eb7
......@@ -9,13 +9,6 @@ The following software have components provided under the terms of this license:
- Android SDK (from https://www.android.com/)
========================================================================
Apache-1.1
========================================================================
The following software have components provided under the terms of this license:
- StAX (from http://stax.codehaus.org/)
========================================================================
Apache-2.0
========================================================================
......@@ -462,6 +455,10 @@ The following software have components provided under the terms of this license:
- Guava: Google Core Libraries for Java (from https://github.com/google/guava.git)
- HPPC Collections (from http://labs.carrotsearch.com)
- Hibernate Validator Engine (from )
- IBM COS Java SDK for Amazon S3 (from https://github.com/ibm/ibm-cos-sdk-java)
- IBM COS Java SDK for COS KMS (from https://github.com/ibm/ibm-cos-sdk-java)
- IBM COS SDK For Java (from https://github.com/ibm/ibm-cos-sdk-java)
- IBM COS SDK for Java - Core (from https://github.com/ibm/ibm-cos-sdk-java)
- Identity and Access Management (IAM) API v1-rev247-1.23.0 (from )
- J2ObjC Annotations (from https://github.com/google/j2objc/)
- J2ObjC Annotations (from https://github.com/google/j2objc/)
......@@ -539,11 +536,10 @@ The following software have components provided under the terms of this license:
- Microsoft Application Insights Java SDK Spring Boot starter (from https://github.com/Microsoft/ApplicationInsights-Java)
- Microsoft Application Insights Java SDK Web Module (from https://github.com/Microsoft/ApplicationInsights-Java)
- Microsoft Application Insights Log4j 2 Appender (from https://github.com/Microsoft/ApplicationInsights-Java)
- Mockito (from http://www.mockito.org)
- Mockito (from http://mockito.org)
- Mockito (from http://mockito.org)
- Mockito (from http://www.mockito.org)
- Mockito (from http://mockito.org)
- Mockito (from http://www.mockito.org)
- Netty Reactive Streams Implementation (from )
- Netty/Buffer (from http://netty.io/)
- Netty/Buffer (from http://netty.io/)
......@@ -571,14 +567,12 @@ The following software have components provided under the terms of this license:
- Non-Blocking Reactive Foundation for the JVM (from https://github.com/reactor/reactor)
- OAuth 2.0 SDK with OpenID Connect extensions (from https://bitbucket.org/connect2id/oauth-2.0-sdk-with-openid-connect-extensions)
- Objenesis (from http://objenesis.org)
- Objenesis (from http://objenesis.org)
- OkHttp (from )
- OkHttp (from )
- OkHttp Logging Interceptor (from )
- OkHttp URLConnection (from )
- OkHttp URLConnection (from )
- Okio (from )
- Okio (from )
- OpenCensus (from https://github.com/census-instrumentation/opencensus-java)
- OpenCensus (from https://github.com/census-instrumentation/opencensus-java)
- OpenCensus (from https://github.com/census-instrumentation/opencensus-java)
......@@ -587,7 +581,6 @@ The following software have components provided under the terms of this license:
- QpidJMS Client (from )
- Reactive Streams Netty driver (from https://github.com/reactor/reactor-netty)
- Retrofit (from )
- Simple XML (from http://simple.sourceforge.net)
- SnakeYAML (from http://www.snakeyaml.org)
- Spring AOP (from https://github.com/spring-projects/spring-framework)
- Spring Beans (from https://github.com/spring-projects/spring-framework)
......@@ -621,8 +614,6 @@ The following software have components provided under the terms of this license:
- Spring Transaction (from https://github.com/spring-projects/spring-framework)
- Spring Web (from https://github.com/spring-projects/spring-framework)
- Spring Web MVC (from https://github.com/spring-projects/spring-framework)
- StAX (from http://stax.codehaus.org/)
- StAX API (from http://stax.codehaus.org/)
- T-Digest (from https://github.com/tdunning/t-digest)
- Woodstox (from https://github.com/FasterXML/woodstox)
- aggs-matrix-stats (from https://github.com/elastic/elasticsearch)
......@@ -652,6 +643,7 @@ The following software have components provided under the terms of this license:
- io.grpc:grpc-protobuf-lite (from https://github.com/grpc/grpc-java)
- io.grpc:grpc-stub (from https://github.com/grpc/grpc-java)
- ion-java (from https://github.com/amznlabs/ion-java/)
- ion-java (from https://github.com/amznlabs/ion-java/)
- jackson-databind (from http://github.com/FasterXML/jackson)
- jackson-databind (from http://github.com/FasterXML/jackson)
- java-cloudant (from https://cloudant.com)
......@@ -667,7 +659,6 @@ The following software have components provided under the terms of this license:
- lettuce (from http://github.com/mp911de/lettuce/wiki)
- micrometer-core (from https://github.com/micrometer-metrics/micrometer)
- micrometer-registry-azure-monitor (from https://github.com/micrometer-metrics/micrometer)
- minio (from https://github.com/minio/minio-java)
- org.xmlunit:xmlunit-core (from http://www.xmlunit.org/)
- parent-join (from https://github.com/elastic/elasticsearch)
- parent-join (from https://github.com/elastic/elasticsearch)
......@@ -733,7 +724,6 @@ The following software have components provided under the terms of this license:
- Hamcrest Core (from http://hamcrest.org/)
- Lucene Common Analyzers (from )
- Lucene Common Analyzers (from )
- StAX (from http://stax.codehaus.org/)
========================================================================
BSD-3-Clause
......@@ -761,7 +751,6 @@ The following software have components provided under the terms of this license:
- Microsoft Application Insights Java SDK Web Module (from https://github.com/Microsoft/ApplicationInsights-Java)
- Microsoft Application Insights Log4j 2 Appender (from https://github.com/Microsoft/ApplicationInsights-Java)
- Mockito (from http://www.mockito.org)
- Mockito (from http://mockito.org)
- Mockito (from http://www.mockito.org)
- NanoHttpd-Core (from )
- Netty/Codec/HTTP (from )
......@@ -786,7 +775,6 @@ CC-BY-3.0
========================================================================
The following software have components provided under the terms of this license:
- "Java Concurrency in Practice" book annotations (from http://jcip.net/)
- FindBugs-jsr305 (from http://findbugs.sourceforge.net/)
========================================================================
......@@ -929,7 +917,6 @@ The following software have components provided under the terms of this license:
- Microsoft Application Insights Java SDK Spring Boot starter (from https://github.com/Microsoft/ApplicationInsights-Java)
- Microsoft Application Insights Java SDK Web Module (from https://github.com/Microsoft/ApplicationInsights-Java)
- Microsoft Application Insights Log4j 2 Appender (from https://github.com/Microsoft/ApplicationInsights-Java)
- SpotBugs Annotations (from https://spotbugs.github.io/)
========================================================================
LGPL-2.1-or-later
......@@ -940,7 +927,6 @@ The following software have components provided under the terms of this license:
- Java Native Access Platform (from https://github.com/java-native-access/jna)
- Javassist (from http://www.javassist.org/)
- SnakeYAML (from http://www.snakeyaml.org)
- SpotBugs Annotations (from https://spotbugs.github.io/)
========================================================================
LGPL-3.0-only
......@@ -996,10 +982,9 @@ The following software have components provided under the terms of this license:
- Microsoft Azure client library for KeyVault Secrets (from https://github.com/Azure/azure-sdk-for-java)
- Microsoft Azure common module for Storage (from https://github.com/Azure/azure-sdk-for-java)
- Mockito (from http://www.mockito.org)
- Mockito (from http://mockito.org)
- Mockito (from http://mockito.org)
- Mockito (from http://www.mockito.org)
- Mockito (from http://mockito.org)
- Mockito (from http://mockito.org)
- Netty/Codec/HTTP (from )
- Netty/Codec/HTTP (from )
- Netty/Common (from )
......@@ -1085,6 +1070,5 @@ unknown
The following software have components provided under the terms of this license:
- Byte Buddy (without dependencies) (from )
- Objenesis (from http://objenesis.org)
......@@ -122,13 +122,8 @@ Instructions for running the Azure integration tests can be found [here](./provi
### GCP
These tests integrate with other datalake services as well as Google datasources in your currently set default GCloud project (normally a sandbox when runing locally). They require the service to be running locally (see above) and for you to be a datalake-admin in the datalake identity service for your currenly set default project.
They can then be run/debugged directly in your IDE of choice using the GUI or via the commandline using
```sh
gradlew build
```
Instructions for running the GCP integration tests can be found [here](./provider/storage-gcp/README.md).
### AWS
......
pipeline {
agent {
kubernetes {
cloud 'openshift'
label 'maven-openjdk18'
yaml """
spec:
containers:
- name: jnlp
image: quay.io/openshift/origin-jenkins-agent-maven:v4.0.0
volumeMounts:
- mountPath: "/tmp"
name: "workspace-volume"
readOnly: false
workingDir: "/tmp"
securityContext:
privileged: false
tty: false
resources:
limits:
cpu: 200m
memory: 2Gi
requests:
cpu: 200m
memory: 2Gi
restartPolicy: "Never"
"""
}
}
environment {
//Cluster environment variable(CLS_ENV). Like QA, DEV, PERF, PROD etc.
CLS_ENV = "dev"
//Service variable(CORE_SERVICE). Like indexer, search, delivery, storage, legal etc.
CORE_SERVICE = "storage"
//GitHub repo URL credential ID for Environment variable files which saved as Secure text in Jenkins Credential.
GIT_ENV_VAR_PATH_URL = credentials('GitRepo-URL-For-Environment-variables')
//Personal token variable ID which saved as Secure text in Jenkins Credential. Like: GitHub-PRIVATE-TOKEN.
PRIVATE_TOKEN = credentials('GitHub-PRIVATE-TOKEN')
def runShell = sh (returnStdout: true, script: "curl --header 'PRIVATE-TOKEN: $PRIVATE_TOKEN' ''$GIT_ENV_VAR_PATH_URL'%2F'$CORE_SERVICE'_'$CLS_ENV'_env.json/raw?ref=master' -s -o env.json")
}
stages {
stage('Integration_test') {
environment {
def readContent = readJSON file: 'env.json'
AUTH_USER_ACCESS = "${readContent['AUTH_USER_ACCESS']}"
AUTH_USER_ACCESS_PASSWORD = "${readContent['AUTH_USER_ACCESS_PASSWORD']}"
AUTH_USER_NO_ACCESS = "${readContent['AUTH_USER_NO_ACCESS']}"
AUTH_USER_NO_ACCESS_PASSWORD = "${readContent['AUTH_USER_NO_ACCESS_PASSWORD']}"
DEPLOY_ENV = "${readContent['DEPLOY_ENV']}"
DOMAIN = "${readContent['DOMAIN']}"
ENTITLEMENT_URL = "${readContent['ENTITLEMENT_URL']}"
KEYCLOAK_CLIENT_ID = "${readContent['KEYCLOAK_CLIENT_ID']}"
KEYCLOAK_CLIENT_SECRET = "${readContent['KEYCLOAK_CLIENT_SECRET']}"
KEYCLOAK_REALM = "${readContent['KEYCLOAK_REALM']}"
KEYCLOAK_URL = "${readContent['KEYCLOAK_URL']}"
LEGAL_URL = "${readContent['LEGAL_URL']}"
PROJECT_ID = "${readContent['PROJECT_ID']}"
PUBSUB_TOKEN = "${readContent['PUBSUB_TOKEN']}"
STORAGE_URL = "${readContent['STORAGE_URL']}"
TENANT_NAME = "${readContent['TENANT_NAME']}"
}
steps {
script {
sh 'mvn -f testing/storage-test-core/pom.xml install'
sh 'mvn -f testing/storage-test-ibm/pom.xml clean test -pl :storage-test-ibm -amd'
}
}
}
}
}
# Pipeline Support Commands
```bash
AZURE_SERVICE="storage"
REPO_BRANCH="master"
TAG="latest"
PARTIAL=${REPO_BRANCH/\//-}
BRANCH=${PARTIAL/./-}
echo "--set image.branch=$BRANCH --set image.tag=$TAG"
# Install the Service
helm upgrade -i osdu-gitlab-$AZURE_SERVICE chart --set image.branch=$BRANCH --set image.tag=$TAG
pod=$(kubectl get pod |grep $AZURE_SERVICE | tail -1 | awk '{print $1}')
status=$(kubectl wait --for=condition=Ready pod/$pod --timeout=60s)
if [[ "$status" != *"met"* ]]; then echo "POD didn't start correctly" ; exit 1 ; fi
```
# Copyright © Microsoft Corporation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: v2
name: storage
appVersion: "latest"
description: Helm Chart for installing storage service.
version: 0.1.0
type: application
# Copyright © Microsoft Corporation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Release.Name }}
namespace: osdu
spec:
replicas: {{ .Values.global.replicaCount }}
selector:
matchLabels:
app: {{ .Chart.Name }}
template:
metadata:
labels:
app: {{ .Chart.Name }}
aadpodidbinding: osdu-identity
spec:
volumes:
- name: azure-keyvault
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: azure-keyvault
containers:
- name: {{ .Chart.Name }}
image: {{ .Values.image.repository }}/{{ .Chart.Name }}-{{ .Values.image.branch }}:{{ .Values.image.tag | default .Chart.AppVersion }}
imagePullPolicy: Always
ports:
- containerPort: 80
readinessProbe:
httpGet:
path: /api/storage/v2/swagger-ui.html
port: 80
volumeMounts:
- name: azure-keyvault
mountPath: "/mnt/azure-keyvault"
readOnly: true
env:
- name: spring_application_name
value: storage
- name: server.servlet.contextPath
value: /api/storage/v2/
- name: server_port
value: "80"
- name: ACCEPT_HTTP # TEMPORARY UNTIL HTTPS
value: "true"
- name: KEYVAULT_URI
valueFrom:
configMapKeyRef:
name: osdu-svc-properties
key: ENV_KEYVAULT
- name: AZURE_TENANT_ID
valueFrom:
configMapKeyRef:
name: osdu-svc-properties
key: ENV_TENANT_ID
- name: AZURE_CLIENT_ID
valueFrom:
secretKeyRef:
name: clientid
key: clientid
- name: AZURE_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: clientpassword
key: clientpassword
- name: aad_client_id
valueFrom:
secretKeyRef:
name: appid
key: appid
- name: azure_activedirectory_session_stateless
value: "true"
- name: azure_activedirectory_AppIdUri
value: "api://$(aad_client_id)"
- name: appinsights_key
valueFrom:
secretKeyRef:
name: appinsights
key: appinsights
- name: cosmosdb_account
valueFrom:
configMapKeyRef:
name: osdu-svc-properties
key: ENV_COSMOSDB_HOST
- name: cosmosdb_key
valueFrom:
secretKeyRef:
name: cosmos
key: cosmos
- name: cosmosdb_database
value: osdu-db
- name: servicebus_namespace_name
valueFrom:
configMapKeyRef:
name: osdu-svc-properties
key: ENV_SERVICEBUS_NAMESPACE
- name: servicebus_connection_string
valueFrom:
secretKeyRef:
name: servicebus
key: servicebus
- name: servicebus_topic_name
value: recordstopic
- name: storage_account
valueFrom:
configMapKeyRef:
name: osdu-svc-properties
key: ENV_STORAGE_ACCOUNT
- name: storage_account_key
valueFrom:
secretKeyRef:
name: storage
key: storage
- name: entitlements_service_endpoint
value: http://entitlements-azure/entitlements/v1
- name: entitlements_service_api_key
value: "OBSOLETE"
- name: legal_service_endpoint
value: http://legal/api/legal/v1
- name: legal_service_region
value: southcentralus
- name: search_service_endpoint
value: http://legal/api/legal/v1
# Copyright © Microsoft Corporation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: v1
kind: Service
metadata:
name: {{ .Chart.Name }}
namespace: osdu
spec:
type: ClusterIP
ports:
- protocol: TCP
port: 80
targetPort: 80
selector:
app: {{ .Chart.Name }}
# Copyright © Microsoft Corporation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
global:
replicaCount: 1
image:
repository: community.opengroup.org:5555/osdu/platform/system/storage
branch: master
tag: latest
---
# Source: storage/templates/service.yaml
# Copyright © Microsoft Corporation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: v1
kind: Service
metadata:
name: storage
namespace: osdu
spec:
type: ClusterIP
ports:
- protocol: TCP
port: 80
targetPort: 80
selector:
app: storage
---
# Source: storage/templates/deployment.yaml
# Copyright © Microsoft Corporation
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: apps/v1
kind: Deployment
metadata:
name: osdu-gitlab-storage
namespace: osdu
spec:
replicas: 1
selector:
matchLabels:
app: storage
template:
metadata:
labels:
app: storage
aadpodidbinding: osdu-identity
spec:
volumes:
- name: azure-keyvault
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: azure-keyvault
containers:
- name: storage
image: community.opengroup.org:5555/osdu/platform/system/storage/storage-master:latest
imagePullPolicy: Always
ports:
- containerPort: 80
readinessProbe:
httpGet:
path: /api/storage/v2/swagger-ui.html
port: 80
volumeMounts:
- name: azure-keyvault
mountPath: "/mnt/azure-keyvault"
readOnly: true
env:
- name: spring_application_name
value: storage
- name: server.servlet.contextPath
value: /api/storage/v2/
- name: server_port
value: "80"
- name: ACCEPT_HTTP # TEMPORARY UNTIL HTTPS
value: "true"
- name: KEYVAULT_URI
valueFrom:
configMapKeyRef:
name: osdu-svc-properties
key: ENV_KEYVAULT
- name: AZURE_TENANT_ID
valueFrom:
configMapKeyRef:
name: osdu-svc-properties
key: ENV_TENANT_ID
- name: AZURE_CLIENT_ID
valueFrom:
secretKeyRef:
name: clientid
key: clientid
- name: AZURE_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: clientpassword
key: clientpassword
- name: aad_client_id
valueFrom:
secretKeyRef:
name: appid
key: appid
- name: azure_activedirectory_session_stateless
value: "true"
- name: azure_activedirectory_AppIdUri
value: "api://$(aad_client_id)"
- name: appinsights_key
valueFrom:
secretKeyRef:
name: appinsights
key: appinsights
- name: cosmosdb_account
valueFrom:
configMapKeyRef:
name: osdu-svc-properties
key: ENV_COSMOSDB_HOST
- name: cosmosdb_key
valueFrom:
secretKeyRef:
name: cosmos
key: cosmos
- name: cosmosdb_database
value: osdu-db
- name: servicebus_namespace_name
valueFrom:
configMapKeyRef:
name: osdu-svc-properties
key: ENV_SERVICEBUS_NAMESPACE
- name: servicebus_connection_string
valueFrom:
secretKeyRef:
name: servicebus
key: servicebus
- name: servicebus_topic_name
value: recordstopic
- name: storage_account
valueFrom:
configMapKeyRef:
name: osdu-svc-properties
key: ENV_STORAGE_ACCOUNT
- name: storage_account_key
valueFrom:
secretKeyRef:
name: storage
key: storage
- name: entitlements_service_endpoint
value: http://entitlements-azure/entitlements/v1
- name: entitlements_service_api_key
value: "OBSOLETE"