README.md

Register Service

os-register-gcp is a Spring Boot service that hosts CRUD APIs that allows consumers to register a push endpoint that can be triggered when data change events happen within the OSDU R2 ecosystem.

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.

Requirements

• Java 8
• Maven 3.6.0+
• GCloud command line tool
• GCloud access to opendes project

General Tips

Environment Variable Management The following tools make environment variable configuration simpler

• direnv - for a shell/terminal environment
• EnvFile - for Intellij IDEA

Lombok This project uses Lombok for code generation. You may need to configure your IDE to take advantage of this tool.

• Intellij configuration
• VSCode configuration

Installation

In order to run the service locally or remotely, you will need to have the following environment variables defined.

name	value	description	sensitive?	source
LOG_PREFIX	service	Logging prefix	no	-
SERVER_SERVLET_CONTEXPATH	/api/register/v1	Register context path	no	-
ENTITLEMENTS_API	ex https://entitlements.com/entitlements/v1	Entitlements API endpoint	no	output of infrastructure deployment
STORAGE_API	ex https://os-storage-dot-opendes.appspot.com/api/storage/v2	Storage API endpoint	no	output of infrastructure deployment
RECORDS_CHANGE_PUBSUB_ENDPOINT	ex https://os-notification-dot-opendes.appspot.com/api/notification/v1/push-handlers/records-changed	Notification API endpoint 'records-changed'	no	output of infrastructure deployment
GOOGLE_CLOUD_PROJECT	ex opendes	Google Cloud Project Id	no	output of infrastructure deployment
SUBSCRIBER_SECRET	ex7a786376626e	HMAC_SECRET from notification int tests in HEX , pattern(^[a-zA-Z0-9]{8,30}+$)	yes	output of infrastructure deployment
SUBSCRIBER_PRIVATE_KEY_ID	********	Private key id of DE_OPS_TESTER from notification int tests	yes	output of infrastructure deployment
INTEGRATION_TEST_AUDIENCES	ex *****.apps.googleusercontent.com	Client ID for getting access to cloud resources	yes	https://console.cloud.google.com/apis/credentials
SERVICE_IDENTITY	ex osdu-gcp-sa	Service account identity "osdu-gcp-sa@iam.gserviceaccount.com"	yes	https://console.cloud.google.com/apis/credentials
ENVIRONMENT	ex dev	Service environment config	no	-
GCLOUD_REGION	ex us-central1	cloud region	no	-
GOOGLE_AUDIENCES	ex *****.apps.googleusercontent.com	Client ID for getting access to cloud resources	yes	https://console.cloud.google.com/apis/credentials
PARTITION_API	ex http://localhost:8081/api/partition/v1	Partition service endpoint	no	-

System Environment required to run service

name	value	description	sensitive?	source
SPRING_PROFILES_ACTIVE	local	spring active profile	no

Run Locally

Check that maven is installed:

$ mvn --version
Apache Maven 3.6.0
Maven home: /usr/share/maven
Java version: 1.8.0_212, vendor: AdoptOpenJDK, runtime: /usr/lib/jvm/jdk8u212-b04/jre
...

You will need to configure access to the remote maven repository that holds the OSDU dependencies. This file should live within ~/.m2/settings.xml:

$ cat ~/.m2/settings.xml
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
    <servers>
        <server>
            <id>os-core</id>
            <username>slb-des-ext-collaboration</username>
            <!-- Treat this auth token like a password. Do not share it with anyone, including Microsoft support. -->
            <password>${VSTS_FEED_TOKEN}</password>
        </server>
    </servers>
</settings>

• Update the Google cloud SDK to the latest version:

gcloud components update

• Set Google Project Id:

gcloud config set project <YOUR-PROJECT-ID>

• Perform a basic authentication in the selected project:

gcloud auth application-default login

• Navigate to register service's root folder and run:

mvn jetty:run
## Testing
* Navigate to register service's root folder and run:
 
```bash
mvn clean install   

• If you wish to see the coverage report then go to testing/target/site/jacoco-aggregate and open index.html

• If you wish to build the project without running tests

mvn clean install -DskipTests

After configuring your environment as specified above, you can follow these steps to build and run the application. These steps should be invoked from the repository root.

cd provider/register-gcp/ && mvn spring-boot:run -Dspring-boot.run.profiles=local 

Testing

Navigate to register service's root folder and run all the tests:

# build + test + install core service code
$ (cd register-core/ && mvn clean install)

Test the application

After the service has started it should be accessible via a web browser by visiting http://localhost:8080/api/register/v1/swagger-ui.html. If the request does not fail, you can then run the integration tests.

Running E2E Tests

This section describes how to run cloud OSDU E2E tests (testing/register-test-gcp).

You will need to have the following environment variables defined.

name	value	description	sensitive?	source
DE_OPS_TESTER	********	A base64 encoded google service account json credentials with ops level authorization for OSDU services	yes	https://console.cloud.google.com/iam-admin/serviceaccounts
DE_ADMIN_TESTER	********	A base64 encoded google service account json credentials with admin level authorization for OSDU services	yes	https://console.cloud.google.com/iam-admin/serviceaccounts
DE_EDITOR_TESTER	********	A base64 encoded google service account json credentials with editor level authorization for OSDU services	yes	https://console.cloud.google.com/iam-admin/serviceaccounts
DE_NO_ACCESS_TESTER	********	A base64 encoded google service account json credentials with no authorization for OSDU services	yes	https://console.cloud.google.com/iam-admin/serviceaccounts
REGISTER_BASE_URL	ex https://os-register-dot-opendes.appspot.com/	Register API endpoint	no	output of infrastructure deployment
ENVIRONMENT	ex local OR dev	'local' for local testing or 'dev' for dev testing	no	-
SUBSCRIBER_SECRET	ex ********	String in hex , must match pattern ^[a-zA-Z0-9]{8,30}+$ & be in register variable SUBSCRIBER_SECRET	yes	-
INTEGRATION_TEST_AUDIENCE	ex *****.apps.googleusercontent.com	Client ID for getting access to cloud resources	yes	https://console.cloud.google.com/apis/credentials
CLIENT_TENANT	ex nonexistenttenant	Client tenant, it's supposed to be tenant that we do not have access to, it can be not existing tenant	no	-
OSDU_TENANT	ex osdu	Osdu tenant	no	-
SUBSCRIPTION_ID	********	A base64 encoded string of subscribed topic + subscriber url ex records-changedhttp://localhost:8081/api/register/v1/test/challenge/1	no	-
REGISTER_CUSTOM_PUSH_URL	exhttps://os-register-dot-opendes.appspot.com/api/register/v1/test/challenge/1	Register push url, that will act as subscriber	no	-

Entitlements configuration for integration accounts

DE_OPS_TESTER	DE_ADMIN_TESTER	DE_EDITOR_TESTER	DE_NO_ACCESS_TESTER
service.entitlements.user users.datalake.ops data.test1 data.integration.test users@{tenant1}@{domain}.com	service.entitlements.user users.datalake.admins data.test1 data.integration.test users@{tenant1}@{domain}.com	service.entitlements.user users.datalake.editors data.test1 data.integration.test users@{tenant1}@{domain}.com	service.entitlements.user data.test1 data.integration.test users@{tenant1}@{domain}.com

Execute following command to build code and run all the integration tests:

# Note: this assumes that the environment variables for integration tests as outlined
#       above are already exported in your environment.
$ (cd testing/register-test-core/ && mvn clean test)
$ (cd testing/register-test-gcp/ && mvn clean test)

Deployment

• Data-Lake Register Google Cloud Endpoints on App Engine Flex environment
  • Edit the app.yaml
    • Open the app.yaml file in editor, and replace the YOUR-PROJECT-ID PROJECT line with Google Cloud Platform project Id. Also update INTEGRATION_TEST_AUDIENCE, SUBSCRIBER_PRIVATE_KEY_ID, SPRING_PROFILES_ACTIVE, ENTITLEMENTS_API, STORAGE_API, RECORDS_CHANGE_PUBSUB_ENDPOINT, GOOGLE_CLOUD_PROJECT and STORAGE_API based on your deployment
  • Google Documentation: https://cloud.google.com/cloud-build/docs/deploying-builds/deploy-appengine

Cloud KMS Setup

Enable cloud KMS on master project

Create king ring and key in the master project

    gcloud services enable cloudkms.googleapis.com
    export KEYRING_NAME="csqp"
    export CRYPTOKEY_NAME="registerService"
    gcloud kms keyrings create $KEYRING_NAME --location global
    gcloud kms keys create $CRYPTOKEY_NAME --location global \
    		--keyring $KEYRING_NAME \
    		--purpose encryption

Add Cloud KMS CryptoKey Encrypter/Decrypter role to the App Engine default service account of the master project through IAM - Role tab

Add Cloud KMS Encrypt/Decrypt role to the App Engine default service account of master project

Licence

Copyright © Google LLC Copyright © EPAM Systems

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.