README.md

Service Configuration for Google Cloud

Table of Contents

• Overview
• Environment variables
• Common properties for all environments
• For Mappers to activate drivers
• For Google Cloud only
• GCS configuration
• Google cloud service account configuration

Overview

Environment variables

Common properties for all environments

name	value	description	sensitive?	source
APP_ENTITLEMENTS	ex https://entitlements.com/entitlements/v1	Entitlements API endpoint	no	output of infrastructure deployment
APP_REGISTER	ex https://register.com/api/register/v1	Storage API endpoint	no	output of infrastructure deployment
PARTITION_API	ex http://localhost:8081/api/partition/v1	Partition service endpoint	no	-
SERVICE_TOKEN_PROVIDER	ex GCP or OPENID	Service token provider	no	-

System Environment required to run service

name	value	description	sensitive?	source
SPRING_PROFILES_ACTIVE	gcp	spring active profile	no
REDIS_HOST	ex 127.0.0.1	Redis host	no	-
REDIS_PORT	ex 6379	Redis port	no	-
REDIS_PASSWORD	ex ``	Redis password	yes	-
REDIS_WITH_SSL	ex true or false	Redis host ssl config	no

For Mappers to activate drivers

name	value	description
OQMDRIVER	pubsub	to activate OQM driver for Google PubSub
OQMDRIVER	rabbitmq	to activate OQM driver for Rabbit MQ

For Google Cloud only

name	value	description	sensitive?	source
APP_PROJECT	ex opendes	Google Cloud Project Id	no	output of infrastructure deployment

service account IAM roles

Also, the following IAM roles should be assigned to the service's Google service account (SA)

IAM role	The purpose
Service Account Token Creator	To write yourself JWT for requesting neighbor microservices
Pub/Sub Editor	To fetch available PubSub topics and subscriptions and be able to create them

Testing

After the service has started it should be accessible via a web browser by visiting http://localhost:8080/api/notification/v1/swagger-ui.html. If the request does not fail, you can then run the integration tests.

Dependencies needed to run the integration tests

• Java 8
• Maven
• Values for the following environment variables in Config.java

name	value	description	sensitive?	source
DE_OPS_TESTER	*****	Service account base64 encoded string for API calls. Note: this user must have entitlements configured already, also Private key id of this account must be set in Register service variable SUBSCRIBER_PRIVATE_KEY_ID	yes	https://console.cloud.google.com/iam-admin/serviceaccounts
INTEGRATION_TESTER	*****	Service account base64 encoded string for API calls.	yes	https://console.cloud.google.com/iam-admin/serviceaccounts
HMAC_SECRET	ex7a786376626e	String in hex , must match pattern ^[a-zA-Z0-9]{8,30}+$ & be in register variable SUBSCRIBER_SECRET	yes	-
REGISTER_BASE_URL	http://localhost:8081/api/register/v1	Register service url	no	-
OSDU_TENANT	ex osdu	osdu tenant	no	-
INTEGRATION_AUDIENCE	ex osdu	integration audience	no	-
TOPIC_ID	ex records-changed	PubSub topic id	no	https://console.cloud.google.com/cloudpubsub/topic
STORAGE_HOST	http://localhost:8081/api/storage/v1/	Storage service url	no	-
LEGAL_HOST	http://localhost:8081/api/legal/v1/	Legal service url	no	-
GROUP_ID	osdu-gcp.go3-nrg.projects.epam.com	Group id, used in storage record ACL. Full group will be "data.default.viewers@{{data-partition-id}}.{{group_id}}"	no	-

Entitlements configuration for integration accounts

DE_OPS_TESTER	DE_ADMIN_TESTER	DE_EDITOR_TESTER	DE_NO_ACCESS_TESTER
notification.pubsub service.entitlements.user users users.datalake.ops	service.entitlements.user users users.datalake.admins	service.entitlements.user users users.datalake.editors	service.entitlements.user users

Above variables should be configured in the release pipeline to run integration tests. You should also replace them with proper values if you wish to run tests locally.

Commands to run tests

• Integration tests are refactored into two pieces: Core and Provider. Core contains business logic for tests and is a dependency for executing the tests from provider module. To build the core module, simply navigate to notification-test-core directory and run mvn clean install. This will build the core module
• Next, to execute the integration tests, navigate to the provider module and execute mvn test

# (cd testing/notification-test-core/ && mvn clean install)
# Note: this assumes that the environment variables for integration tests as outlined
#       above are already exported in your environment.
$ (cd testing/notification-test-gc/ && mvn clean test)

Navigate to notification service's root folder and run all the tests:

# build + test + install core service code
$ (cd notification-core/ && mvn clean install)

PubSub configuration

At PubSub should be created set of topics and subscriptions. (see Overview)

topic name	subscription name pattern	description	sensitive?
register-subscriber-control	notification-control-sub	Register subscriber control topic	yes
{topic-name} e.g. records-changed	notification-{topic-name}-service	Service topics from topics.json on Register service	yes
{topic-name}-publish e.g. records-changed-publish	notification-{topic-name}-publish	Publish topics in relation 1-on-1 for each service topic	yes

Control topic name can be overridden by:

• through the Spring Boot property oqm-register-subscriber-control-topic-name
• environment variable OQM_REGISTER_SUBSCRIBER_CONTROL_TOPIC_NAME

Important. Dead lettering configuration

Default values of subscription settings Dead lettering = Disabled, and Retry policy = Retry immediately leads to a significant increase in paying bills.

To avoid such a situation need to configure the dead letter topic and set parameters:

• Maximum delivery attempts = 5
• Retry policy = Retry after exponential backoff delay
• Minimum backoff duration = 10 seconds
• Maximum backoff duration = 600 seconds

On the subscription dead lettering settings tab:

• Select an active dead letter topic
• Assign Publisher role
• Assign Subscriber role
• Dead letter topic should not be the source topic

Notification service uses a common dead-lettering-notification topic because dead lettering logic is not supported yet.

Example for records-changed topic: