Newer
Older
![Eldor Pulatov [EPAM/GCP]'s avatar](https://secure.gravatar.com/avatar/1e4bd21cab9a546e73714ab2775ef99fb969bdbf3e14880232a7af0550462f8f?s=144&d=identicon "Eldor Pulatov [EPAM/GCP]")
# Service Configuration for Google Cloud
Dmitrii Novikov (EPAM)
committed
## Table of Contents <a name="TOC"></a>
* [Overview](#overview)
* [Environment variables](#environment-variables)
* [Common properties for all environments](#common-properties-for-all-environments)
* [For Mappers to activate drivers](#for-mappers-to-activate-drivers)
* [For Google Cloud only](#for-google-cloud-only)
Dmitrii Novikov (EPAM)
committed
* [GCS configuration](#GCS-configuration)
* [Google cloud service account configuration](#Google-cloud-service-account-configuration)
## Overview
Dmitrii Novikov (EPAM)
committed
## Environment variables
### Common properties for all environments
| name | value | description | sensitive? | source |
| --- | --- | --- | --- | --- |
| `APP_ENTITLEMENTS` | ex `https://entitlements.com/entitlements/v1` | Entitlements API endpoint | no | output of infrastructure deployment |
| `APP_REGISTER` | ex `https://register.com/api/register/v1` | Storage API endpoint | no | output of infrastructure deployment |
| `PARTITION_API` | ex `http://localhost:8081/api/partition/v1` | Partition service endpoint | no | - |
| `SERVICE_TOKEN_PROVIDER` | ex `GCP` or `OPENID` | Service token provider | no | - |
**System Environment required to run service**
| name | value | description | sensitive? | source |
| --- | --- | --- | --- | --- |
| `SPRING_PROFILES_ACTIVE` | `gcp` | spring active profile | no |
| `REDIS_HOST` | ex `127.0.0.1` | Redis host | no | - |
| `REDIS_PORT` | ex `6379` | Redis port | no | - |
| `REDIS_PASSWORD` | ex `` | Redis password | yes | - |
| `REDIS_WITH_SSL` | ex `true` or `false` | Redis host ssl config | no | |
Dmitrii Novikov (EPAM)
committed
### For Mappers to activate drivers
| name | value | description |
|-----------|-----------|---------------------------------------------------------|
| OQMDRIVER | pubsub | to activate **OQM** driver for **Google PubSub** |
| OQMDRIVER | rabbitmq | to activate **OQM** driver for **Rabbit MQ** |
#### For Google Cloud only
Dmitrii Novikov (EPAM)
committed
| name | value | description | sensitive? | source |
|------------------------------|---------------------------------------|--------------------------------------------------------------------|------------|---------------------------------------------------|
| `APP_PROJECT` | ex `opendes` | Google Cloud Project Id | no | output of infrastructure deployment |
##### service account IAM roles
Dmitrii Novikov (EPAM)
committed
Also, the following IAM roles should be assigned to the service's Google service account (SA)
| IAM role | The purpose |
|----------|-------------------------------------------------------------------------------|
| Service Account Token Creator | To write yourself JWT for requesting neighbor microservices |
| Pub/Sub Editor | To fetch available PubSub topics and subscriptions and be able to create them |
![Riabokon Stanislav(EPAM)[GCP]'s avatar](https://secure.gravatar.com/avatar/b616272b81055d50e0facfaf0b22cb0d438cd3e3de42663c349fc6f075ed1051?s=144&d=identicon "Riabokon Stanislav(EPAM)[GCP]")
## Testing
After the service has started it should be accessible via a web browser by visiting [http://localhost:8080/api/notification/v1/swagger-ui.html](http://localhost:8080/swagger-ui.html). If the request does not fail, you can then run the integration tests.
### Dependencies needed to run the integration tests
* Java 8
* Maven
* Values for the following environment variables in Config.java
![Kanstantsin Dubrouski [EPAM / GCP]'s avatar](https://secure.gravatar.com/avatar/a219cfad41cbeca2f3dcaa9432871d27f3cefc621d9e36eb57b335fc3be03ecc?s=144&d=identicon "Kanstantsin Dubrouski [EPAM / GCP]")
| name | value | description | sensitive? | source |
|------------------------|-----------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------|--------------------------------------------------------------|
| `DE_OPS_TESTER` | `*****` | Service account base64 encoded string for API calls. Note: this user must have entitlements configured already, also **Private key id** of this account must be set in Register service variable SUBSCRIBER_PRIVATE_KEY_ID | yes | <https://console.cloud.google.com/iam-admin/serviceaccounts> |
| `INTEGRATION_TESTER` | `*****` | Service account base64 encoded string for API calls. | yes | <https://console.cloud.google.com/iam-admin/serviceaccounts> |
| `HMAC_SECRET` | ex`7a786376626e` | String in hex , must match pattern ^[a-zA-Z0-9]{8,30}+$ & be in register variable SUBSCRIBER_SECRET | yes | - |
| `REGISTER_BASE_URL` | `http://localhost:8081/api/register/v1` | Register service url | no | - |
| `OSDU_TENANT` | ex `osdu` | osdu tenant | no | - |
| `INTEGRATION_AUDIENCE` | ex `osdu` | integration audience | no | - |
| `TOPIC_ID` | ex `records-changed` | PubSub topic id | no | <https://console.cloud.google.com/cloudpubsub/topic> |
| `STORAGE_HOST` | `http://localhost:8081/api/storage/v1/` | Storage service url | no | - |
| `LEGAL_HOST` | `http://localhost:8081/api/legal/v1/` | Legal service url | no | - |
| `GROUP_ID` | `osdu-gcp.go3-nrg.projects.epam.com` | Group id, used in storage record ACL. Full group will be "data.default.viewers@{{data-partition-id}}.{{group_id}}" | no | - |
**Entitlements configuration for integration accounts**
| DE_OPS_TESTER | DE_ADMIN_TESTER | DE_EDITOR_TESTER | DE_NO_ACCESS_TESTER |
| --- | --- | --- | --- |
|notification.pubsub<br/>service.entitlements.user<br/>users<br/>users.datalake.ops</br>| service.entitlements.user<br/>users<br/>users.datalake.admins</br> | service.entitlements.user<br/>users<br/>users.datalake.editors</br> | service.entitlements.user<br/>users<br/>|
Above variables should be configured in the release pipeline to run integration tests. You should also replace them with proper values if you wish to run tests locally.
### Commands to run tests
* Integration tests are refactored into two pieces: Core and Provider. Core contains business logic for tests and is a dependency for executing the tests from provider module. To build the core module, simply navigate to `notification-test-core` directory and run `mvn clean install`. This will build the core module
* Next, to execute the integration tests, navigate to the provider module and execute `mvn test`
```bash
# (cd testing/notification-test-core/ && mvn clean install)
# Note: this assumes that the environment variables for integration tests as outlined
# above are already exported in your environment.
$ (cd testing/notification-test-gc/ && mvn clean test)
```
Navigate to notification service's root folder and run all the tests:
```bash
# build + test + install core service code
$ (cd notification-core/ && mvn clean install)
```
At PubSub should be created set of topics and subscriptions. (see [Overview](#overview))
| topic name | subscription name pattern | description | sensitive? |
|-------------------------------------------------------|--------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------|
| `register-subscriber-control` | `notification-control-topic-{data-partition-id}` | Register subscriber control topic | yes |
| `{topic-name}` e.g. `records-changed` | `notification-{topic-name}-service` | Service topics from [topics.json](https://community.opengroup.org/osdu/platform/system/register/-/blob/master/provider/register-gc/src/main/resources/topics.json) on Register service | yes |
| `{topic-name}-publish` e.g. `records-changed-publish` | `notification-{topic-name}-publish}` | Publish topics in relation 1-on-1 for each service topic | yes |
Dmitrii Novikov (EPAM)
committed
Control topic name can be overridden by:
Dmitrii Novikov (EPAM)
committed
* through the Spring Boot property `oqm-register-subscriber-control-topic-name`
* environment variable `OQM_REGISTER_SUBSCRIBER_CONTROL_TOPIC_NAME`
### Important. Dead lettering configuration
Default values of subscription settings `Dead lettering = Disabled`,
and `Retry policy = Retry immediately` leads to a **significant increase in paying bills**.
To avoid such a situation need to configure the dead letter topic and set
parameters:
- Maximum delivery attempts = 5
- Retry policy = Retry after exponential backoff delay
- Minimum backoff duration = 10 seconds
- Maximum backoff duration = 600 seconds
On the subscription dead lettering settings tab:
- Select an active dead letter topic
- Assign Publisher role
- Assign Subscriber role
- Dead letter topic should not be the source topic
Notification service uses a common `dead-lettering-notification` topic because dead lettering logic is not supported yet.