diff --git a/provider/notification-gcp/README.md b/provider/notification-gcp/README.md index 80a59b3287f04ee392f40d1e1d83fd8b0dc23384..9b8c66c14ea75955424b9bf819ade2247c625798 100644 --- a/provider/notification-gcp/README.md +++ b/provider/notification-gcp/README.md @@ -4,13 +4,39 @@ notification-gcp is a [Spring Boot](https://spring.io/projects/spring-boot) serv ## Getting Started These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system. -### Requirements +# Features of implementation +This is a universal solution created using EPAM OQM mapper technology. +It allows you to work with various implementations of message brokers. + +## Limitations of the current version + +In the current version, the OQM mapper is equipped with 2 drivers to the message brokers: + +- Google PubSub; +- RabbitMQ + +## Extensibility + +To use any other message broker, implement a driver for it. With an extensible set of drivers, the solution is unrestrictedly universal and portable without modification to the main code. + +Mapper support "multitenancy" with flexibility in how it is implemented. +It switches between datasources of different tenants due to the work of a bunch of classes that implement the following interfaces: + +- Destination - takes a description of the current context, e.g., "data-partition-id = opendes" +- DestinationResolver – accepts Destination, finds the resource, connects, and returns Resolution +- DestinationResolution – contains a ready-made connection, the mapper uses it to get to data + +# Settings and Configuration +## Requirements +### Mandatory * Java 8 * [Maven 3.6.0+](https://maven.apache.org/download.cgi) + +### for Google Cloud only * GCloud command line tool * GCloud access to opendes project -### General Tips +## General Tips **Environment Variable Management** The following tools make environment variable configuration simpler @@ -22,16 +48,45 @@ This project uses [Lombok](https://projectlombok.org/) for code generation. You - [Intellij configuration](https://projectlombok.org/setup/intellij) - [VSCode configuration](https://projectlombok.org/setup/vscode) -### Installation -In order to run the service locally or remotely, you will need to have the following environment variables defined. +## Mapper tuning mechanisms + +This service uses specific implementation of DestinationResolver based on the tenant information provided by the OSDU Partition service. +A total of 2 resolvers are implemented, which are divided into two groups: +### for universal technologies: +- for RabbitMQ: mappers/oqm/MqTenantOqmDestinationResolver.java + +#### Their algorithms are as follows: +- incoming Destination carries data-partition-id +- resolver accesses the Partition service and gets PartitionInfo +- from PartitionInfo resolver retrieves properties for the connection: URL, username, password etc. +- resolver creates a data source, connects to the resource, remembers the datasource +- resolver gives the datasource to the mapper in the Resolution object + +### for native Google Cloud technologies: +- for PubSub: mappers/oqm/PsTenantOqmDestinationResolver.java + +#### Their algorithms are similar, +Except that they do not receive special properties from the Partition service for connection, because the location of the resources is unambiguously known - they are in the GCP project. And credentials are also not needed - access to data is made on behalf of the Google Identity SA under which the service itself is launched. Therefore, resolver takes only the value of the **projectId** property from PartitionInfo and uses it to connect to a resource in the corresponding GCP project. + +# Configuration + +## Service Configuration +In order to run the service locally or remotely, define the following environment variables. +Most of them are common to all hosting environments, but there are properties that are only necessary when running in Google Cloud. + +#### Common properties for all environments | name | value | description | sensitive? | source | | --- | --- | --- | --- | --- | | `APP_ENTITLEMENTS` | ex `https://entitlements.com/entitlements/v1` | Entitlements API endpoint | no | output of infrastructure deployment | | `APP_REGISTER` | ex `https://register.com/api/register/v1` | Storage API endpoint | no | output of infrastructure deployment | +| `PARTITION_API` | ex `http://localhost:8081/api/partition/v1` | Partition service endpoint | no | - | + +#### For Google Cloud only +| name | value | description | sensitive? | source | +|------------------------------|---------------------------------------|--------------------------------------------------------------------|------------|---------------------------------------------------| | `APP_PROJECT` | ex `opendes` | Google Cloud Project Id | no | output of infrastructure deployment | | `APP_AUDIENCES` | ex `*****.apps.googleusercontent.com` | Client ID for getting access to cloud resources | yes | https://console.cloud.google.com/apis/credentials | -| `PARTITION_API` | ex `http://localhost:8081/api/partition/v1` | Partition service endpoint | no | - | **System Environment required to run service** @@ -39,7 +94,115 @@ In order to run the service locally or remotely, you will need to have the follo | --- | --- | --- | --- | --- | | `SPRING_PROFILES_ACTIVE` | `local` | spring active profile | no | -### Run Locally +## Configuring mappers Datasources +When using non-Google-Cloud-native technologies, property sets must be defined on the Partition service as part of PartitionInfo for each Tenant. + +#### for OQM - RabbitMQ: +**prefix:** `register.oqm.rabbitmq` +It can be overridden by: + +- through the Spring Boot property `oqm.rabbitmq.partitionPropertiesPrefix` +- environment variable `OQM_RABBITMQ_PARTITIONPROPERTIESPREFIX` + +**Propertyset** (for two types of connection: messaging and admin operations): + +| Property | Description | +| --- | --- | +| register.oqm.rabbitmq.amqp.host | messaging hostnameorIP | +| register.oqm.rabbitmq.amqp.port | - port | +| register.oqm.rabbitmq.amqp.path | - path | +| register.oqm.rabbitmq.amqp.username | - username | +| register.oqm.rabbitmq.amqp.password | - password | +| register.oqm.rabbitmq.admin.schema | admin host schema | +| register.oqm.rabbitmq.admin.host | - host name | +| register.oqm.rabbitmq.admin.port | - port | +| register.oqm.rabbitmq.admin.path | - path | +| register.oqm.rabbitmq.admin.username | - username | +| register.oqm.rabbitmq.admin.password | - password | + +<details><summary>Example of a single tenant definition</summary> + +``` + +curl -L -X PATCH 'https://dev.osdu.club/api/partition/v1/partitions/opendes' -H 'data-partition-id: opendes' -H 'Authorization: Bearer ...' -H 'Content-Type: application/json' --data-raw '{ + "properties": { + "register.oqm.rabbitmq.amqp.host": { + "sensitive": false, + "value": "localhost" + }, + "register.oqm.rabbitmq.amqp.port": { + "sensitive": false, + "value": "5672" + }, + "register.oqm.rabbitmq.amqp.path": { + "sensitive": false, + "value": "" + }, + "register.oqm.rabbitmq.amqp.username": { + "sensitive": false, + "value": "guest" + }, + "register.oqm.rabbitmq.amqp.password": { + "sensitive": true, + "value": "guest" + }, + + "register.oqm.rabbitmq.admin.schema": { + "sensitive": false, + "value": "http" + }, + "register.oqm.rabbitmq.admin.host": { + "sensitive": false, + "value": "localhost" + }, + "register.oqm.rabbitmq.admin.port": { + "sensitive": false, + "value": "9002" + }, + "register.oqm.rabbitmq.admin.path": { + "sensitive": false, + "value": "/api" + }, + "register.oqm.rabbitmq.admin.username": { + "sensitive": false, + "value": "guest" + }, + "register.oqm.rabbitmq.admin.password": { + "sensitive": true, + "value": "guest" + } + } +}' + +``` + +</details> + +## Interaction with message brokers + +### Specifics of work through PULL subscription + +To receive messages from brokers, this solution uses the PULL-subscriber mechanism in the Notification service. +This is its cardinal difference from other implementations that use PUSH-subscribers (webhooks). +This opens a wide choice when choosing brokers. + +When using PULL-subscribers, there is a need to restore Notification service subscribers for each Subscription +at the start of Notification service, as well as in the runtime, +upon registration of a new Subscription by the Register service. + +To do this, a special "command" topic is involved: + +- the default topic name is `register-subscriber-control`. + +If necessary, the name of the topic can be overridden through: + +- Spring Boot property `oqm.registerSubscriberControlTopicName` +- environment variable `OQM_REGISTERSUBSCRIBERCONTROLTOPICNAME` + +A topic is created, in its absence, when any of Register or Notification services starts. + +# Run and test the service +## Running Locally Check that maven is installed: ```bash $ mvn --version @@ -145,9 +308,9 @@ After the service has started it should be accessible via a web browser by visit **Entitlements configuration for integration accounts** - | DE_OPS_TESTER | DE_ADMIN_TESTER | DE_EDITOR_TESTER | DE_NO_ACCESS_TESTER | - | --- | --- | --- | --- | - |notification.pubsub<br/>service.entitlements.user<br/>users<br/>users.datalake.ops</br>| service.entitlements.user<br/>users<br/>users.datalake.admins</br> | service.entitlements.user<br/>users<br/>users.datalake.editors</br> | service.entitlements.user<br/>users<br/>| +| DE_OPS_TESTER | DE_ADMIN_TESTER | DE_EDITOR_TESTER | DE_NO_ACCESS_TESTER | +| --- | --- | --- | --- | +|notification.pubsub<br/>service.entitlements.user<br/>users<br/>users.datalake.ops</br>| service.entitlements.user<br/>users<br/>users.datalake.admins</br> | service.entitlements.user<br/>users<br/>users.datalake.editors</br> | service.entitlements.user<br/>users<br/>| Above variables should be configured in the release pipeline to run integration tests. You should also replace them with proper values if you wish to run tests locally. diff --git a/provider/notification-gcp/src/main/java/org/opengroup/osdu/notification/provider/gcp/mappers/oqm/OqmConfigurationProperties.java b/provider/notification-gcp/src/main/java/org/opengroup/osdu/notification/provider/gcp/mappers/oqm/OqmConfigurationProperties.java new file mode 100644 index 0000000000000000000000000000000000000000..3167c7f084f2c37ebb10c3a78cb0e9531eb4052c --- /dev/null +++ b/provider/notification-gcp/src/main/java/org/opengroup/osdu/notification/provider/gcp/mappers/oqm/OqmConfigurationProperties.java @@ -0,0 +1,23 @@ +package org.opengroup.osdu.notification.provider.gcp.mappers.oqm; + +import lombok.Getter; +import lombok.Setter; +import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; +import org.springframework.boot.context.properties.ConfigurationProperties; +import org.springframework.context.annotation.Configuration; + +/** + * @author Rostislav_Dublin + * @since 28.11.2021 + */ + +@Configuration +@ConditionalOnProperty(name = "oqmDriver") +@ConfigurationProperties(prefix = "oqm") +@Getter +@Setter +public class OqmConfigurationProperties { + + private String registerSubscriberControlTopicName = "register-subscriber-control"; + +}