Skip to content

(GONRG-3831) GCP Register: OSM and OQM mappers

Rostislav Dublin (EPAM) requested to merge gcp-oqm-osm into master

Type of change

  • Bug Fix
  • Feature

Does this introduce a change in the core logic?

  • [NO]

Does this introduce a change in the cloud provider implementation, if so which cloud?

  • AWS
  • Azure
  • GCP
  • IBM

Does this introduce a breaking change?

  • [YES/NO]

What is the current behavior?

Register works with KV store and Message Broker directly

What is the new/expected behavior?

Register will work with KV and MQ through mapper/driver abstraction

Have you added/updated Unit Tests and Integration Tests?

yes

Any other useful information

Features of implementation

This is a universal solution created using EPAM OSM and OQM mappers technology. It allows you to work with various implementations of data stores and message brokers.

Limitations of the current version

In the current version, the mappers are equipped with several drivers to the stores and the message broker:

  • OSM (mapper for KV-data): Google Datastore; Postgres
  • OBM (mapper to message brokers): Google PubSub; RabbitMQ

Extensibility

To use any other store or message broker, implement a driver for it. With an extensible set of drivers, the solution is unrestrictedly universal and portable without modification to the main code.

Mapper tuning mechanisms

This service uses specific implementations of DestinationResolvers based on the tenant information provided by the OSDU Partition service. A total of 4 resolvers are implemented, which are divided into two groups:

for universal technologies:
  • for Postgres: mappers/osm/PgTenantOsmDestinationResolver.java
  • for RabbitMQ: mappers/oqm/MqTenantOqmDestinationResolver.java
Their algorithms are as follows:
  • incoming Destination carries data-partition-id
  • resolver accesses the Partition service and gets PartitionInfo
  • from PartitionInfo resolver retrieves properties for the connection: URL, username, password etc.
  • resolver creates a data source, connects to the resource, remembers the datasource
  • resolver gives the datasource to the mapper in the Resolution object
for native Google Cloud technologies:
  • for Datastore: mappers/osm/DsTenantOsmDestinationResolver.java
  • for PubSub: mappers/oqm/PsTenantOqmDestinationResolver.java
Their algorithms are similar,

Except that they do not receive special properties from the Partition service for connection, because the location of the resources is unambiguously known - they are in the GCP project. And credentials are also not needed - access to data is made on behalf of the Google Identity SA under which the service itself is launched. Therefore, resolver takes only the value of the projectId property from PartitionInfo and uses it to connect to a resource in the corresponding GCP project.

Edited by Rostislav Dublin (EPAM)

Merge request reports