Commit 1aa6604b authored by Artem Dobrynin (EPAM)'s avatar Artem Dobrynin (EPAM) Committed by Riabokon Stanislav(EPAM)[GCP]
Browse files

Migrate Partition Service to Anthos (new approach) (GONRG-3881)

parent 27080fd6
# Partition Service
os-partition-gcp is a [Spring Boot](https://spring.io/projects/spring-boot) service that is responsible for creating and retrieving partition specific properties on behalf of other services whether they are secret values or not.
## Features of implementation
This is a universal solution created using EPAM OSM mapper technology. It allows you to work with various
implementations of KV stores.
## Limitations of the current version
In the current version, the mappers have been equipped with several drivers to the stores:
OSM (mapper for KV-data): Google Datastore; Postgres
## Extensibility
To use any other store or message broker, implement a driver for it. With an extensible set of drivers, the solution is
unrestrictedly universal and portable without modification to the main code.
Mappers support "multitenancy" with flexibility in how it is implemented. They switch between datasources of different
tenants due to the work of a bunch of classes that implement the following interfaces:
* Destination - takes a description of the current context, e.g., "data-partition-id = opendes";
* DestinationResolver – accepts Destination, finds the resource, connects, and returns Resolution;
* DestinationResolution – contains a ready-made connection, the mapper uses it to get the data.
## Getting Started
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.
......@@ -12,6 +33,34 @@ Pre-requisites
* Lombok 1.16 or later
* Maven
## Mapper tuning mechanisms
This service uses specific implementations of DestinationResolvers. A total of 2 resolvers have been implemented, which are divided into two groups:
### for universal technologies:
- for Postgres: osm/config/resolver/OsmPostgresDestinationResolver.java
#### Their algorithms are as follows:
- incoming Destination carries data-partition-id
- resolver accesses the Partition service and gets PartitionInfo
- from PartitionInfo resolver retrieves properties for the connection: URL, username, password etc.
- resolver creates a data source, connects to the resource, remembers the datasource
- resolver gives the datasource to the mapper in the Resolution object
### for native Google Cloud technologies:
- for Datastore: osm/config/resolver/OsmDatastoreDestinationResolver.java
#### Their algorithms are similar,
Except that they do not receive special properties from the Partition service for connection, because the location of
the resources is unambiguously known - they are in the GCP project. And credentials are also not needed - access to data
is made on behalf of the Google Identity SA under which the service itself is launched. Therefore, resolver takes only
the value of the **projectId** property from PartitionInfo and uses it to connect to a resource in the corresponding GCP
project.
### Installation
In order to run the service locally or remotely, you will need to have the following environment variables defined.
......@@ -26,7 +75,61 @@ In order to run the service locally or remotely, you will need to have the follo
| `GOOGLE_APPLICATION_CREDENTIALS` | ex `/path/to/directory/service-key.json` | Service account credentials, you only need this if running locally | yes | https://console.cloud.google.com/iam-admin/serviceaccounts |
| `KEY_RING` | ex `csqp` | A key ring holds keys in a specific Google Cloud location and permit us to manage access control on groups of keys | yes | https://cloud.google.com/kms/docs/resource-hierarchy#key_rings |
| `KMS_KEY` | ex `partitionService` | A key exists on one key ring linked to a specific location. | yes | https://cloud.google.com/kms/docs/resource-hierarchy#key_rings |
| `PARTITION_PROPERTY_KIND` | ex `PartitionProperty` | Kind name to store the properties. | no | - |
| `PARTITION_NAMESPACE` | ex `partition` | Namespace for database. | no | - |
| `osmDriver` | ex `postgres` or `datastore` | Osm driver mode that defines which storage will be used | no | - |
| `osm.postgres.url` | ex `jdbc:postgresql://127.0.0.1:5432/postgres` | Postgres server URL | no | - |
| `osm.postgres.username` | ex `postgres` | Postgres admin username | no | - |
| `osm.postgres.password` | ex `postgres` | Postgres admin password | yes | - |
## Configuring mappers' Datasources
When using non-Google-Cloud-native technologies, property sets must be defined on the Partition service as part of
PartitionInfo for each Tenant.
They are specific to each storage technology:
#### for OSM - Postgres:
**database structure**
OSM works with data logically organized as "partition"->"namespace"->"kind"->"record"->"columns". The above sequence
describes how it is named in Google Datastore, where "partition" maps to "GCP project".
This is how **Postgres** OSM driver does. Notice, the above hierarchy has been kept, but Postgres uses alternative entities
for it.
| Datastore hierarchy level | | Postgres alternative used |
|---------------------------|-----|----------------------------|
| partition (GCP project) | == | Postgres server URL |
| namespace | == | Schema |
| kind | == | Table |
| record | == | '<multiple table records>' |
| columns | == | id, data (jsonb) |
As we can see in the above table, Postgres uses different approach in storing business data in records. Not like
Datastore, which segments data into multiple physical columns, Postgres organises them into the single JSONB "data"
column. It allows provisioning new data registers easily not taking care about specifics of certain registers structure.
In the current OSM version (as on December'21) the Postgres OSM driver is not able to create new tables in runtime.
So this is a responsibility of DevOps / CICD to provision all required SQL tables (for all required data kinds) when on new
environment or tenant provisioning when using Postgres. Detailed instructions (with examples) for creating new tables is
in the **OSM module Postgres driver README.md** `org/opengroup/osdu/core/gcp/osm/translate/postgresql/README.md`
As a quick shortcut, this example snippet can be used by DevOps DBA:
* `exampleschema` equals to `PARTITION_NAMESPACE`
* `ExampleKind` equals to `PARTITION_PROPERTY_KIND`
```postgres-psql
--CREATE SCHEMA "<exampleschema>";
CREATE TABLE <exampleschema>."<ExampleKind>"(
id text COLLATE pg_catalog."default" NOT NULL,
pk bigint NOT NULL GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
data jsonb NOT NULL,
CONSTRAINT <ExampleKind>_id UNIQUE (id)
);
CREATE INDEX <ExampleKind>_datagin ON <exampleschema>."<ExampleKind>" USING GIN (data);
```
### Run Locally
Check that maven is installed:
......
......@@ -28,7 +28,12 @@
<dependency>
<groupId>org.opengroup.osdu</groupId>
<artifactId>core-lib-gcp</artifactId>
<version>0.7.0</version>
<version>0.13.0-rc3</version>
</dependency>
<dependency>
<groupId>org.opengroup.osdu</groupId>
<artifactId>osm</artifactId>
<version>0.13.0-SNAPSHOT</version>
</dependency>
<dependency>
......@@ -134,6 +139,16 @@
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.google.http-client</groupId>
<artifactId>google-http-client</artifactId>
<version>1.39.0</version>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<plugins>
<plugin>
......
......@@ -17,13 +17,17 @@
package org.opengroup.osdu.partition.provider.gcp;
import org.opengroup.osdu.core.gcp.di.GcpPartitionClientFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.gcp.data.datastore.repository.config.EnableDatastoreRepositories;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.FilterType;
import org.springframework.transaction.annotation.EnableTransactionManagement;
@ComponentScan({"org.opengroup.osdu"})
@ComponentScan(basePackages = {"org.opengroup.osdu"}, excludeFilters =
@ComponentScan.Filter(type = FilterType.ASSIGNABLE_TYPE, classes = {GcpPartitionClientFactory.class})
)
@SpringBootApplication
@EnableDatastoreRepositories
@EnableTransactionManagement
......
/*
* Copyright 2020-2021 Google LLC
* Copyright 2020-2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.partition.provider.gcp.config;
import lombok.Getter;
import lombok.Setter;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;
@Configuration
@ConditionalOnProperty(name = "osmDriver", havingValue = "postgres")
@ConfigurationProperties(prefix = "osm.postgres")
@Getter
@Setter
public class PostgresOsmConfiguration {
private String url;
private String username;
private String password;
private Integer maximumPoolSize = 40;
private Integer minimumIdle = 0;
private Integer idleTimeout = 30000;
private Integer maxLifetime = 1800000;
private Integer connectionTimeout = 30000;
}
......@@ -43,6 +43,10 @@ public class PropertiesConfiguration {
private String serviceAccountTail;
private String partitionPropertyKind;
private String partitionNamespace;
@PostConstruct
public void setUp() {
if (Objects.isNull(serviceAccountTail) || serviceAccountTail.isEmpty()) {
......
......@@ -17,25 +17,18 @@
package org.opengroup.osdu.partition.provider.gcp.model;
import com.google.cloud.datastore.Key;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import org.opengroup.osdu.partition.model.Property;
import org.springframework.cloud.gcp.data.datastore.core.mapping.Entity;
import org.springframework.cloud.gcp.data.datastore.core.mapping.Field;
import org.springframework.data.annotation.Id;
@Entity(name = "PartitionProperty")
@Data
@NoArgsConstructor
@AllArgsConstructor
public class PartitionPropertyEntity {
@Id
private Key key;
private String id;
@Field(name = "partition_id")
private String partitionId;
private String name;
......
/*
* Copyright 2020-2021 Google LLC
* Copyright 2020-2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.partition.provider.gcp.osm.config.mapper;
import com.google.cloud.datastore.Key;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import org.opengroup.osdu.core.gcp.osm.persistence.IdentityTranslator;
import org.opengroup.osdu.core.gcp.osm.translate.Instrumentation;
import org.opengroup.osdu.core.gcp.osm.translate.TypeMapper;
import org.opengroup.osdu.partition.provider.gcp.model.PartitionPropertyEntity;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Scope;
import org.springframework.stereotype.Component;
import java.util.Collections;
import java.util.HashMap;
import static org.springframework.beans.factory.config.BeanDefinition.SCOPE_SINGLETON;
@Component
@Scope(SCOPE_SINGLETON)
@ConditionalOnProperty(name = "osmDriver")
public class TypeMapperImpl extends TypeMapper {
public TypeMapperImpl(){
super(ImmutableList.of(
new Instrumentation<>(PartitionPropertyEntity.class,
new HashMap<String, String>() {{
put("partitionId", "partition_id");
}},
ImmutableMap.of(),
new IdentityTranslator<>(
PartitionPropertyEntity::getId,
(p, o) -> p.setId(((Key)o).getName())
),
Collections.singletonList("id")
))
);
}
}
/*
* Copyright 2020-2021 Google LLC
* Copyright 2020-2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.partition.provider.gcp.osm.config.provider;
import lombok.RequiredArgsConstructor;
import org.opengroup.osdu.core.gcp.osm.model.Destination;
import org.opengroup.osdu.core.gcp.osm.model.Kind;
import org.opengroup.osdu.core.gcp.osm.model.Namespace;
import org.opengroup.osdu.partition.provider.gcp.config.PropertiesConfiguration;
import org.springframework.stereotype.Service;
@Service
@RequiredArgsConstructor
public class OsmPartitionDestinationProvider {
private final PropertiesConfiguration config;
public Destination getDestination() {
return Destination.builder()
.partitionId(config.getGoogleCloudProject())
.namespace(new Namespace(config.getPartitionNamespace()))
.kind(new Kind(config.getPartitionPropertyKind()))
.build();
}
}
/*
* Copyright 2020-2021 Google LLC
* Copyright 2020-2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.partition.provider.gcp.osm.config.resolver;
import com.google.api.gax.retrying.RetrySettings;
import com.google.cloud.TransportOptions;
import com.google.cloud.datastore.Datastore;
import com.google.cloud.datastore.DatastoreOptions;
import com.google.cloud.http.HttpTransportOptions;
import lombok.RequiredArgsConstructor;
import org.opengroup.osdu.core.gcp.osm.model.Destination;
import org.opengroup.osdu.core.gcp.osm.translate.datastore.DsDestinationResolution;
import org.opengroup.osdu.core.gcp.osm.translate.datastore.DsDestinationResolver;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Scope;
import org.springframework.stereotype.Component;
import org.threeten.bp.Duration;
import java.io.IOException;
import static org.springframework.beans.factory.config.BeanDefinition.SCOPE_SINGLETON;
@Component
@Scope(SCOPE_SINGLETON)
@ConditionalOnProperty(name = "osmDriver", havingValue = "datastore")
@RequiredArgsConstructor
public class OsmDatastoreDestinationResolver implements DsDestinationResolver {
protected static final RetrySettings RETRY_SETTINGS = RetrySettings.newBuilder().setMaxAttempts(6).setInitialRetryDelay(Duration.ofSeconds(10L)).setMaxRetryDelay(Duration.ofSeconds(32L)).setRetryDelayMultiplier(2.0D).setTotalTimeout(Duration.ofSeconds(50L)).setInitialRpcTimeout(Duration.ofSeconds(50L)).setRpcTimeoutMultiplier(1.0D).setMaxRpcTimeout(Duration.ofSeconds(50L)).build();
protected static final TransportOptions TRANSPORT_OPTIONS = HttpTransportOptions.newBuilder().setReadTimeout(30000).build();
private final Datastore partitionDatastore;
/**
* Takes provided Destination with partitionId set to needed tenantId, returns its TenantInfo.projectId.
*
* @param destination to resolve
* @return resolution results
*/
@Override
public DsDestinationResolution resolve(Destination destination) {
String projectId = destination.getPartitionId();
Datastore datastore =
partitionDatastore == null
? DatastoreOptions.newBuilder()
.setRetrySettings(RETRY_SETTINGS)
.setTransportOptions(TRANSPORT_OPTIONS)
.setProjectId(projectId)
.setNamespace(destination.getNamespace().getName())
.build()
.getService()
: partitionDatastore;
return DsDestinationResolution.builder()
.projectId(datastore.getOptions().getProjectId())
.datastore(datastore)
.build();
}
@Override
public void close() throws IOException {
//Method stub
}
}
\ No newline at end of file
/*
* Copyright 2020-2021 Google LLC
* Copyright 2020-2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.partition.provider.gcp.osm.config.resolver;
import com.zaxxer.hikari.HikariDataSource;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.opengroup.osdu.core.gcp.osm.model.Destination;
import org.opengroup.osdu.core.gcp.osm.translate.postgresql.PgDestinationResolution;
import org.opengroup.osdu.core.gcp.osm.translate.postgresql.PgDestinationResolver;
import org.opengroup.osdu.partition.provider.gcp.config.PostgresOsmConfiguration;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.jdbc.DataSourceBuilder;
import org.springframework.context.annotation.Scope;
import org.springframework.stereotype.Component;
import javax.annotation.PreDestroy;
import javax.sql.DataSource;
import java.util.HashMap;
import java.util.Map;
import static org.springframework.beans.factory.config.BeanDefinition.SCOPE_SINGLETON;
@Component
@Scope(SCOPE_SINGLETON)
@ConditionalOnProperty(name = "osmDriver", havingValue = "postgres")
@RequiredArgsConstructor
@Slf4j
public class OsmPostgresDestinationResolver implements PgDestinationResolver {
private final PostgresOsmConfiguration properties;
private static final String DRIVER_CLASS_NAME = "org.postgresql.Driver";
private final Map<String, DataSource> dataSourceCache = new HashMap<>();
@Override
public PgDestinationResolution resolve(Destination destination) {
String partitionId = destination.getPartitionId();
DataSource dataSource = dataSourceCache.get(partitionId);
if (dataSource == null || (dataSource instanceof HikariDataSource && ((HikariDataSource) dataSource).isClosed())) {
synchronized (dataSourceCache) {
dataSource = dataSourceCache.get(partitionId);
if (dataSource == null || (dataSource instanceof HikariDataSource && ((HikariDataSource) dataSource).isClosed())) {
dataSource = buildDataSourceFromProperties(partitionId);
}
}
}
return PgDestinationResolution.builder()
.datasource(dataSource)
.build();
}
private DataSource buildDataSourceFromProperties(String partitionId) {
DataSource dataSource;
dataSource = DataSourceBuilder.create()
.driverClassName(DRIVER_CLASS_NAME)
.url(properties.getUrl())
.username(properties.getUsername())
.password(properties.getPassword())
.build();
HikariDataSource hikariDataSource = (HikariDataSource) dataSource;
hikariDataSource.setMaximumPoolSize(properties.getMaximumPoolSize());
hikariDataSource.setMinimumIdle(properties.getMinimumIdle());
hikariDataSource.setIdleTimeout(properties.getIdleTimeout());
hikariDataSource.setMaxLifetime(properties.getMaxLifetime());
hikariDataSource.setConnectionTimeout(properties.getConnectionTimeout());
dataSourceCache.put(partitionId, dataSource);
return dataSource;
}
@PreDestroy
@Override
public void close() {
log.info("On pre-destroy. {} DataSources to shutdown", dataSourceCache.size());
for (DataSource dataSource : dataSourceCache.values()) {
if (dataSource instanceof HikariDataSource && !((HikariDataSource) dataSource).isClosed()) {
((HikariDataSource) dataSource).close();
}
}
}
}
/*
* Copyright 2020-2021 Google LLC
* Copyright 2020-2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.partition.provider.gcp.osm.repository;