Merge branch 'gcp/feature/osm-impl' into 'master'

Migrate Partition Service to Anthos (new approach) (GONRG-3881)

See merge request !130

provider/partition-gcp/README.md

 # Partition Service
 os-partition-gcp is a [Spring Boot](https://spring.io/projects/spring-boot) service that is responsible for creating and retrieving partition specific properties on behalf of other services whether they are secret values or not.
 
+## Features of implementation
+This is a universal solution created using EPAM OSM mapper technology. It allows you to work with various
+implementations of KV stores.
+
+## Limitations of the current version
+In the current version, the mappers have been equipped with several drivers to the stores:
+
+OSM (mapper for KV-data): Google Datastore; Postgres
+
+
+## Extensibility
+To use any other store or message broker, implement a driver for it. With an extensible set of drivers, the solution is
+unrestrictedly universal and portable without modification to the main code.
+Mappers support "multitenancy" with flexibility in how it is implemented. They switch between datasources of different
+tenants due to the work of a bunch of classes that implement the following interfaces:
+
+* Destination - takes a description of the current context, e.g., "data-partition-id = opendes";
+* DestinationResolver – accepts Destination, finds the resource, connects, and returns Resolution;
+* DestinationResolution – contains a ready-made connection, the mapper uses it to get the data.
+
+
 ## Getting Started
 These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.
 
@@ -12,6 +33,34 @@ Pre-requisites
 * Lombok 1.16 or later
 * Maven
 
+## Mapper tuning mechanisms
+
+This service uses specific implementations of DestinationResolvers. A total of 2 resolvers have been implemented, which are divided into two groups:
+
+### for universal technologies:
+
+- for Postgres: osm/config/resolver/OsmPostgresDestinationResolver.java
+
+#### Their algorithms are as follows:
+
+- incoming Destination carries data-partition-id
+- resolver accesses the Partition service and gets PartitionInfo
+- from PartitionInfo resolver retrieves properties for the connection: URL, username, password etc.
+- resolver creates a data source, connects to the resource, remembers the datasource
+- resolver gives the datasource to the mapper in the Resolution object
+
+### for native Google Cloud technologies:
+
+- for Datastore: osm/config/resolver/OsmDatastoreDestinationResolver.java
+
+#### Their algorithms are similar,
+
+Except that they do not receive special properties from the Partition service for connection, because the location of
+the resources is unambiguously known - they are in the GCP project. And credentials are also not needed - access to data
+is made on behalf of the Google Identity SA under which the service itself is launched. Therefore, resolver takes only
+the value of the **projectId** property from PartitionInfo and uses it to connect to a resource in the corresponding GCP
+project.
+
 ### Installation
 In order to run the service locally or remotely, you will need to have the following environment variables defined.
 
@@ -26,7 +75,61 @@ In order to run the service locally or remotely, you will need to have the follo
 | `GOOGLE_APPLICATION_CREDENTIALS` | ex `/path/to/directory/service-key.json` | Service account credentials, you only need this if running locally | yes | https://console.cloud.google.com/iam-admin/serviceaccounts |
 | `KEY_RING` | ex `csqp` | A key ring holds keys in a specific Google Cloud location and permit us to manage access control on groups of keys | yes | https://cloud.google.com/kms/docs/resource-hierarchy#key_rings |
 | `KMS_KEY` | ex `partitionService` | A key exists on one key ring linked to a specific location. | yes | https://cloud.google.com/kms/docs/resource-hierarchy#key_rings |
-
+| `PARTITION_PROPERTY_KIND` | ex `PartitionProperty` | Kind name to store the properties. | no | - |
+| `PARTITION_NAMESPACE` | ex `partition` | Namespace for database. | no | - |
+| `osmDriver` | ex `postgres` or `datastore` | Osm driver mode that defines which storage will be used | no | - |
+| `osm.postgres.url` | ex `jdbc:postgresql://127.0.0.1:5432/postgres` | Postgres server URL | no | - |
+| `osm.postgres.username` | ex `postgres` | Postgres admin username | no | - |
+| `osm.postgres.password` | ex `postgres` | Postgres admin password | yes | - |
+
+## Configuring mappers' Datasources
+
+When using non-Google-Cloud-native technologies, property sets must be defined on the Partition service as part of
+PartitionInfo for each Tenant.
+
+They are specific to each storage technology:
+
+#### for OSM - Postgres:
+
+**database structure**
+OSM works with data logically organized as "partition"->"namespace"->"kind"->"record"->"columns". The above sequence
+describes how it is named in Google Datastore, where "partition" maps to "GCP project".
+
+This is how **Postgres** OSM driver does. Notice, the above hierarchy has been kept, but Postgres uses alternative entities
+for it.
+
+| Datastore hierarchy level |     | Postgres alternative used  |
+|---------------------------|-----|----------------------------|
+| partition (GCP project)   | ==  | Postgres server URL        |
+| namespace                 | ==  | Schema                     |
+| kind                      | ==  | Table                      |
+| record                    | ==  | '<multiple table records>' |
+| columns                   | ==  | id, data (jsonb)           |
+
+As we can see in the above table, Postgres uses different approach in storing business data in records. Not like
+Datastore, which segments data into multiple physical columns, Postgres organises them into the single JSONB "data"
+column. It allows provisioning new data registers easily not taking care about specifics of certain registers structure.
+In the current OSM version (as on December'21) the Postgres OSM driver is not able to create new tables in runtime.
+
+So this is a responsibility of DevOps / CICD to provision all required SQL tables (for all required data kinds) when on new
+environment or tenant provisioning when using Postgres. Detailed instructions (with examples) for creating new tables is
+in the **OSM module Postgres driver README.md** `org/opengroup/osdu/core/gcp/osm/translate/postgresql/README.md`
+
+As a quick shortcut, this example snippet can be used by DevOps DBA:
+
+* `exampleschema` equals to `PARTITION_NAMESPACE` 
+* `ExampleKind` equals to `PARTITION_PROPERTY_KIND`
+
+```postgres-psql
+--CREATE SCHEMA "<exampleschema>";
+CREATE TABLE <exampleschema>."<ExampleKind>"(
+    id text COLLATE pg_catalog."default" NOT NULL,
+    pk bigint NOT NULL GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+    data jsonb NOT NULL,
+    CONSTRAINT <ExampleKind>_id UNIQUE (id)
+);
+CREATE INDEX <ExampleKind>_datagin ON <exampleschema>."<ExampleKind>" USING GIN (data);
+```
 
 ### Run Locally
 Check that maven is installed:

provider/partition-gcp/pom.xml

@@ -28,7 +28,12 @@
     <dependency>
       <groupId>org.opengroup.osdu</groupId>
       <artifactId>core-lib-gcp</artifactId>
-      <version>0.7.0</version>
+      <version>0.13.0-rc3</version>
+    </dependency>
+    <dependency>
+      <groupId>org.opengroup.osdu</groupId>
+      <artifactId>osm</artifactId>
+      <version>0.13.0-SNAPSHOT</version>
     </dependency>
 
     <dependency>
@@ -134,6 +139,16 @@
     </dependency>
   </dependencies>
 
+  <dependencyManagement>
+    <dependencies>
+      <dependency>
+        <groupId>com.google.http-client</groupId>
+        <artifactId>google-http-client</artifactId>
+        <version>1.39.0</version>
+      </dependency>
+    </dependencies>
+  </dependencyManagement>
+
   <build>
     <plugins>
       <plugin>

provider/partition-gcp/src/main/java/org/opengroup/osdu/partition/provider/gcp/PartitionGcpApplication.java

@@ -17,13 +17,17 @@
 
 package org.opengroup.osdu.partition.provider.gcp;
 
+import org.opengroup.osdu.core.gcp.di.GcpPartitionClientFactory;
 import org.springframework.boot.SpringApplication;
 import org.springframework.boot.autoconfigure.SpringBootApplication;
 import org.springframework.cloud.gcp.data.datastore.repository.config.EnableDatastoreRepositories;
 import org.springframework.context.annotation.ComponentScan;
+import org.springframework.context.annotation.FilterType;
 import org.springframework.transaction.annotation.EnableTransactionManagement;
 
-@ComponentScan({"org.opengroup.osdu"})
+@ComponentScan(basePackages = {"org.opengroup.osdu"}, excludeFilters =
+  @ComponentScan.Filter(type = FilterType.ASSIGNABLE_TYPE, classes = {GcpPartitionClientFactory.class})
+)
 @SpringBootApplication
 @EnableDatastoreRepositories
 @EnableTransactionManagement

provider/partition-gcp/src/main/java/org/opengroup/osdu/partition/provider/gcp/config/PostgresOsmConfiguration.java

+/*
+ * Copyright 2020-2021 Google LLC
+ * Copyright 2020-2021 EPAM Systems, Inc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.opengroup.osdu.partition.provider.gcp.config;
+
+import lombok.Getter;
+import lombok.Setter;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.boot.context.properties.ConfigurationProperties;
+import org.springframework.context.annotation.Configuration;
+
+@Configuration
+@ConditionalOnProperty(name = "osmDriver", havingValue = "postgres")
+@ConfigurationProperties(prefix = "osm.postgres")
+@Getter
+@Setter
+public class PostgresOsmConfiguration {
+
+    private String url;
+    private String username;
+    private String password;
+
+    private Integer maximumPoolSize = 40;
+    private Integer minimumIdle = 0;
+    private Integer idleTimeout = 30000;
+    private Integer maxLifetime = 1800000;
+    private Integer connectionTimeout = 30000;
+
+}

provider/partition-gcp/src/main/java/org/opengroup/osdu/partition/provider/gcp/config/PropertiesConfiguration.java

@@ -43,6 +43,10 @@ public class PropertiesConfiguration {
 
     private String serviceAccountTail;
 
+    private String partitionPropertyKind;
+
+    private String partitionNamespace;
+
     @PostConstruct
     public void setUp() {
         if (Objects.isNull(serviceAccountTail) || serviceAccountTail.isEmpty()) {

provider/partition-gcp/src/main/java/org/opengroup/osdu/partition/provider/gcp/model/PartitionPropertyEntity.java

@@ -17,25 +17,18 @@
 
 package org.opengroup.osdu.partition.provider.gcp.model;
 
-import com.google.cloud.datastore.Key;
 import lombok.AllArgsConstructor;
 import lombok.Data;
 import lombok.NoArgsConstructor;
 import org.opengroup.osdu.partition.model.Property;
-import org.springframework.cloud.gcp.data.datastore.core.mapping.Entity;
-import org.springframework.cloud.gcp.data.datastore.core.mapping.Field;
-import org.springframework.data.annotation.Id;
 
-@Entity(name = "PartitionProperty")
 @Data
 @NoArgsConstructor
 @AllArgsConstructor
 public class PartitionPropertyEntity {
 
-  @Id
-  private Key key;
+  private String id;
 
-  @Field(name = "partition_id")
   private String partitionId;
 
   private String name;

provider/partition-gcp/src/main/java/org/opengroup/osdu/partition/provider/gcp/osm/config/mapper/TypeMapperImpl.java

+/*
+ * Copyright 2020-2021 Google LLC
+ * Copyright 2020-2021 EPAM Systems, Inc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.opengroup.osdu.partition.provider.gcp.osm.config.mapper;
+
+import com.google.cloud.datastore.Key;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import org.opengroup.osdu.core.gcp.osm.persistence.IdentityTranslator;
+import org.opengroup.osdu.core.gcp.osm.translate.Instrumentation;
+import org.opengroup.osdu.core.gcp.osm.translate.TypeMapper;
+import org.opengroup.osdu.partition.provider.gcp.model.PartitionPropertyEntity;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.context.annotation.Scope;
+import org.springframework.stereotype.Component;
+
+import java.util.Collections;
+import java.util.HashMap;
+
+import static org.springframework.beans.factory.config.BeanDefinition.SCOPE_SINGLETON;
+
+@Component
+@Scope(SCOPE_SINGLETON)
+@ConditionalOnProperty(name = "osmDriver")
+public class TypeMapperImpl extends TypeMapper {
+    public TypeMapperImpl(){
+        super(ImmutableList.of(
+                new Instrumentation<>(PartitionPropertyEntity.class,
+                        new HashMap<String, String>() {{
+                            put("partitionId", "partition_id");
+                        }},
+                        ImmutableMap.of(),
+                        new IdentityTranslator<>(
+                                PartitionPropertyEntity::getId,
+                                (p, o) -> p.setId(((Key)o).getName())
+                        ),
+                        Collections.singletonList("id")
+                ))
+        );
+    }
+}

provider/partition-gcp/src/main/java/org/opengroup/osdu/partition/provider/gcp/osm/config/provider/OsmPartitionDestinationProvider.java

+/*
+ * Copyright 2020-2021 Google LLC
+ * Copyright 2020-2021 EPAM Systems, Inc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.opengroup.osdu.partition.provider.gcp.osm.config.provider;
+
+import lombok.RequiredArgsConstructor;
+import org.opengroup.osdu.core.gcp.osm.model.Destination;
+import org.opengroup.osdu.core.gcp.osm.model.Kind;
+import org.opengroup.osdu.core.gcp.osm.model.Namespace;
+import org.opengroup.osdu.partition.provider.gcp.config.PropertiesConfiguration;
+import org.springframework.stereotype.Service;
+
+@Service
+@RequiredArgsConstructor
+public class OsmPartitionDestinationProvider {
+    private final PropertiesConfiguration config;
+
+    public Destination getDestination() {
+         return Destination.builder()
+                    .partitionId(config.getGoogleCloudProject())
+                    .namespace(new Namespace(config.getPartitionNamespace()))
+                    .kind(new Kind(config.getPartitionPropertyKind()))
+                .build();
+    }
+}

provider/partition-gcp/src/main/java/org/opengroup/osdu/partition/provider/gcp/osm/config/resolver/OsmDatastoreDestinationResolver.java

+/*
+ * Copyright 2020-2021 Google LLC
+ * Copyright 2020-2021 EPAM Systems, Inc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.opengroup.osdu.partition.provider.gcp.osm.config.resolver;
+
+import com.google.api.gax.retrying.RetrySettings;
+import com.google.cloud.TransportOptions;
+import com.google.cloud.datastore.Datastore;
+import com.google.cloud.datastore.DatastoreOptions;
+import com.google.cloud.http.HttpTransportOptions;
+import lombok.RequiredArgsConstructor;
+import org.opengroup.osdu.core.gcp.osm.model.Destination;
+import org.opengroup.osdu.core.gcp.osm.translate.datastore.DsDestinationResolution;
+import org.opengroup.osdu.core.gcp.osm.translate.datastore.DsDestinationResolver;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.context.annotation.Scope;
+import org.springframework.stereotype.Component;
+import org.threeten.bp.Duration;
+
+import java.io.IOException;
+
+import static org.springframework.beans.factory.config.BeanDefinition.SCOPE_SINGLETON;
+
+@Component
+@Scope(SCOPE_SINGLETON)
+@ConditionalOnProperty(name = "osmDriver", havingValue = "datastore")
+@RequiredArgsConstructor
+public class OsmDatastoreDestinationResolver implements DsDestinationResolver {
+
+    protected static final RetrySettings RETRY_SETTINGS = RetrySettings.newBuilder().setMaxAttempts(6).setInitialRetryDelay(Duration.ofSeconds(10L)).setMaxRetryDelay(Duration.ofSeconds(32L)).setRetryDelayMultiplier(2.0D).setTotalTimeout(Duration.ofSeconds(50L)).setInitialRpcTimeout(Duration.ofSeconds(50L)).setRpcTimeoutMultiplier(1.0D).setMaxRpcTimeout(Duration.ofSeconds(50L)).build();
+    protected static final TransportOptions TRANSPORT_OPTIONS = HttpTransportOptions.newBuilder().setReadTimeout(30000).build();
+    private final Datastore partitionDatastore;
+
+    /**
+     * Takes provided Destination with partitionId set to needed tenantId, returns its TenantInfo.projectId.
+     *
+     * @param destination to resolve
+     * @return resolution results
+     */
+    @Override
+    public DsDestinationResolution resolve(Destination destination) {
+        String projectId = destination.getPartitionId();
+
+        Datastore datastore =
+                partitionDatastore == null
+                        ? DatastoreOptions.newBuilder()
+                                    .setRetrySettings(RETRY_SETTINGS)
+                                    .setTransportOptions(TRANSPORT_OPTIONS)
+                                    .setProjectId(projectId)
+                                    .setNamespace(destination.getNamespace().getName())
+                                .build()
+                            .getService()
+                        : partitionDatastore;
+
+        return DsDestinationResolution.builder()
+                .projectId(datastore.getOptions().getProjectId())
+                .datastore(datastore)
+                .build();
+    }
+
+    @Override
+    public void close() throws IOException {
+        //Method stub
+    }
+}
\ No newline at end of file

provider/partition-gcp/src/main/java/org/opengroup/osdu/partition/provider/gcp/osm/config/resolver/OsmPostgresDestinationResolver.java

+/*
+ * Copyright 2020-2021 Google LLC
+ * Copyright 2020-2021 EPAM Systems, Inc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.opengroup.osdu.partition.provider.gcp.osm.config.resolver;
+
+import com.zaxxer.hikari.HikariDataSource;
+import lombok.RequiredArgsConstructor;
+import lombok.extern.slf4j.Slf4j;
+import org.opengroup.osdu.core.gcp.osm.model.Destination;
+import org.opengroup.osdu.core.gcp.osm.translate.postgresql.PgDestinationResolution;
+import org.opengroup.osdu.core.gcp.osm.translate.postgresql.PgDestinationResolver;
+import org.opengroup.osdu.partition.provider.gcp.config.PostgresOsmConfiguration;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
+import org.springframework.boot.jdbc.DataSourceBuilder;
+import org.springframework.context.annotation.Scope;
+import org.springframework.stereotype.Component;
+
+import javax.annotation.PreDestroy;
+import javax.sql.DataSource;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.springframework.beans.factory.config.BeanDefinition.SCOPE_SINGLETON;
+
+
+@Component
+@Scope(SCOPE_SINGLETON)
+@ConditionalOnProperty(name = "osmDriver", havingValue = "postgres")
+@RequiredArgsConstructor
+@Slf4j
+public class OsmPostgresDestinationResolver implements PgDestinationResolver {
+
+    private final PostgresOsmConfiguration properties;
+
+    private static final String DRIVER_CLASS_NAME = "org.postgresql.Driver";
+
+    private final Map<String, DataSource> dataSourceCache = new HashMap<>();
+
+    @Override
+    public PgDestinationResolution resolve(Destination destination) {
+
+        String partitionId = destination.getPartitionId();
+
+        DataSource dataSource = dataSourceCache.get(partitionId);
+        if (dataSource == null || (dataSource instanceof HikariDataSource && ((HikariDataSource) dataSource).isClosed())) {
+            synchronized (dataSourceCache) {
+                dataSource = dataSourceCache.get(partitionId);
+                if (dataSource == null || (dataSource instanceof HikariDataSource && ((HikariDataSource) dataSource).isClosed())) {
+                    dataSource = buildDataSourceFromProperties(partitionId);
+                }
+            }
+        }
+
+        return PgDestinationResolution.builder()
+                .datasource(dataSource)
+                .build();
+    }
+
+    private DataSource buildDataSourceFromProperties(String partitionId) {
+        DataSource dataSource;
+        dataSource = DataSourceBuilder.create()
+                .driverClassName(DRIVER_CLASS_NAME)
+                .url(properties.getUrl())
+                .username(properties.getUsername())
+                .password(properties.getPassword())
+                .build();
+
+        HikariDataSource hikariDataSource = (HikariDataSource) dataSource;
+        hikariDataSource.setMaximumPoolSize(properties.getMaximumPoolSize());
+        hikariDataSource.setMinimumIdle(properties.getMinimumIdle());
+        hikariDataSource.setIdleTimeout(properties.getIdleTimeout());
+        hikariDataSource.setMaxLifetime(properties.getMaxLifetime());
+        hikariDataSource.setConnectionTimeout(properties.getConnectionTimeout());
+
+        dataSourceCache.put(partitionId, dataSource);
+        return dataSource;
+    }
+
+    @PreDestroy
+    @Override
+    public void close() {
+        log.info("On pre-destroy. {} DataSources to shutdown", dataSourceCache.size());
+        for (DataSource dataSource : dataSourceCache.values()) {
+            if (dataSource instanceof HikariDataSource && !((HikariDataSource) dataSource).isClosed()) {
+                ((HikariDataSource) dataSource).close();
+            }
+        }
+    }
+}

provider/partition-gcp/src/main/java/org/opengroup/osdu/partition/provider/gcp/osm/repository/OsmPartitionPropertyRepository.java

+/*
+ * Copyright 2020-2021 Google LLC
+ * Copyright 2020-2021 EPAM Systems, Inc
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */