Skip to content
Snippets Groups Projects
Commit dc9a9eb6 authored by Riabokon Stanislav(EPAM)[GCP]'s avatar Riabokon Stanislav(EPAM)[GCP]
Browse files

Removed gcp reference module (GONRG-4669)

parent 655d4b27
No related branches found
No related tags found
1 merge request!236Removed gcp reference module (GONRG-4669)
Showing
with 0 additions and 1329 deletions
......@@ -129,7 +129,6 @@
<module>provider/partition-aws</module>
<module>provider/partition-ibm</module>
<module>provider/partition-gcp</module>
<module>provider/partition-reference</module>
</modules>
<profiles>
......
FROM openjdk:8-slim
WORKDIR /app
ARG PROVIDER_NAME
ENV PROVIDER_NAME $PROVIDER_NAME
ARG PORT
ENV PORT $PORT
# Copy the jar to the production image from the builder stage.
COPY provider/partition-${PROVIDER_NAME}/target/partition-${PROVIDER_NAME}-*-spring-boot.jar partition-${PROVIDER_NAME}.jar
# Run the web service on container startup.
CMD java -Djava.security.egd=file:/dev/./urandom -Dserver.port=${PORT} -jar /app/partition-${PROVIDER_NAME}.jar
# Partition Service
os-partition-reference is a [Spring Boot](https://spring.io/projects/spring-boot) service that is responsible for creating and retrieving partition specific properties on behalf of other services whether they are secret values or not.
## Getting Started
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.
### Prerequisites
Pre-requisites
* GCloud SDK with java (latest version)
* JDK 8
* Lombok 1.16 or later
* Maven
### Installation
In order to run the service locally or remotely, you will need to have the following environment variables defined.
| name | value | description | sensitive? | source |
| --- | --- | --- | --- | --- |
| `LOG_PREFIX` | `service` | Logging prefix | no | - |
| `SERVER_SERVLET_CONTEXPATH` | `/api/partition/v1` | Servlet context path | no | - |
| `AUTHORIZE_API` | ex `https://entitlements.com/entitlements/v1` | Entitlements API endpoint | no | output of infrastructure deployment |
| `GOOGLE_CLOUD_PROJECT` | ex `osdu-cicd-epam` | Google Cloud Project Id| no | output of infrastructure deployment |
| `GOOGLE_AUDIENCES` | ex `*****.apps.googleusercontent.com` | Client ID for getting access to cloud resources | yes | https://console.cloud.google.com/apis/credentials |
| `PARTITION_ADMIN_ACCOUNTS` | ex `admin@domen.iam.gserviceaccount.com,osdu-gcp-sa,workload-identity` | List of partition admin account emails, could be in full form like `admin@domen.iam.gserviceaccount.com` or in `starts with` pattern like `osdu-gcp-sa`| no | - |
| `GOOGLE_APPLICATION_CREDENTIALS` | ex `/path/to/directory/service-key.json` | Service account credentials, you only need this if running locally | yes | https://console.cloud.google.com/iam-admin/serviceaccounts |
| `KEY_RING` | ex `csqp` | A key ring holds keys in a specific Google Cloud location and permit us to manage access control on groups of keys | yes | https://cloud.google.com/kms/docs/resource-hierarchy#key_rings |
| `KMS_KEY` | ex `partitionService` | A key exists on one key ring linked to a specific location. | yes | https://cloud.google.com/kms/docs/resource-hierarchy#key_rings |
| `MONGO_DB_URL` | ex `localhost:27017` | Mongo DB Url| yes | output of infrastructure deployment |
| `MONGO_DB_USER` | ex `mongouser` | Mongo DB userName| yes | output of infrastructure deployment |
| `MONGO_DB_PASSWORD` | ex `mongopassword` | Mongo DB userPassword. If password contains `@` symbols, it must be encoded to URL-encoded format. | yes | output of infrastructure deployment |
| `MONGO_DB_NAME` | ex `mongoDBName` | Mongo DB DbName| yes | output of infrastructure deployment |
### Run Locally
Check that maven is installed:
```bash
$ mvn --version
Apache Maven 3.6.0
Maven home: /usr/share/maven
Java version: 1.8.0_212, vendor: AdoptOpenJDK, runtime: /usr/lib/jvm/jdk8u212-b04/jre
...
```
You may need to configure access to the remote maven repository that holds the OSDU dependencies. This file should live within `~/.mvn/community-maven.settings.xml`:
```bash
$ cat ~/.m2/settings.xml
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
<servers>
<server>
<id>community-maven-via-private-token</id>
<!-- Treat this auth token like a password. Do not share it with anyone, including Microsoft support. -->
<!-- The generated token expires on or before 11/14/2019 -->
<configuration>
<httpHeaders>
<property>
<name>Private-Token</name>
<value>${env.COMMUNITY_MAVEN_TOKEN}</value>
</property>
</httpHeaders>
</configuration>
</server>
</servers>
</settings>
```
* Update the Google cloud SDK to the latest version:
```bash
gcloud components update
```
* Set Google Project Id:
```bash
gcloud config set project <YOUR-PROJECT-ID>
```
* Perform a basic authentication in the selected project:
```bash
gcloud auth application-default login
```
* Navigate to partition service's root folder and run:
```bash
mvn clean install
```
* If you wish to see the coverage report then go to target/site/jacoco/index.html and open index.html
* If you wish to build the project without running tests
```bash
mvn clean install -DskipTests
```
After configuring your environment as specified above, you can follow these steps to build and run the application. These steps should be invoked from the *repository root.*
```bash
cd provider/partition-reference/ && mvn spring-boot:run
```
## Testing
Navigate to partition service's root folder and run all the tests:
```bash
# build + install integration test core
$ (cd testing/partition-test-core/ && mvn clean install)
```
### Running E2E Tests
This section describes how to run cloud OSDU E2E tests (testing/partition-test-gcp).
You will need to have the following environment variables defined.
| name | value | description | sensitive? | source |
| --- | --- | --- | --- | --- |
| `ENVIRONMENT` | ex `dev` | | no | |
| `PARTITION_BASE_URL` | ex `http://localhost:8080/` | service base URL | yes | |
| `CLIENT_TENANT` | ex `opendes` | name of the client partition | yes | |
| `MY_TENANT` | ex `opendes` | name of the OSDU partition | yes | |
| `INTEGRATION_TESTER` | `********` | Service account for API calls. Note: this user must be `PARTITION_ADMIN_ACCOUNT` | yes | https://console.cloud.google.com/iam-admin/serviceaccounts |
| `NO_DATA_ACCESS_TESTER` | `********` | Service account base64 encoded string without data access | yes | https://console.cloud.google.com/iam-admin/serviceaccounts |
| `INTEGRATION_TEST_AUDIENCE` | `********` | client application ID | yes | https://console.cloud.google.com/apis/credentials |
Execute following command to build code and run all the integration tests:
```bash
# Note: this assumes that the environment variables for integration tests as outlined
# above are already exported in your environment.
$ (cd testing/partition-test-gcp/ && mvn clean test)
```
## Deployment
GKE Google Documentation: https://cloud.google.com/build/docs/deploying-builds/deploy-gke
Anthos Google Documentation: https://cloud.google.com/anthos/multicluster-management/gateway/tutorials/cloud-build-integration
#### Cloud KMS Setup
Enable cloud KMS on master project.
Create king ring and key in the ***master project***
```bash
gcloud services enable cloudkms.googleapis.com
export KEYRING_NAME="csqp"
export CRYPTOKEY_NAME="partionService"
gcloud kms keyrings create $KEYRING_NAME --location global
gcloud kms keys create $CRYPTOKEY_NAME --location global \
--keyring $KEYRING_NAME \
--purpose encryption
```
Add **Cloud KMS CryptoKey Encrypter/Decrypter** role to the used **service account** by Partition Service of the ***master project*** through IAM - Role tab.
Add "Cloud KMS Encrypt/Decrypt" role to the used **service account** by Partition Service of the ***master project*** through IAM - Role tab.
## Licence
Copyright © Google LLC
Copyright © EPAM Systems
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
version: "3"
services:
os-storage-app:
build:
args:
JAR_FILE: target/partition-reference-0.12.0-spring-boot.jar
context: ""
dockerfile: ../Dockerfile
image: us.gcr.io/osdu-anthos-02/os-partition/anthos-partition-reference
ports:
- "8080:8080"
apiVersion: v1
data:
AUTHORIZE_API: ${AUTHORIZE_API}
MONGO_DB_URL: ${MONGO_DB_URL}
MONGO_DB_USER: ${MONGO_DB_USER}
MONGO_DB_NAME: ${MONGO_DB_NAME}
GOOGLE_CLOUD_PROJECT: ${GOOGLE_CLOUD_PROJECT}
GOOGLE_AUDIENCES: ${GOOGLE_AUDIENCES}
KEY_RING: ${KEY_RING}
KMS_KEY: ${KMS_KEY}
PARTITION_ADMIN_ACCOUNTS: ${PARTITION_ADMIN_ACCOUNTS}
LOG_LEVEL: ${LOG_LEVEL}
kind: ConfigMap
metadata:
labels:
app: partition-reference
name: partition-config
namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
generateName: partition-reference-anthos
labels:
app: partition-reference
name: partition-reference
namespace: default
spec:
selector:
matchLabels:
app: partition-reference
replicas: 1
template:
metadata:
labels:
app: partition-reference
spec:
containers:
- env:
- name: AUTHORIZE_API
valueFrom:
configMapKeyRef:
key: AUTHORIZE_API
name: partition-config
- name: PARTITION_ADMIN_ACCOUNTS
valueFrom:
configMapKeyRef:
key: PARTITION_ADMIN_ACCOUNTS
name: partition-config
- name: LOG_LEVEL
valueFrom:
configMapKeyRef:
key: LOG_LEVEL
name: partition-config
- name: MONGO_DB_URL
valueFrom:
configMapKeyRef:
key: MONGO_DB_URL
name: partition-config
- name: MONGO_DB_USER
valueFrom:
configMapKeyRef:
key: MONGO_DB_USER
name: partition-config
- name: MONGO_DB_PASSWORD
valueFrom:
secretKeyRef:
name: partition-secret
key: mongo.db.password
- name: MONGO_DB_NAME
valueFrom:
configMapKeyRef:
key: MONGO_DB_NAME
name: partition-config
- name: GOOGLE_CLOUD_PROJECT
valueFrom:
configMapKeyRef:
key: GOOGLE_CLOUD_PROJECT
name: partition-config
- name: GOOGLE_AUDIENCES
valueFrom:
configMapKeyRef:
key: GOOGLE_AUDIENCES
name: partition-config
- name: KEY_RING
valueFrom:
configMapKeyRef:
key: KEY_RING
name: partition-config
- name: KMS_KEY
valueFrom:
configMapKeyRef:
key: KMS_KEY
name: partition-config
image: us.gcr.io/osdu-anthos-02/os-partition/anthos-partition-reference:9966597-dirty
name: partition-reference
---
apiVersion: v1
kind: Service
metadata:
name: partition-reference
namespace: default
spec:
ports:
- protocol: TCP
port: 80
targetPort: 8080
selector:
app: partition-reference
type: LoadBalancer
---
apiVersion: v1
data:
mongo.db.password: ${mongo.db.password}
kind: Secret
metadata:
name: partition-secret
namespace: default
type: Opaque
\ No newline at end of file
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.opengroup.osdu</groupId>
<artifactId>partition</artifactId>
<version>0.16.0-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>
<groupId>org.opengroup.osdu</groupId>
<artifactId>partition-reference</artifactId>
<description>Partition service on Hybrid Cloud</description>
<version>0.16.0-SNAPSHOT</version>
<packaging>jar</packaging>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.13.2.2</version>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<!-- Internal packages -->
<dependency>
<groupId>org.opengroup.osdu</groupId>
<artifactId>partition-core</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.opengroup.osdu</groupId>
<artifactId>core-lib-gcp</artifactId>
<version>0.10.0</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</dependency>
<dependency>
<groupId>ch.qos.logback.contrib</groupId>
<artifactId>logback-json-classic</artifactId>
<version>0.1.5</version>
</dependency>
<!-- Test Dependencies -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.powermock</groupId>
<artifactId>powermock-api-mockito2</artifactId>
<version>2.0.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.powermock</groupId>
<artifactId>powermock-module-junit4</artifactId>
<version>2.0.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>3.0.0</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
<configuration>
<classifier>spring-boot</classifier>
<mainClass>
org.opengroup.osdu.partition.provider.reference.PartitionReferenceApplication
</mainClass>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.7.7.201606060606</version>
<executions>
<execution>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<execution>
<id>report</id>
<phase>prepare-package</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
/*
Copyright 2002-2021 Google LLC
Copyright 2002-2021 EPAM Systems, Inc
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package org.opengroup.osdu.partition.provider.reference;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.mongo.MongoAutoConfiguration;
import org.springframework.context.annotation.ComponentScan;
@ComponentScan({"org.opengroup.osdu"})
@SpringBootApplication(exclude = {MongoAutoConfiguration.class})
public class PartitionReferenceApplication {
public static void main(String[] args) {
SpringApplication.run(PartitionReferenceApplication.class, args);
}
}
/*
Copyright 2002-2021 Google LLC
Copyright 2002-2021 EPAM Systems, Inc
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package org.opengroup.osdu.partition.provider.reference.cache;
import org.opengroup.osdu.core.common.cache.VmCache;
import org.opengroup.osdu.core.gcp.multitenancy.credentials.GcsCredential;
import org.springframework.stereotype.Component;
@Component
public class CredentialsCache extends VmCache<String, GcsCredential> {
public CredentialsCache() {
super(30, 1000);
}
}
/*
Copyright 2002-2021 Google LLC
Copyright 2002-2021 EPAM Systems, Inc
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package org.opengroup.osdu.partition.provider.reference.cache;
import org.opengroup.osdu.core.common.cache.VmCache;
import org.opengroup.osdu.core.common.model.entitlements.Groups;
import org.springframework.stereotype.Component;
@Component
public class GroupCache extends VmCache<String, Groups> {
public GroupCache() {
super(30, 1000);
}
}
/*
Copyright 2002-2021 Google LLC
Copyright 2002-2021 EPAM Systems, Inc
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package org.opengroup.osdu.partition.provider.reference.cache;
import java.util.List;
import lombok.RequiredArgsConstructor;
import org.opengroup.osdu.core.common.cache.VmCache;
import org.opengroup.osdu.partition.model.PartitionInfo;
import org.opengroup.osdu.partition.provider.reference.config.PropertiesConfiguration;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
@RequiredArgsConstructor
public class VmCacheConfiguration {
private final PropertiesConfiguration properties;
@Bean(name = "partitionListCache")
public VmCache<String, List<String>> partitionListCache() {
return new VmCache<>(this.properties.getCacheExpiration() * 60,
this.properties.getCacheMaxSize());
}
@ConfigurationProperties
@Bean(name = "partitionServiceCache")
public VmCache<String, PartitionInfo> partitionServiceCache() {
return new VmCache<>(this.properties.getCacheExpiration() * 60,
this.properties.getCacheMaxSize());
}
}
/*
* Copyright 2021 Google LLC
* Copyright 2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.partition.provider.reference.config;
import com.google.api.client.googleapis.auth.oauth2.GoogleIdTokenVerifier;
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.json.jackson2.JacksonFactory;
import java.io.IOException;
import java.security.GeneralSecurityException;
import java.util.Collections;
import lombok.RequiredArgsConstructor;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
@RequiredArgsConstructor
public class IDTokenVerifierConfiguration {
private final PropertiesConfiguration configuration;
@Bean
public GoogleIdTokenVerifier buildTokenVerifier() throws GeneralSecurityException, IOException {
return new GoogleIdTokenVerifier.Builder(
GoogleNetHttpTransport.newTrustedTransport(),
JacksonFactory.getDefaultInstance())
.setAudience(Collections.singleton(configuration.getGoogleAudiences()))
.build();
}
}
/*
* Copyright 2021 Google LLC
* Copyright 2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.partition.provider.reference.config;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;
@Configuration
@ConfigurationProperties
@Data
public class MongoDBConfigProperties {
private String mongoDbUrl;
private String mongoDbUser;
private String mongoDbPassword;
private String mongoDbName;
}
/*
Copyright 2002-2021 Google LLC
Copyright 2002-2021 EPAM Systems, Inc
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package org.opengroup.osdu.partition.provider.reference.config;
import java.util.List;
import java.util.Objects;
import javax.annotation.PostConstruct;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;
@Configuration
@ConfigurationProperties
@Data
public class PropertiesConfiguration {
private String googleAudiences;
private List<String> partitionAdminAccounts;
private String googleCloudProject;
private int cacheExpiration;
private int cacheMaxSize;
private String serviceAccountTail;
@PostConstruct
public void setUp() {
if (Objects.isNull(serviceAccountTail) || serviceAccountTail.isEmpty()) {
this.serviceAccountTail = googleCloudProject + ".iam.gserviceaccount.com";
}
}
}
/*
* Copyright 2021 Google LLC
* Copyright 2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.partition.provider.reference.model;
import java.util.HashMap;
import java.util.Map;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import org.opengroup.osdu.partition.model.Property;
import org.springframework.data.mongodb.core.mapping.Document;
@Data
@AllArgsConstructor
@NoArgsConstructor
@Document(collection = "Partitions")
public class PartitionPropertyDocument {
private String partitionId;
private Map<String, Property> properties = new HashMap<>();
}
\ No newline at end of file
/*
* Copyright 2021 Google LLC
* Copyright 2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.partition.provider.reference.persistence;
import com.mongodb.client.MongoCollection;
import org.bson.Document;
import org.opengroup.osdu.partition.provider.reference.util.MongoClientHandler;
import org.springframework.stereotype.Component;
@Component
public class MongoDdmsClient {
private MongoClientHandler mongoClientHandler;
public MongoDdmsClient(MongoClientHandler mongoClientHandler) {
this.mongoClientHandler = mongoClientHandler;
}
public MongoCollection<Document> getMongoCollection(String dbName, String collectionName) {
return mongoClientHandler.getMongoClient().getDatabase(dbName)
.getCollection(collectionName);
}
}
/*
Copyright 2002-2021 Google LLC
Copyright 2002-2021 EPAM Systems, Inc
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package org.opengroup.osdu.partition.provider.reference.repository;
import static com.mongodb.client.model.Filters.eq;
import com.google.gson.Gson;
import com.mongodb.client.FindIterable;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.model.ReplaceOptions;
import com.mongodb.client.result.DeleteResult;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
import java.util.Optional;
import org.bson.Document;
import org.opengroup.osdu.partition.model.PartitionInfo;
import org.opengroup.osdu.partition.model.Property;
import org.opengroup.osdu.partition.provider.reference.model.PartitionPropertyDocument;
import org.opengroup.osdu.partition.provider.reference.persistence.MongoDdmsClient;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Repository;
@Repository
public class PartitionPropertyEntityRepository {
protected static final String PARTITIONS = "Partitions";
protected static final String PARTITIONS_DATABASE = "PartitionsDB";
protected static final String PARTITION_ID = "partitionId";
private final MongoDdmsClient mongoDdmsClient;
@Autowired
public PartitionPropertyEntityRepository(MongoDdmsClient mongoDdmsClient) {
this.mongoDdmsClient = mongoDdmsClient;
}
public Optional<PartitionInfo> createPartition(String partitionId, PartitionInfo partitionInfo) {
if (Objects.nonNull(partitionInfo) && Objects.nonNull(partitionId)) {
MongoCollection<Document> mongoCollection = mongoDdmsClient
.getMongoCollection(PARTITIONS_DATABASE, PARTITIONS);
PartitionPropertyDocument partitionPropertyDocument = convertToPartitionPropertyDocument(
partitionId, partitionInfo);
mongoCollection.replaceOne(eq(PARTITION_ID, partitionId),
Document.parse(new Gson().toJson(partitionPropertyDocument)),
new ReplaceOptions().upsert(true));
return Optional.ofNullable(partitionInfo);
}
return Optional.empty();
}
public Optional<PartitionInfo> updatePartition(String partitionId, PartitionInfo partitionInfo) {
if (Objects.nonNull(partitionInfo) && Objects.nonNull(partitionId)) {
MongoCollection<Document> mongoCollection = mongoDdmsClient
.getMongoCollection(PARTITIONS_DATABASE, PARTITIONS);
Document result = mongoCollection.find(eq(PARTITION_ID, partitionId)).first();
PartitionPropertyDocument partitionPropertyDocument = new Gson().fromJson(result.toJson(),
PartitionPropertyDocument.class);
partitionPropertyDocument.getProperties().putAll(partitionInfo.getProperties());
mongoCollection.replaceOne(eq(PARTITION_ID, partitionId),
Document.parse(new Gson().toJson(partitionPropertyDocument)),
new ReplaceOptions().upsert(true));
return Optional.ofNullable(partitionInfo);
}
return Optional.empty();
}
public Optional<PartitionInfo> findByPartitionId(String partitionId) {
if (Objects.nonNull(partitionId)) {
MongoCollection<Document> mongoCollection = mongoDdmsClient
.getMongoCollection(PARTITIONS_DATABASE, PARTITIONS);
Document result = mongoCollection.find(eq(PARTITION_ID, partitionId)).first();
if (Objects.nonNull(result)) {
PartitionPropertyDocument partitionPropertyDocument = new Gson().fromJson(result.toJson(),
PartitionPropertyDocument.class);
PartitionInfo partitionInfo = convertToPartitionInfo(partitionPropertyDocument);
return Optional.ofNullable(partitionInfo);
}
}
return Optional.empty();
}
public List<String> getAllPartitions() {
MongoCollection<Document> mongoCollection = mongoDdmsClient
.getMongoCollection(PARTITIONS_DATABASE, PARTITIONS);
FindIterable<Document> results = mongoCollection.find();
List<String> partitionsIds = new ArrayList<>();
for (Document document : results) {
PartitionPropertyDocument partitionPropertyDocument = new Gson().fromJson(document.toJson(),
PartitionPropertyDocument.class);
partitionsIds.add(partitionPropertyDocument.getPartitionId());
}
return partitionsIds;
}
public boolean isDeletedPartitionInfoByPartitionId(String partitionId) {
if (Objects.nonNull(partitionId)) {
MongoCollection<Document> mongoCollection = mongoDdmsClient
.getMongoCollection(PARTITIONS_DATABASE, PARTITIONS);
DeleteResult result = mongoCollection.deleteMany(eq(PARTITION_ID, partitionId));
return result.wasAcknowledged();
}
return false;
}
public Optional<Property> findByPartitionIdAndName(String partitionId, String key) {
if (Objects.nonNull(partitionId) && Objects.nonNull(key)) {
MongoCollection<Document> mongoCollection = mongoDdmsClient
.getMongoCollection(PARTITIONS_DATABASE, PARTITIONS);
Document result = mongoCollection.find(eq(PARTITION_ID, partitionId)).first();
if (Objects.nonNull(result)) {
PartitionPropertyDocument partitionPropertyDocument = new Gson().fromJson(result.toJson(),
PartitionPropertyDocument.class);
Property property = partitionPropertyDocument.getProperties().get(key);
if (Objects.nonNull(property)) {
return Optional.ofNullable(property);
}
}
}
return Optional.empty();
}
private PartitionPropertyDocument convertToPartitionPropertyDocument(
String partitionId, PartitionInfo partitionInfo) {
PartitionPropertyDocument partitionPropertyDocument = new PartitionPropertyDocument();
partitionPropertyDocument.setPartitionId(partitionId);
partitionPropertyDocument.setProperties(partitionInfo.getProperties());
return partitionPropertyDocument;
}
private PartitionInfo convertToPartitionInfo(
PartitionPropertyDocument partitionPropertyDocument) {
PartitionInfo partitionInfo = new PartitionInfo();
partitionInfo.setProperties(partitionPropertyDocument.getProperties());
return partitionInfo;
}
}
/*
Copyright 2002-2021 Google LLC
Copyright 2002-2021 EPAM Systems, Inc
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package org.opengroup.osdu.partition.provider.reference.security;
import com.google.api.client.googleapis.auth.oauth2.GoogleIdToken;
import com.google.api.client.googleapis.auth.oauth2.GoogleIdTokenVerifier;
import java.util.List;
import java.util.Objects;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.opengroup.osdu.core.common.model.http.AppException;
import org.opengroup.osdu.core.common.model.http.DpsHeaders;
import org.opengroup.osdu.partition.provider.interfaces.IAuthorizationService;
import org.opengroup.osdu.partition.provider.reference.config.PropertiesConfiguration;
import org.springframework.stereotype.Component;
import org.springframework.web.context.annotation.RequestScope;
@Slf4j
@Component
@RequestScope
@RequiredArgsConstructor
public class AuthorizationService implements IAuthorizationService {
private final PropertiesConfiguration configuration;
private final DpsHeaders headers;
private final GoogleIdTokenVerifier verifier;
@Override
public boolean isDomainAdminServiceAccount() {
if (Objects.isNull(headers.getAuthorization()) || headers.getAuthorization().isEmpty()) {
throw AppException.createUnauthorized("No JWT token. Access is Forbidden");
}
String email = null;
try {
String authorization = headers.getAuthorization().replace("Bearer ", "");
GoogleIdToken googleIdToken = verifier.verify(authorization);
if (Objects.isNull(googleIdToken)) {
log.warn("Not valid token provided");
throw AppException.createUnauthorized("Unauthorized. The JWT token could not be validated");
}
email = googleIdToken.getPayload().getEmail();
List<String> partitionAdminAccounts = configuration.getPartitionAdminAccounts();
if (Objects.nonNull(partitionAdminAccounts) && !partitionAdminAccounts.isEmpty()) {
return isAllowedAccount(email);
} else {
if (StringUtils.endsWith(email, configuration.getServiceAccountTail())) {
return true;
} else {
throw AppException.createUnauthorized(
String.format("Unauthorized. The user %s is not Service Principal", email));
}
}
} catch (AppException e){
throw e;
} catch (Exception ex) {
log.warn(String.format("User %s is not unauthorized. %s.", email, ex));
throw AppException.createUnauthorized("Unauthorized. The JWT token could not be validated");
}
}
private boolean isAllowedAccount(String accountEmail) {
if (StringUtils.endsWith(accountEmail, configuration.getServiceAccountTail())) {
for (String partitionAdmin : configuration.getPartitionAdminAccounts()) {
if (partitionAdmin.equals(accountEmail)) {
return true;
}
if (StringUtils.startsWith(accountEmail, partitionAdmin)) {
return true;
}
}
}
throw AppException
.createUnauthorized(String.format("Unauthorized. The user %s is untrusted.", accountEmail));
}
}
/*
Copyright 2002-2021 Google LLC
Copyright 2002-2021 EPAM Systems, Inc
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package org.opengroup.osdu.partition.provider.reference.security;
import org.springframework.context.annotation.Configuration;
import org.springframework.security.config.annotation.method.configuration.EnableGlobalMethodSecurity;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.builders.WebSecurity;
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter;
@EnableWebSecurity
@Configuration
@EnableGlobalMethodSecurity(prePostEnabled = true)
public class SecurityConfigReference extends WebSecurityConfigurerAdapter {
@Override
protected void configure(HttpSecurity http) throws Exception {
http.httpBasic().disable()
.csrf().disable();
}
@Override
public void configure(WebSecurity web) throws Exception {
web.ignoring().antMatchers("/api-docs")
.antMatchers("/swagger");
}
}
/*
Copyright 2002-2021 Google LLC
Copyright 2002-2021 EPAM Systems, Inc
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package org.opengroup.osdu.partition.provider.reference.service;
import lombok.RequiredArgsConstructor;
import org.apache.http.HttpStatus;
import org.opengroup.osdu.core.common.cache.ICache;
import org.opengroup.osdu.core.common.model.http.AppException;
import org.opengroup.osdu.core.common.provider.interfaces.IKmsClient;
import org.opengroup.osdu.partition.logging.AuditLogger;
import org.opengroup.osdu.partition.model.PartitionInfo;
import org.opengroup.osdu.partition.model.Property;
import org.opengroup.osdu.partition.provider.interfaces.IPartitionService;
import org.opengroup.osdu.partition.provider.reference.repository.PartitionPropertyEntityRepository;
import org.springframework.stereotype.Service;
import java.util.Collections;
import java.util.List;
import java.util.Optional;
@Service
@RequiredArgsConstructor
public class PartitionServiceImpl implements IPartitionService {
private static final String UNKNOWN_ERROR_REASON = "unknown error";
private static final String PARTITION_LIST_KEY = "getAllPartitions";
private final PartitionPropertyEntityRepository partitionPropertyEntityRepository;
private final IKmsClient kmsClient;
private final AuditLogger auditLogger;
private final ICache<String, PartitionInfo> partitionServiceCache;
private final ICache<String, List<String>> partitionListCache;
@Override
public PartitionInfo createPartition(String partitionId, PartitionInfo partitionInfo) {
if (partitionServiceCache.get(partitionId) != null)
throw new AppException(HttpStatus.SC_CONFLICT, "partition exist", "Partition with same id exist");
if (partitionPropertyEntityRepository.findByPartitionId(partitionId).isPresent()) {
throw new AppException(HttpStatus.SC_CONFLICT, UNKNOWN_ERROR_REASON,
"Partition already exists.");
}
partitionInfo.getProperties()
.forEach((key, property) -> encryptPartitionPropertyEntityIfNeeded(property));
partitionPropertyEntityRepository.createPartition(partitionId, partitionInfo);
PartitionInfo pi = getPartition(partitionId);
if (pi != null) {
partitionServiceCache.put(partitionId, pi);
partitionListCache.clearAll();
}
return pi;
}
@Override
public PartitionInfo updatePartition(String partitionId, PartitionInfo partitionInfo) {
if (partitionInfo.getProperties().containsKey("id")) {
this.auditLogger.updatePartitionSecretFailure(Collections.singletonList(partitionId));
throw new AppException(HttpStatus.SC_BAD_REQUEST, "can not update id",
"the field id can not be updated");
}
if (!partitionPropertyEntityRepository.findByPartitionId(partitionId).isPresent()) {
this.auditLogger.updatePartitionSecretFailure(Collections.singletonList(partitionId));
throw new AppException(HttpStatus.SC_NOT_FOUND, UNKNOWN_ERROR_REASON,
"An attempt to update not existing partition.");
}
partitionInfo.getProperties().forEach((key, value) -> {
Optional<Property> property = this.partitionPropertyEntityRepository
.findByPartitionIdAndName(partitionId, key);
if (property.isPresent()) {
property.get().setSensitive(value.isSensitive());
property.get().setValue(value.getValue());
encryptPartitionPropertyEntityIfNeeded(property.get());
}
encryptPartitionPropertyEntityIfNeeded(value);
});
partitionPropertyEntityRepository.updatePartition(partitionId, partitionInfo);
PartitionInfo pi = getPartition(partitionId);
if(pi != null) {
partitionServiceCache.put(partitionId, pi);
}
return pi;
}
@Override
public PartitionInfo getPartition(String partitionId) {
PartitionInfo pi = (PartitionInfo) partitionServiceCache.get(partitionId);
if (pi == null) {
Optional<PartitionInfo> result = partitionPropertyEntityRepository.findByPartitionId(
partitionId);
if (!result.isPresent()) {
throw new AppException(HttpStatus.SC_NOT_FOUND, UNKNOWN_ERROR_REASON, "Partition does not exist.");
}
result.get().getProperties()
.forEach((key, property) -> decryptPartitionPropertyIfNeeded(property));
pi = result.get();
partitionServiceCache.put(partitionId, pi);
}
return pi;
}
@Override
public boolean deletePartition(String partitionId) {
if (!partitionPropertyEntityRepository.findByPartitionId(partitionId).isPresent()) {
this.auditLogger.deletePartitionFailure(Collections.singletonList(partitionId));
throw new AppException(HttpStatus.SC_NOT_FOUND, UNKNOWN_ERROR_REASON,
"An attempt to delete not existing partition.");
}
if (partitionPropertyEntityRepository.isDeletedPartitionInfoByPartitionId(partitionId)) {
if (partitionServiceCache.get(partitionId) != null) {
partitionServiceCache.delete(partitionId);
}
partitionListCache.clearAll();
return true;
}
return false;
}
@Override
public List<String> getAllPartitions() {
List<String> partitions = (List<String>)partitionListCache.get(PARTITION_LIST_KEY);
if (partitions == null) {
partitions = partitionPropertyEntityRepository.getAllPartitions();
if (partitions != null) {
partitionListCache.put(PARTITION_LIST_KEY, partitions);
}
}
return partitions;
}
private void encryptPartitionPropertyEntityIfNeeded(Property property) {
if (property.isSensitive()) {
String propertyValue = property.getValue().toString();
property.setValue("this.kmsClient.encryptString " + propertyValue);
}
}
private void decryptPartitionPropertyIfNeeded(Property property) {
if (property.isSensitive()) {
String propertyValue = property.getValue().toString();
property.setValue("this.kmsClient.decryptString " + propertyValue);
}
}
}
/*
* Copyright 2021 Google LLC
* Copyright 2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.partition.provider.reference.util;
import com.mongodb.ConnectionString;
import com.mongodb.MongoClientSettings;
import com.mongodb.client.MongoClient;
import com.mongodb.client.MongoClients;
import org.apache.http.HttpStatus;
import org.opengroup.osdu.core.common.model.http.AppException;
import org.opengroup.osdu.partition.provider.reference.config.MongoDBConfigProperties;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
@Component
public class MongoClientHandler {
private static final String MONGO_PREFIX = "mongodb://";
private static final String MONGO_OPTIONS = "retryWrites=true&w=majority&maxIdleTimeMS=10000";
private MongoClient mongoClient = null;
private MongoDBConfigProperties mongoDBConfigProperties;
private MongoClient getOrInitMongoClient() {
if (mongoClient != null) {
return mongoClient;
}
final String connectionString = String.format("%s%s:%s@%s/?%s",
MONGO_PREFIX,
mongoDBConfigProperties.getMongoDbUser(),
mongoDBConfigProperties.getMongoDbPassword(),
mongoDBConfigProperties.getMongoDbUrl(),
MONGO_OPTIONS);
ConnectionString connString = new ConnectionString(connectionString);
MongoClientSettings settings = MongoClientSettings.builder()
.applyConnectionString(connString)
.retryWrites(true)
.build();
try {
mongoClient = MongoClients.create(settings);
} catch (Exception ex) {
throw new AppException(HttpStatus.SC_INTERNAL_SERVER_ERROR, "Error connecting MongoDB",
ex.getMessage(), ex);
}
return mongoClient;
}
public MongoClient getMongoClient() {
if (mongoClient == null) {
getOrInitMongoClient();
}
return mongoClient;
}
@Autowired
public void setMongoDBConfigProperties(MongoDBConfigProperties mongoDBConfigProperties) {
this.mongoDBConfigProperties = mongoDBConfigProperties;
}
}
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment