Skip to content
Snippets Groups Projects
Commit 9b08b26a authored by Riabokon Stanislav(EPAM)[GCP]'s avatar Riabokon Stanislav(EPAM)[GCP]
Browse files

Merge branch 'gcp-reference-remove' into 'master'

Removed reference gcp module (GONRG-4669)

See merge request osdu/platform/system/schema-service!343
parents 1d8c25e7 1c31709a
Branches
Tags
1 merge request!343Removed reference gcp module (GONRG-4669)
Pipeline #124744 passed
Showing
with 0 additions and 1208 deletions
......@@ -60,7 +60,6 @@
<module>provider/schema-gcp</module>
<module>provider/schema-ibm</module>
<module>provider/schema-azure</module>
<module>provider/schema-reference</module>
</modules>
<repositories>
......
# Schema Service
schema-reference is a Maven multi-module project service.
## Getting Started
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.
### Prerequisites (Infra and access required)
Pre-requisites
* GCloud SDK with java (latest version)
* JDK 8
* Lombok 1.16 or later
* Maven
### Installation
In order to run the service locally or remotely, you will need to have the following environment variables defined.
| name | value | description | sensitive? | source |
| --- | --- | --- | --- | --- |
| `MINIO_ENDPOINT_URL` | `http://127.0.0.1:9000` | MinIo url| no | - |
| `MINIO_ACCESS_KEY` | ex `adminadmin` | MinIo access key | yes | - |
| `MINIO_SECRET_KEY` | ex `adminadmin` | MinIo secret key | yes | - |
| `MONGO_DB_URL` | ex `127.0.0.1:27017` | Shared account id | no | - |
| `MONGO_DB_USER` | ex `admin` | MongoDB user | yes | - |
| `MONGO_DB_PASSWORD` | ex `admin` | MongoDB password | yes | - |
| `MB_RABBITMQ_URI` | ex `amqp://guest:guest@127.0.0.1:5672/%2F` | RabbitMq uri | yes | - |
| `STATUS_QUEUE_NAME` | ex `schema-changed` | RabbitMq queue name | no | - |
| `AUTHORIZE_API` | `https://dev.osdu-gcp.go3-nrg.projects.epam.com/api/entitlements/v2/` | Entitlements API | no | - |
### Run Locally
Check that maven is installed:
```bash
$ mvn --version
Apache Maven 3.6.0
Maven home: /usr/share/maven
Java version: 1.8.0_212, vendor: AdoptOpenJDK, runtime: /usr/lib/jvm/jdk8u212-b04/jre
...
```
You may need to configure access to the remote maven repository that holds the OSDU dependencies. This file should live within `~/.mvn/community-maven.settings.xml`:
```bash
$ cat ~/.m2/settings.xml
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
<servers>
<server>
<id>community-maven-via-private-token</id>
<!-- Treat this auth token like a password. Do not share it with anyone, including Microsoft support. -->
<!-- The generated token expires on or before 11/14/2019 -->
<configuration>
<httpHeaders>
<property>
<name>Private-Token</name>
<value>${env.COMMUNITY_MAVEN_TOKEN}</value>
</property>
</httpHeaders>
</configuration>
</server>
</servers>
</settings>
```
* Update the Google cloud SDK to the latest version:
```bash
gcloud components update
```
* Set Google Project Id:
```bash
gcloud config set project <YOUR-PROJECT-ID>
```
* Perform a basic authentication in the selected project:
```bash
gcloud auth application-default login
```
Once the above Prerequisite are done, we can follow the below steps to run the service locally,
1. Navigate to the root of the schema project, os-schema. For building the project using command line, run below command :
```bash
mvn clean install
```
This will build the core project as well as all the underlying projects. If we want to build projects for specific cloud vendor, we can use mvn --projects command. For example, if we want to build only for reference, we can use below command :
```bash
mvn --projects schema-core,provider/schema-reference clean install
```
2. Run schema service in command line. We need to select which cloud vendor specific schema-service we want to run. For example, if we want to run schema-service for reference, run the below command :
```bash
# Running reference :
java -jar provider\schema-reference\target\os-schema-reference-0.0.1-spring-boot.jar
3. The port and path for the service endpoint can be configured in ```application.properties``` in the provider folder as following. If not specified, then the web container (ex. Tomcat) default is used:
```bash
server.servlet.contextPath=/api/schema-service/v1/
server.port=8080
You can access the service APIs by following the service contract in [schema.yaml](docs/api/schema.yaml)
## Testing
### Running E2E Tests
#TBD
## Deployment
Schema Service is compatible with Cloud Run.
* To deploy into Cloud run, please, use this documentation:
https://cloud.google.com/run/docs/quickstarts/build-and-deploy
## License
Copyright © Google LLC
Copyright © EPAM Systems
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
\ No newline at end of file
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Copyright 2021 Google LLC
~ Copyright 2021 EPAM Systems, Inc
~
~ Licensed under the Apache License, Version 2.0 (the "License");
~ you may not use this file except in compliance with the License.
~ You may obtain a copy of the License at
~
~ https://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<artifactId>os-schema</artifactId>
<groupId>org.opengroup.osdu</groupId>
<version>0.16.0-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>
<artifactId>schema-reference</artifactId>
<description>Schema Service GCP Anthos</description>
<packaging>jar</packaging>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.opengroup.osdu</groupId>
<artifactId>os-core-common</artifactId>
<version>${os-core-common.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.opengroup.osdu</groupId>
<artifactId>os-schema-core</artifactId>
<version>0.16.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongo-java-driver</artifactId>
<version>3.12.10</version>
</dependency>
<dependency>
<groupId>io.minio</groupId>
<artifactId>minio</artifactId>
<version>8.0.0</version>
</dependency>
<dependency>
<groupId>com.rabbitmq</groupId>
<artifactId>amqp-client</artifactId>
<version>5.13.1</version>
</dependency>
<dependency>
<groupId>ch.qos.logback.contrib</groupId>
<artifactId>logback-json-classic</artifactId>
<version>0.1.5</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
<configuration>
<classifier>spring-boot</classifier>
<mainClass>org.opengroup.osdu.schema.ApplicationReference</mainClass>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
/*
* Copyright 2021 Google LLC
* Copyright 2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.schema;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.ComponentScan;
@SpringBootApplication
@ComponentScan({"org.opengroup.osdu"})
@EnableConfigurationProperties
public class ApplicationReference {
public static void main(String[] args) {
SpringApplication.run(ApplicationReference.class);
}
}
/*
* Copyright 2021 Google LLC
* Copyright 2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.schema.config;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;
@Data
@Configuration
@ConfigurationProperties
public class MinIoConfigProperties {
private String minIoEndpointUrl;
private String minIoAccessKey;
private String minIoSecretKey;
}
/*
* Copyright 2021 Google LLC
* Copyright 2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.schema.config;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;
@Data
@Configuration
@ConfigurationProperties
public class MongoDBConfigProperties {
private String mongoDbUrl;
private String mongoDbUser;
private String mongoDbPassword;
private String mongoDbName;
}
/*
* Copyright 2021 Google LLC
* Copyright 2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.schema.config;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;
@Data
@Configuration
@ConfigurationProperties
public class RabbitMqConfigProperties {
private String mbRabbitMqUri;
private String statusQueueName;
}
/*
* Copyright 2021 Google LLC
* Copyright 2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.schema.constants;
public class SchemaAnthosConstants {
public static final String SCHEMA_DATABASE = "schema-db";
public static final String AUTHORITY_COLLECTION = "authority-collection";
public static final String ENTITY_TYPE_COLLECTION = "entitytype-collection";
public static final String SCHEMA_COLLECTION = "schema-collection";
public static final String SOURCE_COLLECTION = "source-collection";
public static final String MINIO_SCHEMA_BUCKET = "schema-bucket";
public static final String SCHEMA_IDENTITY = "schemaIdentity.";
private SchemaAnthosConstants() {
}
}
/*
* Copyright 2021 Google LLC
* Copyright 2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.schema.factory;
import io.minio.MinioClient;
import javax.annotation.PostConstruct;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.opengroup.osdu.schema.config.MinIoConfigProperties;
import org.springframework.context.annotation.Lazy;
import org.springframework.stereotype.Component;
@Lazy
@Slf4j
@Component
@RequiredArgsConstructor
public class CloudObjectStorageFactory {
private final MinIoConfigProperties minIoConfigProperties;
private MinioClient minioClient;
@PostConstruct
public void init() {
minioClient = MinioClient.builder()
.endpoint(minIoConfigProperties.getMinIoEndpointUrl())
.credentials(minIoConfigProperties.getMinIoAccessKey(), minIoConfigProperties.getMinIoSecretKey())
.build();
log.info("Minio client initialized");
}
public MinioClient getClient() {
return this.minioClient;
}
}
/*
* Copyright 2021 Google LLC
* Copyright 2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.schema.factory;
import com.mongodb.ConnectionString;
import com.mongodb.MongoClientSettings;
import com.mongodb.client.MongoClient;
import com.mongodb.client.MongoClients;
import java.util.Objects;
import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.opengroup.osdu.schema.config.MongoDBConfigProperties;
import org.springframework.context.annotation.Lazy;
import org.springframework.stereotype.Component;
@Lazy
@Slf4j
@Component
@RequiredArgsConstructor
public class MongoClientFactory {
private static final String MONGO_PREFIX = "mongodb://";
private static final String MONGO_OPTIONS = "retryWrites=true&w=majority&maxIdleTimeMS=10000";
private final MongoDBConfigProperties mongoDBConfigProperties;
private MongoClient mongoClient;
@PostConstruct
private MongoClient init() {
String connectionString = String.format("%s%s:%s@%s/?%s",
MONGO_PREFIX,
mongoDBConfigProperties.getMongoDbUser(),
mongoDBConfigProperties.getMongoDbPassword(),
mongoDBConfigProperties.getMongoDbUrl(),
MONGO_OPTIONS);
ConnectionString connString = new ConnectionString(connectionString);
MongoClientSettings settings = MongoClientSettings.builder()
.applyConnectionString(connString)
.retryWrites(true)
.build();
try {
mongoClient = MongoClients.create(settings);
} catch (Exception ex) {
log.error("Error connecting MongoDB", ex);
}
return mongoClient;
}
public MongoClient getMongoClient() {
return mongoClient;
}
@PreDestroy
public void destroy() {
if (Objects.nonNull(mongoClient)) {
mongoClient.close();
}
}
}
/*
* Copyright 2021 Google LLC
* Copyright 2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.schema.factory;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import java.io.IOException;
import java.net.URISyntaxException;
import java.security.KeyManagementException;
import java.security.NoSuchAlgorithmException;
import java.util.Objects;
import java.util.concurrent.TimeoutException;
import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.opengroup.osdu.schema.config.RabbitMqConfigProperties;
import org.springframework.context.annotation.Lazy;
import org.springframework.stereotype.Component;
@Lazy
@Slf4j
@Component
@RequiredArgsConstructor
public class RabbitMQFactoryImpl {
private final RabbitMqConfigProperties rabbitMqConfigProperties;
private Channel channel;
@PostConstruct
private void init() {
ConnectionFactory factory = new ConnectionFactory();
try {
String mbRabbitMqUri = rabbitMqConfigProperties.getMbRabbitMqUri();
log.debug(String.format("RabbitMQ Uri = %s", mbRabbitMqUri));
factory.setUri(mbRabbitMqUri);
factory.setAutomaticRecoveryEnabled(true);
Connection conn = factory.newConnection();
this.channel = conn.createChannel();
log.debug("RabbitMQ Channel was created.");
channel.queueDeclare(rabbitMqConfigProperties.getStatusQueueName(), true, false, false, null);
log.debug(String.format("Queue [ %s ] was declared.", rabbitMqConfigProperties.getStatusQueueName()));
} catch (KeyManagementException | NoSuchAlgorithmException | URISyntaxException | IOException | TimeoutException e) {
log.error(e.getMessage(), e);
}
}
public Channel getClient() {
return channel;
}
@PreDestroy
public void destroy() throws IOException, TimeoutException {
if (Objects.nonNull(channel) && channel.isOpen()) {
channel.close();
}
}
}
/*
Copyright 2021 Google LLC
Copyright 2021 EPAM Systems, Inc
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package org.opengroup.osdu.schema.messagebus;
import com.google.gson.Gson;
import com.rabbitmq.client.Channel;
import java.io.IOException;
import java.util.Collections;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.opengroup.osdu.schema.config.RabbitMqConfigProperties;
import org.opengroup.osdu.schema.factory.RabbitMQFactoryImpl;
import org.opengroup.osdu.schema.logging.AuditLogger;
import org.opengroup.osdu.schema.provider.interfaces.messagebus.IMessageBus;
import org.springframework.stereotype.Component;
@Component
@Slf4j
@RequiredArgsConstructor
public class MessageBusImpl implements IMessageBus {
private final RabbitMQFactoryImpl rabbitMQFactory;
private final RabbitMqConfigProperties mqConfigProperties;
private final AuditLogger auditLogger;
@Override
public void publishMessage(String schemaId, String eventType) {
log.info(String.format("Generating event of type %s", eventType));
SchemaPubSubInfo schemaPubSubMsg = new SchemaPubSubInfo(schemaId, eventType);
String message = new Gson().toJson(schemaPubSubMsg);
Channel client = rabbitMQFactory.getClient();
String queueNameWithPrefix = mqConfigProperties.getStatusQueueName();
try {
client.basicPublish("", queueNameWithPrefix, null, message.getBytes());
log.info(String.format("[x] Sent '%s' to queue [%s]", message, queueNameWithPrefix));
this.auditLogger.schemaNotificationSuccess(Collections.singletonList(schemaId));
} catch (IOException e) {
log.error(String.format("Unable to publish message to [%s]", queueNameWithPrefix));
log.error(e.getMessage(), e);
this.auditLogger.schemaNotificationFailure(Collections.singletonList(schemaId));
}
}
}
/*
Copyright 2021 Google LLC
Copyright 2021 EPAM Systems, Inc
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package org.opengroup.osdu.schema.messagebus;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
@Data
@NoArgsConstructor
@AllArgsConstructor
public class SchemaPubSubInfo {
private String kind;
private String op;
}
/*
* Copyright 2021 Google LLC
* Copyright 2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.schema.schemainfostore;
import static com.mongodb.client.model.Filters.eq;
import static org.opengroup.osdu.schema.constants.SchemaAnthosConstants.AUTHORITY_COLLECTION;
import static org.opengroup.osdu.schema.constants.SchemaAnthosConstants.SCHEMA_DATABASE;
import com.google.gson.Gson;
import com.mongodb.client.MongoCollection;
import java.util.Objects;
import lombok.RequiredArgsConstructor;
import org.bson.Document;
import org.opengroup.osdu.schema.exceptions.ApplicationException;
import org.opengroup.osdu.schema.exceptions.BadRequestException;
import org.opengroup.osdu.schema.exceptions.NotFoundException;
import org.opengroup.osdu.schema.factory.MongoClientFactory;
import org.opengroup.osdu.schema.model.Authority;
import org.opengroup.osdu.schema.provider.interfaces.schemainfostore.IAuthorityStore;
import org.springframework.stereotype.Service;
@Service
@RequiredArgsConstructor
public class AuthorityStore implements IAuthorityStore {
private final MongoClientFactory mongoClientFactory;
@Override
public Authority get(String authorityId) throws NotFoundException, ApplicationException {
MongoCollection<Document> collection = mongoClientFactory.getMongoClient().getDatabase(SCHEMA_DATABASE).getCollection(AUTHORITY_COLLECTION);
Document authorityDoc = collection.find(eq("authorityId", authorityId)).first();
if (Objects.isNull(authorityDoc)) {
throw new NotFoundException("bad input parameter");
}
return new Gson().fromJson(authorityDoc.toJson(), Authority.class);
}
@Override
public Authority create(Authority authority) throws ApplicationException, BadRequestException {
MongoCollection<Document> collection = mongoClientFactory.getMongoClient().getDatabase(SCHEMA_DATABASE).getCollection(AUTHORITY_COLLECTION);
collection.insertOne(Document.parse(new Gson().toJson(authority)));
return authority;
}
}
/*
* Copyright 2021 Google LLC
* Copyright 2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.schema.schemainfostore;
import static com.mongodb.client.model.Filters.eq;
import static org.opengroup.osdu.schema.constants.SchemaAnthosConstants.ENTITY_TYPE_COLLECTION;
import static org.opengroup.osdu.schema.constants.SchemaAnthosConstants.SCHEMA_DATABASE;
import com.google.gson.Gson;
import com.mongodb.client.MongoCollection;
import java.util.Objects;
import lombok.RequiredArgsConstructor;
import org.bson.Document;
import org.opengroup.osdu.schema.exceptions.ApplicationException;
import org.opengroup.osdu.schema.exceptions.BadRequestException;
import org.opengroup.osdu.schema.exceptions.NotFoundException;
import org.opengroup.osdu.schema.factory.MongoClientFactory;
import org.opengroup.osdu.schema.model.EntityType;
import org.opengroup.osdu.schema.provider.interfaces.schemainfostore.IEntityTypeStore;
import org.springframework.stereotype.Service;
@Service
@RequiredArgsConstructor
public class EntityTypeStore implements IEntityTypeStore {
private final MongoClientFactory mongoClientFactory;
@Override
public EntityType get(String entityTypeId) throws NotFoundException, ApplicationException {
MongoCollection<Document> collection = mongoClientFactory.getMongoClient().getDatabase(SCHEMA_DATABASE).getCollection(ENTITY_TYPE_COLLECTION);
Document entityType = collection.find(eq("entityTypeId", entityTypeId)).first();
if (Objects.isNull(entityType)) {
throw new NotFoundException("bad input parameter");
}
return new Gson().fromJson(entityType.toJson(), EntityType.class);
}
@Override
public EntityType create(EntityType entityType) throws BadRequestException, ApplicationException {
MongoCollection<Document> collection = mongoClientFactory.getMongoClient().getDatabase(SCHEMA_DATABASE).getCollection(ENTITY_TYPE_COLLECTION);
collection.insertOne(Document.parse(new Gson().toJson(entityType)));
return entityType;
}
}
/*
* Copyright 2021 Google LLC
* Copyright 2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.schema.schemainfostore;
import static com.mongodb.client.model.Accumulators.max;
import static com.mongodb.client.model.Aggregates.group;
import static com.mongodb.client.model.Aggregates.match;
import static com.mongodb.client.model.Filters.and;
import static com.mongodb.client.model.Filters.eq;
import static org.opengroup.osdu.schema.constants.SchemaAnthosConstants.SCHEMA_COLLECTION;
import static org.opengroup.osdu.schema.constants.SchemaAnthosConstants.SCHEMA_DATABASE;
import static org.opengroup.osdu.schema.constants.SchemaAnthosConstants.SCHEMA_IDENTITY;
import com.google.gson.Gson;
import com.mongodb.client.AggregateIterable;
import com.mongodb.client.FindIterable;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.result.DeleteResult;
import com.mongodb.client.result.UpdateResult;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Date;
import java.util.List;
import java.util.Objects;
import java.util.stream.Collectors;
import java.util.stream.StreamSupport;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.bson.Document;
import org.bson.conversions.Bson;
import org.opengroup.osdu.schema.constants.SchemaConstants;
import org.opengroup.osdu.schema.exceptions.ApplicationException;
import org.opengroup.osdu.schema.exceptions.BadRequestException;
import org.opengroup.osdu.schema.exceptions.NotFoundException;
import org.opengroup.osdu.schema.factory.MongoClientFactory;
import org.opengroup.osdu.schema.model.QueryParams;
import org.opengroup.osdu.schema.model.SchemaIdentity;
import org.opengroup.osdu.schema.model.SchemaInfo;
import org.opengroup.osdu.schema.model.SchemaRequest;
import org.opengroup.osdu.schema.provider.interfaces.schemainfostore.ISchemaInfoStore;
import org.springframework.stereotype.Service;
@Service
@Slf4j
@RequiredArgsConstructor
public class SchemaInfoStore implements ISchemaInfoStore {
private final MongoClientFactory mongoClientFactory;
@Override
public SchemaInfo updateSchemaInfo(SchemaRequest schema) throws ApplicationException, BadRequestException {
MongoCollection<Document> collection = mongoClientFactory.getMongoClient().getDatabase(SCHEMA_DATABASE).getCollection(SCHEMA_COLLECTION);
SchemaInfo newSchemaInfo = schema.getSchemaInfo();
newSchemaInfo.setDateCreated(new Date());
UpdateResult updateResult =
collection.replaceOne(eq(newSchemaInfo.getSchemaIdentity().getId()),
Document.parse(new Gson().toJson(newSchemaInfo)));
if (!updateResult.wasAcknowledged()) {
log.error(SchemaConstants.OBJECT_INVALID);
throw new ApplicationException("Invalid object, updating failed");
}
return newSchemaInfo;
}
@Override
public SchemaInfo createSchemaInfo(SchemaRequest schema) throws ApplicationException, BadRequestException {
MongoCollection<Document> collection = mongoClientFactory.getMongoClient().getDatabase(SCHEMA_DATABASE).getCollection(SCHEMA_COLLECTION);
SchemaInfo schemaInfo = schema.getSchemaInfo();
checkIfSupersededSchemaExists(schemaInfo);
schemaInfo.setDateCreated(new Date());
collection.insertOne(Document.parse(new Gson().toJson(schemaInfo)).append("_id", schemaInfo.getSchemaIdentity().getId()));
return schemaInfo;
}
private void checkIfSupersededSchemaExists(SchemaInfo schemaInfo) throws ApplicationException, BadRequestException {
SchemaIdentity supersededBy = schemaInfo.getSupersededBy();
if (Objects.nonNull(supersededBy)) {
try {
getSchemaInfo(supersededBy.getId());
} catch (NotFoundException e) {
log.error(SchemaConstants.INVALID_SUPERSEDEDBY_ID);
throw new BadRequestException(SchemaConstants.INVALID_SUPERSEDEDBY_ID);
}
}
}
@Override
public SchemaInfo getSchemaInfo(String schemaId) throws ApplicationException, NotFoundException {
MongoCollection<Document> collection = mongoClientFactory.getMongoClient().getDatabase(SCHEMA_DATABASE).getCollection(SCHEMA_COLLECTION);
Document schemaIdentity = collection.find(eq(schemaId)).first();
if (Objects.isNull(schemaIdentity)) {
throw new NotFoundException("bad input parameter");
}
return new Gson().fromJson(schemaIdentity.toJson(), SchemaInfo.class);
}
@Override
public String getLatestMinorVerSchema(SchemaInfo schemaInfo) throws ApplicationException {
MongoCollection<Document> collection = mongoClientFactory.getMongoClient().getDatabase(SCHEMA_DATABASE).getCollection(SCHEMA_COLLECTION);
AggregateIterable<Document> documents = collection.aggregate(Arrays.asList(match(and(
eq(SchemaConstants.AUTHORITY, schemaInfo.getSchemaIdentity().getAuthority()),
eq(SchemaConstants.ENTITY_TYPE, schemaInfo.getSchemaIdentity().getEntityType()),
eq(SchemaConstants.MAJOR_VERSION, schemaInfo.getSchemaIdentity().getSchemaVersionMajor()),
eq(SchemaConstants.SOURCE, schemaInfo.getSchemaIdentity().getSource())
)), group(SchemaConstants.MINOR_VERSION, max(SchemaConstants.MINOR_VERSION, "$_id"))));
return documents.first().toJson();
}
@Override
public List<SchemaInfo> getSchemaInfoList(QueryParams queryParams, String tenantId) throws ApplicationException {
MongoCollection<Document> collection = mongoClientFactory.getMongoClient().getDatabase(SCHEMA_DATABASE).getCollection(SCHEMA_COLLECTION);
FindIterable<Document> documents = collection.find(getFilters(queryParams));
return StreamSupport.stream(documents.spliterator(), false)
.map(document -> new Gson().fromJson(document.toJson(), SchemaInfo.class))
.collect(Collectors.toList());
}
@Override
public boolean isUnique(String schemaId, String tenantId) throws ApplicationException {
MongoCollection<Document> collection = mongoClientFactory.getMongoClient().getDatabase(SCHEMA_DATABASE).getCollection(SCHEMA_COLLECTION);
Document schemaInfo = collection.find(eq(schemaId)).first();
return Objects.isNull(schemaInfo);
}
@Override
public boolean cleanSchema(String schemaId) throws ApplicationException {
MongoCollection<Document> collection = mongoClientFactory.getMongoClient().getDatabase(SCHEMA_DATABASE).getCollection(SCHEMA_COLLECTION);
DeleteResult deleteResult = collection.deleteOne(eq(schemaId));
return deleteResult.wasAcknowledged();
}
private Bson getFilters(QueryParams queryParams) {
ArrayList<Bson> filters = new ArrayList<>();
if (Objects.nonNull(queryParams.getAuthority())) {
filters.add(eq(SCHEMA_IDENTITY + SchemaConstants.AUTHORITY, queryParams.getAuthority()));
}
if (Objects.nonNull(queryParams.getSource())) {
filters.add(eq(SCHEMA_IDENTITY + SchemaConstants.SOURCE, queryParams.getSource()));
}
if (Objects.nonNull(queryParams.getEntityType())) {
filters.add(eq(SCHEMA_IDENTITY + SchemaConstants.ENTITY_TYPE, queryParams.getEntityType()));
}
if (Objects.nonNull(queryParams.getSchemaVersionMajor())) {
filters.add(eq(SCHEMA_IDENTITY + SchemaConstants.MAJOR_VERSION, queryParams.getSchemaVersionMajor()));
}
if (Objects.nonNull(queryParams.getSchemaVersionMinor())) {
filters.add(eq(SCHEMA_IDENTITY + SchemaConstants.MINOR_VERSION, queryParams.getSchemaVersionMinor()));
}
if (Objects.nonNull(queryParams.getSchemaVersionPatch())) {
filters.add(eq(SCHEMA_IDENTITY + SchemaConstants.PATCH_VERSION, queryParams.getSchemaVersionPatch()));
}
if (Objects.nonNull(queryParams.getStatus())) {
filters.add(eq(SCHEMA_IDENTITY + SchemaConstants.STATUS, queryParams.getStatus().toUpperCase()));
}
return filters.isEmpty() ? new Document() : and(filters);
}
}
/*
* Copyright 2021 Google LLC
* Copyright 2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.schema.schemainfostore;
import static com.mongodb.client.model.Filters.eq;
import static org.opengroup.osdu.schema.constants.SchemaAnthosConstants.SCHEMA_DATABASE;
import static org.opengroup.osdu.schema.constants.SchemaAnthosConstants.SOURCE_COLLECTION;
import com.google.gson.Gson;
import com.mongodb.client.MongoCollection;
import java.util.Objects;
import lombok.RequiredArgsConstructor;
import org.bson.Document;
import org.opengroup.osdu.schema.exceptions.ApplicationException;
import org.opengroup.osdu.schema.exceptions.BadRequestException;
import org.opengroup.osdu.schema.exceptions.NotFoundException;
import org.opengroup.osdu.schema.factory.MongoClientFactory;
import org.opengroup.osdu.schema.model.Source;
import org.opengroup.osdu.schema.provider.interfaces.schemainfostore.ISourceStore;
import org.springframework.stereotype.Service;
@Service
@RequiredArgsConstructor
public class SourceStore implements ISourceStore {
private final MongoClientFactory mongoClientFactory;
@Override
public Source get(String sourceId) throws NotFoundException, ApplicationException {
MongoCollection<Document> collection = mongoClientFactory.getMongoClient().getDatabase(SCHEMA_DATABASE).getCollection(SOURCE_COLLECTION);
Document sourceDoc = collection.find(eq("sourceId", sourceId)).first();
if (Objects.isNull(sourceDoc)) {
throw new NotFoundException("bad input parameter");
}
return new Gson().fromJson(sourceDoc.toJson(), Source.class);
}
@Override
public Source create(Source source) throws BadRequestException, ApplicationException {
MongoCollection<Document> collection = mongoClientFactory.getMongoClient().getDatabase(SCHEMA_DATABASE).getCollection(SOURCE_COLLECTION);
collection.insertOne(Document.parse(new Gson().toJson(source)));
return source;
}
}
/*
* Copyright 2021 Google LLC
* Copyright 2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.schema.schemastore;
import static org.opengroup.osdu.schema.constants.SchemaAnthosConstants.MINIO_SCHEMA_BUCKET;
import io.minio.GetObjectArgs;
import io.minio.MinioClient;
import io.minio.ObjectWriteResponse;
import io.minio.PutObjectArgs;
import io.minio.RemoveObjectArgs;
import io.minio.errors.ErrorResponseException;
import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.nio.charset.StandardCharsets;
import java.security.InvalidKeyException;
import java.util.HashMap;
import java.util.Map;
import java.util.stream.Collectors;
import lombok.RequiredArgsConstructor;
import org.opengroup.osdu.schema.constants.SchemaConstants;
import org.opengroup.osdu.schema.exceptions.ApplicationException;
import org.opengroup.osdu.schema.exceptions.NotFoundException;
import org.opengroup.osdu.schema.factory.CloudObjectStorageFactory;
import org.opengroup.osdu.schema.provider.interfaces.schemastore.ISchemaStore;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.stereotype.Service;
@Service
@RequiredArgsConstructor
public class SchemaStore implements ISchemaStore {
private final CloudObjectStorageFactory storageFactory;
@Override
public String createSchema(String filePath, String content) throws ApplicationException {
MinioClient client = storageFactory.getClient();
Map<String, String> headers = new HashMap<>();
headers.put("Content-Type", MediaType.APPLICATION_OCTET_STREAM_VALUE);
headers.put("X-Amz-Storage-Class", "REDUCED_REDUNDANCY");
byte[] bytes = content.getBytes(StandardCharsets.UTF_8);
try {
ObjectWriteResponse objectWriteResponse = client.putObject(
PutObjectArgs.builder()
.bucket(MINIO_SCHEMA_BUCKET)
.object(filePath + SchemaConstants.JSON_EXTENSION)
.stream(new ByteArrayInputStream(bytes), bytes.length, -1)
.headers(headers)
.build());
return objectWriteResponse.etag();
} catch (InvalidKeyException e) {
throw new ApplicationException(HttpStatus.BAD_REQUEST, e.getMessage());
} catch (ErrorResponseException e) {
throw new ApplicationException(HttpStatus.INTERNAL_SERVER_ERROR, e.errorResponse().message());
} catch (Exception e) {
throw new ApplicationException(SchemaConstants.INTERNAL_SERVER_ERROR);
}
}
@Override
public String getSchema(String dataPartitionId, String filePath) throws NotFoundException, ApplicationException {
MinioClient client = storageFactory.getClient();
try {
InputStream stream =
client.getObject(GetObjectArgs.builder().bucket(MINIO_SCHEMA_BUCKET).object(filePath + SchemaConstants.JSON_EXTENSION).build());
return new BufferedReader(new InputStreamReader(stream, StandardCharsets.UTF_8)).lines().collect(Collectors.joining("\n"));
} catch (InvalidKeyException e) {
throw new ApplicationException(HttpStatus.BAD_REQUEST, e.getMessage());
} catch (ErrorResponseException e) {
if (e.response().code() == HttpStatus.NOT_FOUND.value()) {
throw new NotFoundException(HttpStatus.NOT_FOUND, e.errorResponse().message());
} else {
throw new ApplicationException(HttpStatus.INTERNAL_SERVER_ERROR, e.errorResponse().message());
}
} catch (Exception e) {
throw new ApplicationException(SchemaConstants.INTERNAL_SERVER_ERROR);
}
}
@Override
public boolean cleanSchemaProject(String schemaId) throws ApplicationException {
MinioClient client = storageFactory.getClient();
try {
client.removeObject(RemoveObjectArgs.builder().bucket(MINIO_SCHEMA_BUCKET).object(schemaId).build());
return true;
} catch (InvalidKeyException e) {
throw new ApplicationException(HttpStatus.BAD_REQUEST, e.getMessage());
} catch (ErrorResponseException e) {
throw new ApplicationException(HttpStatus.INTERNAL_SERVER_ERROR, e.errorResponse().message());
} catch (Exception e) {
throw new ApplicationException(SchemaConstants.INTERNAL_SERVER_ERROR);
}
}
}
/*
* Copyright 2021 Google LLC
* Copyright 2021 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.schema.security;
import org.springframework.security.config.annotation.method.configuration.EnableGlobalMethodSecurity;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter;
@EnableWebSecurity
@EnableGlobalMethodSecurity(prePostEnabled = true)
public class WebSecurity extends WebSecurityConfigurerAdapter {
@Override
protected void configure(HttpSecurity http) throws Exception {
http.httpBasic().disable().csrf().disable();
}
}
#
# Copyright 2021 Google LLC
# Copyright 2021 EPAM Systems, Inc
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
server.servlet.context-path=/api/schema-service/v1
logging.level.org.springframework.web=${LOG_LEVEL:DEBUG}
server.port=8080
minio-endpoint-url=http://127.0.0.1:9000
minio-access-key=adminadmin
minio-secret-key=adminadmin
mongo-db-url=127.0.0.1:27017
mongo-db-user=admin
mongo-db-password=admin
mb-rabbitmq-uri=amqp://guest:guest@127.0.0.1:5672/%2F
status-queue-name=schema-changed
AUTHORIZE_API=https://dev.osdu-gcp.go3-nrg.projects.epam.com/api/entitlements/v2/
AUTHORIZE_API_KEY=
LOG_PREFIX=schema
\ No newline at end of file
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment