Commit 28de0833 authored by Rucha Deshpande's avatar Rucha Deshpande
Browse files

Merge branch 'aws-partition-mongo-update' into 'master'

partition mongo update

See merge request !132
parents 1a78ef84 6f0e6372
Pipeline #85694 passed with stages
in 42 minutes and 42 seconds
This diff is collapsed.
......@@ -3,6 +3,7 @@
The Partition service is responsible for creating and retrieving partition specific properties on behalf of other services whether they are secret values or not. It is a Maven multi-module project with each cloud implementation placed in its submodule.
## Running Locally - AWS
Instructions for running the AWS implementation locally can be found [here](./provider/partition-aws/README.md)
## Running Locally - Azure
Instructions for running the Azure implementation locally can be found [here](./provider/partition-azure/README.md)
## Running Locally - GCP
......
# Partition Service
The AWS Partition service is a [Spring Boot](https://spring.io/projects/spring-boot) service that creates, reads, updates, and destroys partition properties. The partition properties are stored in a [MongoDB database](https://www.mongodb.com/) and encrypted by [AWS KMS](https://aws.amazon.com/kms/) if the properties are flagged as sensitive.
## Running Locally
The following instructions are the minimum requirements for running the AWS partition service locally.
### Prerequisites
* [JDK 8](https://docs.aws.amazon.com/corretto/latest/corretto-8-ug/downloads-list.html)
* [Maven 3.6.0+](https://maven.apache.org/download.cgi)
* IDE ([IntelliJ](https://www.jetbrains.com/idea/download/) is preferred)
* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
* [Postman](https://www.postman.com/)
* [MongoDB](https://docs.mongodb.com/v4.0/installation/) on local machine or a MongoDB connection string for test database.
* [Mongo Compass](https://www.mongodb.com/products/compass) <Optional>
* OSDU Instance deployed on AWS
### Service Configuration
The following environment variables need to be defined to run the service locally.
| Name | Example Value | Description | Sensitive? | Source |
| --- | --- | --- | --- | --- |
| `LOCAL_MODE` | `true` | A required flag to indicate to the authorization service that partition service is running locally versus in the cluster | no | - |
| `AWS_REGION` | ex `us-east-1` | The region where resources needed by the service are deployed | no | - |
| `AWS_ACCESS_KEY_ID` | - | The AWS Access Key for a user with access to Backend Resources required by the service | yes | [temporary security credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html) |
| `AWS_SECRET_ACCESS_KEY` | - | The AWS Secret Key for a user with access to Backend Resources required by the service | yes | [temporary security credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html) |
| `AWS_SESSION_TOKEN` | - | AWS Session token needed if using an SSO user session to authenticate | yes | [temporary security credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html) |
| `ENVIRONMENT` | ex `osdu-dev` | The name of the environment stack name | no | Output by infrastructure deployment |
| `ENTITLEMENTS_BASE_URL` | ex `https://alias.dev.osdu.aws` | The base URL of OSDU | no | Output by infrastructure deployment |
| `MONGODB_ENDPOINT` | ex `mongodbname.robot.mongodb.net` or `localhost` | MongoDB database endpoint used for local development | yes | https://www.mongodb.com/ |
| `MONGODB_AUTH_DATABASE` | ex `partitions` | The database name within the MongoDB instance - defaults to `partitions` | no | - |
| `MONGODB_USERNAME` | ex `admin` | MongoDB username used as part of the connection string | yes | - |
| `MONGODB_PASSWORD` | - | MongoDB password used as part of the connection string | yes | - |
| `MONGODB_USE_SRV_ENDPOINT` | `false` or `true` | To run the service locally, set this to false | no | - |
| `KEY_ARN` | ex `arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab` | A symmetric AWS KMS encryption key ARN with the appropriate key user policy that allows encrypt/decrypt access to the IAM role configured in the following step | yes | https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html |
### MongoDB Setup Locally
* Navigate to where the mongo sever is installed and start the server using:
``C:\Program Files\MongoDB\Server\5.0\bin>mongod``
The server will start on the default port 27017
* Launch MongoDB Compass and create a new connection to localhost: 27017
![New Connection](docs/img/newconn.png)
* Next add a db user using mongo shell using the following commands:
![Add new db user](docs/img/mongo_createuser.png)
### Run Locally
Check that maven is installed:
```bash
mvn --version
Apache Maven 3.8.3
Maven home: C:\opt\apache-maven-3.8.3
Java version: 1.8.0_302, vendor: Amazon.com Inc., runtime: C:\Program Files\Amazon Corretto\jdk1.8.0_302\jre
```
You may need to configure access to the remote maven repository that holds the OSDU dependencies. Copy one of the below files' content to your .m2 folder
* For development against the OSDU GitLab environment, leverage: `<REPO_ROOT>~/.mvn/community-maven.settings.xml`
* For development in an AWS Environment, leverage: `<REPO_ROOT>/provider/legal-aws/maven/settings.xml`
* Navigate to the AWS partition service's root folder and run:
```bash
mvn clean package -pl partition-core,provider/partition-aws
```
* If you wish to build the project without running tests
```bash
mvn clean package -pl partition-core,provider/partition-aws -DskipTests
```
After configuring your environment as specified above, you can follow these steps to run the application. These steps should be invoked from the *repository root.*
NOTE: If not on osx/linux: Replace `*` with version numbers as defined in the provider/partition-aws/pom.xml file
```bash
java -jar provider/partition-aws/target/partition-aws-*.*.*-SNAPSHOT-spring-boot.jar
```
Alternatively, if using IntelliJ, you can configure your environment variables and run configuration by selecting Run>Edit Configurations and fill in the below information:
Main Class: org.opengroup.osdu.partition.provider.aws.PartitionApplication
Use Classpath of Module: partition-aws
Environment Variables: (As defined above)
To run the configuration, select Run>Run and select your configuration.
### Hitting Partitions API
The service will be accessible at [http://localhost:8080/api/partition/v1/partitions/](http://localhost:8080/api/partition/v1/partitions/). A /info endpoint is available at [http://localhost:8080/api/partition/v1/info/](http://localhost:8080/api/partition/v1/info/).
A bearer access token is required to authorize all partitions API requests, except for /info. To generate an access token a POST request must be sent to the following URL:
{{auth_token_url}}?grant_type=client_credentials&client_id={{client_id}}&scope={{scope}}. The request must use "Basic Auth" with the client ID and secret passed in as parameters. The table below explains where to find these parameters:
| Parameter | Value | Sensitive? | Source |
| --- | --- | --- | --- |
| `auth_token_url` | ex `https://osdu-dev-888733619319.auth.us-east-1.amazoncognito.com/oauth2/token` | no | Found in AWS SSM under resource path /osdu/{resource_prefix}/client-credentials-client-id |
| `client_id` | - | yes | Found in AWS SSM under resource path /osdu/{resource_prefix}/client-credentials-client-id |
| `client_secret` | - | yes | Found in AWS Secrets Manager under resource path /osdu/{resource_prefix}/client_credentials_secret |
| `scope` | ex `osduOnAws/osduOnAWSService` | no | Found in AWS SSM under resource path /osdu/{resource_prefix}/oauth-custom-scope |
All partitions API requests should use Bearer Token auth using the access token returned from hitting the above endpoint.
## Testing
### Running Unit Tests
Navigate to the partition service's root folder:
```bash
cd provider/partition-aws
```
Install the project dependencies and run unit tests:
```bash
mvn clean install
```
### Running Integration Tests
Execute following command to build code and run all the integration tests from the root folder:
```bash
mvn clean package -f testing/pom.xml -pl partition-test-core,partition-test-aws -DskipTests
mvn test -f testing/partition-test-aws/pom.xml
```
## Licence
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
......@@ -57,7 +57,7 @@
<dependency>
<groupId>org.opengroup.osdu.core.aws</groupId>
<artifactId>os-core-lib-aws</artifactId>
<version>0.12.3</version>
<version>0.13.0-rc3</version>
</dependency>
<dependency>
<groupId>org.opengroup.osdu</groupId>
......@@ -87,6 +87,11 @@
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-cognitoidentity</artifactId>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-encryption-sdk-java</artifactId>
<version>2.3.3</version>
</dependency>
<!-- Third party Apache 2.0 license packages -->
<dependency>
......@@ -104,9 +109,12 @@
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-commons</artifactId>
<version>2.1.10.RELEASE</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>
<dependency>
<groupId>javax.inject</groupId>
<artifactId>javax.inject</artifactId>
......@@ -161,6 +169,10 @@
<artifactId>spring-security-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
</dependency>
</dependencies>
<build>
......@@ -192,6 +204,26 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>pl.project13.maven</groupId>
<artifactId>git-commit-id-plugin</artifactId>
<version>4.0.5</version>
<executions>
<execution>
<goals>
<goal>revision</goal>
</goals>
</execution>
</executions>
<configuration>
<verbose>true</verbose>
<dateFormat>yyyy-MM-dd'T'HH:mm:ssZ</dateFormat>
<generateGitPropertiesFile>true</generateGitPropertiesFile>
<generateGitPropertiesFilename>
${project.build.outputDirectory}/git.properties
</generateGitPropertiesFilename>
</configuration>
</plugin>
</plugins>
</build>
......
......@@ -14,11 +14,15 @@
package org.opengroup.osdu.partition.provider.aws;
import org.opengroup.osdu.core.aws.mongodb.config.MongoDBAutoconfigExclude;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.FilterType;
@ComponentScan({"org.opengroup.osdu"})
@ComponentScan(
basePackages = {"org.opengroup.osdu"},
excludeFilters = {@ComponentScan.Filter(type = FilterType.ASSIGNABLE_TYPE, value = MongoDBAutoconfigExclude.class)})
@SpringBootApplication
public class PartitionApplication {
......
// Copyright © 2020, Amazon Web Services
// Copyright © 2021 Amazon Web Services
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
......@@ -12,23 +12,16 @@
// See the License for the specific language governing permissions and
// limitations under the License.
package org.opengroup.osdu.partition.provider.aws;
package org.opengroup.osdu.partition.provider.aws.model;
import javax.annotation.PostConstruct;
import org.springframework.data.mongodb.repository.MongoRepository;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
import java.util.List;
import java.util.Optional;
@Component
public class AwsServiceConfig {
@Value("${aws.resource.prefix}")
public String environment;
public interface IPartitionRepository extends MongoRepository<Partition, String> {
public String ssmPartitionPrefix;
@PostConstruct
public void init() {
ssmPartitionPrefix = "/osdu/" + environment + "/partition/partitions/";
}
public Optional<Partition> findById(String id);
public void deleteById(String id);
public List<Partition> findAll();
}
// Copyright © 2021 Amazon Web Services
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package org.opengroup.osdu.partition.provider.aws.model;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import org.opengroup.osdu.partition.model.Property;
import org.springframework.data.mongodb.core.mapping.MongoId;
import javax.validation.constraints.NotEmpty;
import java.util.HashMap;
import java.util.Map;
@Data
@AllArgsConstructor
@NoArgsConstructor
public class Partition {
@NotEmpty
@MongoId
String id;
Map<String, Property> properties = new HashMap<>();
}
......@@ -27,4 +27,4 @@ public class BasicAuthSecurityConfig extends WebSecurityConfigurerAdapter {
((HttpSecurity)http.httpBasic().disable())
.csrf().disable();
}
}
}
\ No newline at end of file
......@@ -14,12 +14,18 @@
package org.opengroup.osdu.partition.provider.aws.service;
import java.util.*;
import lombok.RequiredArgsConstructor;
import org.apache.http.HttpStatus;
import org.bson.types.Binary;
import org.opengroup.osdu.core.common.logging.JaxRsDpsLog;
import org.opengroup.osdu.core.common.model.http.AppException;
import org.opengroup.osdu.partition.model.PartitionInfo;
import org.opengroup.osdu.partition.model.Property;
import org.opengroup.osdu.partition.provider.aws.util.SSMHelper;
import org.opengroup.osdu.partition.provider.aws.model.Partition;
import org.opengroup.osdu.partition.provider.aws.model.IPartitionRepository;
import org.opengroup.osdu.partition.provider.aws.util.AwsKmsEncryptionClient;
import org.opengroup.osdu.partition.provider.interfaces.IPartitionService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
......@@ -31,179 +37,158 @@ import java.util.Map;
* AWS implementation doesn't use cache.
*/
@Service
@RequiredArgsConstructor
public class PartitionServiceImpl implements IPartitionService {
@Autowired
private JaxRsDpsLog logger;
@Autowired
private SSMHelper ssmHelper;
public PartitionServiceImpl() {
}
private final IPartitionRepository repository;
private final AwsKmsEncryptionClient awsKmsEncryptionClient;
@Override
public PartitionInfo createPartition(String partitionId, PartitionInfo partitionInfo) {
if (ssmHelper.partitionExists(partitionId)) {
// throw error if partition already exists
if (repository.findById(partitionId).isPresent()) {
throw new AppException(HttpStatus.SC_CONFLICT, "partition exist", "Partition with same id exist");
}
try {
for (Map.Entry<String, Property> entry : partitionInfo.getProperties().entrySet()) {
ssmHelper.createOrUpdateSecret(partitionId, entry.getKey(), entry.getValue().getValue());
}
Partition savedPartition = repository.save(new Partition(partitionId, encryptSensitiveProperties(partitionInfo, partitionId)));
/**
* SSM parameters are not immediately available after pushing to System Manager.
* This API is expected to return a 200 response meaning that the parameters should be available immediately.
* This logic is added to validate when the parameters become available before returning the 200 response.
* The performance hit is acceptable because partitions are only created as an early operation and shouldn't affect
* the performance of runtime workflows
*/
int retryCount = 10;
boolean partitionReady = false;
while (!partitionReady && retryCount > 0) {
retryCount--;
List<String> partitionCheck = ssmHelper.getSsmParamsPathsForPartition(partitionId);
if (partitionCheck.size() == partitionInfo.getProperties().size())
partitionReady = true;
else
Thread.sleep(500);
}
return new PartitionInfo(savedPartition.getProperties());
}
String rollbackSuccess = "Failed";
if (!partitionReady) {
try {
ssmHelper.deletePartitionSecrets(partitionId);
rollbackSuccess = "Succeeded";
}
catch (Exception e){
@Override
public PartitionInfo updatePartition(String partitionId, PartitionInfo partitionInfo) {
}
Optional<Partition> partition = repository.findById(partitionId);
throw new AppException(HttpStatus.SC_INTERNAL_SERVER_ERROR, "Partition Creation Failed", "One or more secrets couldn't be stored. Rollback " + rollbackSuccess);
}
// throw error if search did not return a result
if (!partition.isPresent()) {
throw new AppException(HttpStatus.SC_NOT_FOUND, "Partition does not exist", "Partition does not exist");
}
catch (AppException appE) {
throw appE;
// throw error if the client tries to update the id
if (partitionInfo.getProperties().containsKey("id")) {
throw new AppException(HttpStatus.SC_BAD_REQUEST, "Cannot update id", "the field id cannot be updated");
}
catch (Exception e) {
try {
Thread.sleep(2000); //wait for any existing ssm parameters that got added to normalize
ssmHelper.deletePartitionSecrets(partitionId);
}
catch (Exception deleteE) {
//if the partition didnt get created at all deletePartition will throw an exception. Eat it so we return the creation exception.
}
Map<String, Property> updatedProperties = partition.get().getProperties();
Map<String, Property> encryptedPropertiesToAdd = encryptSensitiveProperties(partitionInfo, partitionId);
logger.error("Failed to create partition due to key creation failure in ssm", e.getMessage());
throw new AppException(HttpStatus.SC_INTERNAL_SERVER_ERROR, "Partition Creation Failure", e.getMessage(), e);
for (Map.Entry<String, Property> e : encryptedPropertiesToAdd.entrySet()) {
updatedProperties.put(e.getKey(), e.getValue());
}
// throw error if save was unsuccessful
try {
repository.save(new Partition(partitionId, updatedProperties));
} catch (Exception e) {
logger.error("Failed to update partition", e.getMessage());
throw new AppException(HttpStatus.SC_INTERNAL_SERVER_ERROR, "Partition update Failure", e.getMessage(), e);
}
return partitionInfo;
}
@Override
public PartitionInfo updatePartition(String partitionId, PartitionInfo partitionInfo) {
public PartitionInfo getPartition(String partitionId) {
Optional<Partition> partition = repository.findById(partitionId);
if (!ssmHelper.partitionExists(partitionId)) {
throw new AppException(HttpStatus.SC_NOT_FOUND, "partition does not exist", "Partition doesn't exist");
// throw error if partition doesn't exist
if (!partition.isPresent()) {
throw new AppException(HttpStatus.SC_NOT_FOUND, "Partition not found", String.format("%s partition not found", partitionId));
}
if(partitionInfo.getProperties().containsKey("id")) {
throw new AppException(HttpStatus.SC_BAD_REQUEST, "Cannot update id", "the field id cannot be updated");
}
PartitionInfo partitionInfo;
try {
for (Map.Entry<String, Property> entry : partitionInfo.getProperties().entrySet()) {
ssmHelper.createOrUpdateSecret(partitionId, entry.getKey(), entry.getValue().getValue());
}
/**
* SSM parameters are not immediately available after pushing to System Manager.
* This API is expected to return a 200 response meaning that the parameters should be available immediately.
* This logic is added to validate when the parameters become available before returning the 200 response.
* The performance hit is acceptable because partitions are only created as an early operation and shouldn't affect
* the performance of runtime workflows
*/
int retryCount = 10;
boolean partitionReady = false;
while (!partitionReady && retryCount > 0) {
retryCount--;
List<String> partitionCheck = ssmHelper.getSsmParamsPathsForPartition(partitionId);
if (partitionCheck.size() == partitionInfo.getProperties().size())
partitionReady = true;
else
Thread.sleep(500);
}
partitionInfo = decryptSensitiveProperties(partition.get().getProperties(), partitionId);
} catch (ClassCastException e) {
throw new AppException(HttpStatus.SC_INTERNAL_SERVER_ERROR, "Corrupt data", String.format("%s contains unreadable data", partitionId));
} catch (IllegalStateException e) {
throw new AppException(HttpStatus.SC_INTERNAL_SERVER_ERROR, "Illegal database modification", String.format("Partition %s has been modified without permission", partitionId));
}
String rollbackSuccess = "Failed";
if (!partitionReady) {
try {
ssmHelper.deletePartitionSecrets(partitionId);
rollbackSuccess = "Succeeded";
}
catch (Exception e){
return partitionInfo;
}
}
throw new AppException(HttpStatus.SC_INTERNAL_SERVER_ERROR, "Partition update Failed", "One or more secrets couldn't be stored. Rollback " + rollbackSuccess);
@Override
public boolean deletePartition(String partitionId) {
}
}
catch (AppException appE) {
throw appE;
// throw error if partition doesn't exist
if (!repository.findById(partitionId).isPresent()) {
throw new AppException(HttpStatus.SC_NOT_FOUND, "Partition not found", String.format("%s partition not found", partitionId));
}
catch (Exception e) {
try {
Thread.sleep(2000); //wait for any existing ssm parameters that got added to normalize
ssmHelper.deletePartitionSecrets(partitionId);
}
catch (Exception deleteE) {
//if the partition didnt get created at all deletePartition will throw an exception. Eat it so we return the creation exception.
}
repository.deleteById(partitionId);
logger.error("Failed to update partition due to key creation failure in ssm", e.getMessage());
throw new AppException(HttpStatus.SC_INTERNAL_SERVER_ERROR, "Partition update Failure", e.getMessage(), e);
}
return partitionInfo;
return true;
}
@Override
public PartitionInfo getPartition(String partitionId) {
public List<String> getAllPartitions() {
Map<String,Property> secrets = ssmHelper.getPartitionSecrets(partitionId);
List<Partition> partitionList = repository.findAll();
List<String> partitions = new ArrayList<>();
//throw error if partition doesn't exist
if (secrets.size() <= 0) {
throw new AppException(HttpStatus.SC_NOT_FOUND, "partition not found", String.format("%s partition not found", partitionId));
// populate list of ids, i.e. partition names
for (Partition p: partitionList) {
String id = p.getId();
partitions.add(id);
}
return PartitionInfo.builder().properties(secrets).build();
return partitions;
}
@Override