Commit a7c309cf authored by Rostislav Dublin (EPAM)'s avatar Rostislav Dublin (EPAM)
Browse files

Closes GONRG-1108 by adding flexibility to INTEGRATION_TESTING and...

Closes GONRG-1108 by adding flexibility to INTEGRATION_TESTING and NO_DATA_ACCESS_TESTER ENV VARS pass ways (filename or content, plain or base64)
parent b7c657b8
......@@ -55,4 +55,4 @@ eslint-sast:
- when: never
osdu-gcp-test:
allow_failure: false
allow_failure: false
\ No newline at end of file
......@@ -21,6 +21,12 @@ All references on repositories settings are external to `pom.xml` and should be
To build against Community GitLab repositories, use `.mvn/community-maven.settings.xml` settings:
`mvn clean compile test --settings .mvn/community-maven.settings.xml`
## Running integration tests
Integration tests are located in a separate project for each cloud in the ```testing``` directory under the project root directory.
### GCP
Instructions for running the GCP integration tests can be found [here](./provider/delivery-gcp/README.md).
## Publish
All references on Binary Storage required to publish Maven artifacts are external to `pom.xml` and should be configured through Maven `settings.xml` file.
There are two profiles available in `.mvn/community-maven.settings.xml` that can be used to publish artifacts to Community GitLab:
......@@ -28,4 +34,5 @@ There are two profiles available in `.mvn/community-maven.settings.xml` that can
2. `GitLab-Authenticate-With-Private-Token` - profile for local development and manual publishing of artifacts. To activate the profile, the developer should have `COMMUNITY_MAVEN_TOKEN` env variable with a configured personal GitLab access token. Please see [GitLab documentation](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) on how to set-up Personal Access token in GitLab:
```
export COMMUNITY_MAVEN_TOKEN='Your personal GitLab access token'
mvn deploy --settings .mvn/community-maven.settings.xml
\ No newline at end of file
mvn deploy --settings .mvn/community-maven.settings.xml
......@@ -70,7 +70,8 @@
<module>provider/delivery-ibm</module>
<module>provider/delivery-gcp</module>
<module>provider/delivery-azure</module>
</modules>
<module>provider/delivery-reference</module>
</modules>
<repositories>
<repository>
......
# delivery-gcp
## Running Locally
### Requirements
In order to run this service locally, you will need the following:
- [Maven 3.6.0+](https://maven.apache.org/download.cgi)
- [AdoptOpenJDK8](https://adoptopenjdk.net/)
- Infrastructure dependencies, deployable through the relevant [infrastructure template](https://community.opengroup.org/osdu/platform/deployment-and-operations/infra-gcp-provisioning)
### Environment Variables
In order to run the service locally, you will need to have the following environment variables defined.
| name | value | description | sensitive? | source |
| --- | --- | --- | --- | --- |
| `LOG_PREFIX` | `delivery` | Logging prefix | no | - |
| `GCP_REDIS_HOST` | `localhost` | Redis host | no | - |
| `GCP_SEARCH_QUERY_URL` | ex `http://localhost:8080/api/search/v2/query` | Search query endpoint | no | - |
| `GCP_ENTITLEMENTS_URL` | ex `http://localhost:8080/entitlements/v1` | Entitlements API endpoint | no | - |
**Required to run integration tests**
| name | value | description | sensitive? | source |
| --- | --- | --- | --- | --- |
| `DELIVERY_HOST` | `http://localhost:8080/api/delivery/v2/` | Delivery endpoint | no | - |
| `SEARCH_HOST` | `http://localhost:8080/api/search/v2/` | Search endpoint | no | - |
| `STORAGE_HOST` | `http://localhost:8080/api/storage/v2/` | Storage endpoint | no | - |
| `ENTITLEMENTS_DOMAIN` | `example.com` | OSDU R2 to run tests under | no | - |
| `TENANT` | `opendes` | Tenant name | no | - |
| `DEFAULT_DATA_PARTITION_ID_TENANT1` | `opendes` | Data partition id | no | - |
| `DEFAULT_DATA_PARTITION_ID_TENANT2` | `opendes` | Data partition id | no | - |
| `SEARCH_INTEGRATION_TESTER` | `********` | Service account for API calls, passed as a filename or JSON content, plain or Base64 encoded. Note: this user must have entitlements configured already | yes | https://console.cloud.google.com/iam-admin/serviceaccounts |
| `INTEGRATION_TEST_AUDIENCE` | `********` | Client application ID | yes | https://console.cloud.google.com/apis/credentials |
| `LEGAL_TAG` | ex `opendes-osdu-demo-legaltag` | Currently existing, not expired legal tag name| no | create one with POST {{legal_url}}api/legal/v1/legaltags |
| `OTHER_RELEVANT_DATA_COUNTRIES` | ex `US` | - | no | - |
| `GCLOUD_PROJECT` | ex `osdu-cicd-epam` | Google cloud project id | no | - |
| `GCP_DEPLOY_FILE` | `********` | Service account for test data tear down, passed as a filename or JSON content, plain or Base64 encoded. Must have cloud storage role configured | yes | https://console.cloud.google.com/iam-admin/serviceaccounts |
**Entitlements configuration for integration accounts**
| SEARCH_INTEGRATION_TESTER |
| --- |
| users<br/>service.entitlements.user<br/>service.storage.admin<br/>service.legal.user<br/>service.search.user<br/>service.delivery.viewer |
**Cloud roles configuration for integration accounts**
| GCP_DEPLOY_FILE|
| --- |
| storage.admin access to the Google Cloud Storage |
### Configure Maven
Check that maven is installed:
```bash
$ mvn --version
Apache Maven 3.6.0
Maven home: /usr/share/maven
Java version: 1.8.0_212, vendor: AdoptOpenJDK, runtime: /usr/lib/jvm/jdk8u212-b04/jre
...
```
You may need to configure access to the remote maven repository that holds the OSDU dependencies. This file should live within `~/.mvn/community-maven.settings.xml`:
```bash
$ cat ~/.m2/settings.xml
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
<servers>
<server>
<id>community-maven-via-private-token</id>
<!-- Treat this auth token like a password. Do not share it with anyone, including Microsoft support. -->
<!-- The generated token expires on or before 11/14/2019 -->
<configuration>
<httpHeaders>
<property>
<name>Private-Token</name>
<value>${env.COMMUNITY_MAVEN_TOKEN}</value>
</property>
</httpHeaders>
</configuration>
</server>
</servers>
</settings>
```
### Build and run the application
After configuring your environment as specified above, you can follow these steps to build and run the application. These steps should be invoked from the *repository root.*
```bash
cd provider/delivery-gcp/ && mvn spring-boot:run
```
### Test the application
After the service has started it should be accessible via a web browser by visiting [http://localhost:8080/swagger-ui.html](http://localhost:8080/swagger-ui.html). If the request does not fail, you can then run the integration tests.
```bash
# build + install integration test core
$ (cd testing/delivery-test-core/ && mvn clean install)
# build + run GCP integration tests.
#
# Note: this assumes that the environment variables for integration tests as outlined
# above are already exported in your environment.
$ (cd testing/delivery-test-gcp/ && mvn clean test)
```
## License
Copyright 2020 Google LLC
Copyright 2020 EPAM Systems, Inc
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
......@@ -101,6 +101,25 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.7.7.201606060606</version>
<executions>
<execution>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<execution>
<id>report</id>
<phase>prepare-package</phase>
<goals>
<goal>report</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
......
# Use the official AdoptOpenJDK for a base image.
# https://hub.docker.com/_/openjdk
FROM openjdk:8-slim
WORKDIR /app
COPY target/delivery-reference-0.0.1-SNAPSHOT.jar delivery.jar
# Run the web service on container startup.
CMD java -Djava.security.egd=file:/dev/./urandom -Dserver.port=8080 -jar /app/delivery.jar
apiVersion: v1
data:
GCP_SEARCH_QUERY_URL: ${GCP_SEARCH_QUERY_URL}
GCP_ENTITLEMENTS_URL: ${GCP_ENTITLEMENTS_URL}
MINIO_ENDPOINT_URL : ${MINIO_ENDPOINT_URL}
MINIO_REGION : ${MINIO_REGION}
GCP_REDIS_HOST: ${GCP_REDIS_HOST}
kind: ConfigMap
metadata:
labels:
app: delivery
name: delivery-config
namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
generateName: delivery
labels:
app: delivery
name: delivery
namespace: default
spec:
selector:
matchLabels:
app: delivery
replicas: 1
template:
metadata:
labels:
app: delivery
spec:
containers:
- env:
- name: GCP_SEARCH_QUERY_URL
valueFrom:
configMapKeyRef:
key: GCP_SEARCH_QUERY_URL
name: delivery-config
- name: GCP_ENTITLEMENTS_URL
valueFrom:
configMapKeyRef:
key: GCP_ENTITLEMENTS_URL
name: delivery-config
- name: MINIO_ENDPOINT_URL
valueFrom:
configMapKeyRef:
key: MINIO_ENDPOINT_URL
name: delivery-config
- name: MINIO_REGION
valueFrom:
configMapKeyRef:
key: MINIO_REGION
name: delivery-config
- name: GCP_REDIS_HOST
valueFrom:
configMapKeyRef:
key: GCP_REDIS_HOST
name: delivery-config
- name: MINIO_ACCESS_KEY
valueFrom:
secretKeyRef:
name: delivery-secret
key: minio.access_key
- name: MINIO_SECRET_KEY
valueFrom:
secretKeyRef:
name: delivery-secret
key: minio.secret_key
image: us.gcr.io/osdu-anthos-02/os-delivery/anthos-delivery:latest
name: os-delivery-1-sha256-1
---
apiVersion: v1
kind: Service
metadata:
name: delivery
namespace: default
spec:
ports:
- protocol: TCP
port: 80
targetPort: 8080
selector:
app: delivery
type: LoadBalancer
---
apiVersion: v1
data:
minio.access_key : ${MINIO_ACCESS_KEY}
minio.secret_key : ${MINIO_SECRET_KEY}
kind: Secret
metadata:
name: delivery-secret
namespace: default
type: Opaque
\ No newline at end of file
<?xml version="1.0" encoding="UTF-8"?>
<!--
Copyright 2020 Google LLC
Copyright 2020 EPAM Systems, Inc
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<artifactId>os-delivery</artifactId>
<groupId>org.opengroup.osdu</groupId>
<version>0.0.2-SNAPSHOT</version>
<relativePath>../..</relativePath>
</parent>
<artifactId>delivery-reference</artifactId>
<description>Delivery service reference</description>
<version>0.0.1</version>
<dependencies>
<dependency>
<groupId>org.opengroup.osdu</groupId>
<artifactId>delivery-core</artifactId>
<version>0.0.2-SNAPSHOT</version>
</dependency>
<!-- Testing packages -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>1.10.19</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.powermock</groupId>
<artifactId>powermock-module-junit4</artifactId>
<version>2.0.2</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.minio</groupId>
<artifactId>minio</artifactId>
<version>7.1.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>2.1.9.RELEASE</version>
<configuration>
<mainClass>org.opengroup.osdu.delivery.DeliveryApplication</mainClass>
</configuration>
<executions>
<execution>
<goals>
<goal>repackage</goal>
</goals>
<configuration>
<classifier>SNAPSHOT</classifier>
<mainClass>org.opengroup.osdu.delivery.DeliveryApplication</mainClass>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
\ No newline at end of file
apiVersion: skaffold/v2beta5
kind: Config
metadata:
name: delivery-reference
build:
artifacts:
- image: us.gcr.io/osdu-anthos-02/os-delivery/anthos-delivery
deploy:
kubectl:
manifests:
- kubernetes/deployments/deployment-delivery-service.yml
package org.opengroup.osdu.delivery.provider.reference;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.ComponentScan;
@SpringBootApplication
@ComponentScan({"org.opengroup.osdu"})
public class DeliveryReferenceApplication {
public static void main(String[] args) {
SpringApplication.run(DeliveryReferenceApplication.class, args);
}
}
/*
* Copyright 2020 Google LLC
* Copyright 2020 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.delivery.provider.reference.cache;
import org.opengroup.osdu.core.common.cache.RedisCache;
import org.opengroup.osdu.core.common.model.entitlements.Groups;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;
@Component
public class GroupCache extends RedisCache<String, Groups> {
public GroupCache(
@Value("${gcp.redis.host}") final String redisHost,
@Value("${gcp.redis.port}") final Integer redisPort,
@Value("${gcp.redis.exp.time}") final Integer expTimeSec) {
super(redisHost, redisPort, expTimeSec, String.class, Groups.class);
}
}
package org.opengroup.osdu.delivery.provider.reference.factory;
import io.minio.MinioClient;
import io.minio.errors.InvalidEndpointException;
import io.minio.errors.InvalidPortException;
import javax.annotation.PostConstruct;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Lazy;
import org.springframework.stereotype.Component;
@Component
@Lazy
public class CloudObjectStorageFactory {
private static final Logger logger = LoggerFactory.getLogger(CloudObjectStorageFactory.class);
@Value("${minio.endpoint_url}")
private String endpointURL;
@Value("${minio.access_key}")
private String accessKey;
@Value("${minio.secret_key}")
private String secretKey;
@Value("${minio.region:us-east-1}")
private String region;
@Value("${minio.prefix:local-dev}")
private String bucketNamePrefix;
private MinioClient minioClient;
private String bucketName;
public CloudObjectStorageFactory() { }
@PostConstruct
public void init() throws InvalidEndpointException, InvalidPortException {
this.minioClient = new MinioClient(this.endpointURL, this.accessKey, this.secretKey, this.region);
logger.info("Minio client initialized");
}
public MinioClient getClient() {
return this.minioClient;
}
}
/*
* Copyright 2020 Google LLC
* Copyright 2020 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.delivery.provider.reference.security;
import org.springframework.security.config.annotation.method.configuration.EnableGlobalMethodSecurity;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter;
@EnableWebSecurity
@EnableGlobalMethodSecurity(prePostEnabled = true)
public class BasicAuthSecurityConfig extends WebSecurityConfigurerAdapter {
@Override
protected void configure(HttpSecurity http) throws Exception {
http
.httpBasic().disable()
.csrf().disable();
}
}
/*
* Copyright 2020 Google LLC
* Copyright 2020 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.opengroup.osdu.delivery.provider.reference.security;
import org.springframework.security.core.Authentication;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;
@Controller
public class WhoamiController {
@RequestMapping(value = "/whoami")
@ResponseBody
public String whoami() {
final Authentication auth = SecurityContextHolder.getContext().getAuthentication();
String userName = auth.getName();
String roles = String.valueOf(auth.getAuthorities());
String details = String.valueOf(auth.getPrincipal());
return "user: " + userName + "<BR>" +
"roles: " + roles + "<BR>" +
"details: " + details + "<BR>";
}
}
/*
* Copyright 2020 Google LLC
* Copyright 2020 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at