Commit d9375598 authored by David Diederich's avatar David Diederich
Browse files

Initial Import

parents
# IDEs and editors
/.idea
.project
.classpath
.c9/
*.launch
.factorypath
.settings/
*.sublime-workspace
*.iml
openapi.json
target/
# IDE - VSCode
.vscode/*
#IDE - Webstorm
/**/vcs.xml
/**/works
#Docker
docker-compose.override.yml
# Misc
/cloc
/bin
/pkg
/src/vendor
/src/api/vendor
/deploy
/cobertura
/testresults
/.cover
# Runtime files created during local run
/app/*
# Python, virtualenv
/venv
testing/*/venv/
testing/*/env/
#.pyc files in all directories
**/*.pyc
\ No newline at end of file
Copyright 2017-2019, Schlumberger
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
# Spatial Reference Catalog Service
## This repository contains
1. The Java implementation of the Spatial Reference Catalog (aka crs-catalog). The Java code is located in the ```src``` folder. To open the Java project, open ```pom.xml```.
1. Tests are located in ```src/test/java/org/opengroup/osdu/crs/...```
1. The openapi specification file is crs_catalog_openapi.json.
1. Python integration and health tests in the ```testing``` folder.
See also the test's [README.md](testing/README.md)
## Prerequisites
1. The project builds with [maven](https://maven.apache.org/). Make sure maven is installed locally.
1. The project requires the [Lombok](https://projectlombok.org/) plug-in installed for your IDE.
### Build service and run unit tests
```sh
mvn clean install
```
## Running Azure Catalog Service locally
### Configure Maven Settings
To obtain maven dependencies from the **Azure DevOps Artifacts** we need to configure the value for variable ${VSTS_FEED_TOKEN} described in `maven\settings.xml`:
- **Get token:**
We can use personal token generated in VSTS on [Personal Access Tokens](https://dev.azure.com/slb-swt/_usersSettings/tokens) > New token > Organization: slb-des-ext-collaboration > Create
- **Set token in your local home folder:**
Open or create `USER_HOME_FOLDER\.m2\settings.xml` and paste your personal token in `<password></password>`section.
```xml
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
<servers>
<server>
<id>os-core</id>
<username>slb-des-ext-collaboration</username>
<password></password>
</server>
</servers>
</settings>
```
#### Build and run Catalog Service locally using bash
- Set the required environments described in [Release/deployment](##Release/deployment) section
- Navigate to the Catalog Service's root folder ```os-crs-catalog-service```
- Build core and run unit tests in command line:
```bash
mvn clean install -P crs-catalog-core
# To run without tests add -Dmaven.test.skip=true
```
- Navigate to the Catalog Service's Azure
- Build services in command line:
```bash
mvn clean package -P crs-catalog-aks,crs-catalog-core
```
- Navigate to the Catalog Service's root folder ```os-crs-catalog-service```
- Run application with command
```bash
java -jar provider/crs-catalog-azure/crs-catalog-aks/target/crs-catalog-aks-1.0.0.jar
```
#### Running Azure Catalog Service using IntelliJ IDEA
Navigate to the **Create Run/Debug Configuration** tool
Select **'Add New Configuration'** and select **Application**
Type the next commands into the suggested fields:
- Working directory: ```{path_to_the_catalog}/os-crs-catalog-service```
- Main class: ```org.opengroup.osdu.crs.CRSAKSApplication```
- Use classpath of module: ```crs-catalog-aks```
***Note: If you don't see "crs-catalog-aks" in the dropdown menu - find appropriate pom.xml and click "Add as a Maven project"***
- Environment variables: Set the required environments described in [Release/deployment](##Release/deployment) section
Execute **Run** or **Debug** for configured Application
### Debug locally - e.g. using Postman
In the Postman Settings / General, turn SSL certificate validation off when running locally.
Similarly, when not using Postman but client code, set the configuration ```verify_ssl``` false (see [instructions](https://github.com/swagger-api/swagger-codegen/issues/7778))
Run application using debug mode and use [Postman](https://www.getpostman.com/)
to send a GET request to obtaining the Swagger API documentation
```
http://localhost:8080/api/crs/catalog/v2/swagger.json
```
or send a Get request to catalog:
```
http://localhost:8080/api/crs/catalog/v2
```
Open the Swagger-UI:
```
http://localhost:8080/api/crs/catalog/swagger-ui.html
```
Headers for Postman:
| Key | Value |
|----------|----------|
| Authorization | Bearer `<token>` |
| data-partition-id | $MY_TENANT (see [testing\README.md](testing/README.md)) |
### Build and run the Docker container locally
1. Run the `maven run` command to have the .jar file generated.
1. Have the Azure subscription set up
1. Open a Powershell
1. Install the Azure CLI locally
1. Authenticate yourself to Azure Container Registry (acr) with the following command:
```az acr login --name delfi```
1. Execute the following command to build the container image:
```docker build -t crs-catalog .```
1. Execute the following command to build the container image:
```docker run -t --rm -p 8080:8080 crs-catalog```
1. Use Postman or curl to try out the endpoints
## Release/deployment
VSTS release definition is located at provider\crs-catalog-azure\crs-catalog-aks\devops, which
requires the following environment variables:
| Variable | Contents |
|----------|----------|
| ENTITLEMENT_URL | Required |
| CRS_CATALOG_FILENAME | Required, e.g. `{path_to_the_catalog}/os-crs-catalog-service/data/crs_catalog_v2.json` |
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.opengroup.osdu.crs-catalog-service</groupId>
<artifactId>crs-catalog-service</artifactId>
<version>1.0.0</version>
</parent>
<groupId>org.opengroup.osdu.crs-catalog-service</groupId>
<artifactId>crs-catalog-core</artifactId>
<version>1.0.0</version>
<packaging>jar</packaging>
<name>crs-catalog-core</name>
<description>CRS catalog service core</description>
<properties>
<app.version>1</app.version>
<app.id>crs-catalog-core</app.id>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<objectify.version>5.1.22</objectify.version>
<slf4j.version>1.7.25</slf4j.version>
<spring.version>5.2.4.RELEASE</spring.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.showDeprecation>true</maven.compiler.showDeprecation>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<springfox-swagger2.version>2.9.2</springfox-swagger2.version>
</properties>
<prerequisites>
<maven>3.1.0</maven>
</prerequisites>
<repositories>
<repository>
<id>os-core</id>
<url>https://pkgs.dev.azure.com/slb-des-ext-collaboration/_packaging/os-core/maven/v1</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
<dependencies>
<!-- Compile/runtime dependencies -->
<dependency>
<groupId>javax.inject</groupId>
<artifactId>javax.inject</artifactId>
<version>1</version>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-config</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jetty</artifactId>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger2</artifactId>
<version>${springfox-swagger2.version}</version>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger-ui</artifactId>
<version>${springfox-swagger2.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
<groupId>javax.jdo</groupId>
<artifactId>jdo-api</artifactId>
<version>3.1</version>
</dependency>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
<version>2.8.5</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.2</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.opengroup.osdu</groupId>
<artifactId>os-core-common</artifactId>
<version>0.0.13</version>
</dependency>
<dependency>
<groupId>com.googlecode.objectify</groupId>
<artifactId>objectify</artifactId>
<version>${objectify.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4j.version}</version>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.2.3</version>
</dependency>
<!-- Test Dependencies -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.powermock</groupId>
<artifactId>powermock-module-junit4</artifactId>
<version>${powermock.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.powermock</groupId>
<artifactId>powermock-api-mockito2</artifactId>
<version>${powermock.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>3.3.3</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.codehaus.mojo</groupId>
<artifactId>cobertura-maven-plugin</artifactId>
<version>2.7</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<!-- for hot reload of the web application-->
<outputDirectory>${project.build.directory}/${project.build.finalName}/WEB-INF/classes</outputDirectory>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>versions-maven-plugin</artifactId>
<version>2.7</version>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>display-dependency-updates</goal>
<goal>display-plugin-updates</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<version>3.7.0</version>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.21.0</version>
<configuration>
<redirectTestOutputToFile>true</redirectTestOutputToFile>
<!--
<testFailureIgnore>true</testFailureIgnore>
-->
</configuration>
</plugin>
<plugin>
<groupId>org.whitesource</groupId>
<artifactId>whitesource-maven-plugin</artifactId>
<version>18.11.1</version>
<configuration>
<orgToken>${orgToken}</orgToken>
<wssUrl>https://app-eu.whitesourcesoftware.com/agent</wssUrl>
<aggregateModules>true</aggregateModules>
<product>DE</product>
<aggregateProjectName>de-crs-catalog-service</aggregateProjectName>
<forceCheckAllDependencies>true</forceCheckAllDependencies>
<checkPolicies>false</checkPolicies>
<forceUpdate>true</forceUpdate>
<failOnError>false</failOnError>
<excludes>
<exclude>integration-tests</exclude>
</excludes>
</configuration>
</plugin>
</plugins>
</build>
</project>
// Copyright 2017-2019, Schlumberger
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package org.opengroup.osdu.crs.api;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
@RestController
@RequestMapping("/_ah")
public class HealthCheck {
@GetMapping("/liveness_check")
public ResponseEntity<String> livenessCheck() {
return new ResponseEntity<String>("CRS Catalog service is alive", HttpStatus.OK);
}
@GetMapping("/readiness_check")
public ResponseEntity<String> readinessCheck() {
return new ResponseEntity<String>("CRS Catalog service is ready", HttpStatus.OK);
}
}
\ No newline at end of file
package org.opengroup.osdu.crs.api;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
@Controller
public class Home {
@RequestMapping(value = {"/", "/swagger"})
public String swagger() {
return "redirect:swagger-ui.html";
}
}
\ No newline at end of file
package org.opengroup.osdu.crs.configuration;
import org.opengroup.osdu.crs.middleware.AuthenticationRequestFilter;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.lang.Nullable;
import org.springframework.security.config.annotation.method.configuration.EnableGlobalMethodSecurity;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.builders.WebSecurity;
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter;
import org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter;
import org.springframework.web.servlet.HandlerExceptionResolver;
import static org.springframework.security.config.http.SessionCreationPolicy.STATELESS;
@EnableWebSecurity
@EnableGlobalMethodSecurity(prePostEnabled = true)
public class SecurityConfiguration extends WebSecurityConfigurerAdapter {
private static final String[] ALLOWED_URLS = {
"/",
"/v2/api-docs",
"/swagger-resources/**",
"/swagger-ui.html",
"/webjars/**",
"/_ah/**",
"/error",
"/favicon.ico",
"/csrf" // Required to prevent errors in logs while Swagger is trying to discover a valid csrf token. Should be deleted after the issue on the Swagger's side https://github.com/springfox/springfox/issues/2578 is resolved
};
private final HandlerExceptionResolver handlerExceptionResolver;
private final String entUrl;
public SecurityConfiguration(HandlerExceptionResolver handlerExceptionResolver,
@Value("${osdu.entitlement.url}") String entUrl) {
this.handlerExceptionResolver = handlerExceptionResolver;
this.entUrl = entUrl;
}
@Override
protected void configure(HttpSecurity http) throws Exception {
AuthenticationRequestFilter authhenticationFilter = new AuthenticationRequestFilter(entUrl, handlerExceptionResolver);
http.csrf().disable()
.sessionManagement().sessionCreationPolicy(STATELESS)
.and()
.authorizeRequests()
.anyRequest().authenticated()
.and()
.addFilterAfter(authhenticationFilter, UsernamePasswordAuthenticationFilter.class);
}
@Override
public void init(WebSecurity web) throws Exception {
web.ignoring().mvcMatchers(ALLOWED_URLS);
super.init(web);
}
}
package org.opengroup.osdu.crs.configuration;
import org.opengroup.osdu.core.common.model.http.DpsHeaders;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.util.List;
import springfox.documentation.builders.ParameterBuilder;
import springfox.documentation.builders.PathSelectors;
import springfox.documentation.builders.RequestHandlerSelectors;
import springfox.documentation.schema.ModelRef;
import springfox.documentation.service.ApiKey;
import springfox.documentation.service.AuthorizationScope;
import springfox.documentation.service.Parameter;
import springfox.documentation.service.SecurityReference;
import springfox.documentation.spi.DocumentationType;
import springfox.documentation.spi.service.contexts.SecurityContext;
import springfox.documentation.spring.web.plugins.Docket;
import springfox.documentation.swagger2.annotations.EnableSwagger2;
import static java.util.Collections.singletonList;
import static org.apache.commons.lang3.StringUtils.startsWithIgnoreCase;
import static org.springframework.http.HttpHeaders.AUTHORIZATION;
@Configuration
@EnableSwagger2
public class SwaggerConfiguration {
private static final String BEARER_AUTH_KEY_NAME = "Bearer Authorization";
public static final String PASS_AS_HEADER = "header";
@Bean
public Docket api(List<Parameter> globalParameters) {
return new Docket(DocumentationType.SWAGGER_2)
.select()
.apis(RequestHandlerSelectors.any())
.paths(s -> !startsWithIgnoreCase(s, "/error"))
.build()
.globalOperationParameters(globalParameters)
.securitySchemes(singletonList(bearerAuth()))
.securityContexts(singletonList(securityContext()));
}
@Bean
public Parameter dataPatitionParameter() {
ParameterBuilder builder = new ParameterBuilder();
builder.name(DpsHeaders.DATA_PARTITION_ID)
.description("tenant")
.defaultValue("opendes")
.modelRef(new ModelRef("string"))
.parameterType(PASS_AS_HEADER)
.required(true)
.build();
return builder.build();
}
private ApiKey bearerAuth() {
return new ApiKey(BEARER_AUTH_KEY_NAME, AUTHORIZATION, PASS_AS_HEADER);
}
private SecurityContext securityContext() {
return SecurityContext.builder()
.securityReferences(defaultAuth())
.forPaths(PathSelectors.any())
.build();
}
private List<SecurityReference> defaultAuth() {
AuthorizationScope[] authorizationScopes = new AuthorizationScope[0];
return singletonList(
new SecurityReference(BEARER_AUTH_KEY_NAME, authorizationScopes)
);
}
}
package org.opengroup.osdu.crs.middleware;
import org.opengroup.osdu.core.common.entitlements.EntitlementsAPIConfig;
import org.opengroup.osdu.core.common.entitlements.EntitlementsFactory;
import org.opengroup.osdu.core.common.entitlements.IEntitlementsFactory;
import org.opengroup.osdu.core.common.entitlements.IEntitlementsService