Commit 4ddd2e72 authored by Jason's avatar Jason
Browse files

Merge branch 'azure-updating-file-service' into 'master'

Azure updating file service to fix pipelines

See merge request !26
parents 89617303 7c9c3acb
Pipeline #15462 passed with stages
in 20 minutes and 45 seconds
......@@ -14,8 +14,6 @@ Apache-2.0
========================================================================
The following software have components provided under the terms of this license:
- AMQP 1.0 JMS Spring Boot AutoConfiguration (from https://repo1.maven.org/maven2/org/amqphub/spring/amqp-10-jms-spring-boot-autoconfigure)
- AMQP 1.0 JMS Spring Boot Starter (from https://repo1.maven.org/maven2/org/amqphub/spring/amqp-10-jms-spring-boot-starter)
- ASM Core (from )
- ASM based accessors helper used by json-smart (from )
- Adapter: RxJava (from )
......@@ -26,11 +24,8 @@ The following software have components provided under the terms of this license:
- Apache Commons FileUpload (from http://commons.apache.org/proper/commons-fileupload/)
- Apache Commons Lang (from http://commons.apache.org/proper/commons-lang/)
- Apache Commons Logging (from http://commons.apache.org/proper/commons-logging/)
- Apache Commons Logging (from http://commons.apache.org/proper/commons-logging/)
- Apache Commons Text (from http://commons.apache.org/proper/commons-text/)
- Apache Commons Validator (from http://commons.apache.org/proper/commons-validator/)
- Apache Geronimo JMS Spec 2.0 (from http://geronimo.apache.org/maven/${siteId}/${version})
- Apache Groovy (from http://groovy-lang.org)
- Apache Groovy (from http://groovy-lang.org)
- Apache HttpAsyncClient (from http://hc.apache.org/httpcomponents-asyncclient)
- Apache HttpClient (from http://hc.apache.org/httpcomponents-client)
......@@ -55,7 +50,6 @@ The following software have components provided under the terms of this license:
- Commons Digester (from http://commons.apache.org/digester/)
- Commons IO (from http://commons.apache.org/io/)
- Commons IO (from http://commons.apache.org/io/)
- Commons IO (from http://commons.apache.org/io/)
- Commons Lang (from http://commons.apache.org/lang/)
- Converter: Jackson (from )
- Elastic JNA Distribution (from https://github.com/java-native-access/jna)
......@@ -85,10 +79,6 @@ The following software have components provided under the terms of this license:
- Guava: Google Core Libraries for Java (from https://github.com/google/guava.git)
- HPPC Collections (from http://labs.carrotsearch.com)
- Hibernate Validator Engine (from )
- IBM COS Java SDK for Amazon S3 (from https://github.com/ibm/ibm-cos-sdk-java)
- IBM COS Java SDK for COS KMS (from https://github.com/ibm/ibm-cos-sdk-java)
- IBM COS SDK For Java (from https://github.com/ibm/ibm-cos-sdk-java)
- IBM COS SDK for Java - Core (from https://github.com/ibm/ibm-cos-sdk-java)
- J2ObjC Annotations (from https://github.com/google/j2objc/)
- J2ObjC Annotations (from https://github.com/google/j2objc/)
- JBoss Logging 3 (from http://www.jboss.org)
......@@ -99,6 +89,7 @@ The following software have components provided under the terms of this license:
- JSONassert (from https://github.com/skyscreamer/JSONassert)
- Jackson 2 extensions to the Google HTTP Client Library for Java. (from https://github.com/google/google-http-java-client.git/google-http-client-jackson2)
- Jackson dataformat: CBOR (from http://github.com/FasterXML/jackson-dataformats-binary)
- Jackson dataformat: CBOR (from http://github.com/FasterXML/jackson-dataformats-binary)
- Jackson datatype: JSR310 (from http://wiki.fasterxml.com/JacksonModuleJSR310)
- Jackson datatype: JSR310 (from http://wiki.fasterxml.com/JacksonModuleJSR310)
- Jackson-annotations (from http://github.com/FasterXML/jackson)
......@@ -122,7 +113,6 @@ The following software have components provided under the terms of this license:
- Java Native Access Platform (from https://github.com/java-native-access/jna)
- Java UUID Generator (from http://wiki.fasterxml.com/JugHome)
- Javassist (from http://www.javassist.org/)
- Javassist (from http://www.javassist.org/)
- Joda-Time (from http://www.joda.org/joda-time/)
- Json Path (from https://github.com/jayway/JsonPath)
- Lucene Common Analyzers (from )
......@@ -141,12 +131,14 @@ The following software have components provided under the terms of this license:
- Lucene Spatial Extras (from )
- Lucene Suggest (from )
- MapStruct Core (from )
- Metrics Core (from https://github.com/dropwizard/metrics)
- Microsoft Application Insights Java SDK Core (from https://github.com/Microsoft/ApplicationInsights-Java)
- Microsoft Application Insights Java SDK Spring Boot starter (from https://github.com/Microsoft/ApplicationInsights-Java)
- Microsoft Application Insights Java SDK Web Module (from https://github.com/Microsoft/ApplicationInsights-Java)
- Microsoft Application Insights Log4j 2 Appender (from https://github.com/Microsoft/ApplicationInsights-Java)
- Microsoft Azure Netty HTTP Client Library (from https://github.com/Azure/azure-sdk-for-java)
- Mockito (from http://mockito.org)
- Mockito (from http://mockito.org)
- Netty Reactive Streams Implementation (from )
- Netty/Buffer (from http://netty.io/)
- Netty/Buffer (from http://netty.io/)
......@@ -169,7 +161,6 @@ The following software have components provided under the terms of this license:
- Netty/Transport (from http://netty.io/)
- Netty/Transport/Native/Unix/Common (from )
- Nimbus JOSE+JWT (from https://bitbucket.org/connect2id/nimbus-jose-jwt)
- Nimbus JOSE+JWT (from https://bitbucket.org/connect2id/nimbus-jose-jwt)
- Nimbus LangTag (from https://bitbucket.org/connect2id/nimbus-language-tags)
- Non-Blocking Reactive Foundation for the JVM (from https://github.com/reactor/reactor)
- OAuth 2.0 SDK with OpenID Connect extensions (from https://bitbucket.org/connect2id/oauth-2.0-sdk-with-openid-connect-extensions)
......@@ -188,7 +179,6 @@ The following software have components provided under the terms of this license:
- OpenCensus (from https://github.com/census-instrumentation/opencensus-java)
- PowerMock (from http://www.powermock.org)
- Protocol Buffer extensions to the Google HTTP Client Library for Java. (from )
- QpidJMS Client (from )
- Reactive Object Pool (from https://github.com/reactor/reactor-pool)
- Reactive Streams Netty driver (from https://github.com/reactor/reactor-netty)
- Retrofit (from )
......@@ -241,8 +231,6 @@ The following software have components provided under the terms of this license:
- Spring Data Core (from )
- Spring Expression Language (SpEL) (from https://github.com/spring-projects/spring-framework)
- Spring Expression Language (SpEL) (from https://github.com/spring-projects/spring-framework)
- Spring JMS (from https://github.com/spring-projects/spring-framework)
- Spring Messaging (from https://github.com/spring-projects/spring-framework)
- Spring TestContext Framework (from https://github.com/spring-projects/spring-framework)
- Spring Transaction (from https://github.com/spring-projects/spring-framework)
- Spring Web (from https://github.com/spring-projects/spring-framework)
......@@ -278,11 +266,8 @@ The following software have components provided under the terms of this license:
- io.grpc:grpc-protobuf (from https://github.com/grpc/grpc-java)
- io.grpc:grpc-protobuf-lite (from https://github.com/grpc/grpc-java)
- io.grpc:grpc-stub (from https://github.com/grpc/grpc-java)
- ion-java (from https://github.com/amznlabs/ion-java/)
- jackson-databind (from http://github.com/FasterXML/jackson)
- jackson-databind (from http://github.com/FasterXML/jackson)
- java-cloudant (from https://cloudant.com)
- java-cloudant (from https://cloudant.com)
- javax.inject (from http://code.google.com/p/atinject/)
- jersey-core (from )
- jsr311-api (from https://jsr311.dev.java.net)
......@@ -298,16 +283,11 @@ The following software have components provided under the terms of this license:
- parent-join (from https://github.com/elastic/elasticsearch)
- parent-join (from https://github.com/elastic/elasticsearch)
- perfmark:perfmark-api (from https://github.com/perfmark/perfmark)
- powermock-api-support (from )
- powermock-core (from http://www.powermock.org)
- powermock-module-junit4 (from http://www.powermock.org)
- powermock-module-junit4-common (from )
- powermock-reflect (from )
- proto-google-cloud-datastore-v1 (from https://github.com/googleapis/api-client-staging)
- proto-google-cloud-firestore-admin-v1 (from https://repo1.maven.org/maven2/com/google/api/grpc/proto-google-cloud-firestore-admin-v1)
- proto-google-cloud-firestore-v1 (from https://repo1.maven.org/maven2/com/google/api/grpc/proto-google-cloud-firestore-v1)
- proton-j (from )
- proton-j (from )
- rank-eval (from https://github.com/elastic/elasticsearch)
- rank-eval (from https://github.com/elastic/elasticsearch)
- rest (from https://github.com/elastic/elasticsearch)
......@@ -323,9 +303,6 @@ The following software have components provided under the terms of this license:
- spring-security-config (from http://spring.io/spring-security)
- spring-security-core (from http://spring.io/spring-security)
- spring-security-crypto (from http://spring.io/spring-security)
- spring-security-oauth2-core (from http://spring.io/spring-security)
- spring-security-oauth2-jose (from http://spring.io/spring-security)
- spring-security-oauth2-resource-server (from http://spring.io/spring-security)
- spring-security-rsa (from http://github.com/spring-projects/spring-security-oauth)
- spring-security-test (from http://spring.io/spring-security)
- spring-security-web (from http://spring.io/spring-security)
......@@ -451,6 +428,7 @@ The following software have components provided under the terms of this license:
- AspectJ weaver (from http://www.aspectj.org)
- JUnit (from http://junit.org)
- JUnit (from http://junit.org)
- JUnit Jupiter (Aggregator) (from https://junit.org/junit5/)
- Logback Classic Module (from )
- Logback Core Module (from )
......@@ -536,7 +514,6 @@ The following software have components provided under the terms of this license:
- Java Native Access (from https://github.com/java-native-access/jna)
- Java Native Access Platform (from https://github.com/java-native-access/jna)
- Javassist (from http://www.javassist.org/)
- Javassist (from http://www.javassist.org/)
- Logback Classic Module (from )
- Logback Core Module (from )
- Microsoft Application Insights Java SDK Core (from https://github.com/Microsoft/ApplicationInsights-Java)
......@@ -551,7 +528,6 @@ The following software have components provided under the terms of this license:
- Java Native Access (from https://github.com/java-native-access/jna)
- Java Native Access Platform (from https://github.com/java-native-access/jna)
- Javassist (from http://www.javassist.org/)
- SnakeYAML (from http://www.snakeyaml.org)
========================================================================
......@@ -608,6 +584,7 @@ The following software have components provided under the terms of this license:
- Microsoft Azure client library for KeyVault Secrets (from https://github.com/Azure/azure-sdk-for-java)
- Microsoft Azure common module for Storage (from https://github.com/Azure/azure-sdk-for-java)
- Mockito (from http://mockito.org)
- Mockito (from http://mockito.org)
- Netty/Codec/HTTP (from )
- Netty/Common (from )
- Netty/Common (from )
......@@ -626,7 +603,6 @@ MPL-1.1
========================================================================
The following software have components provided under the terms of this license:
- Javassist (from http://www.javassist.org/)
- Javassist (from http://www.javassist.org/)
========================================================================
......@@ -634,7 +610,6 @@ MPL-2.0
========================================================================
The following software have components provided under the terms of this license:
- Javassist (from http://www.javassist.org/)
- Javassist (from http://www.javassist.org/)
- OkHttp (from )
......@@ -675,7 +650,6 @@ public-domain
========================================================================
The following software have components provided under the terms of this license:
- Apache Groovy (from http://groovy-lang.org)
- Asynchronous Http Client (from )
- Bouncy Castle PKIX, CMS, EAC, TSP, PKCS, OCSP, CMP, and CRMF APIs (from http://www.bouncycastle.org/java.html)
- Bouncy Castle Provider (from http://www.bouncycastle.org/java.html)
......
......@@ -58,7 +58,7 @@ spec:
value: "80"
- name: ACCEPT_HTTP # TEMPORARY UNTIL HTTPS
value: "true"
- name: KEYVAULT_URI
- name: KEYVAULT_URL
valueFrom:
configMapKeyRef:
name: osdu-svc-properties
......@@ -66,43 +66,43 @@ spec:
- name: AZURE_CLIENT_ID
valueFrom:
secretKeyRef:
name: clientid
key: clientid
name: active-directory
key: principal-clientid
- name: AZURE_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: clientpassword
key: clientpassword
name: active-directory
key: principal-clientpassword
- name: AZURE_AD_APP_RESOURCE_ID
valueFrom:
secretKeyRef:
name: appid
key: appid
name: active-directory
key: application-appid
- name: AZURE_TENANT_ID
valueFrom:
configMapKeyRef:
name: osdu-svc-properties
key: ENV_TENANT_ID
secretKeyRef:
name: active-directory
key: tenantid
- name: appinsights_key
valueFrom:
secretKeyRef:
name: appinsights
name: central-logging
key: appinsights
- name: AZURE_STORAGE_ACCOUNT
valueFrom:
configMapKeyRef:
name: osdu-svc-properties
key: ENV_STORAGE_ACCOUNT
secretKeyRef:
name: storage
key: storage-account
- name: cosmosdb_account
valueFrom:
configMapKeyRef:
name: osdu-svc-properties
key: ENV_COSMOSDB_HOST
secretKeyRef:
name: cosmos
key: cosmos-endpoint
- name: cosmosdb_key
valueFrom:
secretKeyRef:
name: cosmos
key: cosmos
key: cosmos-primary-key
- name: cosmosdb_database
value: osdu-db
- name: LOG_PREFIX
......@@ -123,6 +123,3 @@ spec:
value: opendes
- name: osdu_storage_url
value: http://storage/api/storage/v2
- name: keyvault_url
value: $(KEYVAULT_URI)
......@@ -42,6 +42,9 @@ public class AuthorizationFilter {
* otherwise false
*/
public boolean hasPermission(String... requiredRoles) {
if(headers.getPartitionId() == null){
return false;
}
AuthorizationResponse authResponse = authorizationService.authorizeAny(headers, requiredRoles);
headers.put(DpsHeaders.USER_EMAIL, authResponse.getUser());
return true;
......
# file-azure
file-azure is a [Spring Boot](https://spring.io/projects/spring-boot) service that provides internal and external API endpoints to let the application or user fetch any records from the system or request file location data.
file-azure is a [Spring Boot](https://spring.io/projects/spring-boot) service that provides internal and external API endpoints to let the application or user fetch any records from the system or request file location data.
For example, users can request generation of an individual signed URL per file. Using a signed URL, OSDU R2 users will be able to upload their files to the system.
## Running Locally
......@@ -45,17 +45,18 @@ az keyvault secret show --vault-name $KEY_VAULT_NAME --name $KEY_VAULT_SECRET_NA
| `AZURE_CLIENT_ID` | `********` | Identity to run the service locally. This enables access to Azure resources. You only need this if running locally | yes | keyvault secret: `$KEYVAULT_URI/secrets/app-dev-sp-username` |
| `AZURE_TENANT_ID` | `********` | AD tenant to authenticate users from | yes | -- |
| `AZURE_CLIENT_SECRET` | `********` | Secret for `$AZURE_CLIENT_ID` | yes | keyvault secret: `$KEYVAULT_URI/secrets/app-dev-sp-password` |
| `KEYVAULT_URI` | ex `https://foo-keyvault.vault.azure.net/` | URI of KeyVault that holds application secrets | no | output of infrastructure deployment |
| `KEYVAULT_URL` | ex `https://foo-keyvault.vault.azure.net/` | URI of KeyVault that holds application secrets | no | output of infrastructure deployment |
| `appinsights_key` | `********` | API Key for App Insights | yes | output of infrastructure deployment |
| `cosmosdb_database` | ex `foo-db` | The name of the CosmosDB database | no | output of infrastructure deployment |
| `cosmosdb_key` | `********` | Key for CosmosDB | yes | output of infrastructure deployments |
| `cosmosdb_account` | ex `devintosdur2cosmosacct` | Cosmos account name | no | output of infrastructure deployment |
| `SEARCH_HOST` | `https://search.azurewebsites.net/api/search/v2/query` | API endpoint for the search query endpoint | no | output of infrastructure deployment + path to the query endpoint |
| `AZURE_AD_APP_RESOURCE_ID` | `********` | AAD client application ID | yes | output of infrastructure deployment |
| `osdu_entitlements_url` | ex `https://foo-entitlements.azurewebsites.net` | Entitlements API endpoint | no | output of infrastructure deployment |
| `osdu_entitlements_app_key` | `********` | The API key clients will need to use when calling the entitlements | yes | -- |
| `osdu_entitlements_url` | ex `https://foo-osdu.msft-osdu-test.org/entitlements/v1` | Entitlements API endpoint | no | output of infrastructure deployment |
| `osdu_entitlements_app_key` | `OBSOLETE` | This is deprecated | no | -- |
| `spring.application.name` | `file-azure` | Name of application. Needed by App Insights | no | -- |
| `osdu_storage_url` | `https://storage.azurewebsites.net/api/storage/v2` | Storage API endpoint | no | -- |
| `AZURE_STORAGE_ACCOUNT` | ex `foo-storage-account` | Storage account for storing documents | no | output of infrastructure deployment |
| `server_port` | ex `8082` | Port the service will run on | no | -- |
**Required to run integration tests**
......@@ -69,9 +70,10 @@ az keyvault secret show --vault-name $KEY_VAULT_NAME --name $KEY_VAULT_SECRET_NA
| `AZURE_AD_APP_RESOURCE_ID` | `********` | AAD client application ID | yes | output of infrastructure deployment |
| `NO_DATA_ACCESS_TESTER` | `********` | Service principal ID of a service principal without entitlements | yes | `aad-no-data-access-tester-client-id` secret from keyvault |
| `NO_DATA_ACCESS_TESTER_SERVICEPRINCIPAL_SECRET` | `********` | Secret for `$NO_DATA_ACCESS_TESTER` | yes | `aad-no-data-access-tester-secret` secret from keyvault |
| `azure.storage.account-name` | ex `foo-storage-account` | Storage account for storing documents | no | output of infrastructure deployment |
| `AZURE_STORAGE_ACCOUNT` | ex `foo-storage-account` | Storage account for storing documents | no | output of infrastructure deployment |
| `USER_ID` | osdu-user | User ID | no | - |
| `EXIST_FILE_ID` | ex '****' | Existing file Id should be added | no | - |
| `TIME_ZONE` | `UTC+0` | Time zone required for tests to pass | yes | - |
......@@ -86,27 +88,6 @@ Java version: 1.8.0_212, vendor: AdoptOpenJDK, runtime: /usr/lib/jvm/jdk8u212-b0
...
```
You may need to configure access to the remote maven repository that holds the OSDU dependencies. A default file should live within `~/.m2/settings.xml`:
```bash
$ cat ~/.m2/settings.xml
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
<servers>
<server>
<id>file-core</id>
<username>mvn-pat</username>
<!-- Treat this auth token like a password. Do not share it with anyone, including Microsoft support. -->
<!-- The generated token expires on or before 11/14/2019 -->
<password>$PERSONAL_ACCESS_TOKEN_GOES_HERE</password>
</server>
</servers>
</settings>
```
_A settings file is also conveniently located in ./.mvn/community-maven.settings.xml which is also used for CI/CD processes._
### Build, Run and Test the application Locally
After configuring your environment as specified above, you can follow these steps to build and run the application
......
......@@ -66,13 +66,13 @@ public class AzureBootstrapConfig {
@Bean
@Named("COSMOS_ENDPOINT")
public String cosmosEndpoint(SecretClient kv) {
return getKeyVaultSecret(kv, "cosmos-endpoint");
return getKeyVaultSecret(kv, "opendes-cosmos-endpoint");
}
@Bean
@Named("COSMOS_KEY")
public String cosmosKey(SecretClient kv) {
return getKeyVaultSecret(kv, "cosmos-primary-key");
return getKeyVaultSecret(kv, "opendes-cosmos-primary-key");
}
String getKeyVaultSecret(SecretClient kv, String secretName) {
......
......@@ -23,7 +23,7 @@ azure.cosmosdb.database=${cosmosdb_database}
azure.keyvault.url=${keyvault_url}
# Azure Blob Storage configuration
file.location.containerName=osdu
file.location.containerName=opendes
file.location.userId=osdu-user
azure.storage.account=${AZURE_STORAGE_ACCOUNT}
......@@ -45,7 +45,7 @@ logging.level.org.springframework.web=DEBUG
# Server configuration
server.servlet.contextPath=/api/file/v2/
server.port=8080
server.port=${server_port}
# Java configuration
JAVA_HEAP_OPTS=-Xms4096M -Xmx4096M
......
......@@ -169,7 +169,14 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.4.2</version>
<version>2.22.2</version>
<dependencies>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<version>5.5.2</version>
</dependency>
</dependencies>
<executions>
<execution>
<phase>integration-test</phase>
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment