diff --git a/README.md b/README.md index e80218e8823f10ff9a5c1059ebd43545e601f460..256aec03d58f0b9d037f6a1bfd0fb0aeb01eb685 100644 --- a/README.md +++ b/README.md @@ -1,138 +1,15 @@ # Introduction -The Partition Service dynamically pulls the correct connection information at runtime to connect to the correct partition. It is a Maven multi-module project with each cloud implemention placed in its submodule. - - +The Partition Service dynamically pulls the correct connection information at runtime to connect to the correct partition. It is a Maven multi-module project with each cloud implementation placed in its submodule. + ## Running Locally - AWS +## Running Locally - Azure +Instructions for running the Azure implementation locally can be found [here](./provider/partition-azure/README.md) ## Running Locally - GCP ## Running Locally - IBM -## Running Locally - Azure - -### Requirements - -In order to run this service locally, you will need the following: - -- [Maven 3.6.0+](https://maven.apache.org/download.cgi) -- Infrastructure dependencies, deployable through the relevant [os-k8s](https:/update/_git/os-k8s) -- While not a strict dependency, example commands in this document use [bash](https://www.gnu.org/software/bash/) - -### General Tips - -**Environment Variable Management** -The following tools make environment variable configuration simpler - - [direnv](https://direnv.net/) - for a shell/terminal environment - - [EnvFile](https://plugins.jetbrains.com/plugin/7861-envfile) - for [Intellij IDEA](https://www.jetbrains.com/idea/) - -**Lombok** -This project uses [Lombok](https://projectlombok.org/) for code generation. You may need to configure your IDE to take advantage of this tool. - - [Intellij configuration](https://projectlombok.org/setup/intellij) - - [VSCode configuration](https://projectlombok.org/setup/vscode) - -### Environment Variables - -In order to run the service locally, you will need to have the following environment variables defined. - -**Note** The following command can be useful to pull secrets from keyvault: -```bash -az keyvault secret show --vault-name $KEY_VAULT_NAME --name $KEY_VAULT_SECRET_NAME --query value -otsv -``` - -**Required to run service** - -| name | value | description | sensitive? | source | -| --- | --- | --- | --- | --- | -| `client-id` | `********` | AAD client application ID | yes | output of infrastructure deployment | -| `KEYVAULT_URI` | (non-secret) | KeyVault URI | no | variable `AZURE_KEYVAULT_URI` from GitLab variable group `Azure Target Env - {{env}}` -| `AZURE_CLIENT_ID` | `********` | Identity to run the service locally. This enables access to Azure resources. You only need this if running locally | yes | keyvault secret: `$KEYVAULT_URI/secrets/app-dev-sp-username` | -| `AZURE_TENANT_ID` | `********` | AD tenant to authenticate users from | yes | keyvault secret: `$KEYVAULT_URI/secrets/app-dev-sp-tenant-id` | -| `AZURE_CLIENT_SECRET` | `********` | Secret for `$AZURE_CLIENT_ID` | yes | keyvault secret: `$KEYVAULT_URI/secrets/app-dev-sp-password` | - -### Configure Maven - -Check that maven is installed: -```bash -$ mvn --version -Apache Maven 3.6.0 -Maven home: /usr/share/maven -Java version: 1.8.0_212, vendor: AdoptOpenJDK, runtime: /usr/lib/jvm/jdk8u212-b04/jre -... -``` - -You will need to configure access to the remote maven repository that holds the OSDU dependencies. This file should live within `~/.m2/settings.xml`: -```bash -<?xml version="1.0" encoding="UTF-8"?> -<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" - xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" - xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd"> - <servers> - <server> - <id>os-core</id> - <username>os-core</username> - <password>${VSTS_FEED_TOKEN}</password> - </server> - </servers> -</settings> -``` - -### Build and run the application - -After configuring your environment as specified above, you can follow these steps to build and run the application. These steps should be invoked from the *repository root.* - -```bash -# build + test + install core service code -$ mvn clean install - -# build + test + package azure service code -$ mvn clean package -P partition-aks - -# run service -# -# Note: this assumes that the environment variables for running the service as outlined -# above are already exported in your environment. -$ cd provider/partition-azure && mvn spring-boot:run -f pom.xml -``` - -### Test the application - -After the service has started it should be accessible via a web browser by visiting [http://localhost:8080/api/partition/v1/swagger-ui.html](http://localhost:8080/api/partition/v1/swagger-ui.html). If the request does not fail, you can then run the integration tests. - -```bash -# build + install integration test core -$ (cd testing/storage-test-core/ && mvn clean install) - -# build + run Azure integration tests. -# -# Note: this assumes that the environment variables for integration tests as outlined -# above are already exported in your environment. -$ (cd testing/storage-test-azure/ && mvn clean test) -``` - -Send a Get request to partition: -``` -http://localhost:8080/api/partition/v1/partitions/tenant2 -``` -Headers for Postman: -| Key | Value | -|----------|----------| -| Authorization | Bearer `<token>` | - -## Debugging - -Jet Brains - the authors of Intellij IDEA, have written an [excellent guide](https://www.jetbrains.com/help/idea/debugging-your-first-java-application.html) on how to debug java programs. - -### Build and run the Docker container locally -1. Run the `maven run` command to have the .jar file generated. -1. Have the Azure subscription set up -1. Open a Powershell -1. Install the Azure CLI locally -1. Authenticate yourself to Azure Container Registry (acr) with the following command: -```az acr login --name delfi``` -1. Execute the following command to build the container image: -```docker build -t partition .``` -1. Execute the following command to build the container image: -```docker run -t --rm -p 8080:8080 partition``` -1. Use Postman or curl to try out the endpoints +## Running Integration Tests +Instructions for running the integration tests can be found [here](./testing/README.md) ## License Copyright 2017-2020, Schlumberger diff --git a/provider/partition-azure/README.md b/provider/partition-azure/README.md new file mode 100644 index 0000000000000000000000000000000000000000..17894d75513d3569ff0c067a054fdd5e3a1097df --- /dev/null +++ b/provider/partition-azure/README.md @@ -0,0 +1,125 @@ +# Running Locally - Azure +## Requirements + +In order to run this service locally, you will need the following: + +- [Maven 3.6.0+](https://maven.apache.org/download.cgi) +- [Docker](https://www.docker.com/), to start a redis server locally + +## General Tips + +**Environment Variable Management** +The following tools make environment variable configuration simpler + - [direnv](https://direnv.net/) - for a shell/terminal environment + - [EnvFile](https://plugins.jetbrains.com/plugin/7861-envfile) - for [Intellij IDEA](https://www.jetbrains.com/idea/) + +**Lombok** +This project uses [Lombok](https://projectlombok.org/) for code generation. You may need to configure your IDE to take advantage of this tool. + - [Intellij configuration](https://projectlombok.org/setup/intellij) + - [VSCode configuration](https://projectlombok.org/setup/vscode) + +### Environment Variables + +In order to run the service locally, you will need to have the following environment variables defined. + +**Note** The following command can be useful to pull secrets from keyvault: +```bash +az keyvault secret show --vault-name $KEY_VAULT_NAME --name $KEY_VAULT_SECRET_NAME --query value -otsv +``` + +**Required to run service** + +| name | value | description | sensitive? | source | +| --- | --- | --- | --- | --- | +| `client-id` | `********` | AAD client application ID | yes | output of infrastructure deployment | +| `KEYVAULT_URI` | (non-secret) | KeyVault URI | no | variable `AZURE_KEYVAULT_URI` from GitLab variable group `Azure Target Env - {{env}}` +| `appinsights_key` | `********` | Application Insights Instrumentation Key, required to hook AppInsights with Partition application | yes | keyvault secret: `$KEYVAULT_URI/secrets/appinsights-key` | +| `AZURE_CLIENT_ID` | `********` | Identity to run the service locally. This enables access to Azure resources. You only need this if running locally | yes | keyvault secret: `$KEYVAULT_URI/secrets/app-dev-sp-username` | +| `AZURE_TENANT_ID` | `********` | AD tenant to authenticate users from | yes | keyvault secret: `$KEYVAULT_URI/secrets/app-dev-sp-tenant-id` | +| `AZURE_CLIENT_SECRET` | `********` | Secret for `$AZURE_CLIENT_ID` | yes | keyvault secret: `$KEYVAULT_URI/secrets/app-dev-sp-password` | + +### Configure Maven + +Check that maven is installed: +```bash +$ mvn --version +Apache Maven 3.6.0 +Maven home: /usr/share/maven +Java version: 1.8.0_212, vendor: AdoptOpenJDK, runtime: /usr/lib/jvm/jdk8u212-b04/jre +... +``` + +You will need to configure access to the remote maven repository that holds the OSDU dependencies. This file should live within `~/.m2/settings.xml`: +```bash +<?xml version="1.0" encoding="UTF-8"?> +<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd"> + <servers> + <server> + <id>os-core</id> + <username>os-core</username> + <password>${VSTS_FEED_TOKEN}</password> + </server> + </servers> +</settings> +``` + +### Start a Redis instance + +Start a redis server locally with `docker run --name some-redis -p 6379:6379 -d redis` command + +### Build and run the application + +After configuring your environment as specified above, you can follow these steps to build and run the application. These steps should be invoked from the *repository root.* + +```bash +# build + test + install core service code +$ mvn clean install + +# build + test + package azure service code +$ mvn clean package -P partition-aks + +# run service +# +# Note: this assumes that the environment variables for running the service as outlined +# above are already exported in your environment. +$ cd provider/partition-azure && mvn spring-boot:run -f pom.xml +``` + +### Test the application + +After the service has started it should be accessible via a web browser by visiting [http://localhost:8080/api/partition/v1/swagger-ui.html](http://localhost:8080/api/partition/v1/swagger-ui.html). If the request does not fail, you can then run the integration tests. + +see [instructions](../../testing/README.md) on how to run integration tests locally. In addition to common testing environment variables, the `partition-test-azure` module also needs additional environment variables, which are described below: + + ``` + INTEGRATION_TESTER (app-dev-sp-username) + TESTER_SERVICEPRINCIPAL_SECRET (app-dev-sp-password) + NO_DATA_ACCESS_TESTER (aad-no-data-access-tester-client-id) + NO_DATA_ACCESS_TESTER_SERVICEPRINCIPAL_SECRET (aad-no-data-access-tester-secret) + AZURE_AD_TENANT_ID (azure tenant id) + AZURE_AD_APP_RESOURCE_ID (aad-client-id) + AZURE_AD_OTHER_APP_RESOURCE_ID (AD Application ID used for negative testing) + ``` + +A liveness check can also be performed at `http://localhost:8080/api/partition/v1/_ah/liveness_check`. Other apis can be found on the swagger page + +## Debugging + +Jet Brains - the authors of Intellij IDEA, have written an [excellent guide](https://www.jetbrains.com/help/idea/debugging-your-first-java-application.html) on how to debug java programs. + +## License +Copyright 2017-2020, Schlumberger + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0) + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. \ No newline at end of file diff --git a/provider/partition-azure/devops/stages.yml b/provider/partition-azure/devops/stages.yml index 8ea1600127379a49d207d091d45dbf958167f77e..c378b117a0c2d1c6beb5076964916f05edb365f3 100644 --- a/provider/partition-azure/devops/stages.yml +++ b/provider/partition-azure/devops/stages.yml @@ -89,6 +89,16 @@ stages: zip -r testing.zip testing cp testing.zip $(build.artifactstagingdirectory) displayName: 'Zip integration tests and upload it as artifact' + - bash: | + #!/bin/bash + set -e + # Upload redis deployment scripts as artifact + pushd provider/partition-azure/redis-deployment + cp master-deployment.yaml $(build.artifactstagingdirectory) + cp redis-service.yaml $(build.artifactstagingdirectory) + cp sentinel-deployment.yaml $(build.artifactstagingdirectory) + cp slave-deployment.yaml $(build.artifactstagingdirectory) + popd - upload: $(Build.ArtifactStagingDirectory) artifact: drop_${{environment.name}} @@ -142,6 +152,19 @@ stages: set -e kubectl --kubeconfig $(kubeConfig.secureFilePath) rollout restart deployment/partition-aks displayName: 'kubectl rollout restart deployment/partition-aks' + - bash: | + #!/bin/bash + set -e + pushd $(Pipeline.Workspace)/drop_${{environment.name}} + kubectl --kubeconfig $(kubeConfig.secureFilePath) apply -f redis-service.yaml + sleep 5 + kubectl --kubeconfig $(kubeConfig.secureFilePath) apply -f master-deployment.yaml + sleep 5 + kubectl --kubeconfig $(kubeConfig.secureFilePath) apply -f sentinel-deployment.yaml + sleep 5 + kubectl --kubeconfig $(kubeConfig.secureFilePath) apply -f slave-deployment.yaml + popd + displayName: 'deploy redis cluster' - ${{ if or(eq(environment.name, 'devint'), eq(environment.name, 'qa')) }}: - bash: | #!/bin/bash diff --git a/testing/README.md b/testing/README.md index 166f51825a16358aec7a19d42c4cccd64d73f38c..6442bf70ad861374443a8fc21b70f953ac6f3471 100644 --- a/testing/README.md +++ b/testing/README.md @@ -20,16 +20,16 @@ Partition integration tests are refactored so that the business logic for integr * JDK8 * Maven * Azure Devops access to slb-des-ext-collaboration organization. You need to generate a PAT that can access dependencies held in the Azure artifacts -* Values for the following environment variables in Config.java +* Values for the following environment variables in Config.java (in `partition-test-core` module) ``` ENVIRONMENT ('local' for local testing or 'dev' for dev testing) - PARTITION_BASE_URL(service base URL ) - + PARTITION_BASE_URL (service base URL, required only if running integration tests against a cloud endpoint) + CLIENT_TENANT (name of the client partition, required only if running integration tests against a cloud endpoint) (e.g. 'common') + MY_TENANT(name of the OSDU partition, required only if running integration tests against a cloud endpoint) (e.g. 'opendes') ``` Above variables should be configured in the release pipeline to run integration tests. You should also replace them with proper values if you wish to run tests locally. - ### Commands to run tests * Integration tests are refactored into two pieces: Core and Provider. Core contains business logic for tests and is a dependency for executing the tests from provider module. To build the core module, simply navigate to `partition-test-core` directory and run `mvn clean install`. This will build the core module * Next, to execute the integration tests, navigate to the provider module and execute `mvn test`