Skip to content
Snippets Groups Projects
Commit 9d09cabd authored by Alok Joshi's avatar Alok Joshi
Browse files

add azure readme, add redis deployment scripts

parent 91c497e8
No related branches found
No related tags found
1 merge request!3Add azure readme, add redis deployment scripts
Pipeline #8156 failed
Stage: review
Hide whitespace changes
Inline Side-by-side
README.md
+ 6
− 129
View file @ 9d09cabd
# Introduction
The Partition Service dynamically pulls the correct connection information at runtime to connect to the correct partition. It is a Maven multi-module project with each cloud implemention placed in its submodule.
The Partition Service dynamically pulls the correct connection information at runtime to connect to the correct partition. It is a Maven multi-module project with each cloud implementation placed in its submodule.
## Running Locally - AWS
## Running Locally - Azure
Instructions for running the Azure implementation locally can be found [here](./provider/partition-azure/README.md)
## Running Locally - GCP
## Running Locally - IBM
## Running Locally - Azure
### Requirements
In order to run this service locally, you will need the following:
- [Maven 3.6.0+](https://maven.apache.org/download.cgi)
- Infrastructure dependencies, deployable through the relevant [os-k8s](https:/update/_git/os-k8s)
- While not a strict dependency, example commands in this document use [bash](https://www.gnu.org/software/bash/)
### General Tips
**Environment Variable Management**
The following tools make environment variable configuration simpler
- [direnv](https://direnv.net/) - for a shell/terminal environment
- [EnvFile](https://plugins.jetbrains.com/plugin/7861-envfile) - for [Intellij IDEA](https://www.jetbrains.com/idea/)
**Lombok**
This project uses [Lombok](https://projectlombok.org/) for code generation. You may need to configure your IDE to take advantage of this tool.
- [Intellij configuration](https://projectlombok.org/setup/intellij)
- [VSCode configuration](https://projectlombok.org/setup/vscode)
### Environment Variables
In order to run the service locally, you will need to have the following environment variables defined.
**Note** The following command can be useful to pull secrets from keyvault:
```bash
az keyvault secret show --vault-name $KEY_VAULT_NAME --name $KEY_VAULT_SECRET_NAME --query value -otsv
```
**Required to run service**
| name | value | description | sensitive? | source |
| --- | --- | --- | --- | --- |
| `client-id` | `********` | AAD client application ID | yes | output of infrastructure deployment |
| `KEYVAULT_URI` | (non-secret) | KeyVault URI | no | variable `AZURE_KEYVAULT_URI` from GitLab variable group `Azure Target Env - {{env}}`
| `AZURE_CLIENT_ID` | `********` | Identity to run the service locally. This enables access to Azure resources. You only need this if running locally | yes | keyvault secret: `$KEYVAULT_URI/secrets/app-dev-sp-username` |
| `AZURE_TENANT_ID` | `********` | AD tenant to authenticate users from | yes | keyvault secret: `$KEYVAULT_URI/secrets/app-dev-sp-tenant-id` |
| `AZURE_CLIENT_SECRET` | `********` | Secret for `$AZURE_CLIENT_ID` | yes | keyvault secret: `$KEYVAULT_URI/secrets/app-dev-sp-password` |
### Configure Maven
Check that maven is installed:
```bash
$ mvn --version
Apache Maven 3.6.0
Maven home: /usr/share/maven
Java version: 1.8.0_212, vendor: AdoptOpenJDK, runtime: /usr/lib/jvm/jdk8u212-b04/jre
...
```
You will need to configure access to the remote maven repository that holds the OSDU dependencies. This file should live within `~/.m2/settings.xml`:
```bash
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
<servers>
<server>
<id>os-core</id>
<username>os-core</username>
<password>${VSTS_FEED_TOKEN}</password>
</server>
</servers>
</settings>
```
### Build and run the application
After configuring your environment as specified above, you can follow these steps to build and run the application. These steps should be invoked from the *repository root.*
```bash
# build + test + install core service code
$ mvn clean install
# build + test + package azure service code
$ mvn clean package -P partition-aks
# run service
#
# Note: this assumes that the environment variables for running the service as outlined
# above are already exported in your environment.
$ cd provider/partition-azure && mvn spring-boot:run -f pom.xml
```
### Test the application
After the service has started it should be accessible via a web browser by visiting [http://localhost:8080/api/partition/v1/swagger-ui.html](http://localhost:8080/api/partition/v1/swagger-ui.html). If the request does not fail, you can then run the integration tests.
```bash
# build + install integration test core
$ (cd testing/storage-test-core/ && mvn clean install)
# build + run Azure integration tests.
#
# Note: this assumes that the environment variables for integration tests as outlined
# above are already exported in your environment.
$ (cd testing/storage-test-azure/ && mvn clean test)
```
Send a Get request to partition:
```
http://localhost:8080/api/partition/v1/partitions/tenant2
```
Headers for Postman:
| Key | Value |
|----------|----------|
| Authorization | Bearer `<token>` |
## Debugging
Jet Brains - the authors of Intellij IDEA, have written an [excellent guide](https://www.jetbrains.com/help/idea/debugging-your-first-java-application.html) on how to debug java programs.
### Build and run the Docker container locally
1. Run the `maven run` command to have the .jar file generated.
1. Have the Azure subscription set up
1. Open a Powershell
1. Install the Azure CLI locally
1. Authenticate yourself to Azure Container Registry (acr) with the following command:
```az acr login --name delfi```
1. Execute the following command to build the container image:
```docker build -t partition .```
1. Execute the following command to build the container image:
```docker run -t --rm -p 8080:8080 partition```
1. Use Postman or curl to try out the endpoints
## Running Integration Tests
Instructions for running the integration tests can be found [here](./testing/README.md)
## License
Copyright 2017-2020, Schlumberger
......
provider/partition-azure/README.md 0 → 100644
+ 125
− 0
View file @ 9d09cabd
# Running Locally - Azure
## Requirements
In order to run this service locally, you will need the following:
- [Maven 3.6.0+](https://maven.apache.org/download.cgi)
- [Docker](https://www.docker.com/), to start a redis server locally
## General Tips
**Environment Variable Management**
The following tools make environment variable configuration simpler
- [direnv](https://direnv.net/) - for a shell/terminal environment
- [EnvFile](https://plugins.jetbrains.com/plugin/7861-envfile) - for [Intellij IDEA](https://www.jetbrains.com/idea/)
**Lombok**
This project uses [Lombok](https://projectlombok.org/) for code generation. You may need to configure your IDE to take advantage of this tool.
- [Intellij configuration](https://projectlombok.org/setup/intellij)
- [VSCode configuration](https://projectlombok.org/setup/vscode)
### Environment Variables
In order to run the service locally, you will need to have the following environment variables defined.
**Note** The following command can be useful to pull secrets from keyvault:
```bash
az keyvault secret show --vault-name $KEY_VAULT_NAME --name $KEY_VAULT_SECRET_NAME --query value -otsv
```
**Required to run service**
| name | value | description | sensitive? | source |
| --- | --- | --- | --- | --- |
| `client-id` | `********` | AAD client application ID | yes | output of infrastructure deployment |
| `KEYVAULT_URI` | (non-secret) | KeyVault URI | no | variable `AZURE_KEYVAULT_URI` from GitLab variable group `Azure Target Env - {{env}}`
| `appinsights_key` | `********` | Application Insights Instrumentation Key, required to hook AppInsights with Partition application | yes | keyvault secret: `$KEYVAULT_URI/secrets/appinsights-key` |
| `AZURE_CLIENT_ID` | `********` | Identity to run the service locally. This enables access to Azure resources. You only need this if running locally | yes | keyvault secret: `$KEYVAULT_URI/secrets/app-dev-sp-username` |
| `AZURE_TENANT_ID` | `********` | AD tenant to authenticate users from | yes | keyvault secret: `$KEYVAULT_URI/secrets/app-dev-sp-tenant-id` |
| `AZURE_CLIENT_SECRET` | `********` | Secret for `$AZURE_CLIENT_ID` | yes | keyvault secret: `$KEYVAULT_URI/secrets/app-dev-sp-password` |
### Configure Maven
Check that maven is installed:
```bash
$ mvn --version
Apache Maven 3.6.0
Maven home: /usr/share/maven
Java version: 1.8.0_212, vendor: AdoptOpenJDK, runtime: /usr/lib/jvm/jdk8u212-b04/jre
...
```
You will need to configure access to the remote maven repository that holds the OSDU dependencies. This file should live within `~/.m2/settings.xml`:
```bash
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
<servers>
<server>
<id>os-core</id>
<username>os-core</username>
<password>${VSTS_FEED_TOKEN}</password>
</server>
</servers>
</settings>
```
### Start a Redis instance
Start a redis server locally with `docker run --name some-redis -p 6379:6379 -d redis` command
### Build and run the application
After configuring your environment as specified above, you can follow these steps to build and run the application. These steps should be invoked from the *repository root.*
```bash
# build + test + install core service code
$ mvn clean install
# build + test + package azure service code
$ mvn clean package -P partition-aks
# run service
#
# Note: this assumes that the environment variables for running the service as outlined
# above are already exported in your environment.
$ cd provider/partition-azure && mvn spring-boot:run -f pom.xml
```
### Test the application
After the service has started it should be accessible via a web browser by visiting [http://localhost:8080/api/partition/v1/swagger-ui.html](http://localhost:8080/api/partition/v1/swagger-ui.html). If the request does not fail, you can then run the integration tests.
see [instructions](../../testing/README.md) on how to run integration tests locally. In addition to common testing environment variables, the `partition-test-azure` module also needs additional environment variables, which are described below:
```
INTEGRATION_TESTER (app-dev-sp-username)
TESTER_SERVICEPRINCIPAL_SECRET (app-dev-sp-password)
NO_DATA_ACCESS_TESTER (aad-no-data-access-tester-client-id)
NO_DATA_ACCESS_TESTER_SERVICEPRINCIPAL_SECRET (aad-no-data-access-tester-secret)
AZURE_AD_TENANT_ID (azure tenant id)
AZURE_AD_APP_RESOURCE_ID (aad-client-id)
AZURE_AD_OTHER_APP_RESOURCE_ID (AD Application ID used for negative testing)
```
A liveness check can also be performed at `http://localhost:8080/api/partition/v1/_ah/liveness_check`. Other apis can be found on the swagger page
## Debugging
Jet Brains - the authors of Intellij IDEA, have written an [excellent guide](https://www.jetbrains.com/help/idea/debugging-your-first-java-application.html) on how to debug java programs.
## License
Copyright 2017-2020, Schlumberger
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
\ No newline at end of file
provider/partition-azure/devops/stages.yml
+ 23
− 0
View file @ 9d09cabd
......@@ -89,6 +89,16 @@ stages:
zip -r testing.zip testing
cp testing.zip $(build.artifactstagingdirectory)
displayName: 'Zip integration tests and upload it as artifact'
- bash: |
#!/bin/bash
set -e
# Upload redis deployment scripts as artifact
pushd provider/partition-azure/redis-deployment
cp master-deployment.yaml $(build.artifactstagingdirectory)
cp redis-service.yaml $(build.artifactstagingdirectory)
cp sentinel-deployment.yaml $(build.artifactstagingdirectory)
cp slave-deployment.yaml $(build.artifactstagingdirectory)
popd
- upload: $(Build.ArtifactStagingDirectory)
artifact: drop_${{environment.name}}
......@@ -142,6 +152,19 @@ stages:
set -e
kubectl --kubeconfig $(kubeConfig.secureFilePath) rollout restart deployment/partition-aks
displayName: 'kubectl rollout restart deployment/partition-aks'
- bash: |
#!/bin/bash
set -e
pushd $(Pipeline.Workspace)/drop_${{environment.name}}
kubectl --kubeconfig $(kubeConfig.secureFilePath) apply -f redis-service.yaml
sleep 5
kubectl --kubeconfig $(kubeConfig.secureFilePath) apply -f master-deployment.yaml
sleep 5
kubectl --kubeconfig $(kubeConfig.secureFilePath) apply -f sentinel-deployment.yaml
sleep 5
kubectl --kubeconfig $(kubeConfig.secureFilePath) apply -f slave-deployment.yaml
popd
displayName: 'deploy redis cluster'
- ${{ if or(eq(environment.name, 'devint'), eq(environment.name, 'qa')) }}:
- bash: |
#!/bin/bash
......
testing/README.md
+ 4
− 4
View file @ 9d09cabd
......@@ -20,16 +20,16 @@ Partition integration tests are refactored so that the business logic for integr
* JDK8
* Maven
* Azure Devops access to slb-des-ext-collaboration organization. You need to generate a PAT that can access dependencies held in the Azure artifacts
* Values for the following environment variables in Config.java
* Values for the following environment variables in Config.java (in `partition-test-core` module)
```
ENVIRONMENT ('local' for local testing or 'dev' for dev testing)
PARTITION_BASE_URL(service base URL )
PARTITION_BASE_URL (service base URL, required only if running integration tests against a cloud endpoint)
CLIENT_TENANT (name of the client partition, required only if running integration tests against a cloud endpoint) (e.g. 'common')
MY_TENANT(name of the OSDU partition, required only if running integration tests against a cloud endpoint) (e.g. 'opendes')
```
Above variables should be configured in the release pipeline to run integration tests. You should also replace them with proper values if you wish to run tests locally.
### Commands to run tests
* Integration tests are refactored into two pieces: Core and Provider. Core contains business logic for tests and is a dependency for executing the tests from provider module. To build the core module, simply navigate to `partition-test-core` directory and run `mvn clean install`. This will build the core module
* Next, to execute the integration tests, navigate to the provider module and execute `mvn test`
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment