diff --git a/README.md b/README.md index e37e4b10b321079b915014153dd4c17725e8c7d4..596f3c7eaf6b21437d830faf88a560c2f166b2a8 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,6 @@ # Introduction -TODO: Give a short introduction of your project. Let this section explain the objectives or the motivation behind this project. -# Getting Started -TODO: Guide users through getting your code up and running on their own system. In this section you can talk about: -1. Installation process -2. Software dependencies -3. Latest releases -4. API references +## Azure Implementation -# Build and Test -TODO: Describe and show how to build your code and run the tests. - -# Contribute -TODO: Explain how other users and developers can contribute to make your code better. - -If you want to learn more about creating good readme files then refer the following [guidelines](https://docs.microsoft.com/en-us/azure/devops/repos/git/create-a-readme?view=azure-devops). You can also seek inspiration from the below readme files: -- [ASP.NET Core](https://github.com/aspnet/Home) -- [Visual Studio Code](https://github.com/Microsoft/vscode) -- [Chakra Core](https://github.com/Microsoft/ChakraCore) \ No newline at end of file +The [os-indexer-azure README.md](./provider/indexer-azure/README.md) has all the information needed to get started +running the `os-indexer` Azure implementation diff --git a/provider/indexer-azure/.envrc.template b/provider/indexer-azure/.envrc.template new file mode 100644 index 0000000000000000000000000000000000000000..e07a7890090f4b5af016ff3c7161c9b5aefb2065 --- /dev/null +++ b/provider/indexer-azure/.envrc.template @@ -0,0 +1,41 @@ +## +# Needed to run the service +## +export storage_service_url= +export STORAGE_SCHEMA_HOST= +export STORAGE_QUERY_RECORD_HOST= +export STORAGE_QUERY_RECORD_FOR_CONVERSION_HOST= +export servicebus_namespace_name= +export KEYVAULT_URI= +export appinsights_key= +export aad_client_id= +export cosmosdb_account= +export cosmosdb_key= +export cosmosdb_database= +export servicebus_topic_name= +export entitlements_service_endpoint= +export AZURE_CLIENT_ID= +export AZURE_TENANT_ID= +export AZURE_CLIENT_SECRET= + + +## +# Needed to run the integration tests +## +export AZURE_AD_TENANT_ID= +export AZURE_TESTER_SERVICEPRINCIPAL_SECRET= +export INTEGRATION_TESTER= +export AZURE_AD_APP_RESOURCE_ID= +export ELASTIC_USER_NAME= +export ELASTIC_PASSWORD= +export ELASTIC_HOST= +export ELASTIC_PORT= +export aad_client_id= +export DEFAULT_DATA_PARTITION_ID_TENANT1= +export DEFAULT_DATA_PARTITION_ID_TENANT2= +export STORAGE_HOST= +export ENVIRONMENT= +export ENTITLEMENTS_DOMAIN= +export LEGAL_TAG= +export OTHER_RELEVANT_DATA_COUNTRIES= + diff --git a/provider/indexer-azure/README.md b/provider/indexer-azure/README.md index 419f45f6093cd09aaad383ff7654c6cf9ce8bced..258014ddb6f2707175b3055aefefb7b98b7c8692 100644 --- a/provider/indexer-azure/README.md +++ b/provider/indexer-azure/README.md @@ -1,20 +1,158 @@ -# Indexer-azure +# os-indexer-azure -This document is a WIP, but it is intended to describe the steps needed to run this service both locally and in Azure. +os-indexer-azure is a [Spring Boot](https://spring.io/projects/spring-boot) service that is responsible for indexing Records that enable the `os-search` service to execute OSDU R2 domain searches against Elasticsearch. -## Environment Configuration +## Running Locally -TODO: +### Requirements -## Environment Configuration - Local Execution +In order to run this service locally, you will need the following: -Due to the usage of [Managed Identity](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview), you will need the following environment variables defined for running the azure implementation locally: +- [Maven 3.6.0+](https://maven.apache.org/download.cgi) +- [AdoptOpenJDK8](https://adoptopenjdk.net/) +- Infrastructure dependencies, deployable through the relevant [infrastructure template](https://dev.azure.com/slb-des-ext-collaboration/open-data-ecosystem/_git/infrastructure-templates?path=%2Finfra&version=GBmaster&_a=contents) +- While not a strict dependency, example commands in this document use [bash](https://www.gnu.org/software/bash/) + +### General Tips + +**Environment Variable Management** +The following tools make environment variable configuration simpler + - [direnv](https://direnv.net/) - for a shell/terminal environment + - [EnvFile](https://plugins.jetbrains.com/plugin/7861-envfile) - for [Intellij IDEA](https://www.jetbrains.com/idea/) + +**Lombok** +This project uses [Lombok](https://projectlombok.org/) for code generation. You may need to configure your IDE to take advantage of this tool. + - [Intellij configuration](https://projectlombok.org/setup/intellij) + - [VSCode configuration](https://projectlombok.org/setup/vscode) + + +### Environment Variables + +In order to run the service locally, you will need to have the following environment variables defined. + +**Note** The following command can be useful to pull secrets from keyvault: +```bash +az keyvault secret show --vault-name $KEY_VAULT_NAME --name $KEY_VAULT_SECRET_NAME --query value -otsv +``` + +**Required to run service** + +| name | value | description | sensitive? | source | +| --- | --- | --- | --- | --- | +| `storage_service_url` | ex `https://storage.azurewebsites.net` | Endpoint of storage service | no | output of infrastructure deployments | +| `STORAGE_SCHEMA_HOST` | `${storage_service_url}/schemas` | Endpoint of schema API | no | - | +| `STORAGE_QUERY_RECORD_HOST` | `${storage_service_url}/query/records` | Endpoint of records API | no | - | +| `STORAGE_QUERY_RECORD_FOR_CONVERSION_HOST` | `${storage_service_url}/query/records:batch` | Endpoint of records batch API | no | - | +| `servicebus_namespace_name` | ex `foo-sb-namespace` | . | . | . | +| `KEYVAULT_URI` | ex `https://foo-kv.vault.azure.net/` | . | . | . | +| `appinsights_key` | `********` | App Insights key | yes | output of infrastructure deployments | +| `aad_client_id` | `********` | AAD client application ID | yes | output of infrastructure deployment | +| `cosmosdb_account` | ex `devintosdur2cosmosacct` | Cosmos account name | no | output of infrastructure deployment | +| `cosmosdb_key` | `********` | Key for CosmosDB | yes | output of infrastructure deployments | +| `cosmosdb_database` | ex `dev-osdu-r2-db` | Cosmos database for documents | no | output of infrastructure deployment | +| `servicebus_topic_name` | `recordstopic` | Service Bus topic name | no | output of infrastructure deployments | +| `entitlements_service_endpoint` | ex `https://entitlements.azurewebsites.net` | Entitlements service endpoint | no | Service Bus topic name | +| `AZURE_CLIENT_ID` | `********` | Identity to run the service locally. This enables access to Azure resources. You only need this if running locally | yes | keyvault secret: `$KEYVAULT_URI/secrets/app-dev-sp-username` | +| `AZURE_TENANT_ID` | `********` | AD tenant to authenticate users from | yes | keyvault secret: `$KEYVAULT_URI/secrets/app-dev-sp-tenant-id` | +| `AZURE_CLIENT_SECRET` | `********` | Secret for `$AZURE_CLIENT_ID` | yes | keyvault secret: `$KEYVAULT_URI/secrets/app-dev-sp-password` | + +**Required to run integration tests** + +| name | value | description | sensitive? | source | +| --- | --- | --- | --- | --- | +| `AZURE_AD_TENANT_ID` | `********` | AD tenant to authenticate users from | yes | -- | +| `AZURE_TESTER_SERVICEPRINCIPAL_SECRET` | `********` | Secret for `$INTEGRATION_TESTER` | yes | -- | +| `INTEGRATION_TESTER` | `********` | System identity to assume for API calls. Note: this user must have entitlements configured already | no | -- | +| `AZURE_AD_APP_RESOURCE_ID` | `********` | AAD client application ID | yes | output of infrastructure deployment | +| `ELASTIC_USER_NAME` | ex `elastic` | Elasticsearch cluster username | yes | output of infrastructure deployment | +| `ELASTIC_PASSWORD` | `********` | Elasticsearch cluster password | yes | output of infrastructure deployment | +| `ELASTIC_HOST` | ex `foobar.ece.msft-osdu-test.org` | Elasticsearch cluster endpoint | no | output of infrastructure deployment | +| `ELASTIC_PORT` | ex `9243` | Elasticsearch cluster port | no | output of infrastructure deployment | +| `aad_client_id` | `********` | AAD client application ID | yes | output of infrastructure deployment | +| `DEFAULT_DATA_PARTITION_ID_TENANT1` | ex `opendes` | Primary data partition for queries | no | Data in search index | +| `DEFAULT_DATA_PARTITION_ID_TENANT2` | ex `common` | Secondary data partition for queries | no | Data in search index | +| `STORAGE_HOST` | ex `https://storage.azurewebsites.net/` | Storage service endpoint | no | output of infrastructure deployment | +| `ENVIRONMENT` | `CLOUD` | Deployment environment | no | - | +| `ENTITLEMENTS_DOMAIN` | `contoso.com` | OSDU R2 service domain | no | - | +| `LEGAL_TAG` | `opendes-public-usa-dataset-7643990` | Legal tag used for test records | no | Needs to be in DB. The referenced tag should already exist. | +| `OTHER_RELEVANT_DATA_COUNTRIES` | `US` | ? | no | - | + +### Configure Maven + +Check that maven is installed: +```bash +$ mvn --version +Apache Maven 3.6.0 +Maven home: /usr/share/maven +Java version: 1.8.0_212, vendor: AdoptOpenJDK, runtime: /usr/lib/jvm/jdk8u212-b04/jre +... +``` + +You will need to configure access to the remote maven repository that holds the OSDU dependencies. This file should live within `~/.m2/settings.xml`: +```bash +$ cat ~/.m2/settings.xml +<?xml version="1.0" encoding="UTF-8"?> +<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd"> + <servers> + <server> + <id>os-core</id> + <username>mvn-pat</username> + <!-- Treat this auth token like a password. Do not share it with anyone, including Microsoft support. --> + <!-- The generated token expires on or before 11/14/2019 --> + <password>$PERSONAL_ACCESS_TOKEN_GOES_HERE</password> + </server> + </servers> +</settings> +``` + +### Build and run the application + +After configuring your environment as specified above, you can follow these steps to build and run the application. These steps should be invoked from the *repository root.* + +```bash +# build + test + install core service code +$ mvn clean install + +# build + test + package azure service code +$ (cd provider/indexer-azure/ && mvn clean package) + +# run service +# +# Note: this assumes that the environment variables for running the service as outlined +# above are already exported in your environment. +$ java -jar $(find provider/indexer-azure/target/ -name *-spring-boot.jar) +``` + +### Test the application + +After the service has started it should be accessible via a web browser by visiting [http://localhost:8080/swagger-ui.html](http://localhost:8080/swagger-ui.html). If the request does not fail, you can then run the integration tests. + +> **Note**: the integration tests for `os-indexer-azure` work by validating that records submitted to `os-storage-azure` can eventually be queried by `os-search-azure`. This only works if the messages emitted by `os-storage-azure` can be consumed by `os-indexer-queue-azure`, which will submit the indexing request to `os-indexer-azure`. +> +> In order to make sure that the integration tests are running against your local environment, you will need to make sure that the there is an instance of `os-indexer-queue-azure` that is configured to call your deployment of `os-indexer-azure`, and that this instance of `os-indexer-queue-azure` is the only consumer of the Service Bus topic. +> +> There are a few ways to do this: +> - Stop the `os-indexer-queue-azure` function in the Azure portal and run it locally. You'll need to remember to restart the Azure deployed `os-indexer-queue-azure` when you are finished testing +> - Deploy your own infrastructure stack and configure all the services *except* `os-indexer-queue-azure` and `os-indexer-azure` to run in Azure. Then, run `os-indexer-queue-azure` locally +> - Rely on the integration tests to run through the CI/CD pipeline + +```bash +# build + install integration test core +$ (cd testing/indexer-test-core/ && mvn clean install) + +# build + run Azure integration tests. +# +# Note: this assumes that the environment variables for integration tests as outlined +# above are already exported in your environment. +$ (cd testing/indexer-test-azure/ && mvn clean test) +``` + +## Debugging + +Jet Brains - the authors of Intellij IDEA, have written an [excellent guide](https://www.jetbrains.com/help/idea/debugging-your-first-java-application.html) on how to debug java programs. -| variable name | value | -| --- | --- | -| `AZURE_CLIENT_ID` | service principal's app id | -| `AZURE_TENANT_ID` | id of the principal's Azure Active Directory tenant | -| `AZURE_CLIENT_SECRET` | one of the service principal's client secrets | ## Deploying service to Azure