diff --git a/README.md b/README.md index 8bfcb90b5e426329e082c09d3ef9aeb0ad8b23dc..cf670738a775199c38be2b6ecc223b1880948a40 100644 --- a/README.md +++ b/README.md @@ -9,4 +9,8 @@ running the `os-indexer` Azure implementation ## GCP Implementation -All documentation for the GCP implementation of `os-indexer` lives [here](./provider/indexer-gcp/README.md) \ No newline at end of file +All documentation for the GCP implementation of `os-indexer` lives [here](./provider/indexer-gcp/README.md) + +## AWS Implementation + +All documentation for the AWS implementation of `os-indexer` lives [here](./provider/indexer-aws/README.md) \ No newline at end of file diff --git a/provider/indexer-aws/README.md b/provider/indexer-aws/README.md new file mode 100644 index 0000000000000000000000000000000000000000..f858e377a60c0bc51628d60abb07fc449fb1fd90 --- /dev/null +++ b/provider/indexer-aws/README.md @@ -0,0 +1,200 @@ +# Indexer Service +indexer-aws is a [Spring Boot](https://spring.io/projects/spring-boot) service that provides a set of APIs to index storage records against Elasticsearch. It's not user-facing, all APIs are used internally by the platform. + +## Running Locally + +These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. + +### Prerequisites +Pre-requisites + +* JDK 8 (https://docs.aws.amazon.com/corretto/latest/corretto-8-ug/downloads-list.html) +* Maven 3.8.3 or later +* Lombok 1.16 or later +* OSDU Instance deployed on AWS + +### Service Configuration +In order to run the service locally or remotely, you will need to have the following environment variables defined. + +| name | example value | required | description | sensitive? | +| --- | --- | --- | --- | --- | +| `LOCAL_MODE` | `true` | yes | Set to 'true' to use env vars in place of the k8s variable resolver | no | +| `APPLICATION_PORT` | `8080` | yes | The port the service will be hosted on. | no | +| `AWS_REGION` | `us-east-1` | yes | The region where resources needed by the service are deployed | no | +| `AWS_ACCESS_KEY_ID` | `ASIAXXXXXXXXXXXXXX` | yes | The AWS Access Key for a user with access to Backend Resources required by the service | yes | +| `AWS_SECRET_ACCESS_KEY` | `super-secret-key==` | yes | The AWS Secret Key for a user with access to Backend Resources required by the service | yes | +| `AWS_SESSION_TOKEN` | `session-token-xxxxxxxxxx` | no | AWS Session token needed if using an SSO user session to authenticate | yes | +| `ENVIRONMENT` | `osdu-prefix` | yes | The Resource Prefix defined during deployment | no | +| `LOG_LEVEL` | `DEBUG` | yes | The Log Level severity to use (https://www.tutorialspoint.com/log4j/log4j_logging_levels.htm) | no | +| `SSM_ENABLED` | `true` | yes | Set to 'true' to use SSM to resolve config properties, otherwise use env vars | no | +| `SSL_ENABLED` | `false` | no | Set to 'false' to disable SSL for local development | no | +| `ENTITLEMENTS_BASE_URL` | `http://localhost:8081` or `https://some-hosted-url` | yes | Specify the base url for an entitlements service instance. Can be run locally or remote | no | +| `PARTITION_BASE_URL` | `http://localhost:8082` or `https://some-hosted-url` | yes | Specify the base url for a partition service instance. Can be run locally or remote | no | +| `STORAGE_BASE_URL` | `http://localhost:8082` or `https://some-hosted-url` | yes | Specify the base url for a partition service instance. Can be run locally or remote | no | +| `SCHEMA_BASE_URL` | `http://localhost:8082` or `https://some-hosted-url` | yes | Specify the base url for a partition service instance. Can be run locally or remote | no | + + +### Run Locally +Check that maven is installed: + +example: +```bash +$ mvn --version +Apache Maven 3.8.3 (ff8e977a158738155dc465c6a97ffaf31982d739) +Maven home: /usr/local/Cellar/maven/3.8.3/libexec +Java version: 1.8.0_312, vendor: Amazon.com Inc., runtime: /Library/Java/JavaVirtualMachines/amazon-corretto-8.jdk/Contents/Home/jre +... +``` + +You may need to configure access to the remote maven repository that holds the OSDU dependencies. Copy one of the below files' content to your .m2 folder +* For development against the OSDU GitLab environment, leverage: `<REPO_ROOT>~/.mvn/community-maven.settings.xml` +* For development in an AWS Environment, leverage: `<REPO_ROOT>/provider/indexer-aws/maven/settings.xml` + +* Navigate to the service's root folder and run: + +```bash +mvn clean package -pl indexer-core,provider/indexer-aws +``` + +* If you wish to build the project without running tests + +```bash +mvn clean package -pl indexer-core,provider/indexer-aws -DskipTests +``` + +After configuring your environment as specified above, you can follow these steps to run the application. These steps should be invoked from the *repository root.* +<br/> +<br/> +NOTE: If not on osx/linux: Replace `*` with version numbers as defined in the provider/indexer-aws/pom.xml file + +```bash +java -jar provider/indexer-aws/target/indexer-aws-*.*.*-SNAPSHOT-spring-boot.jar +``` + +## Running Elasticsearch locally +For indexer to index anything, it needs to have access to an Elasticsearch cluster. The easiest way to do this is to spin one up locally. +You can spin one up locally using Docker or Kubernetes Helm. What's detailed below is simply downloading executable and running directly. + +Instructions copied from here for longevity: https://www.elastic.co/guide/en/elasticsearch/reference/6.8/getting-started-install.html + +1. Download a distribution from here: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.8.20.tar.gz + +2. Extract the archive +Linux and macOS: `tar -xvf elasticsearch-6.8.20.tar.gz` +Windows PowerShell: `Expand-Archive elasticsearch-6.8.20-windows-x86_64.zip` + +3. Start Elasticsearch from the bin directory: +Linux and macOS: + ``` + cd elasticsearch-6.8.20/bin + ./elasticsearch + ``` + + Windows: + + ``` + cd %PROGRAMFILES%\Elastic\Elasticsearch\bin + .\elasticsearch.exe + ``` + +You should see in the logs that pop up what url and port it runs on. By default you should see http://localhost with port 9300 + +## Testing + + ### Running Integration Tests + This section describes how to run OSDU Integration tests (testing/indexer-test-aws). + + IMPORTANT: You need to setup indexer queue locally first using README in indexer-queue-aws. + + You will need to have the following environment variables defined. + export AWS_COGNITO_AUTH_FLOW=USER_PASSWORD_AUTH + export AWS_COGNITO_AUTH_PARAMS_PASSWORD=$ADMIN_PASSWORD + export AWS_COGNITO_AUTH_PARAMS_USER=$ADMIN_USER + export DEFAULT_DATA_PARTITION_ID_TENANT1=opendes + export DEFAULT_DATA_PARTITION_ID_TENANT2=common + export ENTITLEMENTS_DOMAIN=example.com + export OTHER_RELEVANT_DATA_COUNTRIES=US + export STORAGE_HOST=$STORAGE_URL + export HOST=$SCHEMA_URL + export ELASTIC_HOST=$ELASTIC_HOST + export ELASTIC_PORT=$ELASTIC_PORT + export ELASTIC_PASSWORD=$ELASTIC_PASSWORD + export ELASTIC_USER_NAME=$ELASTIC_USERNAME + + | name | example value | description | sensitive? + | --- | --- | --- | --- | + | `AWS_ACCESS_KEY_ID` | `ASIAXXXXXXXXXXXXXX` | The AWS Access Key for a user with access to Backend Resources required by the service | yes | + | `AWS_SECRET_ACCESS_KEY` | `super-secret-key==` | The AWS Secret Key for a user with access to Backend Resources required by the service | yes | + | `AWS_SESSION_TOKEN` | `session-token-xxxxxxxxx` | AWS Session token needed if using an SSO user session to authenticate | yes | + | `AWS_COGNITO_USER_POOL_ID` | `us-east-1_xxxxxxxx` | User Pool Id for the reference cognito | no | + | `AWS_COGNITO_CLIENT_ID` | `xxxxxxxxxxxx` | Client ID for the Auth Flow integrated with the Cognito User Pool | no | + | `AWS_COGNITO_AUTH_FLOW` | `USER_PASSWORD_AUTH` | Auth flow used by reference cognito deployment | no | + | `DEFAULT_DATA_PARTITION_ID_TENANT1` | `opendes` | Partition used to create and index record | no | + | `DEFAULT_DATA_PARTITION_ID_TENANT2` | `common` | Another needed partition| no | + | `AWS_COGNITO_AUTH_PARAMS_USER` | `int-test-user@testing.com` | Int Test Username | no | + | `AWS_COGNITO_AUTH_PARAMS_USER_NO_ACCESS` | `noaccess@testing.com` | No Access Username | no | + | `AWS_COGNITO_AUTH_PARAMS_PASSWORD` | `some-secure-password` | Int Test User/NoAccessUser Password | yes | + | `ENTITLEMENTS_DOMAIN` | `example.com` | Domain for user's groups | no | + | `OTHER_RELEVANT_DATA_COUNTRIES` | `US` | Used to create demo legal tag | no | + | `STORAGE_HOST` | `http://localhost:8080/api/storage/v2/` | The url where the storage API is hosted | no | + | `HOST` | `http://localhost:8080` | Base url for deployment | no | + | `ELASTIC_HOST` | `localhost` | Url for elasticsearch | no | + | `ELASTIC_PORT` | `9300` | Port for elasticsearch | no | + | `ELASTIC_PASSWORD` | `xxxxxxxxxxxxxxx` | Password for user to access elasticsearch | yes | + | `ELASTIC_USER_NAME` | `xxxxxxxxxxxxxxxx` | Username for user to access elasticsearch | yes | + + + **Creating a new user to use for integration tests** + ``` + aws cognito-idp admin-create-user --user-pool-id ${AWS_COGNITO_USER_POOL_ID} --username ${AWS_COGNITO_AUTH_PARAMS_USER} --user-attributes Name=email,Value=${AWS_COGNITO_AUTH_PARAMS_USER} Name=email_verified,Value=True --message-action SUPPRESS + + aws cognito-idp initiate-auth --auth-flow ${AWS_COGNITO_AUTH_FLOW} --client-id ${AWS_COGNITO_CLIENT_ID} --auth-parameters USERNAME=${AWS_COGNITO_AUTH_PARAMS_USER},PASSWORD=${AWS_COGNITO_AUTH_PARAMS_PASSWORD} + ``` + + **Entitlements group configuration for integration accounts** + <br/> + In order to add user entitlements, run entitlements bootstrap scripts in the entitlements project + + | AWS_COGNITO_AUTH_PARAMS_USER | + | --- | + | service.indexer.admin | + | service.legal.admin | + | service.storage.admin | + + Execute following command to build code and run all the integration tests: + +### Run Tests simulating Pipeline + +* Prior to running tests, scripts must be executed locally to generate pipeline env vars + +```bash +testing/indexer-test-aws/build-aws/prepare-dist.sh + +#Set Neccessary ENV Vars here as defined in run-tests.sh + +dist/testing/integration/build-aws/run-tests.sh +``` + +### Run Tests using mvn +Set required env vars and execute the following: +``` +mvn clean package -f testing/pom.xml -pl indexer-test-core,indexer-test-aws -DskipTests +mvn test -f testing/indexer-test-aws/pom.xml +``` + + + +## License +Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + +[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0) + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. \ No newline at end of file