Indexer Service

indexer-aws is a Spring Boot service that provides a set of APIs to index storage records against Elasticsearch. It's not user-facing, all APIs are used internally by the platform.

Running Locally

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

Prerequisites

Pre-requisites

• JDK 8 (https://docs.aws.amazon.com/corretto/latest/corretto-8-ug/downloads-list.html)
• Maven 3.8.3 or later
• Lombok 1.16 or later
• OSDU Instance deployed on AWS

Service Configuration

In order to run the service locally or remotely, you will need to have the following environment variables defined.

Run Locally

Check that maven is installed:

example:

$ mvn --version
Apache Maven 3.8.3 (ff8e977a158738155dc465c6a97ffaf31982d739)
Maven home: /usr/local/Cellar/maven/3.8.3/libexec
Java version: 1.8.0_312, vendor: Amazon.com Inc., runtime: /Library/Java/JavaVirtualMachines/amazon-corretto-8.jdk/Contents/Home/jre
...

You may need to configure access to the remote maven repository that holds the OSDU dependencies. Copy one of the below files' content to your .m2 folder

• For development against the OSDU GitLab environment, leverage: <REPO_ROOT>~/.mvn/community-maven.settings.xml

• For development in an AWS Environment, leverage: <REPO_ROOT>/provider/indexer-aws/maven/settings.xml

• Navigate to the service's root folder and run:

mvn clean package -pl indexer-core,provider/indexer-aws

• If you wish to build the project without running tests

mvn clean package -pl indexer-core,provider/indexer-aws -DskipTests

After configuring your environment as specified above, you can follow these steps to run the application. These steps should be invoked from the repository root.

NOTE: If not on osx/linux: Replace * with version numbers as defined in the provider/indexer-aws/pom.xml file

java -jar provider/indexer-aws/target/indexer-aws-*.*.*-SNAPSHOT-spring-boot.jar

Running Elasticsearch locally

For indexer to index anything, it needs to have access to an Elasticsearch cluster. The easiest way to do this is to spin one up locally. You can spin one up locally using Docker or Kubernetes Helm. What's detailed below is simply downloading executable and running directly.

Instructions copied from here for longevity: https://www.elastic.co/guide/en/elasticsearch/reference/6.8/getting-started-install.html

1. Download a distribution from here: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.8.20.tar.gz

2. Extract the archive Linux and macOS: tar -xvf elasticsearch-6.8.20.tar.gz Windows PowerShell: Expand-Archive elasticsearch-6.8.20-windows-x86_64.zip

3. Start Elasticsearch from the bin directory: Linux and macOS:

   cd elasticsearch-6.8.20/bin
   ./elasticsearch

   Windows:

   cd %PROGRAMFILES%\Elastic\Elasticsearch\bin
   .\elasticsearch.exe

You should see in the logs that pop up what url and port it runs on. By default you should see http://localhost with port 9300

Testing

Running Integration Tests

This section describes how to run OSDU Integration tests (testing/indexer-test-aws).

IMPORTANT: You need to setup indexer queue locally first using README in indexer-queue-aws.

You will need to have the following environment variables defined. export AWS_COGNITO_AUTH_FLOW=USER_PASSWORD_AUTH export AWS_COGNITO_AUTH_PARAMS_PASSWORD=ADMIN_PASSWORD export AWS_COGNITO_AUTH_PARAMS_USER=ADMIN_USER export DEFAULT_DATA_PARTITION_ID_TENANT1=opendes export DEFAULT_DATA_PARTITION_ID_TENANT2=common export ENTITLEMENTS_DOMAIN=example.com export OTHER_RELEVANT_DATA_COUNTRIES=US export STORAGE_HOST=STORAGE_URL export HOST=SCHEMA_URL export ELASTIC_HOST=ELASTIC_HOST export ELASTIC_PORT=ELASTIC_PORT export ELASTIC_PASSWORD=ELASTIC_PASSWORD export ELASTIC_USER_NAME=ELASTIC_USERNAME

Creating a new user to use for integration tests

aws cognito-idp admin-create-user --user-pool-id ${AWS_COGNITO_USER_POOL_ID} --username ${AWS_COGNITO_AUTH_PARAMS_USER} --user-attributes Name=email,Value=${AWS_COGNITO_AUTH_PARAMS_USER} Name=email_verified,Value=True --message-action SUPPRESS

aws cognito-idp initiate-auth --auth-flow ${AWS_COGNITO_AUTH_FLOW} --client-id ${AWS_COGNITO_CLIENT_ID} --auth-parameters USERNAME=${AWS_COGNITO_AUTH_PARAMS_USER},PASSWORD=${AWS_COGNITO_AUTH_PARAMS_PASSWORD}

Entitlements group configuration for integration accounts
In order to add user entitlements, run entitlements bootstrap scripts in the entitlements project

Execute following command to build code and run all the integration tests:

Run Tests simulating Pipeline

• Prior to running tests, scripts must be executed locally to generate pipeline env vars

testing/indexer-test-aws/build-aws/prepare-dist.sh

#Set Neccessary ENV Vars here as defined in run-tests.sh

dist/testing/integration/build-aws/run-tests.sh 

Run Tests using mvn

Set required env vars and execute the following:

mvn clean package -f testing/pom.xml -pl indexer-test-core,indexer-test-aws -DskipTests
mvn test -f testing/indexer-test-aws/pom.xml

License

Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.