README.md

Indexer Service

os-indexer-reference is a Spring Boot service that is responsible for indexing Records that enable the os-search service to execute OSDU hybrid cloud searches against Elasticsearch.

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.

Prerequisites

Pre-requisites

• GCloud SDK with java (latest version)
• JDK 8
• Lombok 1.16 or later
• Maven

Installation

In order to run the service locally or remotely, you will need to have the following environment variables defined.

name	value	description	sensitive?	source
LOG_PREFIX	service	Logging prefix	no	-
SERVER_SERVLET_CONTEXPATH	/api/indexer/v2	Servlet context path	no	-
AUTHORIZE_API	ex https://entitlements.com/entitlements/v1	Entitlements API endpoint	no	output of infrastructure deployment
LEGALTAG_API	ex https://legal.com/api/legal/v1	Legal API endpoint	no	output of infrastructure deployment
INDEXER_QUEUE_HOST	ex https://os-indexer-queue-dot-opendes.appspot.com/_dps/task-handlers/enqueue	Indexer-Queue API endpoint	no	output of infrastructure deployment
CRS_API	ex https://crs-converter-gae-dot-opendes.appspot.com/api/crs/v1	CRS API endpoint	no	https://console.cloud.google.com/memorystore/redis/instances
STORAGE_HOSTNAME	ex os-storage-dot-opendes.appspot.com	Storage Host	no	output of infrastructure deployment
STORAGE_SCHEMA_HOST	ex https://os-storage-dot-opendes.appspot.com/api/storage/v2/schemas	Storage API endpoint 'schemas'	no	https://console.cloud.google.com/apis/credentials
STORAGE_QUERY_RECORD_FOR_CONVERSION_HOST	ex https://os-storage-dot-opendes.appspot.com/api/storage/v2/query/records:batch	Storage API endpoint 'records'	no	https://console.cloud.google.com/iam-admin/serviceaccounts
STORAGE_QUERY_RECORD_HOST	ex https://os-storage-dot-opendes.appspot.com/api/storage/v2/query/records	Storage API endpoint 'query/records'	no	https://console.cloud.google.com/iam-admin/serviceaccounts
STORAGE_RECORDS_BATCH_SIZE	20	Batch size for storage API POST {endpoint}/query/records:batch	no	-
STORAGE_RECORDS_BY_KIND_BATCH_SIZE	-	Batch size for storage API GET {endpoint}/query/records. If this is not present, defaults to value of STORAGE_RECORDS_BATCH_SIZE	no	-
REDIS_SEARCH_HOST	ex 127.0.0.1	Redis host for search	no	https://console.cloud.google.com/memorystore/redis/instances
REDIS_GROUP_HOST	ex 127.0.0.1	Redis host for groups	no	https://console.cloud.google.com/memorystore/redis/instances
REDIS_SEARCH_PORT	ex 6379	Redis host for search	no	https://console.cloud.google.com/memorystore/redis/instances
GOOGLE_AUDIENCES	ex *****.apps.googleusercontent.com	Client ID for getting access to cloud resources	yes	https://console.cloud.google.com/apis/credentials
GOOGLE_APPLICATION_CREDENTIALS	ex /path/to/directory/service-key.json	Service account credentials, you only need this if running locally	yes	https://console.cloud.google.com/iam-admin/serviceaccounts
security.https.certificate.trust	ex false	Elastic client connection uses TrustSelfSignedStrategy(), if it is 'true'	false	output of infrastructure deployment
indexer.que.service.mail	ex default@iam.gserviceaccount.com	Indexer Que environment service account mail, required if Indexer Que deployed in cloud task mode, to validate token from it	yes	-
SCHEMA_HOST	ex https://os-schema-dot-opendes.appspot.com/api/schema-service/v1/schema	Schema API endpoint	no	output of infrastructure deployment
PARTITION_API	ex https://localhost:8081/api/partition/v1	Partition API endpoint	no	output of infrastructure deployment
MONGO_DB_URL	ex mongodb://localhost:27017	Mongo DB Url	yes	output of infrastructure deployment
MONGO_DB_USER	ex mongouser	Mongo DB userName	yes	output of infrastructure deployment
MONGO_DB_PASSWORD	ex mongopassword	Mongo DB userPassword	yes	output of infrastructure deployment
MONGO_DB_NAME	ex mongoDBName	Mongo DB DbName	yes	output of infrastructure deployment
MB_RABBITMQ_URI	ex amqp://guest:guest@127.0.0.1:5672	MessageBroker RabbitMQ URI	yes	https://console.cloud.google.com/iam-admin/serviceaccounts

Run Locally

Check that maven is installed:

$ mvn --version
Apache Maven 3.6.0
Maven home: /usr/share/maven
Java version: 1.8.0_212, vendor: AdoptOpenJDK, runtime: /usr/lib/jvm/jdk8u212-b04/jre
...

You may need to configure access to the remote maven repository that holds the OSDU dependencies. This file should live within ~/.mvn/community-maven.settings.xml:

$ cat ~/.m2/settings.xml
<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
    <servers>
        <server>
            <id>community-maven-via-private-token</id>
            <!-- Treat this auth token like a password. Do not share it with anyone, including Microsoft support. -->
            <!-- The generated token expires on or before 11/14/2019 -->
             <configuration>
              <httpHeaders>
                  <property>
                      <name>Private-Token</name>
                      <value>${env.COMMUNITY_MAVEN_TOKEN}</value>
                  </property>
              </httpHeaders>
             </configuration>
        </server>
    </servers>
</settings>

• Update the Google cloud SDK to the latest version:

gcloud components update

• Set Google Project Id:

gcloud config set project <YOUR-PROJECT-ID>

• Perform a basic authentication in the selected project: