Skip to content
Snippets Groups Projects
Commit e87903f8 authored by Aliaksandr Ramanovich (EPAM)'s avatar Aliaksandr Ramanovich (EPAM)
Browse files

Merge branch 'GONRG-5894-gc-rename' into 'master'

GONRG-5894 - gc rename

See merge request !455
parents 07ad6a51 7580c316
No related branches found
No related tags found
1 merge request!455GONRG-5894 - gc rename
Pipeline #159090 passed with warnings
Showing
with 137 additions and 109 deletions
......@@ -8,27 +8,27 @@ cli:
project: Indexer
analyze:
modules:
- name: indexer-service
type: mvn
target: pom.xml
path: .
- name: indexer-core
type: mvn
target: indexer-core/pom.xml
path: .
- name: indexer-aws
type: mvn
target: provider/indexer-aws/pom.xml
path: .
- name: indexer-azure
type: mvn
target: provider/indexer-azure/pom.xml
path: .
- name: indexer-gcp
type: mvn
target: provider/indexer-gcp/pom.xml
path: .
- name: indexer-ibm
type: mvn
target: provider/indexer-ibm/pom.xml
path: .
- name: indexer-service
type: mvn
target: pom.xml
path: .
- name: indexer-core
type: mvn
target: indexer-core/pom.xml
path: .
- name: indexer-aws
type: mvn
target: provider/indexer-aws/pom.xml
path: .
- name: indexer-azure
type: mvn
target: provider/indexer-azure/pom.xml
path: .
- name: indexer-gcp
type: mvn
target: provider/indexer-gc/pom.xml
path: .
- name: indexer-ibm
type: mvn
target: provider/indexer-ibm/pom.xml
path: .
......@@ -20,11 +20,10 @@ variables:
IBM_BUILD_SUBDIR: provider/indexer-ibm
IBM_INT_TEST_SUBDIR: testing/indexer-test-ibm
IBM_HELM_CONFIG_PATH: devops/ibm/ibm-indexer-config
IBM_HELM_DEPLOY_PATH: devops/ibm/ibm-indexer-deploy
AZURE_SERVICE: indexer
AZURE_BUILD_SUBDIR: provider/indexer-azure
AZURE_TEST_SUBDIR: testing/indexer-test-azure
......@@ -58,9 +57,9 @@ include:
file: "publishing/pages.yml"
- project: "osdu/platform/ci-cd-pipelines"
file: "cloud-providers/osdu-gcp-global.yml"
file: "cloud-providers/gc-global.yml"
- local: "devops/gcp/pipeline/override-stages.yml"
- local: "devops/gc/pipeline/override-stages.yml"
aws-test-java:
tags: ["aws-internal-test"]
......@@ -359,7 +359,7 @@ The following software have components provided under the terms of this license:
- Apache Maven Wagon :: Providers :: SSH External Provider (from https://repo1.maven.org/maven2/org/apache/maven/wagon/wagon-ssh-external)
- Apache Maven Wagon :: Providers :: SSH Provider (from https://repo1.maven.org/maven2/org/apache/maven/wagon/wagon-ssh)
- Apache Velocity (from http://velocity.apache.org/engine/devel/, http://velocity.apache.org/engine/releases/velocity-1.6.1/, http://velocity.apache.org/engine/releases/velocity-1.6.2/)
- AssertJ fluent assertions (from ${project.parent.url}#${project.artifactId}, https://repo1.maven.org/maven2/org/assertj/assertj-core)
- AssertJ Core (from ${project.organization.url}#${project.artifactId})
- Asynchronous Http Client (from https://repo1.maven.org/maven2/org/asynchttpclient/async-http-client)
- Asynchronous Http Client Netty Utils (from https://repo1.maven.org/maven2/org/asynchttpclient/async-http-client-netty-utils)
- AutoValue Annotations (from https://github.com/google/auto/tree/master/value, https://repo1.maven.org/maven2/com/google/auto/value/auto-value-annotations)
......@@ -619,12 +619,12 @@ The following software have components provided under the terms of this license:
- compiler (from http://github.com/spullara/mustache.java)
- datastore-v1-proto-client (from https://repo1.maven.org/maven2/com/google/cloud/datastore/datastore-v1-proto-client)
- documentdb-bulkexecutor (from http://azure.microsoft.com/en-us/services/documentdb/)
- elasticsearch (from https://github.com/elastic/elasticsearch, https://repo1.maven.org/maven2/org/elasticsearch/elasticsearch)
- elasticsearch-cli (from https://github.com/elastic/elasticsearch)
- elasticsearch-core (from https://github.com/elastic/elasticsearch)
- elasticsearch-geo (from https://github.com/elastic/elasticsearch)
- elasticsearch-secure-sm (from https://github.com/elastic/elasticsearch)
- elasticsearch-x-content (from https://github.com/elastic/elasticsearch)
- elasticsearch (from https://github.com/elastic/elasticsearch, https://github.com/elastic/elasticsearch.git, https://repo1.maven.org/maven2/org/elasticsearch/elasticsearch)
- elasticsearch-cli (from https://github.com/elastic/elasticsearch, https://github.com/elastic/elasticsearch.git)
- elasticsearch-core (from https://github.com/elastic/elasticsearch, https://github.com/elastic/elasticsearch.git)
- elasticsearch-geo (from https://github.com/elastic/elasticsearch.git)
- elasticsearch-secure-sm (from https://github.com/elastic/elasticsearch, https://github.com/elastic/elasticsearch.git)
- elasticsearch-x-content (from https://github.com/elastic/elasticsearch, https://github.com/elastic/elasticsearch.git)
- error-prone annotations (from https://repo1.maven.org/maven2/com/google/errorprone/error_prone_annotations)
- io.grpc:grpc-alts (from https://github.com/grpc/grpc-java)
- io.grpc:grpc-api (from https://github.com/grpc/grpc-java)
......@@ -666,11 +666,11 @@ The following software have components provided under the terms of this license:
- proto-google-cloud-iamcredentials-v1 (from https://github.com/googleapis/google-cloud-java, https://github.com/googleapis/java-iamcredentials/proto-google-cloud-iamcredentials-v1, https://repo1.maven.org/maven2/com/google/api/grpc/proto-google-cloud-iamcredentials-v1)
- proto-google-cloud-logging-v2 (from https://github.com/googleapis/java-logging/proto-google-cloud-logging-v2, https://repo1.maven.org/maven2/com/google/api/grpc/proto-google-cloud-logging-v2)
- proto-google-cloud-pubsub-v1 (from https://github.com/googleapis/googleapis, https://github.com/googleapis/java-pubsub/proto-google-cloud-pubsub-v1)
- proto-google-common-protos (from https://github.com/googleapis/api-client-staging, https://github.com/googleapis/googleapis, https://github.com/googleapis/java-iam/proto-google-common-protos)
- proto-google-iam-v1 (from https://github.com/googleapis/googleapis, https://github.com/googleapis/java-iam/proto-google-iam-v1)
- proto-google-common-protos (from https://github.com/googleapis/api-client-staging, https://github.com/googleapis/gapic-generator-java, https://github.com/googleapis/googleapis, https://github.com/googleapis/java-iam/proto-google-common-protos)
- proto-google-iam-v1 (from https://github.com/googleapis/gapic-generator-java, https://github.com/googleapis/googleapis, https://github.com/googleapis/java-iam/proto-google-iam-v1)
- rank-eval (from https://github.com/elastic/elasticsearch.git)
- resilience4j (from https://github.com/resilience4j/resilience4j, https://resilience4j.readme.io, ttps://resilience4j.readme.io)
- rest (from https://github.com/elastic/elasticsearch)
- rest (from https://github.com/elastic/elasticsearch.git)
- rest-high-level (from https://github.com/elastic/elasticsearch)
- spring-security-crypto (from http://spring.io/spring-security, https://spring.io/projects/spring-security, https://spring.io/spring-security)
- spring-security-oauth2-client (from http://spring.io/spring-security, https://spring.io/projects/spring-security, https://spring.io/spring-security)
......@@ -696,7 +696,7 @@ BSD-2-Clause
========================================================================
The following software have components provided under the terms of this license:
- API Common (from https://github.com/googleapis, https://github.com/googleapis/api-common-java)
- API Common (from https://github.com/googleapis, https://github.com/googleapis/api-common-java, https://repo1.maven.org/maven2/com/google/api/api-common)
- Apache Lucene (module: memory) (from https://lucene.apache.org/)
- Apache Lucene (module: misc) (from https://lucene.apache.org/, https://repo1.maven.org/maven2/org/apache/lucene/lucene-misc)
- Apache Lucene (module: spatial-extras) (from https://lucene.apache.org/, https://repo1.maven.org/maven2/org/apache/lucene/lucene-spatial-extras)
......@@ -729,7 +729,7 @@ BSD-3-Clause
========================================================================
The following software have components provided under the terms of this license:
- API Common (from https://github.com/googleapis, https://github.com/googleapis/api-common-java)
- API Common (from https://github.com/googleapis, https://github.com/googleapis/api-common-java, https://repo1.maven.org/maven2/com/google/api/api-common)
- ASM Analysis (from http://asm.ow2.io/)
- ASM Commons (from http://asm.ow2.io/, https://repo1.maven.org/maven2/org/ow2/asm/asm-commons)
- ASM Core (from http://asm.ow2.io/, http://asm.ow2.org/)
......@@ -1248,7 +1248,7 @@ SunPro
The following software have components provided under the terms of this license:
- Lucene Core (from https://repo1.maven.org/maven2/org/apache/lucene/lucene-core)
- elasticsearch-core (from https://github.com/elastic/elasticsearch)
- elasticsearch-core (from https://github.com/elastic/elasticsearch, https://github.com/elastic/elasticsearch.git)
========================================================================
W3C
......
......@@ -9,8 +9,8 @@ running the `os-indexer` Azure implementation
## Google Cloud Implementation
All documentation for the Google Cloud implementation of `os-indexer` lives [here](./provider/indexer-gcp/README.md)
All documentation for the Google Cloud implementation of `os-indexer` lives [here](./provider/indexer-gc/README.md)
## AWS Implementation
All documentation for the AWS implementation of `os-indexer` lives [here](./provider/indexer-aws/README.md)
\ No newline at end of file
All documentation for the AWS implementation of `os-indexer` lives [here](./provider/indexer-aws/README.md)
apiVersion: v2
name: gcp-indexer-configmap
name: gc-indexer-configmap
description: A Helm chart for Kubernetes
# A chart can be either an 'application' or a 'library' chart.
......@@ -21,4 +21,4 @@ version: 0.1.0
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
# It is recommended to use it with quotes.
appVersion: "1.16.0"
appVersion: "1.19.0"
......@@ -60,7 +60,7 @@ First you need to set variables in **values.yaml** file using any code editor. S
Run this command from within this directory:
```console
helm install gcp-indexer-configmap .
helm install gc-indexer-configmap .
```
## Uninstalling the Chart
......@@ -68,7 +68,7 @@ helm install gcp-indexer-configmap .
To uninstall the helm deployment:
```console
helm uninstall gcp-indexer-configmap
helm uninstall gc-indexer-configmap
```
[Move-to-Top](#configmap-helm-chart)
......@@ -14,7 +14,7 @@ data:
securityHttpsCertificateTrust: "true"
springProfilesActive: "gcp"
storageHost: "http://storage"
# gcp
# Google Cloud
googleAudiences: ""
conf:
appName: "indexer"
......
apiVersion: v2
name: gcp-indexer-deploy
name: gc-indexer-deploy
description: A Helm chart for Kubernetes
# A chart can be either an 'application' or a 'library' chart.
......@@ -21,4 +21,4 @@ version: 0.1.0
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
# It is recommended to use it with quotes.
appVersion: "1.16.0"
appVersion: "1.19.0"
......@@ -56,7 +56,7 @@ First you need to set variables in **values.yaml** file using any code editor. S
Run this command from within this directory:
```console
helm install gcp-indexer-deploy .
helm install gc-indexer-deploy .
```
## Uninstalling the Chart
......@@ -64,7 +64,7 @@ helm install gcp-indexer-deploy .
To uninstall the helm deployment:
```console
helm uninstall gcp-indexer-deploy
helm uninstall gc-indexer-deploy
```
[Move-to-Top](#deploy-helm-chart)
File moved
variables:
GC_SERVICE: indexer
GC_VENDOR: gc
gc-anthos-test:
variables:
GC_VENDOR: anthos
variables:
OSDU_GCP_SERVICE: indexer
OSDU_GCP_VENDOR: gcp
osdu-gcp-anthos-test:
variables:
OSDU_GCP_VENDOR: anthos
......@@ -11,9 +11,9 @@
- [Data Partition provision](#data-partition-provision)
- [Schema change](#schema-change)
- [Schema Service adoption](#schema-service-adoption)
* [R3 Schema Support](#r3-schema-support)
- [R3 Schema Support](#r3-schema-support)
- [Troubleshoot Indexing Issues](#troubleshoot-indexing-issues)
* [Get indexing status](#get-indexing-status)
- [Get indexing status](#get-indexing-status)
## Introduction <a name="introduction"></a>
......@@ -27,28 +27,31 @@ parent, acl, namespace, type, version, legaltags, index to each record at the ti
## Indexer API access <a name="indexer-api-access"></a>
* Required roles
- Required roles
Indexer service requires that users (and service accounts) have dedicated roles in order to use it. Users must be a member of `users.datalake.viewers` or `users.datalake.editors` or `users.datalake.admins`, roles can be assigned using the [Entitlements Service](/solutions/osdu/tutorials/core-services/entitlementsservice). Please look at the API documentation for specific requirements.
In addition to service roles, users __must__ be a member of data groups to access the data.
* Required headers
- Required headers
The OSDU Data Platform stores data in different partitions, depending on the different accounts in the OSDU system.
A user may belong to more than one account. As a user, after logging into the OSDU portal, you need to select the account you wish to be active.
Likewise, when using the Search APIs, you need to specify the active account in the header called `data-partition-id`. The correct `data-partition-id` can be obtained from the CFS services. The `data-partition-id` enables the search within the mapped partition. e.g.
```
data-partition-id: opendes
```
* Optional headers
- Optional headers
The `correlation-id` is a traceable ID to track the journey of a single request. The `correlation-id` can be a GUID on the header with a key. It is best practice to provide the correlation-id so the request can be tracked through all the services.
```
correlation-id: 1e0fef08-22fd-49b1-a5cc-dffa21bc0b70
```
If the service is initiating the request, an ID should be generated. If the `correlation-id` is not provided, then a new ID will be generated by the service so that the request would be traceable.
[Back to table of contents](#TOC)
......@@ -65,12 +68,12 @@ Provides build and git related information.
GET /api/indexer/v2/info HTTP/1.1
```
#### Example response:
#### Example response
```json
{
"groupId": "org.opengroup.osdu",
"artifactId": "indexer-gcp",
"artifactId": "indexer-gc",
"version": "0.10.0-SNAPSHOT",
"buildTime": "2021-07-09T14:29:51.584Z",
"branch": "feature/GONRG-2681_Build_info",
......@@ -140,12 +143,12 @@ will use the same schema and overwrite records with the same ids. Default value
#### Request body
`kind` <br />
&emsp;&emsp;(required, String) Kind to be re-indexed.
&emsp;&emsp;(required, String) Kind to be re-indexed.
[Back to table of contents](#TOC)
## Delete API <a name="delete"></a>
Delete API is used to delete an index for a specific kind.
Only users who belong to the Entitlement groups 'users.datalake.ops' can make calls to this API.
......@@ -165,9 +168,9 @@ curl --request DELETE \
### Data Partition provision <a name="data-partition-provision"></a>
Configures Search backend for a data partition.
Configures Search backend for a data partition.
```http
```http
PUT /api/indexer/v2/partitions/provision HTTP/1.1
```
......@@ -181,6 +184,7 @@ curl --request PUT \
--header 'content-type: application/json' \
--header 'data-partition-id: opendes''
```
</details>
#### Prerequisite
......@@ -244,14 +248,14 @@ Indexer service support r3 schema. These schemas are created via Schema service.
Here is an example following end-to-end workflow can be exercised (please update the schema based on your environment):
* Ingest r3 schema for `opendes:wks:master-data--Wellbore:1.0.0`. Schema service payload can be
- Ingest r3 schema for `opendes:wks:master-data--Wellbore:1.0.0`. Schema service payload can be
found [here](https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/master/testing/indexer-test-core/src/main/resources/testData/r3-index_record_wks_master.schema.json)
.
* Ingest r3 master-data Wellbore record. Storage service payload can be
- Ingest r3 master-data Wellbore record. Storage service payload can be
found [here](https://community.opengroup.org/osdu/platform/system/indexer-service/-/blob/master/testing/indexer-test-core/src/main/resources/testData/r3-index_record_wks_master.json)
* Records can be searched via Search service. Here is sample payload:
- Records can be searched via Search service. Here is sample payload:
```
POST /api/search/v2/query HTTP/1.1
......@@ -322,9 +326,9 @@ Details of the index block:
1) trace: This field collects all the issues related to the indexing and concatenates using '|'. This is a String field.
2) statusCode: This field determines the category of the error. This is integer field. It can have the following values:
* 200 - All OK
* 404 - Schema is missing in Storage
* 400 - Some fields were not properly mapped with the schema defined
- 200 - All OK
- 404 - Schema is missing in Storage
- 400 - Some fields were not properly mapped with the schema defined
3) lastUpdateTime: This field captures the last time the record was updated by the indexer service. This is datetime
field so you can do range queries on this field.
......@@ -346,4 +350,4 @@ The above query will return all records which had problems due to fields mismatc
Please refer to the [Search service](https://community.opengroup.org/osdu/platform/system/search-service/-/blob/master/docs/api/search_openapi.yaml#L28) documentation for examples on different kinds of search
queries.
[Back to table of contents](#TOC)
\ No newline at end of file
[Back to table of contents](#TOC)
......@@ -173,7 +173,7 @@
<module>indexer-core</module>
<module>provider/indexer-aws</module>
<module>provider/indexer-azure</module>
<module>provider/indexer-gcp</module>
<module>provider/indexer-gc</module>
<module>provider/indexer-ibm</module>
</modules>
......
# Indexer Service
os-indexer-gcp is a [Spring Boot](https://spring.io/projects/spring-boot) service that is responsible for indexing Records that enable the `os-search` service to execute OSDU R2 domain searches against Elasticsearch.
Indexer-gc is a [Spring Boot](https://spring.io/projects/spring-boot) service that is responsible for indexing Records that enable the `search` service to execute OSDU R2 domain searches against Elasticsearch.
## Table of Contents <a name="TOC"></a>
* [Getting started](#Getting-started)
* [Mappers](#Mappers)
* [Settings and Configuration](#Settings-and-Configuration)
......@@ -12,15 +14,20 @@ os-indexer-gcp is a [Spring Boot](https://spring.io/projects/spring-boot) servic
* [Licence](#License)
## Getting Started
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.
# Configuration
## Service Configuration
### Anthos:
[Anthos service configuration ](docs/anthos/README.md)
### Google Cloud:
[Google Cloud service configuration ](docs/gcp/README.md)
### Anthos
[Anthos service configuration](docs/anthos/README.md)
### Google Cloud
[Google Cloud service configuration](docs/gc/README.md)
## Mappers
......@@ -28,32 +35,40 @@ This is a universal solution created using EPAM OQM mappers technology. It allow
implementations of message brokers.
For more information about mappers:
- [OQM Readme](https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/oqm/-/blob/master/README.md)
* [OQM Readme](https://community.opengroup.org/osdu/platform/system/lib/cloud/gcp/oqm/-/blob/master/README.md)
### Limitations of the current version
In the current version, the mappers are equipped with several drivers to the stores and the message broker:
- OQM (mapper to message brokers): Google PubSub; RabbitMQ
* OQM (mapper to message brokers): Google PubSub; RabbitMQ
## Settings and Configuration
### Prerequisites
1. Mandatory
- JDK 8
- Lombok 1.16 or later
- Maven
* JDK 8
* Lombok 1.16 or later
* Maven
2. For Google Cloud only
- GCloud SDK with java (latest version)
### Anthos Service Configuration:
[Anthos service configuration ](docs/anthos/README.md)
### Google Cloud Service Configuration:
[Google Cloud service configuration ](docs/gcp/README.md)
* GCloud SDK with java (latest version)
### Anthos Service Configuration
[Anthos service configuration](docs/anthos/README.md)
### Google Cloud Service Configuration
[Google Cloud service configuration](docs/gc/README.md)
## Run service
### Run Locally
Check that maven is installed:
```bash
......@@ -95,6 +110,7 @@ $ cat ~/.m2/settings.xml
```bash
gcloud components update
```
* Set Google Project Id:
```bash
......@@ -129,10 +145,11 @@ mvn clean install -DskipTests
After configuring your environment as specified above, you can follow these steps to build and run the application. These steps should be invoked from the *repository root.*
```bash
cd provider/indexer-gcp/ && mvn spring-boot:run
cd provider/indexer-gc && mvn spring-boot:run
```
## Testing
Navigate to indexer service's root folder and run all the tests:
```bash
......@@ -140,33 +157,39 @@ Navigate to indexer service's root folder and run all the tests:
$ (cd testing/indexer-test-core/ && mvn clean install)
```
### Running E2E Tests
### Running E2E Tests
This section describes how to run cloud OSDU E2E tests.
### Anthos test configuration:
[Anthos service configuration ](docs/anthos/README.md)
### Google Cloud test configuration:
[Google Cloud service configuration ](docs/gcp/README.md)
### Anthos test configuration
[Anthos service configuration](docs/anthos/README.md)
### Google Cloud test configuration
[Google Cloud service configuration](docs/gc/README.md)
## Deployment
* Data-Lake Indexer Google Cloud Endpoints on App Engine Flex environment
* Edit the app.yaml
* Open the [app.yaml](indexer/src/main/appengine/app.yaml) file in editor, and replace the YOUR-PROJECT-ID `GOOGLE_CLOUD_PROJECT` line with Google Cloud Platform project Id. Also update `AUTHORIZE_API`, `CRON_JOB_IP`, `LEGAL_HOSTNAME`, `REGION` and `SECURITY_HTTPS_CERTIFICATE_TRUST` based on your deployment
* Deploy
```sh
mvn appengine:deploy -pl org.opengroup.osdu.indexer:indexer -amd
```
* If you wish to deploy the search service without running tests
```sh
mvn appengine:deploy -pl org.opengroup.osdu.indexer:indexer -amd -DskipTests
```
or
* Google Documentation: https://cloud.google.com/cloud-build/docs/deploying-builds/deploy-appengine
* Google Documentation: <https://cloud.google.com/cloud-build/docs/deploying-builds/deploy-appengine>
#### Memory Store (Redis Instance) Setup
......@@ -179,23 +202,25 @@ The Redis instance must be created under the same region with the App Engine app
```
## Entitlements groups
Storage service account should have entitlements groups listed below:
- service.entitlements.user
- users
- service.schema-service.viewers
- service.storage.admin
- service.search.admin
* service.entitlements.user
* users
* service.schema-service.viewers
* service.storage.admin
* service.search.admin
## Licence
Copyright © Google LLC
Copyright © EPAM Systems
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
[http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0)
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment