Skip to content
Snippets Groups Projects
Commit 4998c110 authored by Zhibin Mai's avatar Zhibin Mai
Browse files

Merge branch 'index_extended_properties' of... 

Merge branch 'index_extended_properties' of https://community.opengroup.org/osdu/platform/system/indexer-service
parents 005fee69 bacf58dc
No related branches found
No related tags found
1 merge request!465Index extended properties defined in property configurations
Hide whitespace changes
Inline Side-by-side
Showing
with 112 additions and 58 deletions
devops/gc/pipeline/override-stages.yml
+ 5
− 0
View file @ 4998c110
......@@ -2,6 +2,11 @@ variables:
GC_SERVICE: indexer
GC_VENDOR: gc
gc-test:
variables:
CUCUMBER_OPTIONS: "--tags '~@* and @indexer-extended'"
gc-baremetal-test:
variables:
GC_VENDOR: baremetal
CUCUMBER_OPTIONS: "--tags '~@* and @indexer-extended'"
provider/indexer-aws/README.md
+ 21
− 20
View file @ 4998c110
......@@ -123,27 +123,28 @@ You should see in the logs that pop up what url and port it runs on. By default
export ELASTIC_PASSWORD=$ELASTIC_PASSWORD
export ELASTIC_USER_NAME=$ELASTIC_USERNAME
| name | example value | description | sensitive?
| --- | --- | --- | --- |
| name | example value | description | sensitive?
| --- |----------------------------------------------------------------------------------------| --- | --- |
| `AWS_ACCESS_KEY_ID` | `ASIAXXXXXXXXXXXXXX` | The AWS Access Key for a user with access to Backend Resources required by the service | yes |
| `AWS_SECRET_ACCESS_KEY` | `super-secret-key==` | The AWS Secret Key for a user with access to Backend Resources required by the service | yes |
| `AWS_SESSION_TOKEN` | `session-token-xxxxxxxxx` | AWS Session token needed if using an SSO user session to authenticate | yes |
| `AWS_COGNITO_USER_POOL_ID` | `us-east-1_xxxxxxxx` | User Pool Id for the reference cognito | no |
| `AWS_COGNITO_CLIENT_ID` | `xxxxxxxxxxxx` | Client ID for the Auth Flow integrated with the Cognito User Pool | no |
| `AWS_COGNITO_AUTH_FLOW` | `USER_PASSWORD_AUTH` | Auth flow used by reference cognito deployment | no |
| `DEFAULT_DATA_PARTITION_ID_TENANT1` | `opendes` | Partition used to create and index record | no |
| `DEFAULT_DATA_PARTITION_ID_TENANT2` | `common` | Another needed partition| no |
| `AWS_COGNITO_AUTH_PARAMS_USER` | `int-test-user@testing.com` | Int Test Username | no |
| `AWS_COGNITO_AUTH_PARAMS_USER_NO_ACCESS` | `noaccess@testing.com` | No Access Username | no |
| `AWS_COGNITO_AUTH_PARAMS_PASSWORD` | `some-secure-password` | Int Test User/NoAccessUser Password | yes |
| `ENTITLEMENTS_DOMAIN` | `example.com` | Domain for user's groups | no |
| `OTHER_RELEVANT_DATA_COUNTRIES` | `US` | Used to create demo legal tag | no |
| `STORAGE_HOST` | `http://localhost:8080/api/storage/v2/` | The url where the storage API is hosted | no |
| `HOST` | `http://localhost:8080` | Base url for deployment | no |
| `ELASTIC_HOST` | `localhost` | Url for elasticsearch | no |
| `ELASTIC_PORT` | `9300` | Port for elasticsearch | no |
| `ELASTIC_PASSWORD` | `xxxxxxxxxxxxxxx` | Password for user to access elasticsearch | yes |
| `ELASTIC_USER_NAME` | `xxxxxxxxxxxxxxxx` | Username for user to access elasticsearch | yes |
| `AWS_SESSION_TOKEN` | `session-token-xxxxxxxxx` | AWS Session token needed if using an SSO user session to authenticate | yes |
| `AWS_COGNITO_USER_POOL_ID` | `us-east-1_xxxxxxxx` | User Pool Id for the reference cognito | no |
| `AWS_COGNITO_CLIENT_ID` | `xxxxxxxxxxxx` | Client ID for the Auth Flow integrated with the Cognito User Pool | no |
| `AWS_COGNITO_AUTH_FLOW` | `USER_PASSWORD_AUTH` | Auth flow used by reference cognito deployment | no |
| `DEFAULT_DATA_PARTITION_ID_TENANT1` | `opendes` | Partition used to create and index record | no |
| `DEFAULT_DATA_PARTITION_ID_TENANT2` | `common` | Another needed partition | no |
| `AWS_COGNITO_AUTH_PARAMS_USER` | `int-test-user@testing.com` | Int Test Username | no |
| `AWS_COGNITO_AUTH_PARAMS_USER_NO_ACCESS` | `noaccess@testing.com` | No Access Username | no |
| `AWS_COGNITO_AUTH_PARAMS_PASSWORD` | `some-secure-password` | Int Test User/NoAccessUser Password | yes |
| `ENTITLEMENTS_DOMAIN` | `example.com` | Domain for user's groups | no |
| `OTHER_RELEVANT_DATA_COUNTRIES` | `US` | Used to create demo legal tag | no |
| `STORAGE_HOST` | `http://localhost:8080/api/storage/v2/` | The url where the storage API is hosted | no |
| `HOST` | `http://localhost:8080` | Base url for deployment | no |
| `ELASTIC_HOST` | `localhost` | Url for elasticsearch | no |
| `ELASTIC_PORT` | `9300` | Port for elasticsearch | no |
| `ELASTIC_PASSWORD` | `xxxxxxxxxxxxxxx` | Password for user to access elasticsearch | yes |
| `ELASTIC_USER_NAME` | `xxxxxxxxxxxxxxxx` | Username for user to access elasticsearch | yes |
| `CUCUMBER_OPTIONS` | `--tags '~@indexer-extended'` OR `--tags '~@* and @indexer-extended'` | By default `--tags '~@* and @indexer-extended'` to enable experimental feature testing | no |
**Creating a new user to use for integration tests**
......@@ -199,4 +200,4 @@ Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
\ No newline at end of file
limitations under the License.
provider/indexer-azure/README.md
+ 1
− 0
View file @ 4998c110
......@@ -85,6 +85,7 @@ az keyvault secret show --vault-name $KEY_VAULT_NAME --name $KEY_VAULT_SECRET_NA
| `ENTITLEMENTS_DOMAIN` | `contoso.com` | OSDU R2 service domain | no | - |
| `LEGAL_TAG` | `opendes-public-usa-dataset-7643990` | Legal tag used for test records | no | Needs to be in DB. The referenced tag should already exist. |
| `OTHER_RELEVANT_DATA_COUNTRIES` | `US` | ? | no | - |
| `CUCUMBER_OPTIONS` | `--tags '~@indexer-extended'` OR `--tags '~@* and @indexer-extended'` | By default `--tags '~@* and @indexer-extended'` to enable experimental feature testing | no | - |
### Configure Maven
......
provider/indexer-gc/docs/anthos/README.md
+ 19
− 18
View file @ 4998c110
......@@ -233,24 +233,25 @@ Give `client-id` and `client-secret` to services, which should be authorized wit
You will need to have the following environment variables defined.
| name | value | description | sensitive? | source |
|--------------------------------------|-----------------------------------------------------------------|---------------------------------------------------------------------------------------------------|-----------------------------------------|-------------------------------------|
| `ELASTIC_PASSWORD` | `********` | Password for Elasticsearch | yes | output of infrastructure deployment |
| `ELASTIC_USER_NAME` | `********` | User name for Elasticsearch | yes | output of infrastructure deployment |
| `ELASTIC_HOST` | ex `elastic.domain.com` | Host Elasticsearch | yes | output of infrastructure deployment |
| `ELASTIC_PORT` | ex `9243` | Port Elasticsearch | yes | output of infrastructure deployment |
| `INDEXER_HOST` | ex `https://os-indexer-dot-opendes.appspot.com/api/indexer/v2/` | Indexer API endpoint | no | output of infrastructure deployment |
| `GROUP_ID` | ex `opendes-gcp.projects.com` | OSDU R2 to run tests under | no | - |
| `OTHER_RELEVANT_DATA_COUNTRIES` | ex `US` | valid legal tag with a other relevant data countries | no | - |
| `LEGAL_TAG` | ex `opendes-demo-legaltag` | valid legal tag with a other relevant data countries from `DEFAULT_OTHER_RELEVANT_DATA_COUNTRIES` | no | - |
| `DEFAULT_DATA_PARTITION_ID_TENANT1` | ex `opendes` | HTTP Header 'Data-Partition-ID' | no | - |
| `DEFAULT_DATA_PARTITION_ID_TENANT2` | ex `opendes` | HTTP Header 'Data-Partition-ID' | no | - |
| `SEARCH_HOST` | ex `http://localhost:8080/api/search/v2/` | Endpoint of search service | no | - |
| `STORAGE_HOST` | ex `http://os-storage-dot-opendes.appspot.com/api/storage/v2/` | Storage API endpoint | no | output of infrastructure deployment |
| `SECURITY_HTTPS_CERTIFICATE_TRUST` | ex `false` | Elastic client connection uses TrustSelfSignedStrategy(), if it is 'true' | false | output of infrastructure deployment |
| `TEST_OPENID_PROVIDER_CLIENT_ID` | `********` | Client Id for `$INTEGRATION_TESTER` | yes | -- |
| `TEST_OPENID_PROVIDER_CLIENT_SECRET` | `********` | | Client secret for `$INTEGRATION_TESTER` | -- |
| `TEST_OPENID_PROVIDER_URL` | `https://keycloak.com/auth/realms/osdu` | OpenID provider url | yes | -- |
| name | value | description | sensitive? | source |
|--------------------------------------|-----------------------------------------------------------------------|---------------------------------------------------------------------------------------------------|-----------------------------------------|-------------------------------------|
| `ELASTIC_PASSWORD` | `********` | Password for Elasticsearch | yes | output of infrastructure deployment |
| `ELASTIC_USER_NAME` | `********` | User name for Elasticsearch | yes | output of infrastructure deployment |
| `ELASTIC_HOST` | ex `elastic.domain.com` | Host Elasticsearch | yes | output of infrastructure deployment |
| `ELASTIC_PORT` | ex `9243` | Port Elasticsearch | yes | output of infrastructure deployment |
| `INDEXER_HOST` | ex `https://os-indexer-dot-opendes.appspot.com/api/indexer/v2/` | Indexer API endpoint | no | output of infrastructure deployment |
| `GROUP_ID` | ex `opendes-gcp.projects.com` | OSDU R2 to run tests under | no | - |
| `OTHER_RELEVANT_DATA_COUNTRIES` | ex `US` | valid legal tag with a other relevant data countries | no | - |
| `LEGAL_TAG` | ex `opendes-demo-legaltag` | valid legal tag with a other relevant data countries from `DEFAULT_OTHER_RELEVANT_DATA_COUNTRIES` | no | - |
| `DEFAULT_DATA_PARTITION_ID_TENANT1` | ex `opendes` | HTTP Header 'Data-Partition-ID' | no | - |
| `DEFAULT_DATA_PARTITION_ID_TENANT2` | ex `opendes` | HTTP Header 'Data-Partition-ID' | no | - |
| `SEARCH_HOST` | ex `http://localhost:8080/api/search/v2/` | Endpoint of search service | no | - |
| `STORAGE_HOST` | ex `http://os-storage-dot-opendes.appspot.com/api/storage/v2/` | Storage API endpoint | no | output of infrastructure deployment |
| `SECURITY_HTTPS_CERTIFICATE_TRUST` | ex `false` | Elastic client connection uses TrustSelfSignedStrategy(), if it is 'true' | false | output of infrastructure deployment |
| `TEST_OPENID_PROVIDER_CLIENT_ID` | `********` | Client Id for `$INTEGRATION_TESTER` | yes | -- |
| `TEST_OPENID_PROVIDER_CLIENT_SECRET` | `********` | | Client secret for `$INTEGRATION_TESTER` | -- |
| `TEST_OPENID_PROVIDER_URL` | `https://keycloak.com/auth/realms/osdu` | OpenID provider url | yes | -- |
| `CUCUMBER_OPTIONS` | `--tags '~@indexer-extended'` OR `--tags '~@* and @indexer-extended'` | By default `--tags '~@indexer-extended'` to disable experimental feature testing | no | -- |
**Entitlements configuration for integration accounts**
......
provider/indexer-gc/docs/gc/README.md
+ 18
− 17
View file @ 4998c110
......@@ -168,23 +168,24 @@ TBD
You will need to have the following environment variables defined.
| name | value | description | sensitive? | source |
|-------------------------------------|----------------------------------------------------------------|---------------------------------------------------------------------------------------------------|------------|------------------------------------------------------------|
| `ELASTIC_PASSWORD` | `********` | Password for Elasticsearch | yes | output of infrastructure deployment |
| `ELASTIC_USER_NAME` | `********` | User name for Elasticsearch | yes | output of infrastructure deployment |
| `ELASTIC_HOST` | ex `elastic.domain.com` | Host Elasticsearch | yes | output of infrastructure deployment |
| `ELASTIC_PORT` | ex `9243` | Port Elasticsearch | yes | output of infrastructure deployment |
| `GCLOUD_PROJECT` | ex `opendes` | Google Cloud Project Id | no | output of infrastructure deployment |
| `INDEXER_HOST` | ex `https://os-indexer-dot-opendes.appspot.com/api/indexer/v2/` | Indexer API endpoint | no | output of infrastructure deployment |
| `GROUP_ID` | ex `opendes-gc.projects.com` | OSDU R2 to run tests under | no | - |
| `OTHER_RELEVANT_DATA_COUNTRIES` | ex `US` | valid legal tag with a other relevant data countries | no | - |
| `LEGAL_TAG` | ex `opendes-demo-legaltag` | valid legal tag with a other relevant data countries from `DEFAULT_OTHER_RELEVANT_DATA_COUNTRIES` | no | - |
| `DEFAULT_DATA_PARTITION_ID_TENANT1` | ex `opendes` | HTTP Header 'Data-Partition-ID' | no | - |
| `DEFAULT_DATA_PARTITION_ID_TENANT2` | ex `opendes` | HTTP Header 'Data-Partition-ID' | no | - |
| `SEARCH_INTEGRATION_TESTER` | `********` | Service account for API calls. Note: this user must have entitlements configured already | yes | <https://console.cloud.google.com/iam-admin/serviceaccounts> |
| `SEARCH_HOST` | ex `http://localhost:8080/api/search/v2/` | Endpoint of search service | no | - |
| `STORAGE_HOST` | ex `http://os-storage-dot-opendes.appspot.com/api/storage/v2/` | Storage API endpoint | no | output of infrastructure deployment |
| `SECURITY_HTTPS_CERTIFICATE_TRUST` | ex `false` | Elastic client connection uses TrustSelfSignedStrategy(), if it is 'true' | false | output of infrastructure deployment |
| name | value | description | sensitive? | source |
|-------------------------------------|-----------------------------------------------------------------------|---------------------------------------------------------------------------------------------------|------------|--------------------------------------------------------------|
| `ELASTIC_PASSWORD` | `********` | Password for Elasticsearch | yes | output of infrastructure deployment |
| `ELASTIC_USER_NAME` | `********` | User name for Elasticsearch | yes | output of infrastructure deployment |
| `ELASTIC_HOST` | ex `elastic.domain.com` | Host Elasticsearch | yes | output of infrastructure deployment |
| `ELASTIC_PORT` | ex `9243` | Port Elasticsearch | yes | output of infrastructure deployment |
| `GCLOUD_PROJECT` | ex `opendes` | Google Cloud Project Id | no | output of infrastructure deployment |
| `INDEXER_HOST` | ex `https://os-indexer-dot-opendes.appspot.com/api/indexer/v2/` | Indexer API endpoint | no | output of infrastructure deployment |
| `GROUP_ID` | ex `opendes-gc.projects.com` | OSDU R2 to run tests under | no | - |
| `OTHER_RELEVANT_DATA_COUNTRIES` | ex `US` | valid legal tag with a other relevant data countries | no | - |
| `LEGAL_TAG` | ex `opendes-demo-legaltag` | valid legal tag with a other relevant data countries from `DEFAULT_OTHER_RELEVANT_DATA_COUNTRIES` | no | - |
| `DEFAULT_DATA_PARTITION_ID_TENANT1` | ex `opendes` | HTTP Header 'Data-Partition-ID' | no | - |
| `DEFAULT_DATA_PARTITION_ID_TENANT2` | ex `opendes` | HTTP Header 'Data-Partition-ID' | no | - |
| `SEARCH_INTEGRATION_TESTER` | `********` | Service account for API calls. Note: this user must have entitlements configured already | yes | <https://console.cloud.google.com/iam-admin/serviceaccounts> |
| `SEARCH_HOST` | ex `http://localhost:8080/api/search/v2/` | Endpoint of search service | no | - |
| `STORAGE_HOST` | ex `http://os-storage-dot-opendes.appspot.com/api/storage/v2/` | Storage API endpoint | no | output of infrastructure deployment |
| `SECURITY_HTTPS_CERTIFICATE_TRUST` | ex `false` | Elastic client connection uses TrustSelfSignedStrategy(), if it is 'true' | false | output of infrastructure deployment |
| `CUCUMBER_OPTIONS` | `--tags '~@indexer-extended'` OR `--tags '~@* and @indexer-extended'` | By default `--tags '~@indexer-extended'` to disable experimental feature testing | no | -- |
**Entitlements configuration for integration accounts**
......
testing/indexer-test-aws/src/test/resources/cucumber.properties 0 → 100644
+ 2
− 0
View file @ 4998c110
# tag indexer-extended disabled by default
cucumber.options=--tags '~@* and @indexer-extended'
testing/indexer-test-azure/src/test/resources/cucumber.properties 0 → 100644
+ 2
− 0
View file @ 4998c110
# tag indexer-extended disabled by default
cucumber.options=--tags '~@* and @indexer-extended'
testing/indexer-test-baremetal/src/test/resources/cucumber.properties 0 → 100644
+ 18
− 0
View file @ 4998c110
#
# Copyright 2020-2023 Google LLC
# Copyright 2020-2023 EPAM Systems, Inc
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# tag indexer-extended disabled by default
cucumber.options=--tags ~@indexer-extended
testing/indexer-test-core/pom.xml
+ 1
− 1
View file @ 4998c110
......@@ -85,7 +85,7 @@
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.2</version>
<version>1.18.26</version>
<scope>provided</scope>
</dependency>
......
testing/indexer-test-core/src/main/resources/features/indexrecord/indexRecord-schema-service.feature
+ 4
− 1
View file @ 4998c110
......@@ -18,7 +18,7 @@ Feature: Indexing of the documents
| test:indexer:index-property--Wellbore:1.0.0 | test-indexer-index-property--wellbore-1.0.0 | index-property-wellbore_v1 |
| test:indexer:index-property--WellLog:1.0.0 | test-indexer-index-property--welllog-1.0.0 | index-property-welllog_v1 |
@indexer-extended
Scenario Outline: Prepare the index property configuration records and clean up index of the extended kinds in the Elastic Search
When I ingest records with the <recordFile> with <acl> for a given <kind>
Then I should get the <number> documents for the <index> in the Elastic Search
......@@ -29,6 +29,7 @@ Feature: Indexing of the documents
| kind | recordFile | number | index | acl | extendedKinds |
| "osdu:wks:reference-data--IndexPropertyPathConfiguration:1.0.0" | "osdu_wks_IndexPropertyPathConfiguration_v1" | 2 | "osdu-wks-reference-data--indexpropertypathconfiguration-1.0.0" | "data.default.viewers@tenant1" | "test:indexer:index-property--Wellbore:1.0.0,test:indexer:index-property--WellLog:1.0.0" |
@indexer-extended
Scenario Outline: Ingest the records of the extended kinds, Index in the Elastic Search and Search string field
When I ingest records with the <recordFile> with <acl> for a given <kind>
Then I should be able to search <number> record with index <index> by extended data field <field> and value <value>
......@@ -38,6 +39,7 @@ Feature: Indexing of the documents
| "test:indexer:index-property--Wellbore:1.0.0" | "index-property-wellbore_v1" | 1 | "test-indexer-index-property--wellbore-1.0.0" | "data.default.viewers@tenant1" | "data.WellUWI" | "123454321" |
| "test:indexer:index-property--WellLog:1.0.0" | "index-property-welllog_v1" | 1 | "test-indexer-index-property--welllog-1.0.0" | "data.default.viewers@tenant1" | "data.WellboreName" | "Facility_123" |
@indexer-extended
Scenario Outline: Ingest the records of the extended kinds, Index in the Elastic Search and Search spatial field
When I ingest records with the <recordFile> with <acl> for a given <kind>
Then I should be able search <number> documents for the <index> by bounding box query with points (<top_left_latitude>, <top_left_longitude>) and (<bottom_right_latitude>, <bottom_right_longitude>) on field <field>
......@@ -47,6 +49,7 @@ Feature: Indexing of the documents
| "test:indexer:index-property--Wellbore:1.0.0" | "index-property-wellbore_v1" | 1 | "test-indexer-index-property--wellbore-1.0.0" | "data.default.viewers@tenant1" | "data.Location" | 30 | -96 | 29 | -95 |
| "test:indexer:index-property--WellLog:1.0.0" | "index-property-welllog_v1" | 1 | "test-indexer-index-property--welllog-1.0.0" | "data.default.viewers@tenant1" | "data.SpatialLocation" | 30 | -96 | 29 | -95 |
@indexer-extended
Scenario: End Stateful Scenarios
Then I set ending stateful scenarios
......
testing/indexer-test-gc/pom.xml
+ 1
− 1
View file @ 4998c110
......@@ -87,7 +87,7 @@
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.2</version>
<version>1.18.26</version>
<scope>provided</scope>
</dependency>
......
testing/indexer-test-gc/src/test/resources/cucumber.properties 0 → 100644
+ 18
− 0
View file @ 4998c110
#
# Copyright 2020-2023 Google LLC
# Copyright 2020-2023 EPAM Systems, Inc
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# tag indexer-extended disabled by default
cucumber.options=--tags ~@indexer-extended
testing/indexer-test-ibm/src/main/resources/cucumber.properties 0 → 100644
+ 2
− 0
View file @ 4998c110
# tag indexer-extended disabled by default
cucumber.options=--tags '~@* and @indexer-extended'
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment