Commit 3c41752f authored by Rustam Lotsmanenko (EPAM)'s avatar Rustam Lotsmanenko (EPAM)
Browse files

implement int tests for anthos env, update readme

parent cbecbb87
Pipeline #110021 failed with stages
in 2 minutes and 4 seconds
......@@ -108,42 +108,12 @@ $ (cd testing/storage-test-core/ && mvn clean install)
```
### Running E2E Tests
This section describes how to run cloud OSDU E2E tests (testing/integration-tests/search-test-gcp).
This section describes how to run cloud OSDU E2E tests.
You will need to have the following environment variables defined.
| name | value | description | sensitive? | source |
| --- | --- | --- | --- | --- |
| `ENTITLEMENTS_HOST` | ex `https://entitlements.com/entitlements/v1` | Entitlements API endpoint | no | output of infrastructure deployment |
| `ELASTIC_PASSWORD` | `********` | Password for Elasticsearch | yes | output of infrastructure deployment |
| `ELASTIC_USER_NAME` | `********` | User name for Elasticsearch | yes | output of infrastructure deployment |
| `ELASTIC_HOST` | ex `elastic.domain.com` | Host Elasticsearch | yes | output of infrastructure deployment |
| `ELASTIC_PORT` | ex `9243` | Port Elasticsearch | yes | output of infrastructure deployment |
| `GCLOUD_PROJECT` | ex `opendes` | Google Cloud Project Id| no | output of infrastructure deployment |
| `INDEXER_HOST` | ex `https://os-indexer-dot-opendes.appspot.com/api/indexer/v2/` | Indexer API endpoint | no | output of infrastructure deployment |
| `DATA_GROUP` | `opendes` | The service account to this group and substitute | no | - |
| `ENTITLEMENTS_DOMAIN` | ex `opendes-gcp.projects.com` | OSDU R2 to run tests under | no | - |
| `INTEGRATION_TEST_AUDIENCE` | `********` | client application ID | yes | https://console.cloud.google.com/apis/credentials || `OTHER_RELEVANT_DATA_COUNTRIES` | ex `US` | a other relevant data countries (for LEGAL_TAG) | no | - |
| `DEFAULT_DATA_PARTITION_ID_TENANT1` | ex `opendes` | HTTP Header 'Data-Partition-ID' | no | - |
| `SEARCH_INTEGRATION_TESTER` | `********` | Service account for API calls. Note: this user must have entitlements configured already | yes | https://console.cloud.google.com/iam-admin/serviceaccounts |
| `SEARCH_HOST` | ex `http://localhost:8080/api/search/v2/` | Endpoint of search service | no | - |
| `LEGAL_TAG` | ex `my-legal-tag` | a valid legal tag | no | - |
| `SECURITY_HTTPS_CERTIFICATE_TRUST` | ex `false` | Elastic client connection uses TrustSelfSignedStrategy(), if it is 'true' | false | output of infrastructure deployment |
**Entitlements configuration for integration accounts**
| INTEGRATION_TESTER | NO_DATA_ACCESS_TESTER |
| --- | --- |
| users<br/>service.entitlements.user<br/>service.search.user<br/>data.test1<br/>data.integration.test<br/>users@{tenant1}@{domain}.com |
Execute following command to build code and run all the integration tests:
```bash
# Note: this assumes that the environment variables for integration tests as outlined
# above are already exported in your environment.
$ (cd testing/search-test-gcp/ && mvn clean test)
```
### Anthos test configuration:
[Anthos service configuration ](docs/anthos/README.md)
### GCP test configuration:
[Gcp service configuration ](docs/gcp/README.md)
## Deployment
......
......@@ -100,3 +100,39 @@ curl -L -X PATCH 'http://partition.com/api/partition/v1/partitions/opendes' -H '
}'
```
### Running E2E Tests
You will need to have the following environment variables defined.
| name | value | description | sensitive? | source |
| --- | --- | --- | --- | --- |
| `ENTITLEMENTS_HOST` | ex `https://entitlements.com/entitlements/v1` | Entitlements API endpoint | no | output of infrastructure deployment |
| `ELASTIC_PASSWORD` | `********` | Password for Elasticsearch | yes | output of infrastructure deployment |
| `ELASTIC_USER_NAME` | `********` | User name for Elasticsearch | yes | output of infrastructure deployment |
| `ELASTIC_HOST` | ex `elastic.domain.com` | Host Elasticsearch | yes | output of infrastructure deployment |
| `ELASTIC_PORT` | ex `9243` | Port Elasticsearch | yes | output of infrastructure deployment |
| `INDEXER_HOST` | ex `https://os-indexer-dot-opendes.appspot.com/api/indexer/v2/` | Indexer API endpoint | no | output of infrastructure deployment |
| `DATA_GROUP` | `opendes` | The service account to this group and substitute | no | - |
| `ENTITLEMENTS_DOMAIN` | ex `opendes-gcp.projects.com` | OSDU R2 to run tests under | no | - |
| `DEFAULT_DATA_PARTITION_ID_TENANT1` | ex `opendes` | HTTP Header 'Data-Partition-ID' | no | - |
| `SEARCH_HOST` | ex `http://localhost:8080/api/search/v2/` | Endpoint of search service | no | - |
| `LEGAL_TAG` | ex `my-legal-tag` | a valid legal tag | no | - |
| `SECURITY_HTTPS_CERTIFICATE_TRUST` | ex `false` | Elastic client connection uses TrustSelfSignedStrategy(), if it is 'true' | false | output of infrastructure deployment |
| `TEST_OPENID_PROVIDER_CLIENT_ID` | `********` | Client Id for `$INTEGRATION_TESTER` | yes | -- |
| `TEST_OPENID_PROVIDER_CLIENT_SECRET` | `********` | | Client secret for `$INTEGRATION_TESTER` | -- |
| `TEST_OPENID_PROVIDER_URL` | `https://keycloak.com/auth/realms/osdu` | OpenID provider url | yes | -- |
**Entitlements configuration for integration accounts**
| INTEGRATION_TESTER | NO_DATA_ACCESS_TESTER |
| --- | --- |
| users<br/>service.entitlements.user<br/>service.search.user<br/>data.test1<br/>data.integration.test<br/>users@{tenant1}@{domain}.com |
Execute following command to build code and run all the integration tests:
```bash
# Note: this assumes that the environment variables for integration tests as outlined
# above are already exported in your environment.
$ (cd testing/search-test-anthos/ && mvn clean test)
```
......@@ -100,3 +100,40 @@ curl -L -X PATCH 'http://partition.com/api/partition/v1/partitions/opendes' -H '
}'
```
### Running E2E Tests
You will need to have the following environment variables defined.
| name | value | description | sensitive? | source |
| --- | --- | --- | --- | --- |
| `ENTITLEMENTS_HOST` | ex `https://entitlements.com/entitlements/v1` | Entitlements API endpoint | no | output of infrastructure deployment |
| `ELASTIC_PASSWORD` | `********` | Password for Elasticsearch | yes | output of infrastructure deployment |
| `ELASTIC_USER_NAME` | `********` | User name for Elasticsearch | yes | output of infrastructure deployment |
| `ELASTIC_HOST` | ex `elastic.domain.com` | Host Elasticsearch | yes | output of infrastructure deployment |
| `ELASTIC_PORT` | ex `9243` | Port Elasticsearch | yes | output of infrastructure deployment |
| `GCLOUD_PROJECT` | ex `opendes` | Google Cloud Project Id| no | output of infrastructure deployment |
| `INDEXER_HOST` | ex `https://os-indexer-dot-opendes.appspot.com/api/indexer/v2/` | Indexer API endpoint | no | output of infrastructure deployment |
| `DATA_GROUP` | `opendes` | The service account to this group and substitute | no | - |
| `ENTITLEMENTS_DOMAIN` | ex `opendes-gcp.projects.com` | OSDU R2 to run tests under | no | - |
| `INTEGRATION_TEST_AUDIENCE` | `********` | client application ID | yes | https://console.cloud.google.com/apis/credentials || `OTHER_RELEVANT_DATA_COUNTRIES` | ex `US` | a other relevant data countries (for LEGAL_TAG) | no | - |
| `DEFAULT_DATA_PARTITION_ID_TENANT1` | ex `opendes` | HTTP Header 'Data-Partition-ID' | no | - |
| `SEARCH_INTEGRATION_TESTER` | `********` | Service account for API calls. Note: this user must have entitlements configured already | yes | https://console.cloud.google.com/iam-admin/serviceaccounts |
| `SEARCH_HOST` | ex `http://localhost:8080/api/search/v2/` | Endpoint of search service | no | - |
| `LEGAL_TAG` | ex `my-legal-tag` | a valid legal tag | no | - |
| `SECURITY_HTTPS_CERTIFICATE_TRUST` | ex `false` | Elastic client connection uses TrustSelfSignedStrategy(), if it is 'true' | false | output of infrastructure deployment |
**Entitlements configuration for integration accounts**
| INTEGRATION_TESTER | NO_DATA_ACCESS_TESTER |
| --- | --- |
| users<br/>service.entitlements.user<br/>service.search.user<br/>data.test1<br/>data.integration.test<br/>users@{tenant1}@{domain}.com |
Execute following command to build code and run all the integration tests:
```bash
# Note: this assumes that the environment variables for integration tests as outlined
# above are already exported in your environment.
$ (cd testing/search-test-gcp/ && mvn clean test)
```
......@@ -36,7 +36,8 @@
<module>search-test-azure</module>
<module>search-test-gcp</module>
<module>search-test-ibm</module>
</modules>
<module>search-test-anthos</module>
</modules>
<repositories>
<repository>
......
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Copyright 2020-2022 Google LLC
~ Copyright 2020-2022 EPAM Systems, Inc
~
~ Licensed under the Apache License, Version 2.0 (the "License");
~ you may not use this file except in compliance with the License.
~ You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>os-search</artifactId>
<groupId>org.opengroup.osdu</groupId>
<version>0.15.0-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
<groupId>org.opengroup.osdu.search</groupId>
<artifactId>search-test-anthos</artifactId>
<version>0.15.0-SNAPSHOT</version>
<properties>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.opengroup.osdu.search</groupId>
<artifactId>search-test-core</artifactId>
<version>0.15.0-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.2</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.nimbusds</groupId>
<artifactId>oauth2-oidc-sdk</artifactId>
<version>9.15</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-to-slf4j</artifactId>
<version>2.11.2</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-jdk14</artifactId>
<version>1.8.0-beta4</version>
</dependency>
</dependencies>
</project>
\ No newline at end of file
/*
* Copyright 2020-2022 Google LLC
* Copyright 2020-2022 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package osdu.step_definitions.info;
import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;
@RunWith(Cucumber.class)
@CucumberOptions(
features = "classpath:features/info/Info.feature",
glue = {"classpath:org.opengroup.osdu.step_definitions/info"},
plugin = {"pretty", "junit:target/cucumber-reports/TEST-info.xml"})
public class RunTest {
}
/*
* Copyright 2020-2022 Google LLC
* Copyright 2020-2022 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package osdu.step_definitions.info;
import cucumber.api.Scenario;
import cucumber.api.java.Before;
import cucumber.api.java.en.Then;
import cucumber.api.java.en.When;
import org.opengroup.osdu.common.info.InfoSteps;
import org.opengroup.osdu.util.Config;
import osdu.util.AnthosHTTPClient;
public class Steps extends InfoSteps {
public Steps() {
super(new AnthosHTTPClient());
}
@Before
public void before(Scenario scenario) {
this.scenario = scenario;
this.httpClient = new AnthosHTTPClient();
}
/******************Inputs being set**************/
@When("^I send get request to version info endpoint")
public void i_send_get_request_to_version_info_endpoint() {
super.i_send_get_request_to_version_info_endpoint();
}
/******************Assert final response**************/
@Then("^I should get version info in response$")
public void i_should_get_version_info_in_response() {
super.i_should_get_version_info_in_response();
}
@Override
protected String getHttpMethod() {
return "GET";
}
@Override
protected String getApi() {
return Config.getSearchBaseURL() + "info";
}
}
/*
* Copyright 2020-2022 Google LLC
* Copyright 2020-2022 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package osdu.step_definitions.query.singlecluster;
import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;
@RunWith(Cucumber.class)
@CucumberOptions(
features = "classpath:features/query/singlecluster/Query.feature",
glue = {"classpath:org.opengroup.osdu.step_definitions/query/singlecluster"},
plugin = {"pretty", "junit:target/cucumber-reports/TEST-query-sc.xml"})
public class RunTest {
}
\ No newline at end of file
/*
* Copyright 2020-2022 Google LLC
* Copyright 2020-2022 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package osdu.step_definitions.query.singlecluster;
import cucumber.api.DataTable;
import cucumber.api.Scenario;
import cucumber.api.java.Before;
import cucumber.api.java.en.Given;
import cucumber.api.java.en.Then;
import cucumber.api.java.en.When;
import java.util.List;
import org.opengroup.osdu.common.query.singlecluster.QuerySteps;
import org.opengroup.osdu.util.Config;
import osdu.util.AnthosHTTPClient;
public class Steps extends QuerySteps {
public Steps() {
super(new AnthosHTTPClient());
}
@Given("^the elastic search is initialized with the following data$")
public void the_elastic_search_is_initialized_with_the_following_data(DataTable dataTable) {
super.the_elastic_search_is_initialized_with_the_following_data(dataTable);
}
@Before
public void before(Scenario scenario) {
this.scenario = scenario;
this.httpClient = new AnthosHTTPClient();
}
/******************Inputs being set**************/
@When("^I limit the count of returned results to (-?\\d+)$")
public void i_limit_the_count_of_returned_results_to(int limit) {
super.i_limit_the_count_of_returned_results_to(limit);
}
@When("^I send None with \"(.*?)\"$")
public void i_send_None_with(String kind) {
super.i_send_None_with(kind);
}
@When("^I send \"(.*?)\" with \"(.*?)\"$")
public void i_send_with(String query, String kind) {
super.i_send_with(query, kind);
}
@When("^I want the results sorted by (.*?)$")
public void i_sort_with(String sortJson) {
super.i_sort_with(sortJson);
}
@When("^I want to aggregate by \"(.*?)\"$")
public void i_aggregate_by(String aggField) throws Throwable {
super.i_aggregate_by(aggField);
}
@When("^I want to search as owner (.*?)$")
public void i_search_as(String isOwner) {
super.i_search_as(isOwner);
}
@When("^I set the fields I want in response as ([\"(a-zA-Z0-9)\",?]*)$")
public void i_set_the_fields_I_want_in_response_as(List<String> returnedFileds) {
super.i_set_the_fields_I_want_in_response_as(returnedFileds);
}
@When("^I set the offset of starting point as None$$|^I limit the count of returned results to None$$")
public void offset_of_starting_point_as_None() {
super.offset_of_starting_point_as_None();
}
@When("^I set the offset of starting point as (-?\\d+)$")
public void i_set_the_offset_of_starting_point_as(int offset) {
super.i_set_the_offset_of_starting_point_as(offset);
}
@When("^I send request to tenant \"(.*?)\"$")
public void i_send_request_to_tenant(String tenant) {
super.i_send_request_to_tenant(tenant);
}
@When("^I apply geographical query on field \"(.*?)\"$")
public void i_apply_geographical_query_on_field(String field) {
super.i_apply_geographical_query_on_field(field);
}
@When("^define bounding box with points \\((-?\\d+), (-?\\d+)\\) and \\((-?\\d+), (-?\\d+)\\)$")
public void define_bounding_box_with_points_and(Double topLatitude, Double topLongitude, Double bottomLatitude, Double
bottomLongitude) {
super.define_bounding_box_with_points_and(topLatitude, topLongitude, bottomLatitude, bottomLongitude);
}
@When("^define bounding box with points \\(None, None\\) and \\((\\d+), (\\d+)\\)$")
public void define_bounding_box_with_points_None_None_and(Double bottomLatitude, Double bottomLongitude) {
super.define_bounding_box_with_points_None_None_and(bottomLatitude, bottomLongitude);
}
@When("^define focus coordinates as \\((-?\\d+), (-?\\d+)\\) and search in a (\\d+) radius$")
public void define_focus_coordinates_as_and_search_in_a_radius(Double latitude, Double longitude, int distance) {
super.define_focus_coordinates_as_and_search_in_a_radius(latitude, longitude, distance);
}
@When("^define intersection polygon with points \\((-?\\d+), (-?\\d+)\\) and \\((-?\\d+), (-?\\d+)\\) and \\((-?\\d+), (-?\\d+)\\) and \\((-?\\d+), (-?\\d+)\\) and \\((-?\\d+), (-?\\d+)\\)")
public void define_intersection_polygon_with_points(Double latitude1, Double longitude1, Double latitude2, Double longitude2,
Double latitude3, Double longitude3, Double latitude4, Double longitude4,
Double latitude5, Double longitude5) {
super.define_intersection_polygon_with_points(latitude1, longitude1, latitude2, longitude2, latitude3, longitude3, latitude4, longitude4, latitude5,
longitude5);
}
@When("^define within polygon with points \\((-?\\d+), (-?\\d+)\\)")
public void define_within_polygon_with_points(Double latitude1, Double longitude1) {
super.define_within_polygon_with_points(latitude1, longitude1);
}
/******************Assert final response**************/
@Then("^I should get in response (\\d+) records$")
public void i_should_get_in_response_records(int resultCount) {
super.i_should_get_in_response_records(resultCount);
}
@Then("^I should get in response (\\d+) records with ([\"(a-zA-Z0-9)\",?]*)$")
public void i_should_get_in_response_records_with_fields(int resultCount, List<String> returnedFields) {
super.i_should_get_in_response_records(resultCount, returnedFields);
}
@Then("^I should get records in right order first record id: \"(.*?)\", last record id: \"(.*?)\"$")
public void i_should_get_records_in_right_order(String firstRecId, String lastRecId) {
super.i_should_get_records_in_right_order(firstRecId, lastRecId);
}
@Then("^I should get (\\d+) unique values")
public void i_should_get_aggregation_with_unique_values(int uniqueValueCount) {
super.i_should_get_aggregation_with_unique_values(uniqueValueCount);
}
@Then("^I should get ([^\"]*) response with reason: \"(.*?)\", message: \"(.*?)\" and errors: \"(.*?)\"$")
public void i_should_get_response_with_reason_message_and_errors(List<Integer> codes, String type, String msg, String error) {
super.i_should_get_response_with_reason_message_and_errors(codes, type, msg, error);
}
@When("^define geo polygon with following points ([^\"]*)$")
public void define_geo_polygon_with_following_points_points_list(List<String> points) {
super.define_geo_polygon_with_following_points_points_list(points);
}
@Override
protected String getHttpMethod() {
return "POST";
}
@Override
protected String getApi() {
return Config.getSearchBaseURL() + "query";
}
}
\ No newline at end of file
/*
* Copyright 2020-2022 Google LLC
* Copyright 2020-2022 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package osdu.step_definitions.querybycursor.singlecluster;
import cucumber.api.CucumberOptions;
import cucumber.api.junit.Cucumber;
import org.junit.runner.RunWith;
@RunWith(Cucumber.class)
@CucumberOptions(
features = "classpath:features/querybycursor/singlecluster/QueryByCursor.feature",
glue = {"classpath:org.opengroup.osdu.step_definitions/querybycursor/singlecluster"},
plugin = {"pretty", "junit:target/cucumber-reports/TEST-querybycursor-sc.xml"})
public class RunTest {
}
/*
* Copyright 2020-2022 Google LLC
* Copyright 2020-2022 EPAM Systems, Inc
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package osdu.step_definitions.querybycursor.singlecluster;
import cucumber.api.DataTable;
import cucumber.api.Scenario;