Merge branch 'docs-and-logs-improvement' into 'master'

Docs and logs improvement (GONRG-1035)

See merge request !31

testing/notification-test-gcp/README.md → provider/notification-gcp/README.md

+#Notification Service
+notification-gcp is a [Spring Boot](https://spring.io/projects/spring-boot) service that allow for interested consumers to subscribe to data and metadata changes using a publish/subscriber pattern.
+
+## Getting Started
+These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.
+
+### Requirements
+* Java 8
+* [Maven 3.6.0+](https://maven.apache.org/download.cgi)
+* GCloud command line tool
+* GCloud access to opendes project
+
+### General Tips
+
+**Environment Variable Management**
+The following tools make environment variable configuration simpler
+ - [direnv](https://direnv.net/) - for a shell/terminal environment
+ - [EnvFile](https://plugins.jetbrains.com/plugin/7861-envfile) - for [Intellij IDEA](https://www.jetbrains.com/idea/)
+
+**Lombok**
+This project uses [Lombok](https://projectlombok.org/) for code generation. You may need to configure your IDE to take advantage of this tool.
+ - [Intellij configuration](https://projectlombok.org/setup/intellij)
+ - [VSCode configuration](https://projectlombok.org/setup/vscode)
+
+### Installation
+In order to run the service locally or remotely, you will need to have the following environment variables defined.
+
+| name | value | description | sensitive? | source |
+| ---  | ---   | ---         | ---        | ---    |
+| `APP_ENTITLEMENTS` | ex `https://entitlements.com/entitlements/v1` | Entitlements API endpoint | no | output of infrastructure deployment |
+| `APP_REGISTER` | ex `https://register.com/api/register/v1` | Storage API endpoint | no | output of infrastructure deployment |
+| `APP_PROJECT` | ex `opendes` | Google Cloud Project Id | no | output of infrastructure deployment |
+| `APP_AUDIENCES` | ex `*****.apps.googleusercontent.com` | Client ID for getting access to cloud resources | yes | https://console.cloud.google.com/apis/credentials |
+
+**System Environment required to run service**
+
+| name | value | description | sensitive? | source |
+| ---  | ---   | ---         | ---        | ---    |
+| `SPRING_PROFILES_ACTIVE` | `local` | spring active profile | no |
+
+### Run Locally
+Check that maven is installed:
+```bash
+$ mvn --version
+Apache Maven 3.6.0
+Maven home: /usr/share/maven
+Java version: 1.8.0_212, vendor: AdoptOpenJDK, runtime: /usr/lib/jvm/jdk8u212-b04/jre
+...
+```
+
+You will need to configure access to the remote maven repository that holds the OSDU dependencies. This file should live within `~/.m2/settings.xml`:
+
+```bash
+$ cat ~/.m2/settings.xml
+<?xml version="1.0" encoding="UTF-8"?>
+<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
+          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
+    <servers>
+        <server>
+            <id>os-core</id>
+            <username>slb-des-ext-collaboration</username>
+            <!-- Treat this auth token like a password. Do not share it with anyone, including Microsoft support. -->
+            <password>${VSTS_FEED_TOKEN}</password>
+        </server>
+    </servers>
+</settings>
+```
+
+* Update the Google cloud SDK to the latest version:
+
+```bash
+gcloud components update
+```
+* Set Google Project Id:
+
+```bash
+gcloud config set project <YOUR-PROJECT-ID>
+```
+
+* Perform a basic authentication in the selected project:
+
+```bash
+gcloud auth application-default login
+```
+
+* Navigate to notification service's root folder and run:
+
+```bash
+mvn jetty:run
+## Testing
+* Navigate to notification service's root folder and run:
+ 
+```bash
+mvn clean install   
+```
+
+* If you wish to see the coverage report then go to testing/target/site/jacoco-aggregate and open index.html
+
+* If you wish to build the project without running tests
+
+```bash
+mvn clean install -DskipTests
+```
+
+After configuring your environment as specified above, you can follow these steps to build and run the application. These steps should be invoked from the *repository root.*
+
+```bash
+cd provider/notification-gcp/ && mvn spring-boot:run -Dspring-boot.run.profiles=local 
+```
+
+## Testing
+Navigate to notification service's root folder and run all the tests:
+
+```bash
+# build + test + install core service code
+$ (cd notification-core/ && mvn clean install)
+```
+
+## Test the application
+
+After the service has started it should be accessible via a web browser by visiting [http://localhost:8080/api/notification/v1/swagger-ui.html](http://localhost:8080/swagger-ui.html). If the request does not fail, you can then run the integration tests.
+
 ###Dependencies needed to run the integration tests 
-* JDK11
+* Java 8
 * Maven
 * Values for the following environment variables in Config.java
   
 | name | value | description | sensitive? | source |
 | ---  | ---   | ---         | ---        | ---    |
-| `DE_OPS_TESTER` | `*****` | Service account base64 encoded string for API calls. Note: this user must have entitlements configured already | yes | https://console.cloud.google.com/iam-admin/serviceaccounts |
+| `DE_OPS_TESTER` | `*****` | Service account base64 encoded string for API calls. Note: this user must have entitlements configured already, also **Private key id** of this account must be set in Register service variable SUBSCRIBER_PRIVATE_KEY_ID  | yes | https://console.cloud.google.com/iam-admin/serviceaccounts |
 | `DE_ADMIN_TESTER` | `*****` | Service account base64 encoded string for API calls. Note: this user must have entitlements configured already | yes | https://console.cloud.google.com/iam-admin/serviceaccounts |
 | `DE_EDITOR_TESTER` | `*****` | Service account base64 encoded string for API calls. Note: this user must have entitlements configured already  | yes | https://console.cloud.google.com/iam-admin/serviceaccounts |
 | `DE_NO_ACCESS_TESTER` | `*****` | Service account base64 encoded string for API calls. Note: this user must have entitlements configured already | yes | https://console.cloud.google.com/iam-admin/serviceaccounts |
-| `ENVIRONMENT` | `dev` OR `local` | Local for running locally with services url's predefined as http://localhost , Dev is configurable environment | no | - |
-| `HMAC_SECRET` | `*****` | sensitive secret to run HMAC tests | yes | - |
+| `ENVIRONMENT` | `dev` OR `local` OR `dev_gke`| Local for running locally with services url's predefined as http://localhost , Dev & Dev_Gke is configurable environment | no | - |
+| `HMAC_SECRET` | ex`7a786376626e` | String in hex , must match pattern ^[a-zA-Z0-9]{8,30}+$ & be in register variable SUBSCRIBER_SECRET | yes | - |
 | `REGISTER_BASE_URL` | `http://localhost:8081/api/register/v1` | Register service url | no | - |
 | `NOTIFICATION_BASE_URL` | `http://localhost:8080/` | Notification service url  | no | - |
 | `INTEGRATION_TEST_AUDIENCE` | `********` | Client application ID | yes | https://console.cloud.google.com/apis/credentials |
 | `CLIENT_TENANT` | ex `opendes` | Client tenant | no | - |
 | `OSDU_TENANT` | ex `osdu` | Osdu tenant | no | - |
 | `TOPIC_ID` | ex `records-changed` | PubSub topic id | no | https://console.cloud.google.com/cloudpubsub/topic |
+| `REGISTER_CUSTOM_PUSH_URL_HMAC` | ex `http://localhost:8081/api/register/v1/test/challenge/hmac-integration-test` | Register testing push url | no | - |
 
  **Entitlements configuration for integration accounts**
  
  | DE_OPS_TESTER | DE_ADMIN_TESTER | DE_EDITOR_TESTER | DE_NO_ACCESS_TESTER | 
  | ---  | ---   | ---  | ---   |
- |notification.pubsub<br/>service.entitlements.user<br/>users<br/>| service.entitlements.user<br/>users<br/> | service.entitlements.user<br/>users<br/> | service.entitlements.user<br/>users<br/>|
+ |notification.pubsub<br/>service.entitlements.user<br/>users<br/>users.datalake.ops</br>| service.entitlements.user<br/>users<br/>users.datalake.admins</br> | service.entitlements.user<br/>users<br/>users.datalake.editors</br> | service.entitlements.user<br/>users<br/>|
 
 Above variables should be configured in the release pipeline to run integration tests. You should also replace them with proper values if you wish to run tests locally.
 
 ###Commands to run tests
 * Integration tests are refactored into two pieces: Core and Provider. Core contains business logic for tests and is a dependency for executing the tests from provider module. To build the core module, simply navigate to `notification-test-core` directory and run `mvn clean install`. This will build the core module
 * Next, to execute the integration tests, navigate to the provider module and execute `mvn test`
-
-
+```bash
+# (cd testing/notification-test-core/ && mvn clean install)
+# Note: this assumes that the environment variables for integration tests as outlined
+#       above are already exported in your environment.
+$ (cd testing/notification-test-gcp/ && mvn clean test)
+```
 ## License
 Copyright © Google LLC
 

testing/README.md

@@ -42,4 +42,4 @@ Notification integration tests are refactored so that the business logic for int
 * Next, to execute the integration tests, navigate to the provider module and execute `mvn test`
 
 ### GCP
-Instructions for running the GCP integration tests can be found [here](./notification-test-gcp/README.md).
+Instructions for running the GCP integration tests can be found [here](../provider/notification-gcp/README.md).
\ No newline at end of file

testing/notification-test-core/src/main/java/org/opengroup/osdu/notification/api/PubsubEndpointGSATests.java

@@ -74,7 +74,12 @@ public abstract class PubsubEndpointGSATests extends BaseTestTemplate {
         gsaSecret.setSecretType("GSA");
         ((GsaSecret) gsaSecret).setValue(gsaSecretValue);
         subscription.setSecret(gsaSecret);
-        subscription = this.subscriptionService.create(subscription);
+        try {
+            subscription = this.subscriptionService.create(subscription);
+        }catch (SubscriptionException e){
+            System.out.println("Subscription exception inner response : " + e.getHttpResponse());
+            throw e;
+        }
 
         Config.Instance().NotificationId = subscription.getNotificationId();
         this.subscriptionId = subscription.getId();

testing/notification-test-core/src/main/java/org/opengroup/osdu/notification/api/PubsubEndpointHMACTests.java

@@ -91,11 +91,15 @@ public abstract class PubsubEndpointHMACTests extends BaseTestTemplate {
         secret.setValue(Config.Instance().hmacSecretValue);
 
         subscription.setSecret(secret);
-        Subscription subscriptionCreated = subscriptionService.create(subscription);
-        notificationId = subscriptionCreated.getNotificationId();
-        subscriptionId = subscriptionCreated.getId();
-
-        Config.Instance().NotificationId = notificationId;
+        try {
+            Subscription subscriptionCreated = subscriptionService.create(subscription);
+            notificationId = subscriptionCreated.getNotificationId();
+            subscriptionId = subscriptionCreated.getId();
+            Config.Instance().NotificationId = notificationId;
+        }catch (SubscriptionException e){
+            System.out.println("Subscription exception inner response : " + e.getHttpResponse());
+            throw e;
+        }
     }
 
     @Override