Skip to content
GitLab
Commit c8452591 authored by Subham Agarwal's avatar Subham Agarwal Committed by ethiraj krishnamanaidu
Browse files

updated code, test folder segratation, readme updated, code coverage increased

parent e742263e
Hide whitespace changes
Inline Side-by-side
README.md
View file @ c8452591
......@@ -2,11 +2,31 @@
The Schema Service is a Maven multi-module project with each cloud implemention placed in its submodule. To build or run Schema Service locally, follow the below steps :
1. Clone the os-schema repository from git . Below is the URL :
### 1. GCP deployment
[https://dev.azure.com/slb-swt/data-management/_git/os-schema/](https://dev.azure.com/slb-swt/data-management/_git/os-schema/)
#### Prerequisite (Infra and access required)
2. Navigate to the root of the schema project, os-schema. For building the project using command line, run below command :
Schema service as per design uses two module from GCP. GCS or Google cloud storage to store actual schemas and Google cloud datastore to store schema metadata. It follows the multi tenancy
concept of DE, which means service is deployed in one GCP project and data is stored in client specific project. And permission to speccfic tenant project is decided based on data-partition-id user passes
as part of request header. So, to make it work from local we must have following setup done as prerequisite,
1. GCP project setup is done and local gcloud sdk configured by activating the account/user and pointing to correct GCP project. You can follow the steps from [here](https://cloud.google.com/deployment-manager/docs/step-by-step-guide/installation-and-setup)
2. Bucket with name <project-id>-schema (e.g opendes-schema) is created in tenant GCS and tenant datafier service account has read/write access to that bucket. Steps to create bucket and grant access can be followed from [here](https://cloud.google.com/storage/docs/creating-buckets)
3. Tenant datafier service account has read/write access to Google cloud datastore in tenant project. You can follow access control on datastore from [here](https://cloud.google.com/datastore/docs/access/iam). Permission required is ```roles/datastore.user```
4. Service-account/user activated as part of step 1 has service token creator role on datafier service-account of the data partition used. Details on service account creator role can be accessed from [here](https://cloud.google.com/iam/docs/service-accounts#the_service_account_token_creator_role)
5. TenantInfo table should be present in service GCP datastore under namespace ```datascosystem``` and kind ```tenantInfo``` and has entry corresponding to data-partition-id passed.
6. User/service-account that will be used to run the service has access to ```service.schema-service.editors``` group in the specified data-partition.
### Local deployment Steps
Once the above Prerequisite are done, we can follow the below steps to run the service locally,
1. Navigate to the root of the schema project, os-schema. For building the project using command line, run below command :
```bash
mvn clean install
```
......@@ -14,16 +34,18 @@ The Schema Service is a Maven multi-module project with each cloud implemention
```bash
mvn --projects schema-core,provider/schema-gcp clean install
```
3. Run schema service in command line. We need to select which cloud vendor specific schema-service we want to run. For example, if we want to run schema-service for GCP, run the below command :
2. Run schema service in command line. We need to select which cloud vendor specific schema-service we want to run. For example, if we want to run schema-service for GCP, run the below command :
```bash
# Running GCP :
java -jar -Dspring.profiles.active=local provider\schema-gcp\target\os-schema-gcp-0.0.1-spring-boot.jar
4. The port and path for the service endpoint can be configured in ```application.properties``` in the provider folder as following. If not specified, then the web container (ex. Tomcat) default is used:
java -jar provider\schema-gcp\target\os-schema-gcp-0.0.1-spring-boot.jar
3. The port and path for the service endpoint can be configured in ```application.properties``` in the provider folder as following. If not specified, then the web container (ex. Tomcat) default is used:
```bash
server.servlet.contextPath=/api/schema-service/v1/
server.port=8080
```
You can access the service APIs by following the service contract in [schema.yaml](https://dev.azure.com/slb-des-ext-collaboration/open-data-ecosystem/_git/os-schema?path=%2Fdocs%2Fapi%2Fschema.yaml)
## Running Automated Integration Test
DevSanity tests are located in a schema-core project in testing directory under the project root directory.
......@@ -35,8 +57,14 @@ They can then be run/debugged directly in your IDE of choice using the GUI or vi
Below command has to be run post building complete project.
cd schema-core
mvn verify -P IntegrationTest
cd testing/schema-test-gcp
mvn verify
## Deploy Shared Schemas
Schema service as part of deployment deploys pre-defined OSDU schemas so end users can get community accepted schemas to refer. Such schemas are present in [folder](https://dev.azure.com/slb-des-ext-collaboration/open-data-ecosystem/_git/os-schema?path=%2Fdeployments%2Fshared-schemas%2Fosdu) and script to deploy the schema are present [here](https://dev.azure.com/slb-des-ext-collaboration/open-data-ecosystem/_git/os-schema?path=%2Fdeployments%2Fscripts).
Details to deploy shared schemas can be found under [README.md](deployments/shared-schemas/README.md)
......
azure-pipelines.yml
View file @ c8452591
......@@ -64,21 +64,20 @@ stages:
docker-compose build $(dockerImageName)
docker tag gcr.io/opendes/$(dockerImageName) gcr.io/opendes/$(dockerImageName):$(tag)
docker push gcr.io/opendes/$(dockerImageName):$(tag)
docker push gcr.io/opendes/$(dockerImageName)
echo 'Push done.'
docker push gcr.io/opendes/$(dockerImageName) echo 'Push done.'
kubectl --kubeconfig $(kubeconfig.secureFilePath) rollout restart deployment/$(deploymentName)
popd
sleep 10
OUTPUT="Alive"
OUTPUT="200 OK"
ENDPOINT=$(SCHEMA_DEV_URL)/health
echo $ENDPOINT
while [ -z "$STATUS" ]; do
STATUS=`curl -v --silent --http1.0 "$ENDPOINT" 2>&1 | grep "$OUTPUT"`
echo $STATUS
if [ -z "$STATUS" ]; then
echo "Endpoint is not up yet."
sleep 10
......@@ -92,9 +91,9 @@ stages:
- task: Maven@3
displayName: 'Running IntegrationTest'
inputs:
mavenPomFile: '$(osProjectName)-core/pom.xml'
mavenPomFile: 'testing/schema-test-gcp/pom.xml'
goals: 'verify'
options: '-P IntegrationTest --settings maven/settings.xml -DVSTS_FEED_TOKEN=$(VSTS_FEED_TOKEN)'
options: '--settings maven/settings.xml -DVSTS_FEED_TOKEN=$(VSTS_FEED_TOKEN)'
publishJUnitResults: false
javaHomeOption: 'JDKVersion'
mavenVersionOption: 'Default'
......@@ -104,7 +103,7 @@ stages:
env:
INTEGRATION_TEST_AUDIENCE: $(INTEGRATION_TEST_AUDIENCE)
INTEGRATION_TESTER : $(INTEGRATION_TESTER)
- task: UsePythonVersion@0
inputs:
versionSpec: '3.x'
......@@ -129,7 +128,7 @@ stages:
export DATA_PARTITION=$(DATA_PARTITION)
python deployments/scripts/DeploySharedSchemas.py -u $(SCHEMA_DEV_URL)/schema
- task: PublishBuildArtifacts@1
displayName: 'Publish Artifact: drop'
inputs:
......@@ -137,7 +136,7 @@ stages:
ArtifactName: 'drop'
publishLocation: 'Container'
condition: succeededOrFailed()
- stage: DeployToQA
condition: and(succeeded(), eq(variables['Build.Reason'], 'Manual'))
variables:
......
docs/api/schema.yaml
View file @ c8452591
......@@ -3,7 +3,7 @@ info:
description: These API takes care of schema management in the Data Ecosystem and offers an
implementation of our schema standard.
version: 1.0.0
title: DE Schema API
title: Schema APIs
tags:
- name: Schema
description: Core Schema Service related methods
......@@ -222,7 +222,7 @@ paths:
schema:
type: integer
minimum: 0
maximum: 50
maximum: 100
example: 10
- in: query
name: offset
......@@ -230,7 +230,7 @@ paths:
schema:
type: integer
minimum: 0
example: 1
example: 0
responses:
'200':
description: Successful response
......
provider/schema-gcp/pom.xml
View file @ c8452591
......@@ -78,7 +78,7 @@
<artifactId>cucumber-guice</artifactId>
<version>5.4.0</version>
<scope>test</scope>
</dependency>
</dependency>
<dependency>
<groupId>com.google.inject</groupId>
......@@ -89,26 +89,9 @@
</dependencies>
<repositories>
<repository>
<id>${gitlab-server}</id>
<url>https://community.opengroup.org/api/v4/groups/17/-/packages/maven</url>
</repository>
</repositories>
<distributionManagement>
<repository>
<id>${gitlab-server}</id>
<url>https://community.opengroup.org/api/v4/projects/26/packages/maven</url>
</repository>
<snapshotRepository>
<id>${gitlab-server}</id>
<url>https://community.opengroup.org/api/v4/projects/26/packages/maven</url>
</snapshotRepository>
</distributionManagement>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
......@@ -124,30 +107,28 @@
</mainClass>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-war-plugin</artifactId>
<configuration>
<failOnMissingWebXml>false</failOnMissingWebXml>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M3</version>
<configuration>
<skipTests>${skipUnitTests}</skipTests>
<forkCount>3</forkCount>
<reuseForks>true</reuseForks>
<argLine>-Xmx1024m -XX:MaxPermSize=256m</argLine>
</configuration>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<id>${gitlab-server}</id>
<url>https://community.opengroup.org/api/v4/groups/17/-/packages/maven</url>
</repository>
</repositories>
<distributionManagement>
<repository>
<id>${gitlab-server}</id>
<url>https://community.opengroup.org/api/v4/projects/26/packages/maven</url>
</repository>
<snapshotRepository>
<id>${gitlab-server}</id>
<url>https://community.opengroup.org/api/v4/projects/26/packages/maven</url>
</snapshotRepository>
</distributionManagement>
</project>
\ No newline at end of file
provider/schema-gcp/src/main/java/org/opengroup/osdu/schema/configuration/DatastoreFactoryBean.java 0 → 100644
View file @ c8452591
package org.opengroup.osdu.schema.configuration;
import org.opengroup.osdu.core.gcp.multitenancy.DatastoreFactory;
import org.opengroup.osdu.core.gcp.multitenancy.TenantFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.config.AbstractFactoryBean;
import org.springframework.stereotype.Component;
@Component
public class DatastoreFactoryBean extends AbstractFactoryBean<DatastoreFactory> {
@Autowired
TenantFactory tenantFactory;
@Override
protected DatastoreFactory createInstance() throws Exception {
return new DatastoreFactory(tenantFactory);
}
@Override
public Class<?> getObjectType() {
return DatastoreFactory.class;
}
}
provider/schema-gcp/src/main/java/org/opengroup/osdu/schema/configuration/StorageFactoryBean.java 0 → 100644
View file @ c8452591
package org.opengroup.osdu.schema.configuration;
import org.opengroup.osdu.core.gcp.multitenancy.GcsMultiTenantAccess;
import org.opengroup.osdu.core.gcp.multitenancy.TenantFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.config.AbstractFactoryBean;
import org.springframework.stereotype.Component;
@Component
public class StorageFactoryBean extends AbstractFactoryBean<GcsMultiTenantAccess> {
@Autowired
TenantFactory tenantFactory;
@Override
protected GcsMultiTenantAccess createInstance() throws Exception {
return new GcsMultiTenantAccess();
}
@Override
public Class<?> getObjectType() {
return GcsMultiTenantAccess.class;
}
}
\ No newline at end of file
provider/schema-gcp/src/main/java/org/opengroup/osdu/schema/credentials/CloudCredentials.java deleted 100644 → 0
View file @ e742263e
package org.opengroup.osdu.schema.credentials;
import java.io.IOException;
import java.security.GeneralSecurityException;
import java.util.Arrays;
import java.util.Collections;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;
import java.util.logging.Level;
import org.apache.commons.lang3.StringUtils;
import org.apache.commons.lang3.time.DateUtils;
import org.apache.http.HttpHeaders;
import org.opengroup.osdu.core.common.http.HttpClient;
import org.opengroup.osdu.core.common.http.HttpRequest;
import org.opengroup.osdu.core.common.http.HttpResponse;
import org.opengroup.osdu.core.common.model.tenant.TenantInfo;
import org.opengroup.osdu.schema.constants.SchemaConstants;
import org.opengroup.osdu.schema.exceptions.ApplicationException;
import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.client.json.jackson2.JacksonFactory;
import com.google.api.services.iam.v1.Iam;
import com.google.api.services.iam.v1.Iam.Projects;
import com.google.api.services.iam.v1.Iam.Projects.ServiceAccounts;
import com.google.api.services.iam.v1.Iam.Projects.ServiceAccounts.SignJwt;
import com.google.api.services.iam.v1.IamScopes;
import com.google.api.services.iam.v1.model.SignJwtRequest;
import com.google.api.services.iam.v1.model.SignJwtResponse;
import com.google.auth.oauth2.AccessToken;
import com.google.auth.oauth2.GoogleCredentials;
import com.google.gson.JsonObject;
import lombok.extern.java.Log;
/**
* This class is used to implement the domain wide delegation using the service
* account token creator role. This class extends GoogleCredentials, and creates
* access token by creating the JWT token without private keys, and signing it
* with Google OAuth2 service.
*
*/
@Log
public class CloudCredentials extends GoogleCredentials {
private static final long serialVersionUID = -8461791038757192780L;
private static final String JWT_AUDIENCE = "https://www.googleapis.com/oauth2/v4/token";
private static final String SERVICE_ACCOUNT_NAME_FORMAT = "projects/%s/serviceAccounts/%s";
private static final String ACCESS_TOKEN = "access_token";
private static final String EXPIRES_IN = "expires_in";
private static final String SCOPE = "https://www.googleapis.com/auth/devstorage.full_control https://www.googleapis.com/auth/datastore";
private static final JsonFactory JSON_FACTORY = new JacksonFactory();
private final TenantInfo tenant;
private Iam iam;
public CloudCredentials(TenantInfo tenant) throws ApplicationException {
if ((tenant == null) || StringUtils.isBlank(tenant.getName()) || StringUtils.isBlank(tenant.getProjectId())
|| StringUtils.isBlank(tenant.getServiceAccount())) {
throw new ApplicationException("Tenant name, project id or service account is not set.");
}
this.tenant = tenant;
}
/**
* Overrides the GoogleCredentials refreshAccessToken. Returns JWT token for the
* service account of the tenant.
*
* @throws IOException
*/
@Override
public AccessToken refreshAccessToken() throws IOException {
try {
Map<String, Object> signJwtPayload = this.getPayload();
String signedJwt = getSignedJwt(signJwtPayload);
return getAccessToken(signedJwt);
} catch (ApplicationException e) {
log.log(Level.SEVERE, "Error : An unexpected error occurred while geting refresh access token : ", e);
throw new IOException("An error occurred when accessing third-party APIs");
}
}
/**
* Creates the payload for JWT signing
*
* @return payload as map
*/
private Map<String, Object> getPayload() {
Map<String, Object> payload = new HashMap<>();
payload.put("scope", SCOPE);
// This will get current time in seconds and add 1 hours (i.e 3600 seconds to
// it)
payload.put("exp", System.currentTimeMillis() / 1000 + 3600);
payload.put("iat", System.currentTimeMillis() / 1000);
payload.put("iss", this.tenant.getServiceAccount());
payload.put("aud", JWT_AUDIENCE);
return payload;
}
/**
* Signs the JWT token of the tenant service account, without private keys.
*
* @param signJwtPayload
* @return the signed JWT
* @throws ApplicationException
*/
private String getSignedJwt(Map<String, Object> signJwtPayload) throws ApplicationException {
try {
SignJwtRequest signJwtRequest = new SignJwtRequest();
signJwtRequest.setPayload(JSON_FACTORY.toString(signJwtPayload));
String serviceAccountName = String.format(SERVICE_ACCOUNT_NAME_FORMAT, this.tenant.getProjectId(),
this.tenant.getServiceAccount());
Iam iamInstance = this.getIam();
Projects projects = iamInstance.projects();
ServiceAccounts serviceAccounts = projects.serviceAccounts();
SignJwt signJwt = serviceAccounts.signJwt(serviceAccountName, signJwtRequest);
SignJwtResponse signJwtResponse = signJwt.execute();
return signJwtResponse.getSignedJwt();
} catch (IOException | GeneralSecurityException e) {
log.log(Level.SEVERE, "Error occoured: ", e);
throw new ApplicationException("An error occurred when accessing third-party APIs");
}
}
/**
* Creates the access token from the signed JWT for the tenant service account
* by signing it with Google OAuth2 service.
*
* @param signedJwt the signed JWT
* @return Access token that is used by Google services such as GCS, PubSub,
* Datastore at the time of the access
* @throws GeneralSecurityException
* @throws ApplicationException
*/
private AccessToken getAccessToken(String signedJwt) throws ApplicationException {
HttpRequest request = HttpRequest.post().url(JWT_AUDIENCE)
.headers(Collections.singletonMap(HttpHeaders.CONTENT_TYPE, "application/x-www-form-urlencoded"))
.body(String.format("%s=%s&%s=%s", "grant_type", "urn:ietf:params:oauth:grant-type:jwt-bearer",
"assertion", signedJwt))
.build();
HttpResponse response = new HttpClient().send(request);
JsonObject jsonResult = response.getAsJsonObject();
if (!jsonResult.has(ACCESS_TOKEN)) {
throw new ApplicationException("An error occurred when accessing third-party APIs");
}
return new AccessToken(jsonResult.get(ACCESS_TOKEN).getAsString(),
DateUtils.addSeconds(new Date(), jsonResult.get(EXPIRES_IN).getAsInt()));
}
/**
* Gets the Iam object of the services project, which is further used for
* signing the JWT
*
* @return the Iam object of the services project
* @throws IOException
* @throws GeneralSecurityException
*/
private Iam getIam() throws GeneralSecurityException, IOException {
if (this.iam == null) {
Iam.Builder builder = new Iam.Builder(GoogleNetHttpTransport.newTrustedTransport(), JSON_FACTORY,
GoogleCredential.getApplicationDefault().createScoped(Arrays.asList(IamScopes.CLOUD_PLATFORM)))
.setApplicationName(SchemaConstants.APPLICATION_NAME);
this.iam = builder.build();
}
return this.iam;
}
}
\ No newline at end of file
provider/schema-gcp/src/main/java/org/opengroup/osdu/schema/credentials/DatastoreFactory.java deleted 100644 → 0
View file @ e742263e
package org.opengroup.osdu.schema.credentials;
import java.util.concurrent.TimeUnit;
import org.opengroup.osdu.core.common.model.tenant.TenantInfo;
import org.opengroup.osdu.core.common.util.Crc32c;
import org.opengroup.osdu.schema.exceptions.ApplicationException;
import org.springframework.stereotype.Component;
import org.threeten.bp.Duration;
import com.google.api.gax.retrying.RetrySettings;
import com.google.cloud.datastore.Datastore;
import com.google.cloud.datastore.DatastoreOptions;
import com.google.common.cache.Cache;
import com.google.common.cache.CacheBuilder;
@Component
public class DatastoreFactory {
private Cache<String, CloudCredentials> cache = CacheBuilder.newBuilder()
.expireAfterAccess(60 * 60, TimeUnit.SECONDS).maximumSize(100).build();
// The numbers used for settings are selected on random basis, We can update
// them with experience/issue faced
private static final RetrySettings RETRY_SETTINGS = RetrySettings.newBuilder().setMaxAttempts(6)
.setInitialRetryDelay(Duration.ofSeconds(1)).setMaxRetryDelay(Duration.ofSeconds(10))
.setRetryDelayMultiplier(2.0).setTotalTimeout(Duration.ofSeconds(50))
.setInitialRpcTimeout(Duration.ofSeconds(50)).setRpcTimeoutMultiplier(1.1)
.setMaxRpcTimeout(Duration.ofSeconds(50)).build();
public Datastore getDatastore(TenantInfo tenantInfo) throws ApplicationException {
String cacheKey = this.getCredentialsCacheKey(tenantInfo.getName());
CloudCredentials credential = this.cache.getIfPresent(cacheKey);
if (credential == null) {
credential = new CloudCredentials(tenantInfo);
this.cache.put(cacheKey, credential);
}
return DatastoreOptions.newBuilder().setRetrySettings(RETRY_SETTINGS).setCredentials(credential)
.setProjectId(tenantInfo.getProjectId()).build().getService();
}
private String getCredentialsCacheKey(String tenantName) {
return Crc32c.hashToBase64EncodedString(String.format("CloudCredential:%s", tenantName));
}
}
\ No newline at end of file
provider/schema-gcp/src/main/java/org/opengroup/osdu/schema/credentials/StorageFactory.java deleted 100644 → 0
View file @ e742263e
package org.opengroup.osdu.schema.credentials;
import java.util.concurrent.TimeUnit;
import org.opengroup.osdu.core.common.model.tenant.TenantInfo;
import org.opengroup.osdu.schema.exceptions.ApplicationException;
import org.springframework.stereotype.Component;
import org.threeten.bp.Duration;
import com.google.api.gax.retrying.RetrySettings;
import com.google.cloud.TransportOptions;
import com.google.cloud.http.HttpTransportOptions;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;
import com.google.common.cache.Cache;
import com.google.common.cache.CacheBuilder;
@Component
public class StorageFactory {
// The numbers used for settings are selected on random basis, We can update
// them with experience/issue faced
private static final RetrySettings RETRY_SETTINGS = RetrySettings.newBuilder().setMaxAttempts(6)
.setInitialRetryDelay(Duration.ofSeconds(1)).setMaxRetryDelay(Duration.ofSeconds(10))
.setRetryDelayMultiplier(2.0).setTotalTimeout(Duration.ofSeconds(50))
.setInitialRpcTimeout(Duration.ofSeconds(50)).setRpcTimeoutMultiplier(1.1)
.setMaxRpcTimeout(Duration.ofSeconds(50)).build();
// The numbers used for settings are selected on random basis, We can update
// them with experience/issue faced
private static final TransportOptions TRANSPORT_OPTIONS = HttpTransportOptions.newBuilder()
.setReadTimeout(40 * 1000).setConnectTimeout(10 * 1000).build();
private Cache<String, CloudCredentials> cache = CacheBuilder.newBuilder()
.expireAfterAccess(60 * 60, TimeUnit.SECONDS).maximumSize(100).build();
public Storage getStorage(TenantInfo tenantInfo) throws ApplicationException {
CloudCredentials credential = this.cache.getIfPresent(tenantInfo.getName());