Commit 7c4d941a authored by Rustam Lotsmanenko (EPAM)'s avatar Rustam Lotsmanenko (EPAM)
Browse files

GONRG-864

Changes:
Docs
parent 5bc8cac9
# Backup service
Backup service provide a set of APIs to backup Schedule, list registered Schedules, list available Backups, restore Asset state from Backup.
Backup service provide a set of APIs to schedule backups for different Assets (Datastore, Storage, ElasticSearch, etc depends on which currently implemented ),
list registered Schedules, list available Backups, restore Asset state from Backup.
## Core Interfaces
###AssetBackupManager
```
void exportBackup(BackupSchedule backupSchedule);
BackupStamp importBackup(String backupId);
void deleteBackups(List<BackupStamp> currentDate);
```
Provide backup managing access to current Asset, based on AssetType of schedule
###SchedulerService
```
void addTaskToScheduler(BackupSchedule backupSchedule);
boolean cancelScheduledTask(BackupSchedule backupSchedule);
void updateTask(BackupSchedule backupSchedule);
```
Holds ThreadPool with scheduled tasks , and provide control
###BackupService
```
BackupSchedule submitBackupSchedule(BackupSchedule backupSchedule);
BackupSchedule updateBackupSchedule(BackupSchedule backupSchedule);
BackupSchedule getBackupSchedule(String id);
BackupStamp submitBackupImportRequest(BackupImportRequest importRequest);
List<BackupSchedule> listSchedules();
EnumMap<Asset, List<BackupStamp>> listBackups();
```
Main service interface
## Data Repositories
###BackupStampRepository
```
EnumMap<Asset, List<BackupStamp>> findBackupStampsByTearDownBeforeCurrentDate(LocalDateTime toDate);
BackupStamp submitBackupStamp(BackupStamp backupStamp);
EnumMap<Asset, List<BackupStamp>> listAvailableBackupStamps();
void deleteBackupStamps(List<BackupStamp> currentDate);
BackupStamp findBackupByStampId(String backupStampId);
```
A backup stamp's repository ,
backup stamp is a model of created backup entity
which provide additional info about it : lifetime, location etc :
```
BackupStamp
String stampId;
String backupRepository;
Asset assetType;
LocalDateTime tearDownDate;
Map<String, String> assetContext;
```
###ScheduleRepository
```
List<BackupSchedule> listBackupSchedules();
BackupSchedule submitSchedule(BackupSchedule backupSchedule);
BackupSchedule updateSchedule(BackupSchedule backupSchedule);
BackupSchedule findByAssetContext(Map<String, String> assetContext);
BackupSchedule save(BackupSchedule backupSchedule);
BackupSchedule findById(String scheduleId);
```
## Diagram
![Diagram](./docs/schema.png)
## GCP Implementation
All documentation for the GCP implementation of Backup service lives [here](./provider/backup-gcp/README.md)
......
@startuml
BackupSchedule -> BackupService : submit schedule
BackupService -> ScheduleRepository : save schedule
BackupService -> SchedulerService : add scheduled task
SchedulerService -> ScheduledTask
ScheduledTask --> ScheduledTask
ScheduledTask -> AssetManager : export backup
AssetManager -> BackupStampsRepository : submit backup stamp
@enduml
## Backup Service
## Introduction <a name="Introduction"></a>
### Submit scheduled backup task
```
POST /backup/v1/submitScheduledTask
POST /backup/v1/schedule
```
```
curl --location --request POST 'localhost:8080/backup/v1/submitScheduledTask' \
--header 'Data-Partition-Id: opendes' \
--header 'Authorization: Bearer <JWT> \
--header 'Content-Type: application/json' \
curl --location --request POST 'localhost:8080/backup/v1/schedule'
--header 'Data-Partition-Id: opendes'
--header 'Authorization: Bearer <JWT>
--header 'Content-Type: application/json'
--data-raw '{
"assetType": "datastore",
"namespace": "opendes",
"kind": "LegalTag",
"backupPeriod": "1",
"active": "true"
"assetType": "DATSTORE",
"backupPeriod": "5",
"active": "true",
"lifetime": 2,
"assetContext": {
"namespace": "opendes",
"kind": "TestKind"
}
}'
```
### Get schedule by id
```
GET /backup/v1/schedule?id=
```
```
curl --location --request GET 'localhost:8080/backup/v1/schedule?id=8de2a9e34bdf489aba20206b8b28ab6e' \
--header 'Data-Partition-Id: opendes'
--header 'Authorization: Bearer <JWT>
```
### Update schedule by id
```
PUT /backup/v1/schedule
```
```
curl --location --request PUT 'localhost:8080/backup/v1/schedule'
--header 'Data-Partition-Id: opendes'
--header 'Content-Type: application/json'
--data-raw '{
"scheduleId":"2049486875f541ad8df6a0e6da777b7f",
"backupPeriod": "5",
"active": "true",
"lifetime": 2,
}'
```
### List backup schedules
```
GET /backup/v1/listSchedules
GET /backup/v1/list_schedules
```
```
curl --location --request GET 'localhost:8080/backup/v1/listSchedules' \
--header 'Data-Partition-Id: opendes' \
curl --location --request GET 'localhost:8080/backup/v1/list_schedules'
--header 'Data-Partition-Id: opendes'
--header 'Authorization: Bearer <JWT>
```
### List available backups
```
GET /backup/v1/listBackups
GET /backup/v1/list_backups
```
```
curl --location --request GET 'localhost:8080/backup/v1/listBackups' \
--header 'Data-Partition-Id: opendes' \
curl --location --request GET 'localhost:8080/backup/v1/list_backups'
--header 'Data-Partition-Id: opendes'
--header 'Authorization: Bearer <JWT>
```
### Import backup
```
POST /backup/v1/submitImport
POST /backup/v1/backup_import
```
```
curl --location --request POST 'localhost:8080/backup/v1/submitImport' \
--header 'Data-Partition-Id: opendes' \
--header 'Authorization: Bearer <JWT> \
--header 'Content-Type: application/json' \
curl --location --request POST 'localhost:8080/backup/v1/backup_import'
--header 'Data-Partition-Id: opendes'
--header 'Authorization: Bearer <JWT>
--header 'Content-Type: application/json'
--data-raw '{
"backupPath": "gs://<backup file path>"
}'
"backupStampId": "4730f8b5dd1145a3b09335c640d455c0",
"asset":"DATASTORE"
}''
```
......@@ -20,11 +20,20 @@ In order to run the service locally or remotely, you will need to have the follo
| name | value | description | sensitive? | source |
| --- | --- | --- | --- | --- |
| `SPRING_CLOUD_GCP_DATASTORE_NAMESPACE` | ex `opendes` | Datastore namespace to backup| no | https://console.cloud.google.com/datastore |
| `OSDU_ENTITLEMENTS_URL` | ex `https://os-entitlements-gcp-jvmvia5dea-uc.a.run.app/entitlements/v1` | Entitlements API endpoint | no | output of infrastructure deployment |
| `GCP_BACKUP_BUCKET` | ex `osdu-cicd-epam-backup-service` | Storage bucket for backups | no | https://console.cloud.google.com/storage |
| `GCP_PROJECT_ID` | ex `osdu-cicd-epam` | GCP project id | no | - |
| `GOOGLE_APPLICATION_CREDENTIALS` | ex `/path/to/directory/service-key.json` | Service account credentials, you only need this if running locally | yes | https://console.cloud.google.com/iam-admin/serviceaccounts |
Schedulers can be configured with following variables, if not defined default values will be used
| name | value | description | default |
| --- | --- | --- | --- |
| `OSDU_SCHEDULER_TEAR-DOWN-TIME-UNIT` | ex `seconds OR minutes OR hours etc` | Time unit for tear down scheduler | `hours` |
| `OSDU_SCHEDULER_TEAR-DOWN-PERIOD` | ex `24` | The period with which expired backups will be deleted | `24` |
| `OSDU_SCHEDULER_BACKUP-TIME-UNIT` | ex `seconds OR minutes OR hours etc` | Time unit for backup export scheduler (Period will be configured with saved schedule) | `hours` |
| `OSDU_SCHEDULER_INITIAL-DELAY` | ex `1` | Delay before start exporting new added assets, helpful for int tests, to not overwhelm backup storage | `1` |
### Run Locally
Check that maven is installed:
......@@ -95,6 +104,55 @@ After configuring your environment as specified above, you can follow these step
```bash
cd provider/backup-gcp && mvn spring-boot:run
```
## Testing
### Running E2E Tests
This section describes how to run cloud OSDU E2E tests (testing/backup-test-gcp).
You will need to have the following environment variables defined.
| name | value | description | sensitive? | source |
| --- | --- | --- | --- | --- |
| `BACKUP_SERVICE_HOST` | ex`http://localhost:8080/backup/v1` | Service endpoint | no | - |
| `DATA_PARTITION_ID` | `opendes` | OSDU tenant used for testing | no | - |
| `INTEGRATION_TESTER` | ex`/path/to/directory/service-key.json` | Service account .json for API calls. Note: this user must have entitlements configured already | yes | https://console.cloud.google.com/iam-admin/serviceaccounts |
| `NO_DATA_ACCESS_TESTER` | ex`/path/to/directory/service-key.json` | Service account .json without data access | yes | https://console.cloud.google.com/iam-admin/serviceaccounts |
| `GCP_DEPLOY_FILE` | ex`/path/to/directory/service-key.json` | Service account for test data tear down, must have cloud role configured | yes | https://console.cloud.google.com/iam-admin/serviceaccounts |
| `INTEGRATION_TEST_AUDIENCE` | `****` | client application ID | yes | https://console.cloud.google.com/apis/credentials |
| `DATASTORE_NAMESPACE` | ex `opendes` | OSDU tenant used for testing | no | - |
**Entitlements configuration for integration accounts**
| INTEGRATION_TESTER | NO_DATA_ACCESS_TESTER |
| --- | --- |
| users<br/>backup.service| users |
**Cloud roles configuration for integration accounts**
| GCP_DEPLOY_FILE|
| --- |
| Cloud Datastore Owner |
Execute following command to build code and run all the integration tests:
```bash
# Note: this assumes that the environment variables for integration tests as outlined
# above are already exported in your environment.
# build + install integration test core
$ (cd testing/backup-test-core/ && mvn clean install)
```
```bash
# build + run GCP integration tests.
$ (cd testing/backup-test-gcp/ && mvn clean test)
```
## Deployment
* To deploy into Cloud run, please, use this documentation:
https://cloud.google.com/run/docs/quickstarts/build-and-deploy
* To deploy into App Engine, please, use this documentation:
https://cloud.google.com/appengine/docs/flexible/java/quickstart
## License
......
server.servlet.contextPath=/backup/v1
server.port=8080
logging.level.org.springframework=INFO
logging.level.org.springframework.boot.autoconfigure=ERROR
log.prefix=backup
spring.cloud.gcp.datastore.namespace=opendes
gcp.backup-bucket=osdu-cicd-epam-backup-service
gcp.project-id=osdu-cicd-epam
#Core
osdu.entitlements.url=https://os-entitlements-gcp-jvmvia5dea-uc.a.run.app/entitlements/v1
osdu.entitlements.app-key=test
osdu.scheduler.tear-down-time-unit=hours
osdu.scheduler.tear-down-period=24
osdu.scheduler.backup-time-unit=minutes
osdu.scheduler.backup-time-unit=hours
osdu.scheduler.initial-delay=1
#GCP
gcp.backup-bucket=osdu-cicd-epam-backup-service
gcp.project-id=osdu-cicd-epam
LOG_PREFIX=${log.prefix}
......@@ -2,12 +2,12 @@ package org.opengroup.osdu.backup;
public class ConfigGCP {
private static final String TARGET_AUDIENCE = "";
private static final String INTEGRATION_TEST_AUDIENCE = "";
private static final String GCP_DEPLOY_FILE = "";
private static final String DATASTORE_NAMESPACE = "";
public static String getTargetAudience() {
return getEnvironmentVariableOrDefaultValue("TARGET_AUDIENCE", TARGET_AUDIENCE);
return getEnvironmentVariableOrDefaultValue("INTEGRATION_TEST_AUDIENCE", INTEGRATION_TEST_AUDIENCE);
}
public static String getDatastoreCredentialsCredentials() {
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment