Commit 17a85d25 authored by Rustam Lotsmanenko (EPAM)'s avatar Rustam Lotsmanenko (EPAM) Committed by Riabokon Stanislav(EPAM)[GCP]
Browse files

Clean up script for deployment framework(GONRG-3087)

parent 33af6e0b
import json
import os
from google.api_core.retry import Retry
from google.cloud import datastore
from Utility import Utility, RunEnv
schema_namespace = os.environ.get("SCHEMA_NAMESPACE")
schema_kind = os.environ.get("SCHEMA_KIND")
default_namespace = "dataecosystem"
default_kind = "schema"
def cleanup_datastore():
if schema_namespace is None:
datastore_client = datastore.Client(namespace=default_namespace)
print(
"SCHEMA_NAMESPACE is empty, using default namespace: " + default_namespace)
else:
datastore_client = datastore.Client(namespace=schema_namespace)
print("SCHEMA_NAMESPACE not empty, using var value: " + schema_namespace)
if schema_kind is None:
print("SCHEMA_KIND is empty, using default kind: " + default_kind)
kind_to_use = default_kind
else:
print("SCHEMA_KIND not empty, using var value: " + schema_kind)
kind_to_use = schema_kind
deployments = Utility.path_to_deployments()
bootstrap_options = json.loads(RunEnv.BOOTSTRAP_OPTIONS)
for option in bootstrap_options:
schema_path = option['folder']
load_sequence = option['load-sequence']
path = os.path.join(deployments, RunEnv.SCHEMAS_FOLDER, schema_path,
load_sequence)
print("Schemas sequence location: " + path)
sequence = Utility.load_json(path)
for item in sequence:
complete_key = datastore_client.key(kind_to_use, item['kind'].replace(
'{{schema-authority}}', RunEnv.DATA_PARTITION))
print("Key to delete: " + complete_key.__str__())
range = Retry(initial=5, maximum=20, multiplier=2, deadline=120)
datastore_client.delete(key=complete_key, retry=range)
if __name__ == '__main__':
cleanup_datastore()
google-cloud-datastore
google.api_core
grpcio
\ No newline at end of file
......@@ -141,3 +141,19 @@ All 120 schemas registered or updated.
```
In case of errors, the list of failed creations/updates are summarized at the end.
### Environment clean up (GCP)
Schema bootstrapping used during new platform configuration, creates schema records in Datastore, that cannot be cleaned up during deinstallation.
If platform deployment must be re-installed, the cleanup script must be runned.
Scripts for cleanup schemas can be found in [DatastoreCleanUp.py](../scripts/DatastoreCleanUp.py)
```bash
pip install -r gcp-deployment-requirements.txt
```
You will need to have the following environment variables defined to run scripts.
| name | value | description | sensitive? | source |
| --- | --- | --- | --- | --- |
| `DATA_PARTITION` | ex `osdu`| Data partition id| no | - |
| `SCHEMA_NAMESPACE` | ex `dataecosystem`| If not specified default `dataecosystem` will be used | no | - |
| `SCHEMA_KIND` | ex `schema`| If not specified default `schema` will be used | no | - |
| `GOOGLE_APPLICATION_CREDENTIALS` | ex`usr/key.json` | Google Service account credentials with delete access to Datastore | yes | - |
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment