diff --git a/.gitignore b/.gitignore
index e7712af8d5150cd6dae63334bae1a39a7281823f..8f7025a30515c00824016b14b17a4d935cacc3fb 100644
--- a/.gitignore
+++ b/.gitignore
@@ -11,6 +11,7 @@ __pycache__/
# Compiled output
dist/
+build/
/tmp
/out-tsc
/bazel-out
diff --git a/frontend/admincli/Makefile b/frontend/admincli/Makefile
index 783759423f3ce4803759ac63dbb8aea62926c6a4..309bc9fc9859e6830bcd77f3b6cd15f45fce54bc 100644
--- a/frontend/admincli/Makefile
+++ b/frontend/admincli/Makefile
@@ -36,9 +36,14 @@ scan:
docker scan $(IMAGE_NAME):$(TAG)
# requires PYTHONPATH to be set to admincli root directory
+tests: test
+
test:
python3.9 -m pytest -v
+testother:
+ python3.9 -m pytest -v -k "not policy"
+
demo: END_RECORDING := "Thanks for watching"
demo:
- rm docs/admincli.cast
diff --git a/frontend/admincli/README.md b/frontend/admincli/README.md
index a3329ad751152ef1101cc63b86f974dc1529ea2c..a2e1a455ab32e56fd08366bb65ce4482fb408f7d 100644
--- a/frontend/admincli/README.md
+++ b/frontend/admincli/README.md
@@ -1,55 +1,76 @@
# Policy Service - Admin CLI
-The Policy Service Admin CLI is an easy to use full featured CLI. This AdminCLI v0.0.1 is new in OSDU Milestone 14, but we believe it's already far beyond POC and ready for production use. As always, please report any issues.
+The Admin CLI is an easy to use full featured CLI.
+This AdminCLI was new in OSDU Milestone 14. It was ready for production use then.
+Now with M15 it's even better. As always, please report any issues.
##### For help:
-`pol.py --help`
-`pol.py ls --help`
+* General help: `pol.py --help`
+* Individual command help is also available, for example: `pol.py ls --help`
-##### The main commands are:
+##### The main policy commands are:
* `add` for adding or updating policies. This is particular useful for automation and loading policies into OSDU partitions,
* `eval` for evaluating policies,
* `ls` for listing and retrieving policies and
* `rm` for deleting policies
-
-##### Additional utils/commands include:
-* `groups` - for showing groups related to your auth context,
-* `health` - retrieves health status of policy service,
-* `info` - retrieves info from policy service,
-* `legal-tags` - get legal tags from legal tag service,
-* `translate` - for helping testing translate which is used by search service via os-core-common
+* `health` - retrieves health status of policy service
+
+##### Additional Policy Developer Utils
+* `compile` Use OPA's Compile API to partially evaluate a query.
+* `config` Diagnostic config on Policy Service.
+* `diff` Compare two policies, show the delta in a context diff format.
+* `opa-add` Add or update a policy directly in OPA ✨ for LOCAL testing/development
+* `opa-rm` delete a policy directly from OPA. 🔥 for LOCAL testing/development
+* `translate` For helping testing translate which is used by search service via os-core-common
+
+##### Additional Utils/commands include:
+* `groups` - Showing groups related to your auth context,
+* `info` - Retrieves info from services,
+* `legal-tags` - Get legal tags from legal tag service,
+* `search` - Search Utility
+* `storage` - Storage and Dataset record retrieval utility
You will need to set the following environmental variables or provide details on command line:
-* `POLICY_URL` or `--url` or `--host`
* `TOKEN` or `--token`
* `DATA_PARTITION` or `--data-partition-id`
-* `ENTITLEMENTS_URL` or `--ent-url` (only used for groups command)
-* `LEGAL_URL` or `--legal-url` (only used for legal-tags command)
+* `BASE_URL` or `--base-url` or `--url` or `--host`
See [setenv.sh](setenv.sh) for an example of these.
-
Please note: *command-line options will override environment variables.*
Command line completion is available:
* `--install-completion`
* `--show-completition`
+##### Development / Testing Notes:
+Individual services can be optionally redirected for development, testing or custom environments:
+* `POLICY_URL` or `--policy-url`
+* `ENTITLEMENTS_URL` or `--ent-url`
+* `STORAGE_URL` or `--storage-url`
+* `SEARCH_URL` or `--search-url`
+* `LEGAL_URL` or `--legal-url`
+* `DATASET_URL` or `--dataset-url`
+These are not required. See [setenv.sh](setenv.sh) for an example of these.
+Please note: *command-line options will override environment variables.*
+
##### Built-in Template Engine
Add, eval and translate support a templating engine that makes it super easy to automate. Hopefully you find this helpful.
-When using --template option the strings in our file input will be replaced
+When using --template option the strings in our file input will be replaced, for example:
* `${data_partition}` will be replaced by the data partition id you are using
* `${name}` will be replaced by the policy id you selected in the command
+See the individual command's help for template support details.
In future releases a compiled version of the CLI may be made available.
-##### Output options for `ls` command:
-The `ls` command support multiple output options `pol.py ls --output <>`
+##### Output options for commands:
+Some commands support multiple output options for example `pol.py ls --output=<>` or `pol.py search --output=<>
* fancy (default with colors and formatting)
* simple
+* excel (supported only on search currently)
* tree (tree output of policies)
-If the policy admincli detects there isn't a tty simple will automatically be selected to make automation easier.
+If the policy admincli detects there isn't a tty simple should automatically be selected to make automation easier.
-In addition to get raw json return from policy service use the `--raw` option.
+In addition many commands support feature to get raw json return from OSDU service using the `--raw` option.
##### Searching:
Search uses standard Regular Expressions (shortened as regex or regexp), sometimes referred to as rational expressions - a sequence of characters that specifies a search pattern in text. This is extremely powerful search facility.
diff --git a/frontend/admincli/pol.py b/frontend/admincli/pol.py
index 323bae1650a5b453801420505e08fcd2f6a81ab8..5322710269b9d84f6fe75cc28b7e3ca7a4d9761d 100755
--- a/frontend/admincli/pol.py
+++ b/frontend/admincli/pol.py
@@ -17,12 +17,14 @@ import os
import sys
import re
import json
+import getpass
from string import Template
import difflib
from rego import ast, walk
import opa
+import search_cli
-__version__ = '0.0.3'
+__version__ = '0.0.4'
__app_name__ = 'OSDU Policy Service AdminCLI'
cli = typer.Typer(rich_markup_mode="rich", help=__app_name__)
@@ -34,6 +36,7 @@ class OutputType(str, Enum):
simple = "simple"
fancy = "fancy"
tree = "tree"
+ excel = "excel"
def _version_callback(value: bool) -> None:
if value:
@@ -46,10 +49,12 @@ def headers(ctx: typer.Context):
"""
headers={'Authorization': 'Bearer ' + ctx.obj.token,
'data-partition-id': ctx.obj.data_partition,
- 'x-user-id': ctx.obj.x_user_id,
'accept': 'application/json',
- 'content-type': 'application/json'
+ 'x-user-id': ctx.obj.x_user_id,
+ #'content-type': 'application/json'
}
+ if ctx.obj.debug:
+ console.print(headers)
return headers
def request_groups(ctx: typer.Context):
@@ -57,16 +62,16 @@ def request_groups(ctx: typer.Context):
return list of groups from entitlements service
"""
try:
- response = requests.get(ctx.obj.entitlements_url, headers = headers(ctx))
+ response = requests.get(ctx.obj.entitlements_url + "/groups", headers = headers(ctx))
except requests.exceptions.RequestException as e:
raise SystemExit(e)
if not response.ok:
if "message" in response.text:
message = json.loads(response.text)["message"]
- error_console.print(message)
+ error_console.print(f"Error: {message}")
else:
- error_console.print(f"An error occurred when talking to {ctx.obj.entitlements_url}")
+ error_console.print(f"Error: An error occurred when talking to {ctx.obj.entitlements_url}")
return ""
jdata = response.json()
if "groups" in jdata:
@@ -81,23 +86,23 @@ def get_policies(ctx: typer.Context):
timeout=10,
headers=headers(ctx))
except requests.exceptions.HTTPError:
- error_console.print(f"endpoint {ctx.obj.url}: HTTPError")
+ error_console.print(f"Error: endpoint {ctx.obj.url}: HTTPError")
return
except requests.exceptions.ConnectionError:
- error_console.print(f"endpoint {ctx.obj.url}: ConnectionError")
+ error_console.print(f"Error: endpoint {ctx.obj.url}: ConnectionError")
return
except requests.exceptions.Timeout:
- error_console.print(f"endpoint {ctx.obj.url}: Timeout")
+ error_console.print(f"Error: endpoint {ctx.obj.url}: Timeout")
return
except requests.exceptions.RequestException:
- error_console.print(f"endpoint {ctx.obj.url}: RequestException")
+ error_console.print(f"Error: endpoint {ctx.obj.url}: RequestException")
return
if "result" in r.text and r.ok:
return(r.json()["result"])
else:
if "detail" in r.text:
- error_console.print(r.json()["detail"])
+ error_console.print(f"Error: {r.json()['detail']}")
raise typer.Exit(1) # non-zero exit status
def delete_partition_policy(ctx: typer.Context, policy_id: str):
@@ -117,7 +122,7 @@ def delete_partition_policy(ctx: typer.Context, policy_id: str):
return True
else:
if "detail" in r.text:
- error_console.print(r.json()["detail"])
+ error_console.print(f"Error: {r.json()['detail']}")
raise typer.Exit(1) # non-zero exit status
def add_partition_policy(ctx: typer.Context, policy_id: str, files: dict):
@@ -135,7 +140,7 @@ def add_partition_policy(ctx: typer.Context, policy_id: str, files: dict):
return r.text
else:
if "detail" in r.text:
- error_console.print(r.json()["detail"])
+ error_console.print(f"Error: {r.json()['detail']}")
raise typer.Exit(1) # non-zero exit status
def get_partition_policy(ctx: typer.Context, policy_id: str, quiet=False):
@@ -153,7 +158,7 @@ def get_partition_policy(ctx: typer.Context, policy_id: str, quiet=False):
return(r.json()["result"])
else:
if not quiet and "detail" in r.text:
- error_console.print(r.json()["detail"])
+ error_console.print(f"Error: {r.json()['detail']}")
def get_instance_policy(ctx: typer.Context, policy_id: str, quiet=False):
"""
@@ -170,7 +175,7 @@ def get_instance_policy(ctx: typer.Context, policy_id: str, quiet=False):
return(r.json()["result"])
else:
if not quiet and "detail" in r.text:
- error_console.print(r.json()["detail"])
+ error_console.print(f"Error: {r.json()['detail']}")
def evaluations_query(ctx: typer.Context, policy_id: str, files: dict):
"""
@@ -194,7 +199,7 @@ def evaluations_query(ctx: typer.Context, policy_id: str, files: dict):
return(r.json()["result"])
else:
if "detail" in r.text:
- error_console.print(r.json()["detail"])
+ error_console.print(f"Error: {r.json()['detail']}")
raise typer.Exit(1) # non-zero exit status
@cli.command(rich_help_panel="Utils")
@@ -203,7 +208,7 @@ def groups(ctx: typer.Context,
domain: bool = False,
domain_suffix: bool = False):
"""
- Show policy related groups of current auth context.
+ Show groups of current auth context.
"""
retgroups = request_groups(ctx)
if all:
@@ -240,7 +245,7 @@ def is_json(myjson):
return False
return True
-@cli.command(rich_help_panel="Developer Utils")
+@cli.command(rich_help_panel="Policy Developer Utils")
def translate(ctx: typer.Context,
policy_id: str= typer.Argument(...),
output: OutputType = typer.Option("fancy", help="Output style"),
@@ -287,13 +292,14 @@ def translate(ctx: typer.Context,
typer.confirm(f"translate above data against {short_policy_id} in {ctx.obj.data_partition}", abort=True)
if not is_json(data):
- error_console.print("Not valid json")
+ error_console.print("Error: Not valid json")
raise typer.Exit(1)
try:
- r = requests.post(ctx.obj.url + "/translate",
- data=data,
- headers=headers(ctx))
+ with console.status("Translating..."):
+ r = requests.post(ctx.obj.url + "/translate",
+ data=data,
+ headers=headers(ctx))
except requests.exceptions.RequestException as e:
raise SystemExit(e)
@@ -304,10 +310,10 @@ def translate(ctx: typer.Context,
print(r.json())
else:
if "detail" in r.text:
- error_console.print(r.json()["detail"], r.status_code)
+ error_console.print(f"Error: {r.json()['detail']} {r.status_code}")
raise typer.Exit(1) # non-zero exit status
-@cli.command()
+@cli.command(rich_help_panel="Policy Commands")
def eval(ctx: typer.Context,
policy_id: str= typer.Argument(None),
file: Path = typer.Option(...,
@@ -340,7 +346,7 @@ def eval(ctx: typer.Context,
This command will allow the policy service to add/replace any auth details in json using the include_auth capability in API.
"""
if policy_id.startswith('osdu/instance/'):
- error_console.print("Instance policies not supported")
+ error_console.print("Error: Instance policies not supported")
raise typer.Abort()
if domain is None:
@@ -481,10 +487,10 @@ def display_all_policies(ctx: typer.Context,
if preview.startswith("p"):
table.add_row(pol['id'], preview.strip())
else:
- print("error")
+ console.print("Error: unexpected preview data")
console.print(table)
-@cli.command("ls")
+@cli.command("ls", rich_help_panel="Policy Commands")
def list(ctx: typer.Context,
policy_list: List[str]= typer.Argument(None),
raw: bool = typer.Option(False, "--raw", help="Full output from policy service"),
@@ -499,6 +505,10 @@ def list(ctx: typer.Context,
If policy_list is - on command-line, the list will be read from standard-in
'''
+ if output == output.excel:
+ error_console.print("Error: Not supported output type")
+ raise typer.Exit(1) # non-zero exit status
+
if policy_list == ['-']:
policy_list = sys.stdin.read().strip().split()
@@ -549,57 +559,54 @@ def list(ctx: typer.Context,
else:
raise typer.Exit(1) # non-zero exit status
-@cli.command(hidden=True, rich_help_panel="Utils")
+@cli.command(hidden=True, rich_help_panel="Policy Developer Utils")
def test(ctx: typer.Context, policy_id: str = typer.Option("", '-p')):
"""
Do some tests. Hidden command
"""
console.print(ctx.obj)
-@cli.command(rich_help_panel="Utils")
+@cli.command(rich_help_panel="Policy Commands")
def health(ctx: typer.Context):
"""
Is Policy service healthy
"""
try:
- r = requests.get(ctx.obj.url + "/health", headers=headers(ctx))
+ with console.status("Getting health..."):
+ r = requests.get(ctx.obj.url + "/health", headers=headers(ctx))
except requests.exceptions.RequestException as e:
raise SystemExit(e)
if r.ok:
- console.print(r.text)
+ console.print(":thumbsup-emoji:",r.text)
else:
error_console.print(r.text)
raise typer.Exit(1) # non-zero exit status
@cli.command(rich_help_panel="Utils")
def info(ctx: typer.Context,
- storage: bool = typer.Option(False, help="Get storage info"),
- search: bool = typer.Option(False, help="Get search info")
+ service: str = typer.Option("policy", "-s", "--service", help="Get search info")
):
"""
- Info on Policy Service, Storage and Search
+ Info on Dataset, Entitlement, Legal, Policy Service, Search, and Storage
"""
- if storage:
- storage_info(ctx=ctx)
- elif search:
- search_info(ctx=ctx)
+ if "storage" in service:
+ get_info(ctx=ctx, base_url=ctx.obj.storage_url)
+ elif "search" in service:
+ get_info(ctx=ctx, base_url=ctx.obj.search_url)
+ elif "entitlement" in service:
+ get_info(ctx=ctx, base_url=ctx.obj.entitlements_url)
+ elif "legal" in service:
+ get_info(ctx=ctx, base_url=ctx.obj.legal_url)
+ elif "policy" in service:
+ get_info(ctx=ctx, base_url=ctx.obj.url)
+ elif "dataset" in service:
+ get_info(ctx=ctx, base_url=ctx.obj.dataset_url)
else:
- policy_info(ctx=ctx)
-
-def policy_info(ctx: typer.Context):
- try:
- r = requests.get(ctx.obj.url + "/info", headers=headers(ctx))
- except requests.exceptions.RequestException as e:
- raise SystemExit(e)
-
- if r.ok:
- console.print(r.json())
- else:
- error_console.print(r.text)
+ error_console.print(f"Error: Unsupported service {service}")
raise typer.Exit(1) # non-zero exit status
-@cli.command(rich_help_panel="Developer Utils")
+@cli.command(rich_help_panel="Policy Developer Utils")
def diff(ctx: typer.Context,
policy_id_1: str = typer.Argument(...),
policy_id_2: str = typer.Argument(...),
@@ -649,12 +656,12 @@ def lookup_policy(ctx: typer.Context, policy_id, quiet=False):
result = get_instance_policy(ctx, short_policy_id, quiet)
if not result:
if not quiet:
- error_console.print(f"Unable to find {policy_id}")
+ error_console.print(f"Error: Unable to find {policy_id}")
raise typer.Exit(1) # non-zero exit status
return result
-@cli.command(rich_help_panel="Developer Utils")
+@cli.command(rich_help_panel="Policy Developer Utils")
def compile(ctx: typer.Context,
file: Path = typer.Option(...,
"-f", "--file",
@@ -715,7 +722,7 @@ def compile(ctx: typer.Context,
error_console.print("Error: unexpected response from policy service. Check if policy exists")
raise typer.Exit(1) # non-zero exit status
-@cli.command(rich_help_panel="Developer Utils")
+@cli.command(rich_help_panel="Policy Developer Utils")
def config(ctx: typer.Context):
"""
Diagnostic config on Policy Service.
@@ -735,17 +742,18 @@ def config(ctx: typer.Context):
@cli.command(rich_help_panel="Utils")
def legal_tags(ctx: typer.Context,
- all: bool = False,
+ raw: bool = False,
limit: int = typer.Option(None, help="Limit number")):
"""
Show legal tags.
"""
try:
- r = requests.get(ctx.obj.legal_url+"?valid=true", headers = headers(ctx))
+ with console.status("Getting legaltags..."):
+ r = requests.get(ctx.obj.legal_url + "/legaltags" + "?valid=true", headers = headers(ctx))
except requests.exceptions.RequestException as e:
raise SystemExit(e)
- if all:
+ if raw:
console.print(json.dumps(r.json(), indent=4))
else:
count = 0
@@ -755,7 +763,7 @@ def legal_tags(ctx: typer.Context,
if count == limit:
break
-@cli.command("opa-add", rich_help_panel="Developer Utils")
+@cli.command("opa-add", rich_help_panel="Policy Developer Utils")
def add_to_opa(ctx: typer.Context,
policy_id: str = typer.Argument(...),
file: Path = typer.Option(...,
@@ -782,7 +790,7 @@ def add_to_opa(ctx: typer.Context,
elif policy_id.startswith('osdu/partition/'):
policy_type = "partition"
else:
- error_console.print("policy_id must start with osdu/instance/ or osdu/partition/")
+ error_console.print("Error: policy_id must start with osdu/instance/ or osdu/partition/")
raise typer.Exit(1) # non-zero exit status
if not policy_id.endswith(".rego"):
@@ -818,7 +826,7 @@ def add_to_opa(ctx: typer.Context,
error_console.print(f"Error adding policy {policy_id} to OPA {r.json()} {r.status_code}")
raise typer.Exit(1) # non-zero exit status
-@cli.command("opa-rm", rich_help_panel="Developer Utils")
+@cli.command("opa-rm", rich_help_panel="Policy Developer Utils")
def delete_from_opa(
ctx: typer.Context,
policy_list: List[str]= typer.Argument(...),
@@ -842,7 +850,7 @@ def delete_from_opa(
elif policy_id.startswith('osdu/partition/'):
policy_type = "partition"
else:
- error_console.print("policy_id must start with osdu/instance/ or osdu/partition/")
+ error_console.print("Error: policy_id must start with osdu/instance/ or osdu/partition/")
raise typer.Exit(1) # non-zero exit status
if not policy_id.endswith(".rego"):
@@ -865,11 +873,13 @@ def delete_from_opa(
def main(
ctx: typer.Context,
token: str = typer.Option(None, '-t', '--token', envvar="TOKEN"),
- url: str = typer.Option(None, '-h', '--url', '--host', envvar="POLICY_URL"),
- entitlements_url: str = typer.Option(None, '-e', '--ent-url', envvar="ENTITLEMENTS_URL"),
- storage_url: str = typer.Option(None, '-e', '--ent-url', envvar="STORAGE_URL", hidden=True),
- search_url: str = typer.Option(None, '-e', '--ent-url', envvar="SEARCH_URL", hidden=True),
- legal_url: str = typer.Option(None, '-l', '--legal-url', envvar="LEGAL_URL"),
+ policy_url: str = typer.Option(None, '-p', '--policy-url', envvar="POLICY_URL"),
+ base_url: str = typer.Option(None, '--host', '--url', '--base-url', envvar="BASE_URL"),
+ entitlements_url: str = typer.Option(None, '-e', '--ent-url', envvar="ENTITLEMENTS_URL", hidden=True),
+ storage_url: str = typer.Option(None, '-s', '--storage-url', envvar="STORAGE_URL", hidden=True),
+ search_url: str = typer.Option(None, '-x', '--ent-url', envvar="SEARCH_URL", hidden=True),
+ legal_url: str = typer.Option(None, '-l', '--legal-url', envvar="LEGAL_URL", hidden=True),
+ dataset_url: str = typer.Option(None, '--dataset-url', envvar="DATASET_URL", hidden=True),
path: str = typer.Option('/api/policy/v1', envvar="POLICY_PATH", hidden=True),
data_partition: str = typer.Option(None, '-d', '--data-partition-id', envvar="DATA_PARTITION"),
x_user_id: str = typer.Option(None, envvar="XUSERID", help="optional user id, added to headers"),
@@ -892,44 +902,59 @@ def main(
error_console.print("Missing data_partition; pass --data-partition or set env[DATA_PARTITION]")
raise typer.Exit(1) # non-zero exit status
- if not url:
- error_console.print("Missing url; pass --url or set env[POLICY_URL]")
- raise typer.Exit(1) # non-zero exit status
+ if not base_url:
+ if not policy_url:
+ error_console.print("Missing url and base_url; pass --url or set env[POLICY_URL]")
+ raise typer.Exit(1) # non-zero exit status
+ base_url = policy_url
+
+ if not policy_url:
+ policy_url = base_url
if not entitlements_url:
- entitlements_url = url + '/api/entitlements/v2/groups'
+ entitlements_url = base_url + '/api/entitlements/v2'
if not storage_url:
- storage_url = url + '/api/storage/v2'
+ storage_url = base_url + '/api/storage/v2'
if not search_url:
- search_url = url + '/api/storage/v2'
+ search_url = base_url + '/api/search/v2'
if not legal_url:
- legal_url = url + "/api/legal/v1/legaltags"
+ legal_url = base_url + "/api/legal/v1"
+
+ if not dataset_url:
+ dataset_url = base_url + "/api/dataset/v1"
+
+ if not x_user_id:
+ x_user_id = getpass.getuser()
ctx.obj = SimpleNamespace(
token = token,
data_partition = data_partition,
- url = url + path,
- diag_url = url + "/diag",
+ url = policy_url + path,
+ diag_url = policy_url + "/diag",
entitlements_url = entitlements_url,
storage_url = storage_url,
search_url = search_url,
+ dataset_url = dataset_url,
debug = debug,
legal_url = legal_url,
x_user_id = x_user_id)
if debug: # hidden option
print(f"token: {ctx.obj.token}")
- print(f"url: {ctx.obj.url}")
+ print(f"base_url: {ctx.obj.base_url}")
+ print(f"policy_url: {ctx.obj.url}")
print(f"entitlements_url: {ctx.obj.entitlements_url}")
print(f"storage_url: {ctx.obj.storage_url}")
print(f"search_url: {ctx.obj.search_url}")
print(f"legal_url: {ctx.obj.legal_url}")
print(f"data_partition: {ctx.obj.data_partition}")
+ print(f"dataset_url: {ctx.obj.dataset_url}")
+ print(f"x-user-id: {ctx.obj.x_user_id}")
-@cli.command()
+@cli.command(rich_help_panel="Policy Commands")
def add(ctx: typer.Context,
policy_id: str = typer.Argument(...),
#file: typer.FileBinaryRead = typer.Option(..., encoding='utf-8')
@@ -954,7 +979,7 @@ def add(ctx: typer.Context,
${name} = short name you give as a policy_id on the command line
"""
if policy_id.startswith('osdu/instance/'):
- error_console.print("Instance policies not supported")
+ error_console.print("Error: Instance policies not supported")
raise typer.Abort()
short_policy_id = os.path.basename(policy_id)
if not short_policy_id.endswith(".rego"):
@@ -972,6 +997,7 @@ def add(ctx: typer.Context,
'name': short_policy_id.removesuffix('.rego')
})
console.print(Panel(data, title="rendered policy", highlight=True))
+
if not force:
typer.confirm(f"add {short_policy_id} in {ctx.obj.data_partition}", abort=True)
bdata = data.encode('utf-8')
@@ -980,7 +1006,7 @@ def add(ctx: typer.Context,
with console.status(f"Adding partition policy {short_policy_id}"):
add_partition_policy(ctx=ctx, policy_id=short_policy_id, files=files)
-@cli.command("rm")
+@cli.command("rm", rich_help_panel="Policy Commands")
def delete(
ctx: typer.Context,
policy_list: List[str]= typer.Argument(...),
@@ -999,7 +1025,7 @@ def delete(
for policy_id in policy_list:
if "osdu/instance/" in policy_id:
- error_console.print("Instance policies not supported")
+ error_console.print("Error: Instance policies not supported")
raise typer.Abort()
short_policy_id = os.path.basename(policy_id)
@@ -1037,12 +1063,13 @@ def delete(
else:
error_console.print(f"Skipping delete of {policy_id}")
-def storage_info(ctx: typer.Context):
+def get_info(ctx: typer.Context, base_url):
"""
- Storage
+ Get info
"""
try:
- r = requests.get(ctx.obj.storage_url + "/info", headers=headers(ctx))
+ with console.status("Getting info..."):
+ r = requests.get(base_url + "/info", headers=headers(ctx))
except requests.exceptions.RequestException as e:
raise SystemExit(e)
@@ -1052,75 +1079,263 @@ def storage_info(ctx: typer.Context):
error_console.print(r.text)
raise typer.Exit(1) # non-zero exit status
-def search_info(ctx: typer.Context):
+@cli.command(rich_help_panel="Utils")
+def storage(ctx: typer.Context,
+ id_list: List[str]= typer.Argument(..., help="IDs to retrieve"),
+ versions: bool = typer.Option(False, "--versions", "-V", help="Show versions"),
+ version: str = typer.Option(None, "-v", "--version", help="Get particular version"),
+ dataset: bool = typer.Option(False, "--dataset", "-D", help="dataset"),
+ get: bool = typer.Option(False, "--get", hidden=True, help="get dataset"),
+ download: bool = typer.Option(False, "--download", "-d", help="download dataset"),
+ raw: bool = typer.Option(False, "--raw", help="json output"),
+ ):
"""
- Storage
+ Storage and Dataset record retrieval utility
"""
+ if id_list == ['-']:
+ id_list = sys.stdin.read().strip().split()
+
+ if not len(id_list):
+ error_console.print("Error: Missing input 'ID_LIST...'")
+ raise typer.Exit(1) # non-zero exit status
+
+ for id in id_list:
+ try:
+ if versions:
+ url = ctx.obj.storage_url + "/records/versions/" + id
+ elif version:
+ url = ctx.obj.storage_url + "/records/" + id + "/" + version
+ elif dataset:
+ url = ctx.obj.dataset_url + "/getDatasetRegistry?id=" + id
+ elif get or download:
+ url = ctx.obj.dataset_url + "/getRetrievalInstructions?id=" + id
+ else:
+ url = ctx.obj.storage_url + "/records/" + id
+
+ with console.status(f"Retrieving record..."):
+ r = requests.get(url,
+ headers=headers(ctx)
+ )
+ except requests.exceptions.RequestException as e:
+ raise SystemExit(e)
+
+ if r.ok:
+ if raw:
+ console.print(r.json())
+ elif get or download:
+ get_dataset(ctx=ctx, data=r.json(), download=download)
+ else:
+ console.print(r.json())
+ else:
+ error_console.print("Error:", r.text, r.status_code)
+ raise typer.Exit(1) # non-zero exit status
+
+def get_filename_from_storage(ctx: typer.Context, id):
try:
- r = requests.get(ctx.obj.search_url + "/info",
- headers=headers(ctx)
+ with console.status(f"Retrieving storage record {id}..."):
+ r = requests.get(ctx.obj.storage_url + "/records/" + id,
+ headers=headers(ctx)
)
except requests.exceptions.RequestException as e:
raise SystemExit(e)
-
+
if r.ok:
- console.print(r.json())
+ name = r.json()["data"]["DatasetProperties"]["FileSourceInfo"]["Name"]
+ return name
+
+def get_dataset(ctx: typer.Context, data, download=False, raw=False):
+ id = data["delivery"][0]["datasetRegistryId"]
+ unsigned_url = data["delivery"][0]["retrievalProperties"]["unsignedUrl"]
+ name = get_filename_from_storage(ctx=ctx, id=id)
+ filename = os.path.basename(name)
+ signed_url = data["delivery"][0]["retrievalProperties"]["signedUrl"]
+
+ if download:
+ try:
+ with console.status(f"Downloading {id} {name}..."):
+ r = requests.get(signed_url)
+ except requests.exceptions.RequestException as e:
+ raise SystemExit(e)
+
+ if r.ok:
+ with console.status(f"Saving {filename}..."):
+ f = open(filename, "wb")
+ f.write(r.content)
+ f.close()
+ console.print(f"Downloaded '{name}' from '{id}' as '{filename}'")
else:
- error_console.print(r.text)
- raise typer.Exit(1) # non-zero exit status
+ console.print(f"unsigned_url: {unsigned_url}")
+ console.print(f"signed_url: {signed_url}")
-@cli.command(hidden=True, rich_help_panel="Developer Utils")
-def search(ctx: typer.Context):
- """
- Search
+@cli.command(rich_help_panel="Utils")
+def search(ctx: typer.Context,
+ id: str = typer.Argument(None),
+ kind: str = typer.Option("*:*:*:*", help="The kind(s) of the record to query"),
+ query: str = typer.Option("", help="Query string based on Lucene query string syntax"),
+ sort_field: str = typer.Option("id", help="Sort field"),
+ sort_order: str = typer.Option("ASC", help="Sort Order", callback=search_cli.search_order_callback),
+ limit: int = typer.Option(10, help="The maximum number of results to return from the given offset"),
+ offset: int = typer.Option(0, help="The starting offset from which to return results"),
+ detail: bool = typer.Option(False, "--detail", help="Show more details"),
+ id_only: bool = typer.Option(False, "--id-only", help="Only show IDs"),
+ output: OutputType = typer.Option("fancy", help="Output style"),
+ file: Path = typer.Option("search.xlsx",
+ "-f", "--file",
+ dir_okay=False,
+ resolve_path=True,
+ help="Excel file for output=excel"
+ ),
+ open_file: bool = typer.Option(False, "--open", help="Open file after saving"),
+ raw: bool = typer.Option(False, "--raw", help="Show raw/json search results"),
+ spatial_filter: Path = typer.Option(None,
+ exists=True,
+ file_okay=True,
+ dir_okay=False,
+ writable=False,
+ readable=True,
+ resolve_path=True,
+ help="Spatial filter json file"
+ ),
+ with_cursor: bool = typer.Option(False, "--with-cursor", help="Use cursors"),
+ cursor: str = typer.Option(None, "--cursor", help="Pass cursor back")
+ ):
"""
+ Search utility
+
+ Kind must follow the convention: {Schema-Authority}:{dataset-name}:{record-type}:{version}
+
+ Example Searches:
+ Wellbore Master Data Instances for Well with ID 1691:
+ --kind="*:*:master-data--Wellbore:*" --query=data.WellID:\\"osdu:master-data--Well:1691:\\"
+
+ Wellbore Trajectory Work Product Components associated with Wellbore ID 1691:
+ --kind=*:*:work-product-component--WellboreTrajectory:* --query=data.WellboreID:\\"osdu:master-data--Wellbore:1691:\\"
+
+ Any record with any field equal "well":
+ --kind="*:*:*:*" --query=well
+
+ Where source is blended or TNO:
+ --kind="*:*:*:*" --query="data.Source:(BLENDED TNO)"
- kind = "osdu:*:*:*"
- kind = "*:*:*:*"
+ Where source is exactly "TNO":
+ --kind="*:*:*:*" --query=data.Source:\\"TNO\\"
- query = 'data.WellID:"osdu:master-data--Well:1000:"'
- offset=0
- limit=0
- queryasowner=False
- aggregateby=''
+ All wellbore logs from 2022 year:
+ --kind="*:*:work-product-component--WellLog:*" --query="createTime:[2022-01-01 TO 2022-12-31]"
+ All well logs deeper than 4000m:
+ --kind="*:*:work-product-component--WellLog:*" query="data.BottomMeasuredDepth:[4000 TO *]"
+
+ All well logs deeper than 2000m or shallower than 4000m:
+ --kind="*:*:work-product-component--WellLog:*" --query="data.BottomMeasuredDepth:(>=2000 OR <=4000)"
+
+ Note: --with-cursor
+ To process the next --with-cursor request, the search service keeps the search context alive for 1 minute,
+ which is the time required to process the next batch of results. Each cursor request sets a new expiry time.
+ The cursor will expire after 1 min and won't return any more results if the requests are not made in specified time.
+ """
+ if id:
+ query=f"id:\"{id}\""
+
+ # Note: above \\ is for help output, so just one \ is needed
+
+ # search_json_data = {
+ # 'kind': kind,
+ # 'query': query,
+ # 'offset': offset,
+ # 'limit': limit,
+ # 'queryAsOwner': queryasowner,
+ # 'aggregateBy': aggregateby,
+ # 'sort': {
+ # 'field': [
+ # 'id',
+ # ],
+ # 'order': [
+ # 'ASC',
+ # ],
+ # },
+ # 'returnedFields': [],
+ # }
+
+ sort_data = {
+ 'field': [ sort_field ],
+ 'order': [ sort_order ]
+ }
search_json_data = {
'kind': kind,
'query': query,
- 'offset': offset,
'limit': limit,
- 'queryAsOwner': queryasowner,
- 'aggregateBy': aggregateby,
- 'sort': {
- 'field': [
- 'id',
- ],
- 'order': [
- 'ASC',
- ],
- },
- 'returnedFields': [],
+ 'offset': offset,
+ 'sort': sort_data
}
- search_json_data = {
- 'kind': kind,
- 'query': "well"
- }
+ if spatial_filter:
+ try:
+ with rich.progress.open(spatial_filter, "r") as f:
+ spatial_filter_data = json.load(f)
+ f.close()
+ except json.decoder.JSONDecodeError as err:
+ error_console.print(f"Error: {spatial_filter} malformed json. {err}")
+ raise typer.Exit(1) # non-zero exit status
+
+ search_json_data["spatialFilter"] = spatial_filter_data
+
+ if with_cursor:
+ url = ctx.obj.search_url + "/query_with_cursor"
+ if cursor:
+ search_json_data["cursor"] = cursor
+ else:
+ url = ctx.obj.search_url + "/query"
+
+ if ctx.obj.debug:
+ console.print(search_json_data)
- console.print(search_json_data)
- console.print(headers(ctx))
try:
- r = requests.post(ctx.obj.search_url + "/query",
- json=search_json_data,
- headers=headers(ctx)
- )
+ with console.status(f"Retrieving search results..."):
+ r = requests.post(url,
+ json=search_json_data,
+ headers=headers(ctx)
+ )
except requests.exceptions.RequestException as e:
raise SystemExit(e)
+ try:
+ # silly math for figuring out terminal size
+ width, height = os.get_terminal_size()
+ except:
+ # not a tty... no worries just simple output
+ output = output.simple
+
if r.ok:
- console.print(r.json())
+ if "results" in r.json():
+ count=len(r.json()["results"])
+
+ if raw:
+ console.print(r.json())
+ elif not count:
+ error_console.print("No results found")
+ raise typer.Exit(2) # non-zero exit status
+ elif id_only:
+ search_cli.display_search_results_idonly(r.json())
+ elif output == output.fancy:
+ search_cli.display_search_results_fancy(r.json(), show_kind=detail)
+ elif output == output.simple:
+ search_cli.display_search_results_simple(r.json())
+ elif output == output.excel:
+ with console.status(f"Saving search results..."):
+ search_cli.display_search_results_excel(r.json(), show_kind=detail, excelfile=file.name)
+ #os.system(f"open -a'Microsoft Excel.app' {file.name}")
+ #os.startfile(f"{file.name}")
+ if open_file:
+ search_cli.open_file(file.name)
+ elif output == output.tree:
+ error_console.print("Error: Not supported output type")
+ raise typer.Exit(1) # non-zero exit status
+
+ if with_cursor:
+ console.print(f":warning-emoji: [bold] Cursor: [blue]{r.json()['cursor']}")
else:
- error_console.print("error:", r.text, r.status_code)
+ error_console.print("Error:", r.text, r.status_code)
raise typer.Exit(1) # non-zero exit status
if __name__ == "__main__":
diff --git a/frontend/admincli/requirements-dev.txt b/frontend/admincli/requirements-dev.txt
index 15270e9f08b7725e0e9e22e7255040c71de23ba0..173d67a4282d4b052dfe1c9ed9a1d2ad33e9ea91 100644
--- a/frontend/admincli/requirements-dev.txt
+++ b/frontend/admincli/requirements-dev.txt
@@ -1,7 +1,5 @@
+XlsxWriter == 3.0.3
+pytest == 7.1.2
requests == 2.25.1
-rich == 12.5.1
+rich == 12.6.0
typer == 0.6.1
-pyinstaller
-pytest
-boto3
-jwt
diff --git a/frontend/admincli/requirements.txt b/frontend/admincli/requirements.txt
index f00240d99e229358aae7babad9caa1461fb202c2..07bee6e41b2cebc56155f6b83100a255812eec9b 100644
--- a/frontend/admincli/requirements.txt
+++ b/frontend/admincli/requirements.txt
@@ -1,5 +1,6 @@
# Automatically generated by https://github.com/damnever/pigar.
+XlsxWriter == 3.0.3
requests == 2.25.1
-rich == 12.5.1
+rich == 12.6.0
typer == 0.6.1
diff --git a/frontend/admincli/search_cli.py b/frontend/admincli/search_cli.py
new file mode 100644
index 0000000000000000000000000000000000000000..d1a69f04ee6fd66858c098317fb6491d0da828be
--- /dev/null
+++ b/frontend/admincli/search_cli.py
@@ -0,0 +1,93 @@
+from rich.table import Table
+from rich.console import Console
+import xlsxwriter
+import os, sys, subprocess
+import typer
+
+console = Console()
+error_console = Console(stderr=True, style="bold red")
+
+def display_search_results_excel(data,
+ show_kind = False,
+ excelfile = "search.xlsx",
+ worksheetname = "Search Results"):
+
+ # Workbook() takes one, non-optional, argument
+ # which is the filename that we want to create.
+ workbook = xlsxwriter.Workbook(excelfile)
+
+ # The workbook object is then used to add new
+ # worksheet via the add_worksheet() method.
+ worksheet = workbook.add_worksheet(worksheetname)
+
+ # Create a new Format object to formats cells
+ # in worksheets using add_format() method.
+ # here we create bold format object.
+ bold = workbook.add_format({'bold': 1})
+
+ headings = ['ID', "Kind", "Authority", "Source", "Type", "Create Time", "Create User"]
+
+ # Write a row of data starting from 'A1' with bold format
+ worksheet.write_row('A1', headings, bold)
+
+ # Start from the first cell below the headers.
+ row = 1
+ col = 0
+ for item in data["results"]:
+ worksheet.write_row(row=row, col=0, data=(
+ item['id'], item["kind"], item["authority"], item["source"], item["type"], item["createTime"], item["createUser"]
+ ))
+ row += 1
+
+ worksheet.set_column(0, 0, 50) # Column A width set to 20.
+ worksheet.set_column(1, 1, 30) # Columns B width set to 30.
+ worksheet.set_column(2, 3, 8) # Columns C-D width set to 8.
+ worksheet.set_column(4, 4, 20) # Columns width set to 25.
+ worksheet.set_column(5, 6, 25) # Columns width set to 25.
+
+ # Finally, close the Excel file
+ workbook.close()
+ console.print(f"Search results ({row-1} records) saved as '{excelfile}'")
+
+def display_search_results_simple(data):
+ for item in data["results"]:
+ created = item["createTime"]
+
+ print(item['id'], item["kind"], item["authority"], item["source"], item["type"], created)
+
+def display_search_results_idonly(data):
+ for item in data["results"]:
+ print(item['id'])
+
+def display_search_results_fancy(data, show_kind=False):
+ count=len(data["results"])
+ if not count:
+ return
+ table = Table(title=f"Search Results ({count})")
+ table.add_column("ID", justify="full", style="cyan", no_wrap=True, min_width=25)
+ if show_kind:
+ table.add_column("Kind", justify="full", style="cyan", no_wrap=True, min_width=25)
+ table.add_column("Authority", justify="center", style="green", no_wrap=True, max_width=10)
+ table.add_column("Source", justify="center", style="green", no_wrap=True, max_width=8)
+ table.add_column("Type", justify="center", style="green", no_wrap=True, min_width=15)
+ table.add_column("Created", justify="center", style="green", no_wrap=True, max_width=11)
+
+ for item in data["results"]:
+ created = item["createTime"]
+ if show_kind:
+ table.add_row(item['id'], item["kind"], item["authority"], item["source"], item["type"], created)
+ else:
+ table.add_row(item['id'], item["authority"], item["source"], item["type"], created)
+ console.print(table)
+
+def open_file(filename):
+ if sys.platform == "win32":
+ os.startfile(filename)
+ else:
+ opener = "open" if sys.platform == "darwin" else "xdg-open"
+ subprocess.call([opener, filename])
+
+def search_order_callback(ctx: typer.Context, value: str):
+ if value != "ASC" and value != 'DESC':
+ raise typer.BadParameter("Only ASC or DESC is allowed")
+ return value
\ No newline at end of file
diff --git a/frontend/admincli/setenv.sh b/frontend/admincli/setenv.sh
index a0b4e03b6f9844e90a630079306e7f37b43a36e3..d9d62cc387bad13f6cb7ae4b9e42f7d570e301fa 100644
--- a/frontend/admincli/setenv.sh
+++ b/frontend/admincli/setenv.sh
@@ -2,8 +2,9 @@
# load by . ./setenv
export DATA_PARTITION=osdu
export TOKEN="$(gcloud auth print-access-token)"
+export BASE_URL="https://policy-dev.osdu.lol"
export POLICY_URL="http://localhost:8080"
-export ENTITLEMENTS_URL="https://policy-dev.osdu.lol/api/entitlements/v2/groups"
-export LEGAL_URL="https://policy-dev.osdu.lol/api/legal/v1/legaltags"
-export STORAGE_URL="https://policy-dev.osdu.lol/api/storage/v2"
-export SEARCH_URL="https://policy-dev.osdu.lol/api/search/v2"
+#export ENTITLEMENTS_URL="https://policy-dev.osdu.lol/api/entitlements/v2"
+#export LEGAL_URL="https://policy-dev.osdu.lol/api/legal/v1"
+#export STORAGE_URL="https://policy-dev.osdu.lol/api/storage/v2"
+#export SEARCH_URL="https://policy-dev.osdu.lol/api/search/v2"
diff --git a/frontend/admincli/tests/create_tag_with_extension.json b/frontend/admincli/tests/create_tag_with_extension.json
new file mode 100644
index 0000000000000000000000000000000000000000..5133e6830348957f6364817fad4e7b00a6ce53df
--- /dev/null
+++ b/frontend/admincli/tests/create_tag_with_extension.json
@@ -0,0 +1,29 @@
+{
+ "name": "opendes-dz-test",
+ "description": "Legal Tag added for Well - updated",
+ "properties": {
+ "countryOfOrigin": [
+ "US",
+ "CA"
+ ],
+ "contractId": "123457",
+ "expirationDate": "2025-12-26",
+ "originator": "Schlumberger",
+ "dataType": "Third Party Data",
+ "securityClassification": "Private",
+ "personalData": "No Personal Data",
+ "exportClassification": "EAR99",
+ "extensionProperties": {
+ "AgreementIdentifier": "dz-test",
+ "EffectiveDate": "2022-06-01T00:00:00",
+ "TerminationDate": "2099-12-31T00:00:00",
+ "AffiliateEnablementIndicator": true,
+ "AgreementParties": [
+ {
+ "AgreementPartyType": "EnabledAffiliate",
+ "AgreementParty": "Shell RDS"
+ }
+ ]
+ }
+ }
+}
diff --git a/frontend/admincli/tests/spatial_filter.json b/frontend/admincli/tests/spatial_filter.json
new file mode 100644
index 0000000000000000000000000000000000000000..89def1a1cf3c29818ff93eb326491e202435a35c
--- /dev/null
+++ b/frontend/admincli/tests/spatial_filter.json
@@ -0,0 +1,10 @@
+{
+ "field": "data.Location",
+ "byDistance": {
+ "point": {
+ "latitude": 37.450727,
+ "longitude": -122.174762
+ },
+ "distance": 1500
+ }
+}
diff --git a/frontend/admincli/tests/spatial_filter2.json b/frontend/admincli/tests/spatial_filter2.json
new file mode 100644
index 0000000000000000000000000000000000000000..5120fd9b799a1b92166258c7cb2d600976136cf9
--- /dev/null
+++ b/frontend/admincli/tests/spatial_filter2.json
@@ -0,0 +1,12 @@
+{
+ "field": "data.Location",
+ "byGeoPolygon": {
+ "points": [
+ {"longitude":-90.65, "latitude":28.56},
+ {"longitude":-90.65, "latitude":35.56},
+ {"longitude":-85.65, "latitude":35.56},
+ {"longitude":-85.65, "latitude":28.56},
+ {"longitude":-90.65, "latitude":28.56}
+ ]
+ }
+}
diff --git a/frontend/admincli/tests/test_cli.py b/frontend/admincli/tests/test_cli.py
index 3956fbc25d71e3d120e5109b316f7525d4be1e4c..742445f681752218fa0f988fefcc59cd4232e1d7 100644
--- a/frontend/admincli/tests/test_cli.py
+++ b/frontend/admincli/tests/test_cli.py
@@ -7,7 +7,15 @@ from pol import cli
runner = CliRunner()
-def test_cli_ls(data_partition):
+def test_cli_policy_add_example(data_partition):
+ result = runner.invoke(cli, ["add", "-f", "tests/example.rego", "-t", "example", "--force"])
+ assert result.exit_code == 0
+
+def test_cli_policy_add_search2(data_partition):
+ result = runner.invoke(cli, ["add", "-f", "../../app/tests/templates/search2.rego", "-t", "search2", "--force"])
+ assert result.exit_code == 0
+
+def test_cli_policy_ls(data_partition):
result = runner.invoke(cli, ["ls"])
assert result.exit_code == 0
assert "osdu/instance/legal.rego" in result.stdout
@@ -15,7 +23,7 @@ def test_cli_ls(data_partition):
assert "osdu/instance/dataauthz.rego" in result.stdout
assert f"osdu/partition/{data_partition}/dataauthz.rego" in result.stdout
-def test_cli_ls_raw(data_partition):
+def test_cli_policy_ls_raw(data_partition):
result = runner.invoke(cli, ["ls", "--raw"])
assert result.exit_code == 0
assert "osdu/instance/legal.rego" in result.stdout
@@ -23,11 +31,11 @@ def test_cli_ls_raw(data_partition):
assert "osdu/instance/dataauthz.rego" in result.stdout
assert f"osdu/partition/{data_partition}/dataauthz.rego" in result.stdout
-def test_cli_ls_output_tree():
+def test_cli_policy_ls_output_tree():
result = runner.invoke(cli, ["ls", "--output=tree"])
assert result.exit_code == 0
-def test_cli_ls_output_fancy(data_partition):
+def test_cli_policy_ls_output_fancy(data_partition):
result = runner.invoke(cli, ["ls", "--output=fancy"])
assert result.exit_code == 0
assert "osdu/instance/legal.rego" in result.stdout
@@ -35,7 +43,7 @@ def test_cli_ls_output_fancy(data_partition):
assert "osdu/instance/dataauthz.rego" in result.stdout
assert f"osdu/partition/{data_partition}/dataauthz.rego" in result.stdout
-def test_cli_ls_output_simple(data_partition):
+def test_cli_policy_ls_output_simple(data_partition):
result = runner.invoke(cli, ["ls", "--output=simple"])
assert result.exit_code == 0
assert "osdu/instance/legal.rego" in result.stdout
@@ -43,84 +51,98 @@ def test_cli_ls_output_simple(data_partition):
assert "osdu/instance/dataauthz.rego" in result.stdout
assert f"osdu/partition/{data_partition}/dataauthz.rego" in result.stdout
-def test_cli_ls_dataauthz(data_partition):
+def test_cli_policy_ls_dataauthz(data_partition):
result = runner.invoke(cli, ["ls", "dataauthz"])
assert f"package osdu.partition[\"{data_partition}\"].dataauthz" in result.stdout
assert result.exit_code == 0
-def test_cli_ls_dataauthz_download(data_partition):
+def test_cli_policy_ls_dataauthz_download(data_partition):
result = runner.invoke(cli, ["ls", "dataauthz", "--download"])
assert f"saved as" in result.stdout
assert result.exit_code == 0
-def test_cli_ls_dataauthz_example_download_raw(data_partition):
- result = runner.invoke(cli, ["ls", "dataauthz", "example", "--raw", "--download"])
+def test_cli_policy_ls_dataauthz_search_download_raw(data_partition):
+ result = runner.invoke(cli, ["ls", "dataauthz", "search", "--raw", "--download"])
assert f"raw saved as" in result.stdout
assert result.exit_code == 0
-def test_cli_ls_stdin_dataauthz(data_partition):
+def test_cli_policy_ls_stdin_dataauthz(data_partition):
result = runner.invoke(cli, ["ls", "-"], input="dataauthz\n")
assert result.exit_code == 0
assert f"package osdu.partition[\"{data_partition}\"].dataauthz" in result.stdout
-def test_cli_ls_more_than_1_legal_dataauthz(data_partition):
+def test_cli_policy_ls_more_than_1_legal_dataauthz(data_partition):
result = runner.invoke(cli, ["ls", "osdu/instance/legal.rego", "dataauthz"])
assert 'package osdu.instance.legal' in result.stdout
assert f"package osdu.partition[\"{data_partition}\"].dataauthz" in result.stdout
assert result.exit_code == 0
-def test_cli_ls_dataauthz_rego(data_partition):
+def test_cli_policy_ls_dataauthz_rego(data_partition):
result = runner.invoke(cli, ["ls", "dataauthz.rego"])
assert f"package osdu.partition[\"{data_partition}\"].dataauthz" in result.stdout
assert result.exit_code == 0
-def test_cli_ls_osdu_partition_osdu_dataauthz(data_partition):
+def test_cli_policy_ls_osdu_partition_osdu_dataauthz(data_partition):
result = runner.invoke(cli, ["ls", "osdu/partition/osdu/dataauthz"])
assert f"package osdu.partition[\"{data_partition}\"].dataauthz" in result.stdout
assert result.exit_code == 0
-def test_cli_ls_osdu_partition_osdu_dataauthz_rego(data_partition):
+def test_cli_policy_ls_osdu_partition_osdu_dataauthz_rego(data_partition):
result = runner.invoke(cli, ["ls", "osdu/partition/osdu/dataauthz.rego"])
assert f"package osdu.partition[\"{data_partition}\"].dataauthz" in result.stdout
assert result.exit_code == 0
-def test_cli_diff(data_partition):
+def test_cli_policy_diff(data_partition):
result = runner.invoke(cli, ["diff", "search", "search2"])
assert result.exit_code == 0
-def test_cli_diff_n(data_partition):
+def test_cli_policy_diff_n(data_partition):
result = runner.invoke(cli, ["diff", "search", "search2", "-n", "0"])
assert result.exit_code == 0
-def test_cli_add_example(data_partition):
- result = runner.invoke(cli, ["add", "-f", "tests/example.rego", "-t", "example", "--force"])
+def test_cli_policy_rm_example(data_partition):
+ result = runner.invoke(cli, ["rm", "example", "--force"])
assert result.exit_code == 0
-def test_cli_rm_example(data_partition):
- result = runner.invoke(cli, ["rm", "example", "--force"])
+def test_cli_policy_health():
+ result = runner.invoke(cli, ["health"])
assert result.exit_code == 0
+ assert "Healthy" in result.stdout
-def test_cli_ls_help():
- result = runner.invoke(cli, ["ls", "--help"])
+def test_cli_info_policy1():
+ result = runner.invoke(cli, ["info"])
assert result.exit_code == 0
+ assert "version" in result.stdout
-def test_cli_health():
- result = runner.invoke(cli, ["health"])
+def test_cli_info_policy2():
+ result = runner.invoke(cli, ["info", "-s", "policy"])
assert result.exit_code == 0
- assert "Healthy" in result.stdout
+ assert "version" in result.stdout
-def test_cli_health_help():
- result = runner.invoke(cli, ["health", "--help"])
+def test_cli_info_dataset():
+ result = runner.invoke(cli, ["info", "-s", "dataset"])
+ assert result.exit_code == 0
+ assert "version" in result.stdout
+
+def test_cli_info_storage():
+ result = runner.invoke(cli, ["info", "-s", "storage"])
assert result.exit_code == 0
+ assert "version" in result.stdout
-def test_cli_info():
- result = runner.invoke(cli, ["info"])
+def test_cli_info_search():
+ result = runner.invoke(cli, ["info", "-s", "search"])
assert result.exit_code == 0
assert "version" in result.stdout
-def test_cli_info_help():
- result = runner.invoke(cli, ["info", "--help"])
+def test_cli_info_entitlement():
+ result = runner.invoke(cli, ["info", "-s", "entitlement"])
assert result.exit_code == 0
+ assert "version" in result.stdout
+
+def test_cli_info_legal():
+ result = runner.invoke(cli, ["info", "-s", "legal"])
+ assert result.exit_code == 0
+ assert "version" in result.stdout
def test_cli_group():
result = runner.invoke(cli, ["groups"])
@@ -143,14 +165,108 @@ def test_cli_legaltags():
result = runner.invoke(cli, ["legal-tags"])
assert result.exit_code == 0
-def test_cli_legaltags_all():
- result = runner.invoke(cli, ["legal-tags", "--all"])
+def test_cli_legaltags_raw():
+ result = runner.invoke(cli, ["legal-tags", "--raw"])
+ assert result.exit_code == 0
+
+def test_cli_search():
+ result = runner.invoke(cli, ["search"])
+ assert result.exit_code == 0
+
+def test_cli_search_raw():
+ result = runner.invoke(cli, ["search", "--raw"])
+ assert "createTime" in result.stdout
+ assert result.exit_code == 0
+
+def test_cli_search_limit1():
+ result = runner.invoke(cli, ["search", "--limit=1"])
+ assert "dataset" or "osdu" or "opendes" in result.stdout
+ assert result.exit_code == 0
+
+def test_cli_search_limit2():
+ result = runner.invoke(cli, ["search", "--limit", "2"])
+ assert "dataset" or "osdu" or "opendes" in result.stdout
+ assert result.exit_code == 0
+
+def test_cli_search_output_simple():
+ result = runner.invoke(cli, ["search", "--output=simple"])
+ assert "dataset" or "osdu" or "opendes" in result.stdout
assert result.exit_code == 0
+def test_cli_search_output_fancy():
+ result = runner.invoke(cli, ["search", "--output=fancy"])
+ assert "dataset" or "osdu" or "opendes" in result.stdout
+ assert result.exit_code == 0
+
+def test_cli_search_output_excel():
+ result = runner.invoke(cli, ["search", "--output=excel"])
+ assert result.exit_code == 0
+
+def test_cli_search_id_only():
+ result = runner.invoke(cli, ["search", "--id-only"])
+ assert "dataset" or "osdu" or "opendes" in result.stdout
+ assert result.exit_code == 0
+
+def test_cli_search_query_well():
+ result = runner.invoke(cli, ["search", "--query", "well"])
+ # Will exit 2 if there isn't well data loaded
+ assert result.exit_code == 0 or result.exit_code == 2, f"unexpected return status {result.exit_code} {result.stdout}"
+
+def test_cli_search_kind():
+ result = runner.invoke(cli, ["search", "--kind", '*:*:*:*'])
+ assert "dataset" or "osdu" or "opendes" in result.stdout
+ assert result.exit_code == 0
+
+def test_cli_search_storage_workflow():
+ # Get an ID from search
+ result = runner.invoke(cli, ["search", "--limit=1", "--id-only"])
+ assert "dataset" or "osdu" or "opendes" in result.stdout
+ id = result.stdout.strip()
+ assert result.exit_code == 0
+ result = runner.invoke(cli, ["storage", id])
+ if "Record not found" in result.stdout:
+ pytest.skip(f"Skipping test - Record {id} not found in storage")
+ assert "data" in result.stdout
+ assert "DatasetProperties" in result.stdout
+ assert "FileSourceInfo" in result.stdout
+ assert "createTime" in result.stdout
+ assert "createUser" in result.stdout
+ assert result.exit_code == 0
+
+def test_cli_search_storage_dataset_workflow():
+ # Get an ID from search
+ result = runner.invoke(cli, ["search", "--limit=1", "--id-only"])
+ assert "dataset" or "osdu" or "opendes" in result.stdout
+ id = result.stdout.strip()
+ result = runner.invoke(cli, ["storage", id, "--dataset"])
+ assert "datasetRegistries" in result.stdout
+ assert result.exit_code == 0
+
+def test_cli_search_storage_dataset_download_workflow():
+ # Get an ID from search
+ result = runner.invoke(cli, ["search", "--limit=1", "--id-only"])
+ assert "dataset" or "osdu" or "opendes" in result.stdout
+ id = result.stdout.strip()
+ result = runner.invoke(cli, ["storage", id, "--get"])
+ if "No DMS handler" in result.stdout:
+ pytest.skip(f"Skipping test - Download not supported for id {id}")
+ assert result.exit_code == 0, f"Expected storage id {id} get to have details on dataset {result.stdout}"
+ result = runner.invoke(cli, ["storage", id, "--download", "--raw"])
+ assert result.exit_code == 0
+ result = runner.invoke(cli, ["storage", id, "--download"])
+ assert "Downloaded" in result.stdout
+ assert result.exit_code == 0
+
+# Test all the helps
+# Help for Policy Commands
def test_cli_help():
result = runner.invoke(cli, ["--help"])
assert result.exit_code == 0
+def test_cli_ls_help():
+ result = runner.invoke(cli, ["ls", "--help"])
+ assert result.exit_code == 0
+
def test_cli_add_help():
result = runner.invoke(cli, ["add", "--help"])
assert result.exit_code == 0
@@ -163,6 +279,11 @@ def test_cli_eval_help():
result = runner.invoke(cli, ["eval", "--help"])
assert result.exit_code == 0
+def test_cli_health_help():
+ result = runner.invoke(cli, ["health", "--help"])
+ assert result.exit_code == 0
+
+# Help for Policy Developer Utils/Commands
def test_cli_translate_help():
result = runner.invoke(cli, ["translate", "--help"])
assert result.exit_code == 0
@@ -171,8 +292,33 @@ def test_cli_diff_help():
result = runner.invoke(cli, ["diff", "--help"])
assert result.exit_code == 0
+def test_cli_compile_help():
+ result = runner.invoke(cli, ["compile", "--help"])
+ assert result.exit_code == 0
+
+def test_cli_opa_add_help():
+ result = runner.invoke(cli, ["opa-add", "--help"])
+ assert result.exit_code == 0
+
+def test_cli_opa_rm_help():
+ result = runner.invoke(cli, ["opa-rm", "--help"])
+ assert result.exit_code == 0
+
+# Help for Utils/Commands
+def test_cli_info_help():
+ result = runner.invoke(cli, ["info", "--help"])
+ assert result.exit_code == 0
+
+def test_cli_search_help():
+ result = runner.invoke(cli, ["search", "--help"])
+ assert result.exit_code == 0
+
+def test_cli_storage_help():
+ result = runner.invoke(cli, ["storage", "--help"])
+ assert result.exit_code == 0
+
def test_cli_show_completion():
result = runner.invoke(cli, ["--show-completion"])
if result.exit_code == 2:
pytest.skip("Skipping test - not supported env")
- assert result.exit_code == 0
+ assert result.exit_code == 0
\ No newline at end of file