Commit b16a9886 authored by Spencer Sutton's avatar Spencer Sutton
Browse files

Data workflow scheduling endpoints

commit 2fabd215 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Fri Jan 08 2021 09:05:12 GMT-0600 (Central Standard Time) 

    Merging with dev


commit 82c9b540 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Thu Jan 07 2021 15:31:21 GMT-0600 (Central Standard Time) 

    Moving to version 0.0.4


commit 288ae20f 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Thu Jan 07 2021 15:30:28 GMT-0600 (Central Standard Time) 

    Query record endpoint


commit 985d71b4 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Mon Jan 04 2021 12:46:29 GMT-0600 (Central Standard Time) 

    Scheduling client


commit 671dd771 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Wed Dec 30 2020 08:52:41 GMT-0600 (Central Standard Time) 

    Adjusting service principal to use ssm for custom scope and token uri


commit 1ba80da5 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Tue Dec 22 2020 15:40:49 GMT-0600 (Central Standard Time) 

    Incrementing to version 3


commit 8bef5ece 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Tue Dec 22 2020 15:06:27 GMT-0600 (Central Standard Time) 

    More bug


commit 2feee09e 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Tue Dec 22 2020 15:03:50 GMT-0600 (Central Standard Time) 

    More bug


commit 33f4893e 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Tue Dec 22 2020 15:00:40 GMT-0600 (Central Standard Time) 

    Fixing more broken unit tests


commit c1675bcd 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Tue Dec 22 2020 14:59:11 GMT-0600 (Central Standard Time) 

    Fixing more broken unit tests


commit 095f7ff8 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Tue Dec 22 2020 14:56:52 GMT-0600 (Central Standard Time) 

    Fixing broken test


commit d93c2b48 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Mon Dec 21 2020 19:37:56 GMT-0600 (Central Standard Time) 

    Renaming test config value


commit 8d846904 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Mon Dec 21 2020 19:26:43 GMT-0600 (Central Standard Time) 

    Updating config in tests


commit 687b27d3 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Mon Dec 21 2020 18:30:34 GMT-0600 (Central Standard Time) 

    Merge branch 'feat/devops' of codecommit::us-east-1://default@os-python-wrapper into feat/entitlements


commit a1640f47 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Mon Dec 21 2020 14:44:22 GMT-0600 (Central Standard Time) 

    Adding a storage endpoint


commit e00d0ee7 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Thu Nov 19 2020 13:36:50 GMT-0600 (Central Standard Time) 

    Adding skip existing command


commit e233a0a7 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Wed Dec 09 2020 10:07:59 GMT-0600 (Central Standard Time) 

    Merge branch 'feat/entitlements' of codecommit::us-east-1://default@os-python-wrapper into feat/entitlements


commit 063260ba 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Thu Nov 19 2020 13:01:27 GMT-0600 (Central Standard Time) 

    Adding comments to buildspec


commit ec7be581 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Wed Dec 09 2020 10:07:38 GMT-0600 (Central Standard Time) 

    Updated dataset


commit 2e724fb8 
Author: Brady Spiva <spivbrad@amazon.com> 
Date: Mon Dec 07 2020 15:10:13 GMT-0600 (Central Standard Time) 

    removing .pyc files; adding .pyc to gitignore


commit 086e5f19 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Thu Nov 19 2020 12:02:45 GMT-0600 (Central Standard Time) 

    typo in twine command


commit 60037b13 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Thu Dec 03 2020 12:36:42 GMT-0600 (Central Standard Time) 

    Updating dataset api


commit a295f6c2 
Author: Brady Spiva <spivbrad@amazon.com> 
Date: Mon Dec 07 2020 15:06:18 GMT-0600 (Central Standard Time) 

    adding legal license


commit 221e46eb 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Thu Nov 19 2020 11:59:22 GMT-0600 (Central Standard Time) 

    Changing the way to use twine


commit 1e1e3be6 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Wed Dec 02 2020 14:49:44 GMT-0600 (Central Standard Time) 

    Adding schema client and data workflow client


commit a9cb71fa 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Thu Nov 19 2020 11:51:08 GMT-0600 (Central Standard Time) 

    Using twine to publish artifact to CodeArtifact


commit b5280136 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Fri Nov 20 2020 13:35:44 GMT-0600 (Central Standard Time) 

    Adding entitlements client


commit 6fd36b54 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Thu Nov 19 2020 11:34:27 GMT-0600 (Central Standard Time) 

    Fixing purposely failed test


commit 8462b083 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Thu Nov 19 2020 11:32:20 GMT-0600 (Central Standard Time) 

    Testing that failed unit test fails build


commit d7dd8682 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Thu Nov 19 2020 11:28:02 GMT-0600 (Central Standard Time) 

    wrong syntax for env vars


commit ed6c461a 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Thu Nov 19 2020 11:22:53 GMT-0600 (Central Standard Time) 

    Setting stubbed env vars for unit tests


commit b64caf8f 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Thu Nov 19 2020 11:16:04 GMT-0600 (Central Standard Time) 

    Forgot the -r on pip install


commit a21b06e9 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Thu Nov 19 2020 11:11:43 GMT-0600 (Central Standard Time) 

    Adding python build stmt


commit 8f84b68d 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Thu Nov 19 2020 11:09:47 GMT-0600 (Central Standard Time) 

    Adding requirements for build


commit 1ee4ff6a 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Thu Nov 19 2020 11:02:43 GMT-0600 (Central Standard Time) 

    Adding discover command to unittest


commit 157d5026 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Thu Nov 19 2020 10:59:32 GMT-0600 (Central Standard Time) 

    Handling of config file for unit tests


commit 645a9210 
Author: Sutton <suttonsp@147dda3a90de.ant.amazon.com> 
Date: Thu Nov 19 2020 10:44:30 GMT-0600 (Central Standard Time) 

    Initial build
parent 96b25318
......@@ -53,7 +53,7 @@ phases:
# publish new artifact to code artifact
- aws codeartifact login --tool twine --domain osdu-dev --domain-owner 888733619319 --repository osdu-python
- python setup.py sdist bdist_wheel
- twine upload --skip-existing --repository codeartifact dist/osdu_api-0.0.3.tar.gz
- twine upload --skip-existing --repository codeartifact dist/osdu_api-0.0.4.tar.gz
artifacts:
......
# Copyright © 2021 Amazon Web Services
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import json
from typing import List
from osdu_api.clients.base_client import BaseClient
from osdu_api.model.data_workflow.workflow_schedule import WorkflowSchedule
from osdu_api.model.data_workflow.get_workflow_schedules_request import GetWorkflowSchedulesRequest
from osdu_api.model.http_method import HttpMethod
'''
Holds the logic for interfacing with Data Workflow's scheduling api
'''
class DataWorkflowSchedulingClient(BaseClient):
def create_workflow_schedule(self, workflow_schedule: WorkflowSchedule, bearer_token=None):
return self.make_request(method=HttpMethod.POST, url='{}{}'.format(self.data_workflow_url, '/scheduling'),
data=workflow_schedule.to_JSON(), bearer_token=bearer_token)
def list_workflow_schedules(self, bearer_token=None):
return self.make_request(method=HttpMethod.GET, url='{}{}'.format(self.data_workflow_url, '/scheduling'), bearer_token=bearer_token)
def get_workflow_schedules(self, get_workflow_schedules_request: GetWorkflowSchedulesRequest, bearer_token=None):
return self.make_request(method=HttpMethod.POST, url='{}{}'.format(self.data_workflow_url, '/scheduling/getSchedules'),
data=get_workflow_schedules_request.to_JSON(), bearer_token=bearer_token)
def delete_workflow_schedule(self, workflow_schedule_name: str, bearer_token=None):
return self.make_request(method=HttpMethod.DELETE, url='{}{}{}'.format(self.data_workflow_url, '/scheduling/', workflow_schedule_name), bearer_token=bearer_token)
......@@ -15,6 +15,7 @@ import json
from typing import List
from osdu_api.clients.base_client import BaseClient
from osdu_api.model.storage.record import Record
from osdu_api.model.storage.query_records_request import QueryRecordsRequest
from osdu_api.model.http_method import HttpMethod
'''
......@@ -73,4 +74,7 @@ class RecordClient(BaseClient):
def delete_record(self, recordId: str, bearer_token = None):
return self.make_request(method=HttpMethod.DELETE, url=('{}{}/{}'.format(self.storage_url, '/records', recordId)), bearer_token=bearer_token)
\ No newline at end of file
def query_records(self, query_records_request: QueryRecordsRequest, bearer_token = None):
return self.make_request(method=HttpMethod.POST, url='{}{}'.format(self.storage_url, '/query/records'),
data=query_records_request.to_JSON(), bearer_token=bearer_token)
......@@ -14,6 +14,7 @@
import json
from osdu_api.clients.data_workflow.data_workflow_client import DataWorkflowClient
from osdu_api.clients.data_workflow.data_workflow_scheduling_client import DataWorkflowSchedulingClient
from osdu_api.clients.dataset_registry.dataset_registry_client import DatasetRegistryClient
from osdu_api.clients.file_dms.file_dms_client import FileDMSClient
from osdu_api.model.file_dms.file import File
......@@ -23,13 +24,25 @@ from osdu_api.model.storage.acl import Acl
from osdu_api.model.storage.legal import Legal
from osdu_api.model.storage.record_ancestry import RecordAncestry
from osdu_api.model.dataset_registry.create_dataset_registries import CreateDatasetRegistries
from osdu_api.model.data_workflow.workflow_schedule import WorkflowSchedule
from osdu_api.model.data_workflow.get_workflow_schedules_request import GetWorkflowSchedulesRequest
data_workflow_client = DataWorkflowClient()
data_workflow_scheduling_client = DataWorkflowSchedulingClient()
dataset_registry_client = DatasetRegistryClient()
file_dms_client = FileDMSClient()
# cd ../.. && pip3 uninstall osdu_api && python3 setup.py sdist bdist_wheel && python3 -m pip install ./dist/osdu_api-0.0.2-py3-none-any.whl && cd osdu_api/examples
def data_workflow_scheduling_example():
workflow_schedule = WorkflowSchedule("test-name", "test-desc", "0 12 * * ? *", "my_first_dag", {})
data_workflow_scheduling_client.create_workflow_schedule(workflow_schedule)
get_workflow_schedules_request = GetWorkflowSchedulesRequest(["test-name"])
response = data_workflow_scheduling_client.get_workflow_schedules(get_workflow_schedules_request)
print(response.content)
def file_dms_example():
location = file_dms_client.get_file_upload_location()
......@@ -80,5 +93,6 @@ def dataset_registry_example():
print(response.status_code)
print(response.content)
dataset_registry_example()
data_workflow_scheduling_example()
# dataset_registry_example()
# file_dms_example()
# Copyright © 2021 Amazon Web Services
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from osdu_api.model.base import Base
'''
Request body to data workflow's start workflow endpoint. Input parameters is a dynamic object
but the API expects "datasetRegistryIds"
'''
class GetWorkflowSchedulesRequest(Base):
def __init__(self, schedule_names: list):
self.scheduleNames = schedule_names
# Copyright © 2021 Amazon Web Services
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from osdu_api.model.base import Base
'''
Request body to data workflow's create schedule workflow input endpoint. Input parameters is a dynamic map object to the dag
'''
class WorkflowSchedule(Base):
def __init__(self, name: str, description: str, cron_schedule: str, dag_name: str, input_parameters: object):
self.name = name
self.description = description
self.cronSchedule = cron_schedule
self.dagName = dag_name
self.inputParameters = input_parameters
# Copyright © 2021 Amazon Web Services
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# from osdu_api.model.legal_compliance import LegalCompliance
from osdu_api.model.base import Base
'''
Query Records Request
'''
class QueryRecordsRequest(Base):
def __init__(self, records: list):
self.records = records
......@@ -18,7 +18,7 @@ with open("README.md", "r") as fh:
setuptools.setup(
name="osdu_api",
version="0.0.3",
version="0.0.4",
author="OSDU team",
description="A package to interface with OSDU microservices",
packages=setuptools.find_packages(),
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment