Commit 018c5f23 authored by Paromita Mitra's avatar Paromita Mitra
Browse files

Merge branch 'master' of...

Merge branch 'master' of https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services into pyhton-ulid_addition
parents ba37fac1 0078dcfe
Pipeline #50611 passed with stages
in 11 minutes and 33 seconds
......@@ -12,6 +12,7 @@ The following software have components provided under the terms of this license:
- boto3 (from https://github.com/boto/boto3)
- botocore (from https://github.com/boto/botocore)
- coverage (from https://coverage.readthedocs.io)
- cryptography (from https://github.com/pyca/cryptography)
- google-api-core (from https://github.com/GoogleCloudPlatform/google-cloud-python)
- google-auth (from https://github.com/GoogleCloudPlatform/google-auth-library-python)
- google-auth-oauthlib (from https://github.com/GoogleCloudPlatform/google-auth-library-python-oauthlib)
......@@ -55,14 +56,16 @@ BSD-2-Clause
========================================================================
The following software have components provided under the terms of this license:
- decorator (from https://github.com/micheles/decorator)
- colorama (from https://github.com/tartley/colorama)
- grpcio (from http://www.grpc.io)
- locket (from http://github.com/mwilliamson/locket.py)
- mock (from https://github.com/testing-cabal/mock)
- numpy (from http://www.numpy.org)
- packaging (from https://github.com/pypa/packaging)
- ply (from http://www.dabeaz.com/ply/)
- pyasn1 (from http://sourceforge.net/projects/pyasn1/)
- pyasn1-modules (from http://sourceforge.net/projects/pyasn1/)
- pycparser (from https://github.com/eliben/pycparser)
- tblib (from https://github.com/ionelmc/python-tblib)
========================================================================
......@@ -76,6 +79,7 @@ The following software have components provided under the terms of this license:
- click (from http://github.com/mitsuhiko/click)
- cloudpickle (from https://github.com/cloudpipe/cloudpickle)
- colorama (from https://github.com/tartley/colorama)
- cryptography (from https://github.com/pyca/cryptography)
- dask (from http://github.com/dask/dask/)
- decorator (from https://github.com/micheles/decorator)
- distributed (from https://distributed.readthedocs.io/en/latest/)
......@@ -87,20 +91,26 @@ The following software have components provided under the terms of this license:
- httpx (from https://github.com/encode/httpx)
- idna (from https://github.com/kjd/idna)
- isodate (from http://cheeseshop.python.org/pypi/isodate)
- locket (from http://github.com/mwilliamson/locket.py)
- mock (from https://github.com/testing-cabal/mock)
- numpy (from http://www.numpy.org)
- oauthlib (from https://github.com/idan/oauthlib)
- packaging (from https://github.com/pypa/packaging)
- pandas (from http://pandas.pydata.org)
- partd (from http://github.com/dask/partd/)
- ply (from http://www.dabeaz.com/ply/)
- protobuf (from https://developers.google.com/protocol-buffers/)
- psutil (from https://github.com/giampaolo/psutil)
- pyarrow (from https://arrow.apache.org/)
- pyasn1 (from http://sourceforge.net/projects/pyasn1/)
- pyasn1-modules (from http://sourceforge.net/projects/pyasn1/)
- pycparser (from https://github.com/eliben/pycparser)
- pyparsing (from http://pyparsing.wikispaces.com/)
- pyrsistent (from http://github.com/tobgu/pyrsistent/)
- python-dateutil (from https://dateutil.readthedocs.org)
- python-rapidjson (from https://github.com/python-rapidjson/python-rapidjson)
- requests-oauthlib (from https://github.com/requests/requests-oauthlib)
- starlette (from https://github.com/encode/starlette)
- tblib (from https://github.com/ionelmc/python-tblib)
- toolz (from http://github.com/pytoolz/toolz/)
- uvicorn (from https://github.com/tomchristie/uvicorn)
- zict (from http://github.com/dask/zict/)
......@@ -111,26 +121,19 @@ CC-BY-4.0
The following software have components provided under the terms of this license:
- adlfs (from https://github.com/hayesgb/adlfs/)
- dask (from http://github.com/dask/dask/)
- distributed (from https://distributed.readthedocs.io/en/latest/)
- fsspec (from http://github.com/intake/filesystem_spec)
- gcsfs (from https://github.com/dask/gcsfs)
- pandas (from http://pandas.pydata.org)
- partd (from http://github.com/dask/partd/)
- toolz (from http://github.com/pytoolz/toolz/)
========================================================================
CC0-1.0
========================================================================
The following software have components provided under the terms of this license:
- dask (from http://github.com/dask/dask/)
- distributed (from https://distributed.readthedocs.io/en/latest/)
- numpy (from http://www.numpy.org)
========================================================================
DOC
CC-BY-SA-3.0
========================================================================
The following software have components provided under the terms of this license:
- dask (from http://github.com/dask/dask/)
- distributed (from https://distributed.readthedocs.io/en/latest/)
- numpy (from http://www.numpy.org)
========================================================================
......@@ -138,6 +141,7 @@ GPL-2.0-only
========================================================================
The following software have components provided under the terms of this license:
- coverage (from https://coverage.readthedocs.io)
- grpcio (from http://www.grpc.io)
========================================================================
......@@ -146,28 +150,22 @@ GPL-2.0-or-later
The following software have components provided under the terms of this license:
- grpcio (from http://www.grpc.io)
- pyparsing (from http://pyparsing.wikispaces.com/)
========================================================================
GPL-3.0-only
========================================================================
The following software have components provided under the terms of this license:
- coverage (from https://coverage.readthedocs.io)
- grpcio (from http://www.grpc.io)
========================================================================
GPL-3.0-or-later
========================================================================
The following software have components provided under the terms of this license:
- pyparsing (from http://pyparsing.wikispaces.com/)
- rfc3986 (from https://rfc3986.readthedocs.org)
========================================================================
ISC
========================================================================
The following software have components provided under the terms of this license:
- click (from http://github.com/mitsuhiko/click)
- grpcio (from http://www.grpc.io)
- requests-oauthlib (from https://github.com/requests/requests-oauthlib)
......@@ -192,6 +190,21 @@ The following software have components provided under the terms of this license:
- chardet (from https://github.com/chardet/chardet)
========================================================================
LGPL-2.1-or-later
========================================================================
The following software have components provided under the terms of this license:
- chardet (from https://github.com/chardet/chardet)
========================================================================
LGPL-3.0-only
========================================================================
The following software have components provided under the terms of this license:
- chardet (from https://github.com/chardet/chardet)
- pycparser (from https://github.com/eliben/pycparser)
========================================================================
MIT
========================================================================
......@@ -218,7 +231,6 @@ The following software have components provided under the terms of this license:
- cachetools (from https://github.com/tkem/cachetools)
- cffi (from http://cffi.readthedocs.org)
- coverage (from https://coverage.readthedocs.io)
- distributed (from https://distributed.readthedocs.io/en/latest/)
- fastapi (from https://github.com/tiangolo/fastapi)
- grpcio (from http://www.grpc.io)
- h11 (from https://github.com/python-hyper/h11)
......@@ -226,6 +238,7 @@ The following software have components provided under the terms of this license:
- jmespath (from https://github.com/jmespath/jmespath.py)
- jsonschema (from http://github.com/Julian/jsonschema)
- msal (from https://github.com/AzureAD/microsoft-authentication-library-for-python)
- msal-extensions (from https://pypi.org/project/msal-extensions/0.1.3/)
- msrest (from https://github.com/Azure/msrest-for-python)
- munch (from http://github.com/Infinidat/munch)
- numpy (from http://www.numpy.org)
......@@ -243,6 +256,7 @@ The following software have components provided under the terms of this license:
- python-rapidjson (from https://github.com/python-rapidjson/python-rapidjson)
- python-ulid (from https://github.com/mdomke/python-ulid)
- pytz (from http://pythonhosted.org/pytz)
- requests-oauthlib (from https://github.com/requests/requests-oauthlib)
- six (from http://pypi.python.org/pypi/six/)
- sniffio (from https://github.com/python-trio/sniffio)
- structlog (from http://www.structlog.org/)
......@@ -252,21 +266,21 @@ The following software have components provided under the terms of this license:
- zipp (from https://github.com/jaraco/zipp)
========================================================================
MIT-CMU
MPL-2.0
========================================================================
The following software have components provided under the terms of this license:
- pyparsing (from http://pyparsing.wikispaces.com/)
- certifi (from http://certifi.io/)
========================================================================
MPL-2.0
NCSA
========================================================================
The following software have components provided under the terms of this license:
- certifi (from http://certifi.io/)
- numpy (from http://www.numpy.org)
========================================================================
NCSA
OPL-1.0
========================================================================
The following software have components provided under the terms of this license:
......@@ -285,9 +299,18 @@ Python-2.0
The following software have components provided under the terms of this license:
- async-timeout (from https://github.com/aio-libs/async_timeout/)
- coverage (from https://coverage.readthedocs.io)
- distributed (from https://distributed.readthedocs.io/en/latest/)
- google-auth (from https://github.com/GoogleCloudPlatform/google-auth-library-python)
- google-auth-oauthlib (from https://github.com/GoogleCloudPlatform/google-auth-library-python-oauthlib)
- numpy (from http://www.numpy.org)
- pandas (from http://pandas.pydata.org)
- ply (from http://www.dabeaz.com/ply/)
- portalocker (from https://github.com/WoLpH/portalocker)
- python-dateutil (from https://dateutil.readthedocs.org)
- pytz (from http://pythonhosted.org/pytz)
- rsa (from https://stuvel.eu/rsa)
- sniffio (from https://github.com/python-trio/sniffio)
- typing-extensions (from https://github.com/python/typing)
- urllib3 (from https://urllib3.readthedocs.io/)
......@@ -312,12 +335,20 @@ The following software have components provided under the terms of this license:
- jsonpath-ng (from https://github.com/h2non/jsonpath-ng)
========================================================================
ZPL-2.1
========================================================================
The following software have components provided under the terms of this license:
- pytz (from http://pythonhosted.org/pytz)
========================================================================
Zlib
========================================================================
The following software have components provided under the terms of this license:
- grpcio (from http://www.grpc.io)
- numpy (from http://www.numpy.org)
========================================================================
public-domain
......@@ -326,7 +357,9 @@ The following software have components provided under the terms of this license:
- botocore (from https://github.com/boto/botocore)
- grpcio (from http://www.grpc.io)
- numpy (from http://www.numpy.org)
- pandas (from http://pandas.pydata.org)
- py (from http://pylib.readthedocs.org/)
- pytz (from http://pythonhosted.org/pytz)
......@@ -43,28 +43,25 @@ def generate_df(columns, index):
df.columns = columns
return df
entity_type_dict={
"well_log": {"entity": "welllogs", "version": "v3"},
"wellbore_trajectory": {"entity": "wellboretrajectories", "version": "v3"},
"log": {"entity": "logs", "version": "v2"},
}
class EntityType(str, Enum):
well_log = "welllogs"
wellbore_trajectory = "wellboretrajectories"
log = "logs"
def build_base_url(entity_type: EntityType) -> str:
if entity_type == 'logs':
version = 'v2'
else:
version = 'v3'
return '{{base_url}}/alpha/ddms/' + version + '/' + entity_type.value
def build_base_url(entity_type: str) -> str:
return '{{base_url}}/alpha/ddms/' + entity_type_dict[entity_type]["version"] + '/' + entity_type_dict[entity_type]["entity"]
def build_base_url_without_dask(entity_type: str) -> str:
return '{{base_url}}/ddms/' + entity_type_dict[entity_type]["version"] + '/' + entity_type_dict[entity_type]["entity"]
@contextmanager
def create_record(env, entity_type: EntityType):
if entity_type == EntityType.well_log:
def create_record(env, entity_type: str):
if entity_type == "well_log":
result = build_request_create_osdu_welllog(False).call(env)
elif entity_type == EntityType.wellbore_trajectory:
elif entity_type == "wellbore_trajectory":
result = build_request_create_osdu_wellboretrajectory(False).call(env)
elif entity_type == EntityType.log:
elif entity_type == "log":
result = build_request_create_log().call(env)
else:
raise RuntimeError()
......@@ -79,11 +76,11 @@ def create_record(env, entity_type: EntityType):
yield record_id
# actually
if entity_type == EntityType.well_log:
if entity_type == "well_log":
build_request_delete_osdu_welllog(record_id).call(env)
elif entity_type == EntityType.wellbore_trajectory:
elif entity_type == "wellbore_trajectory":
build_request_delete_osdu_wellboretrajectory(record_id).call(env)
elif entity_type == EntityType.log:
elif entity_type == "log":
env.set('log_record_id', record_id)
build_request_delete_log().call(env)
......@@ -107,29 +104,31 @@ def build_request(name, method, url, *, payload=None, headers=None) -> RequestRu
return RequestRunner(rq_proto)
def build_request_post_data(entity_type: EntityType, record_id: str, payload) -> RequestRunner:
def build_request_post_data(entity_type: str, record_id: str, payload) -> RequestRunner:
url = build_base_url(entity_type) + f'/{record_id}/data'
return build_request(f'{entity_type} post data', 'POST', url, payload=payload)
def build_request_post_data_without_dask(entity_type: str, record_id: str, payload) -> RequestRunner:
url = build_base_url_without_dask(entity_type) + f'/{record_id}/data'
return build_request(f'{entity_type} post data', 'POST', url, payload=payload)
def build_request_post_chunk(entity_type: EntityType, record_id: str, session_id: str, payload) -> RequestRunner:
def build_request_post_chunk(entity_type: str, record_id: str, session_id: str, payload) -> RequestRunner:
url = build_base_url(entity_type) + f'/{record_id}/sessions/{session_id}/data'
return build_request(f'{entity_type} post data', 'POST', url, payload=payload)
def build_request_get_data(entity_type: EntityType, record_id: str) -> RequestRunner:
def build_request_get_data(entity_type: str, record_id: str) -> RequestRunner:
url = build_base_url(entity_type) + f'/{record_id}/data'
return build_request(f'{entity_type} get data', 'GET', url)
def create_session(env, entity_type: EntityType, record_id: str, overwrite: bool) -> str:
def create_session(env, entity_type: str, record_id: str, overwrite: bool) -> str:
url = build_base_url(entity_type) + f'/{record_id}/sessions'
runner = build_request(f'create {entity_type} session', 'POST', url,
payload={'mode': 'overwrite' if overwrite else 'update'})
return runner.call(env, assert_status=200, headers={"Content-Type": "application/json"}).get_response_obj().id
def complete_session(env, entity_type: EntityType, record_id: str, session_id: str, commit: bool):
def complete_session(env, entity_type: str, record_id: str, session_id: str, commit: bool):
state = "commit" if commit else "abandon"
url = build_base_url(entity_type) + f'/{record_id}/sessions/{session_id}'
runner = build_request(f'{state} session', 'PATCH', url, payload={'state': state})
......@@ -164,10 +163,9 @@ WELLLOG_URL_PREFIX = 'alpha/ddms/v3/welllogs'
@pytest.mark.tag('chunking', 'smoke')
@pytest.mark.parametrize('entity_type', [EntityType.well_log, EntityType.wellbore_trajectory, EntityType.log])
@pytest.mark.parametrize('entity_type', ["well_log", "wellbore_trajectory", "log"])
@pytest.mark.parametrize('serializer', [ParquetSerializer(), JsonSerializer()])
def test_send_one_chunk_without_session(with_wdms_env, entity_type, serializer):
with create_record(with_wdms_env, entity_type) as record_id:
data = generate_df(['MD', 'X'], range(8))
data_to_send = serializer.dump(data)
......@@ -181,7 +179,7 @@ def test_send_one_chunk_without_session(with_wdms_env, entity_type, serializer):
@pytest.mark.tag('chunking', 'smoke')
@pytest.mark.parametrize('entity_type', [EntityType.well_log, EntityType.wellbore_trajectory, EntityType.log])
@pytest.mark.parametrize('entity_type', ["well_log", "wellbore_trajectory", "log"])
@pytest.mark.parametrize('serializer', [ParquetSerializer(), JsonSerializer()])
def test_send_one_chunk_with_session_commit(with_wdms_env, entity_type, serializer):
......@@ -215,7 +213,7 @@ def test_send_one_chunk_with_session_commit(with_wdms_env, entity_type, serializ
@pytest.mark.parametrize("shuffle", [False]) # [False, True]
def test_send_multiple_chunks_with_session_commit(with_wdms_env, shuffle):
# well log on parquet
entity_type = EntityType.well_log
entity_type = "well_log"
serializer = ParquetSerializer()
with create_record(with_wdms_env, entity_type) as record_id:
......@@ -259,7 +257,7 @@ def test_send_multiple_chunks_with_session_commit(with_wdms_env, shuffle):
@pytest.mark.tag('chunking', 'smoke')
def test_get_data_with_offset_filter(with_wdms_env):
# well log on parquet
entity_type = EntityType.well_log
entity_type = "well_log"
serializer = ParquetSerializer()
with create_record(with_wdms_env, entity_type) as record_id:
......@@ -298,7 +296,7 @@ def test_get_data_with_offset_filter(with_wdms_env):
@pytest.mark.tag('chunking', 'smoke')
def test_get_data_with_column_filter(with_wdms_env):
# well log on parquet
entity_type = EntityType.well_log
entity_type = "well_log"
serializer = ParquetSerializer()
with create_record(with_wdms_env, entity_type) as record_id:
......@@ -331,7 +329,7 @@ def test_get_data_with_column_filter(with_wdms_env):
@pytest.mark.tag('chunking', 'smoke')
def test_get_data_with_limit_filter(with_wdms_env):
# well log on parquet
entity_type = EntityType.well_log
entity_type = "well_log"
serializer = ParquetSerializer()
with create_record(with_wdms_env, entity_type) as record_id:
......@@ -366,7 +364,7 @@ def test_get_data_with_limit_filter(with_wdms_env):
@pytest.mark.tag('chunking', 'smoke')
@pytest.mark.parametrize('entity_type', [EntityType.well_log, EntityType.wellbore_trajectory, EntityType.log])
@pytest.mark.parametrize('entity_type', ["well_log", "wellbore_trajectory", "log"])
def test_get_data_with_limit_and_offset_filter(with_wdms_env, entity_type):
serializer = ParquetSerializer()
......@@ -394,7 +392,7 @@ def test_get_data_with_limit_and_offset_filter(with_wdms_env, entity_type):
@pytest.mark.tag('chunking', 'smoke')
@pytest.mark.parametrize('entity_type', [EntityType.well_log, EntityType.wellbore_trajectory])
@pytest.mark.parametrize('entity_type', ["well_log", "wellbore_trajectory", "log"])
@pytest.mark.parametrize('serializer', [ParquetSerializer(), JsonSerializer()])
def test_multiple_overwrite_sessions_in_parallel_then_commit(with_wdms_env, entity_type, serializer):
......@@ -431,7 +429,7 @@ def test_multiple_overwrite_sessions_in_parallel_then_commit(with_wdms_env, enti
@pytest.mark.tag('chunking', 'smoke')
@pytest.mark.parametrize('entity_type', [EntityType.well_log, EntityType.wellbore_trajectory])
@pytest.mark.parametrize('entity_type', ["well_log", "wellbore_trajectory", "log"])
@pytest.mark.parametrize('serializer', [ParquetSerializer(), JsonSerializer()])
def test_multiple_update_sessions_in_parallel_then_commit(with_wdms_env, entity_type, serializer):
......@@ -471,3 +469,20 @@ def test_multiple_update_sessions_in_parallel_then_commit(with_wdms_env, entity_
pd.testing.assert_frame_equal(
expected, serializer.read(result.response.content), check_dtype=False)
# check type set to false since in Json dType is lost so int32 can become int64
@pytest.mark.tag('chunking', 'smoke')
@pytest.mark.parametrize('entity_type', ["log"])
@pytest.mark.parametrize('serializer', [JsonSerializer()])
def test_get_data_from_record_data_without_dask(with_wdms_env, entity_type, serializer):
with create_record(with_wdms_env, entity_type) as record_id:
data = generate_df(['MD', 'X'], range(8))
data_to_send = serializer.dump(data)
headers = {'Content-Type': serializer.mime_type, 'Accept': serializer.mime_type}
build_request_post_data_without_dask(entity_type, record_id, data_to_send).call(with_wdms_env, headers=headers).assert_ok()
result = build_request_get_data(entity_type, record_id).call(with_wdms_env, headers=headers, assert_status=200)
pd.testing.assert_frame_equal(data, serializer.read(result.response.content), check_dtype=False)
# check type set to false since in Json dType is lost so int32 can become int64
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment