Python SDK issueshttps://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues2023-05-26T10:49:50Zhttps://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/17Moving BaseTokenRefresher class from osdu-ingestion-lib project to python-sdk2023-05-26T10:49:50ZValentin GauthierMoving BaseTokenRefresher class from osdu-ingestion-lib project to python-sdkIn some of my project, I would like to be able to access to services (in python) without airflow.
I need a TokenRefresher implementation, like the one in the *osdu-ingestion-lib* project : https://community.opengroup.org/osdu/platform/d...In some of my project, I would like to be able to access to services (in python) without airflow.
I need a TokenRefresher implementation, like the one in the *osdu-ingestion-lib* project : https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/blob/master/osdu_ingestion/libs/refresh_token.py#L30
As this class may be needed without a need to add the ingestion-lib project as a dependency, and it is not specific to airflow, I suggest to move this class in the python-sdk. This will allow any application to depend only on the python-sdk and be able to simply create a token refresher.M18 - Release 0.21Valentin GauthierValentin Gauthierhttps://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/16jsonschema in requirements.txt2023-04-10T17:45:41Zfabian serinjsonschema in requirements.txtHello @Yan_Sushchynski you added `jsonschema==3.2.0`in requirements.txt one month ago.
First of all I see no use of this lib in the whole project, so it is really useful ?
Then I am usin jsonschema for other projects and because of trans...Hello @Yan_Sushchynski you added `jsonschema==3.2.0`in requirements.txt one month ago.
First of all I see no use of this lib in the whole project, so it is really useful ?
Then I am usin jsonschema for other projects and because of transitive dependencies I am blocked with your 3.2.0 version (old version from 2019).
I need a recent version. Do you mind removing it from your requirements.txt as it seems it is not used ?
Thank youhttps://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/14Support no-auth requests from Base Client2022-10-25T13:17:24ZMorris EstepaSupport no-auth requests from Base ClientBase Client currently only makes requests by internally setting the authorization token in the request header. This is causing compatibility issues with signed URLs because they already include authentication information within the URL.
...Base Client currently only makes requests by internally setting the authorization token in the request header. This is causing compatibility issues with signed URLs because they already include authentication information within the URL.
Need add a feature to allow base client to send a request that doesn't automatically include authorization token in the header.M14 - Release 0.17Morris EstepaMorris Estepahttps://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/5ADR: Refactor BaseClient2022-10-18T11:09:53ZSiarhei Khaletski (EPAM)ADR: Refactor BaseClient## Description
SDK as development kit is used to provided utility functions in fast and convenient way.
Each class in SDK has to be self-descriptive: each parameter is clear, everything be able to extend.
The major part of `osdu_api` p...## Description
SDK as development kit is used to provided utility functions in fast and convenient way.
Each class in SDK has to be self-descriptive: each parameter is clear, everything be able to extend.
The major part of `osdu_api` package are clients to OSDU services' API.
Event at the first glance `BasicClient` has a `god-mode` presentation.
What I mean, the class constructor encapsulates a `ConfigManager` logic that was contributed before by GCP/EPAM team.
It means, for instance, that `StorageServiceClient` knows (o should know) about `FileServiceClient` and so forth.
![image](/uploads/fe05a58d30855fc0d8889c0ef0044ff7/image.png)
Furthermore it brings [with hard coded `osdu_api.ini`](https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/blob/master/osdu_api/clients/base_client.py#L54) file in the class constructor. And in accordance with the last updates it absolutely inconvenient to operate this wile within deployment etc.
## Proposal
1. Use `BasicClient` class as an abstract class or as a real base class.
1. Remove configuration initialization from `BasicClass.__init__` method.
1. For each client class specify the client specific attributes for `__init__` method.
## Package version
The update is not backward compatible
## Related MRs:
https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/merge_requests/67
https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/24
https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/21M11 - Release 0.14Ben LasscockBen Lasscockhttps://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/12Refactor/create Python SDK's ConfigManagers to work with Airflow config2022-10-18T11:09:14ZKateryna Kurach (EPAM)Refactor/create Python SDK's ConfigManagers to work with Airflow configThis is a follow-up for: https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/merge_requests/58
Previously GCP team made the effort to decouple the term `Airflow` with Python SDK. We suggest to you create a new `...This is a follow-up for: https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/merge_requests/58
Previously GCP team made the effort to decouple the term `Airflow` with Python SDK. We suggest to you create a new `ConfigManager` class at `osdu-airflow-lib` repository and specify your environment configuration.M14 - Release 0.17harshit aggarwalharshit aggarwalhttps://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/13Missing storageInstructions endpoint in dataset service for Python SDK2022-09-29T13:37:07ZChad LeongMissing storageInstructions endpoint in dataset service for Python SDKAs per the ADR https://community.opengroup.org/osdu/platform/system/dataset/-/issues/29, the dataset API has been updated to follow the REST Convention. The existing APIs are still available, but in sunset period.
In the Python SDK, th...As per the ADR https://community.opengroup.org/osdu/platform/system/dataset/-/issues/29, the dataset API has been updated to follow the REST Convention. The existing APIs are still available, but in sunset period.
In the Python SDK, the dataset client is missing the new `storageInstructions` endpoint https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/blob/master/osdu_api/clients/dataset/dataset_dms_client.py
| Functionality | Existing API (Sunset) | New REST Endpoint | Implemented in Python SDK |
|---------------------|-------------------------|-------------------------|-------------------------|
| Dataset Get-Storage Instructions | GET `/getStorageInstructions?kindSubType={dmsType}` | POST `/storageInstructions?kindSubType={dmsType}` | :x: |
| Dataset Get-Retrieval Instructions | GET `/getRetrievalInstructions?id={dataset-id}` | GET `/retrievalInstructions?id={dataset-id}`| :heavy_check_mark:https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/blob/master/osdu_api/clients/dataset/dataset_dms_client.py#L46 |
| Dataset Batch Get-Retrieval Instructions |POST `/getRetrievalInstructions` | POST `/retrievalInstructions` | :heavy_check_mark:https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/blob/master/osdu_api/clients/dataset/dataset_dms_client.py#L49 |
This is causing issue in the implementation on improvement on manifest ingestion DAG on IBM. https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/17#note_139224M14 - Release 0.17Vadzim Kulybaharshit aggarwalShrikant GargYan Sushchynski (EPAM)Valentin GauthierDevdatta SantraOkoun-Ola Fabien HouetoYaraslau SushchykVadzim Kulybahttps://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/8Implement azure_blob_storage_client module2022-06-13T07:48:11ZVadzim KulybaImplement azure_blob_storage_client modulePrepare interfaces for works with Azure Blob Storage:
- does_file_exist
- download_to_file
- download_file_as_bytes
- upload_filePrepare interfaces for works with Azure Blob Storage:
- does_file_exist
- download_to_file
- download_file_as_bytes
- upload_fileVadzim KulybaVadzim Kulybahttps://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/11Mandatory Usage of Config Ini File for setting environment variables in Pytho...2022-01-23T11:01:10Zharshit aggarwalMandatory Usage of Config Ini File for setting environment variables in Python SDKCurrently all the SDK Clients added for different OSDU services in common python library are reading environment variables from a ini config file via a config parser class [ [Code Reference](https://community.opengroup.org/osdu/platform/...Currently all the SDK Clients added for different OSDU services in common python library are reading environment variables from a ini config file via a config parser class [ [Code Reference](https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/blob/master/osdu_api/configuration/config_manager.py#L46) ]
This config parser class is present for a long time but with this [MR](https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/2) on osdu-ingestion-lib has mandated its usage
We should keep supporting existing way of directly reading environment variables as well because completely relying on ini files will affect the Airflow Deployment for Cloud providers
We should add a fallback in [BaseClient ](https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/blob/master/osdu_api/clients/base_client.py#L52) Class to read environment variables if not present in Configuration
[MR Link](https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/merge_requests/58/diffs) for enabling Airflow Environment variables for Config Objecthttps://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/10Update dataset_dms_client module2022-01-10T07:56:12ZVadzim KulybaUpdate dataset_dms_client moduleNeed to add new retrieval Instructions request method for dataset client, because we have several approaches on get retrieval instructions for datasets:
- https://community.opengroup.org/osdu/platform/system/dataset/-/blob/release/0.12/d...Need to add new retrieval Instructions request method for dataset client, because we have several approaches on get retrieval instructions for datasets:
- https://community.opengroup.org/osdu/platform/system/dataset/-/blob/release/0.12/dataset-core/src/main/java/org/opengroup/osdu/dataset/api/DatasetDmsApi.java#L92
- https://community.opengroup.org/osdu/platform/system/dataset/-/blob/release/0.12/dataset-core/src/main/java/org/opengroup/osdu/dataset/api/DatasetDmsApi.java#L113
- https://community.opengroup.org/osdu/platform/system/dataset/-/blob/v0.12.0/dataset-core/src/main/java/org/opengroup/osdu/dataset/service/DatasetDmsService.java#L31Vadzim KulybaVadzim Kulybahttps://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/9Refactor azure_credentials module2022-01-10T07:56:09ZVadzim KulybaRefactor azure_credentials moduleUpdate _generate_token method:
- update MSI Token generation (add env variables for configurations)
- update populate credentials mechanism for AD (small refactor)Update _generate_token method:
- update MSI Token generation (add env variables for configurations)
- update populate credentials mechanism for AD (small refactor)Vadzim KulybaVadzim Kulybahttps://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/4Python SDK - support of Entitlements [GONRG-2812]2021-11-12T16:04:51ZKateryna Kurach (EPAM)Python SDK - support of Entitlements [GONRG-2812]https://jiraeu.epam.com/browse/GONRG-2812https://jiraeu.epam.com/browse/GONRG-2812Siarhei Khaletski (EPAM)Yan Sushchynski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/7ADR: Move Ingestion Logic from Python SDK to the separate package.2021-11-01T15:47:00ZYan Sushchynski (EPAM)ADR: Move Ingestion Logic from Python SDK to the separate package.The main goal of `Python SDK` was to interface with OSDU microservices' API.
It'd been so, until we moved Manifest Based Ingestion common code from `Ingestion DAG` to it. And now `Python SDK` is responsible not only for interacting wit...The main goal of `Python SDK` was to interface with OSDU microservices' API.
It'd been so, until we moved Manifest Based Ingestion common code from `Ingestion DAG` to it. And now `Python SDK` is responsible not only for interacting with OSDU microservices, but for implementing Manifest Based Ingestion.
We believe that such mixing of two different logics is not proper. Imagine, we want to use Python SDK only for communicating with Entitlements service (or other Service), so we don't need Ingestion logic there at all.
So, it was decided to move Manifest Ingestion logic into separate package that is only responsible for Ingestion process. This new package is cloud-agnostic and it is used mainly in Airflow operators.
The new package's repository is https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib.
There are a few MRs that implement using this new package.
OSDU Ingestion Lib:
https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib/-/merge_requests/1
Python SDK:
https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/merge_requests/47
OSDU Airflow Lib:
https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib/-/merge_requests/6M9 - Release 0.12Debasis ChatterjeeSiarhei Khaletski (EPAM)Kishore BattulaShrikant GargDebasis Chatterjeehttps://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/6Check Manifest Integrity fails on "<data-partition-id>:reference-data--UnitQu...2021-09-30T20:18:46ZYan Sushchynski (EPAM)Check Manifest Integrity fails on "<data-partition-id>:reference-data--UnitQuantity:1" referenceWhen we ingest https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/reference-values/-/blob/master/Manifests/reference-data/OPEN/UnitOfMeasure.1.0.0.json
{
"id": "<data-partition-id>:reference-data--UnitOfMeasure:ppm",...When we ingest https://gitlab.opengroup.org/osdu/subcommittees/data-def/work-products/reference-values/-/blob/master/Manifests/reference-data/OPEN/UnitOfMeasure.1.0.0.json
{
"id": "<data-partition-id>:reference-data--UnitOfMeasure:ppm",
"kind": "<data-partition-id>:wks:reference-data--UnitOfMeasure:1.0.0",
"reason": "Missing parents:
{SRN: osduqa:reference-data--UnitQuantity:1}
"
}M9 - Release 0.12https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/3Python dotenv usage2021-08-24T18:20:07ZSiarhei Khaletski (EPAM)Python dotenv usage## Context
The SDK is configured through the `open_api.ini` [file](https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/blob/master/osdu_api/base_client.py#L51).
The file is parsing within `base_client.py` during...## Context
The SDK is configured through the `open_api.ini` [file](https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/blob/master/osdu_api/base_client.py#L51).
The file is parsing within `base_client.py` during instantiation of `BaseClient` descendants' instances.
There are use cases when it will be nice to have a possibility to configure the SDK by environment variables, for instance, when we use the SDK in CI/CD pipelines or provisioning scripts without usage of `open_api.ini` file.
## Decision
- Use [python-dotenv](https://pypi.org/project/python-dotenv/) for config management
## Scope
- Replace `open_api.ini` file by `.env` file (https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/blob/master/osdu_api/base_client.py#L30);
- Change `__init_` method of `BaseClient` class (https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/blob/master/osdu_api/base_client.py#L49);
- Replace `open_api.ini` file for existing parsers (e.g. [WITSML parser](https://community.opengroup.org/osdu/platform/data-flow/ingestion/energistics-osdu-integration/-/blob/master/build/providers/aws/osdu_api.ini))
Optional:
- Add dynamic access to the parameters, i.e. remove explicit initialization of a instance properties and add dynamic access to any of them;M8 - Release 0.11Siarhei Khaletski (EPAM)Siarhei Khaletski (EPAM)https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/2Using same name for module and python file in same folder2021-02-23T14:42:02ZKishore BattulaUsing same name for module and python file in same folderIn `osdu_api/models` there is a python file named `legal.py` and a folder named `legal`. This is causing the below import statement to fail
```
from osdu_api.model.legal import Legal
```
with the below exception
```
ImportError: cannot i...In `osdu_api/models` there is a python file named `legal.py` and a folder named `legal`. This is causing the below import statement to fail
```
from osdu_api.model.legal import Legal
```
with the below exception
```
ImportError: cannot import name 'Legal'
```https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/1Common validation logic in Python SDK2020-07-15T21:23:42ZElizaveta Zeldina (EPAM)Common validation logic in Python SDKDetails can be found here:
https://community.opengroup.org/osdu/platform/data-flow/home/-/issues/15Details can be found here:
https://community.opengroup.org/osdu/platform/data-flow/home/-/issues/15Dmitriy RudkoDmitriy Rudko