Python SDK issueshttps://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues2024-02-08T13:31:14Zhttps://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/20ADR: Community implementation2024-02-08T13:31:14ZYan Sushchynski (EPAM)ADR: Community implementation
# ADR
## Title
Dynamically Loading the Blob Storage Client and Credentials Implementations
## Context
The community implementation requires maintaining only the core logic within the shared repository. Specific cloud-level implementa...
# ADR
## Title
Dynamically Loading the Blob Storage Client and Credentials Implementations
## Context
The community implementation requires maintaining only the core logic within the shared repository. Specific cloud-level implementations (e.g., AWS, Azure, Google) must be developed separately while adhering to the core logic. This ensures flexibility and interoperability across various cloud platforms.
## Decision
We have opted for a dynamic module import strategy leveraging Python's importlib package. The MR for Python SDK is [here](https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/merge_requests/143). This allows each cloud provider to write their own implementations and install them alongside the basic Python SDK.
The implementations would align with abstract classes defined in the [python-sdk-abstract-classes](https://community.opengroup.org/osdu/platform/deployment-and-operations/base-containers-gcp/python-sdk-abstract-classes) repository.
For instance, Google Cloud's `requirements.txt` may look as follows:
```txt
--extra-index-url=https://community.opengroup.org/api/v4/projects/1460/packages/pypi/simple
osdu-api-abstract
--extra-index-url=https://community.opengroup.org/api/v4/projects/148/packages/pypi/simple
osdu-api==0.25.0.dev804
--extra-index-url=https://community.opengroup.org/api/v4/projects/1461/packages/pypi/simple
osdu-api-gc==0.25.0.dev9
```
The paths to the specific implementations can be defined in two environment variables: `PYTHON_SDK_CREDENTIALS_MODULE` and `PYTHON_SDK_BLOB_STORAGE_MODULE`. These point to the `get_credentials` and `get_blob_storage` functions respectively, which return concrete implementations of the `BaseCredentials` and `BlobStorageClient` abstract classes. See the following example: [get_blob_storage](https://community.opengroup.org/osdu/platform/deployment-and-operations/base-containers-gcp/python-sdk-gc/-/blob/main/osdu_api_gc/blob_storage/__init__.py?ref_type=heads#L18) and [get_credentials](https://community.opengroup.org/osdu/platform/deployment-and-operations/base-containers-gcp/python-sdk-gc/-/blob/main/osdu_api_gc/credentials/__init__.py?ref_type=heads#L18)
In our Google Cloud environment, the environment variables would look like:
```shell
# .env file
PYTHON_SDK_CREDENTIALS_MODULE="osdu_api_gc.credentials"
PYTHON_SDK_BLOB_STORAGE_MODULE="osdu_api_gc.blob_storage"
```
## Class Diagrams
![blob_storage_1_](/uploads/38a770927eb177f598f49bec048e3283/blob_storage_1_.png)
![token_refresher_1_](/uploads/4498faddd98da14fd1e0bf9302318ca1/token_refresher_1_.png)
## Consequences
The dynamic module loading approach offers enhanced flexibility and interoperability across various cloud platforms. It isolates dependencies specific to each cloud provider, ensuring their implementations are developed and installed separately, mitigating any dependency conflicts. The approach calls for strict adherence to defined abstract classes in implementations, facilitating a consistent interface across different modules. However, comprehensive testing and validation are required to ensure custom implementations align with the defined abstract classes, ensuring smooth interaction with the core logic.
Also, unlike Java services, Python services don't use a shared code base.
Services to wrk on:
1. [RAFS](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/rock-and-fluid-sample/rafs-ddms-services/-/tree/main)
2. [Wellbore](https://community.opengroup.org/osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services)
3. [Policy](https://community.opengroup.org/osdu/platform/security-and-compliance/policy)
4. [Python SDK](https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk)https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/18AWS : allow 'osdu_api.ini' file path configuration2024-01-17T12:08:05ZValentin GauthierAWS : allow 'osdu_api.ini' file path configurationFor now, it seems that it is not possible to modify the default file path of the *osdu_api.ini* file when using AWS.
As it is required for the AWS configuration, I suggest a modification of this line : https://community.opengroup.org/osd...For now, it seems that it is not possible to modify the default file path of the *osdu_api.ini* file when using AWS.
As it is required for the AWS configuration, I suggest a modification of this line : https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/blob/master/osdu_api/providers/aws/service_principal_util.py#L63
The modification could be :
```python
config_file_name = os.environ.get("OSDU_API_CONFIG_INI") or "osdu_api.ini"
```
Thus, it will use the "OSDU_API_CONFIG_INI" environment variable as in *DefaultConfigManager* class (see. https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/blob/master/osdu_api/configuration/config_manager.py#L73)
This modification should not break any actual configuration, except if a configuration use 2 differents *osdu_api.ini* files, one used for *DefaultConfiguration* class and one for the AWS configuration (which seems not to be a good practice I think ?).https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/issues/15ADR: Static code analysis for Python libraries2023-10-30T14:43:02ZYan Sushchynski (EPAM)ADR: Static code analysis for Python libraries## Context
Python is a dynamically typed language, so developers don't need to worry about types. This works well if a project is small and a few developers work on it.
However, once the project gets bigger, and involves a lot of engi...## Context
Python is a dynamically typed language, so developers don't need to worry about types. This works well if a project is small and a few developers work on it.
However, once the project gets bigger, and involves a lot of engineers, understanding how code works becomes the cornerstone of the further development. Python has type annotations designed to help developers to understand code. Now, these type annotations in our Python libraries are kind of hints for developers and their IDEs, but following them is not mandatory, and they can be simply ignored.
As a result, we face issues when some methods are called with arguments with wrong types. And these bugs unexpectedly show in runtime under certain conditions.
It is not so rare to get the following runtime error: `AttributeError: 'dict' object has no attribute 'to_JSON'`
However, these bugs could be easily catch with any static analyzer.
## Decision
Add a static analysis step for type checking to CI/CD pipelines right before unit-tests. The step will be run on the container with preinstalled tools for Python static analysis (e.g., [pytype](https://github.com/google/pytype) or [mypy](https://github.com/python/mypy)).
At first, we are going to add static analysis to the following libraries:
1. https://community.opengroup.org/osdu/platform/system/sdks/common-python-sdk/-/tree/master/osdu_api - excluding CSP-specific code from `osdu_api/providers`;
1. https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-ingestion-lib;
1. https://community.opengroup.org/osdu/platform/data-flow/ingestion/osdu-airflow-lib.
Further, we can cover other Python libraries with static analysis.
## Consequences
Pros:
1. It will be much easier to catch subtle bugs without writing extra unit-tests;
2. Developers will be forced to follow type annotations that will make code more readable and understandable.
Cons:
1. The existing code should be refactored to pass static analysis validations;
2. Some developers might find obeying these new rules too strict.M16 - Release 0.19