[ADR] Community implementation for RAFS DDMS

Decision Title

Community Implementation for RAFS

Status

• Proposed
• Trialing
• Under review
• Approved
• Retired

Context & Scope

The service is limited with main cloud providers (GC, AWS, Azure, IBM). And, if we want to add a new implementation (e.g., BareMetal), we will have to create a new folder under the providers directory.

Decision

As a solution, we can introduce a new concept called Community Implementation -- a plugin approach when the plugins should follow some rules; so that developers can write their own implementations without changing the main repository, and connect their implementations at the runtime.

The only requirement of the implementations (plugins) is following the abstract classes from the repository, and having a class of the interface at the following path provider_obm.blob_manager.BlobLoader (example).

So, the class hierarchy will look like this:

classDiagram
    class get_blob_loader {
        <<function>>
    }

    note for get_blob_loader "This function returns a realization of the <i>IBlobLoader</i><br/>The choice is specified with env variables"

    class IBlobLoader {
        <<abstract>>
        +async upload_blob(upload_url: str, blob: bytes)
        +async download_blob(download_url: str): bytes
    }

    class AzureBlobLoader {
        +async upload_blob(upload_url: str, blob: bytes)
        +async download_blob(download_url: str): bytes
    }

    class AWSBlobLoader {
        +async upload_blob(upload_url: str, blob: bytes)
        +async download_blob(download_url: str): bytes
    }

    class GoogleBlobLoader {
        +async upload_blob(upload_url: str, blob: bytes)
        +async download_blob(download_url: str): bytes
    }

    class CimplBlobLoader {
        +async upload_blob(upload_url: str, blob: bytes)
        +async download_blob(download_url: str): bytes
        +_import_blob_loader() Type~BlobLoaderAsync~
    }

    note for CimplBlobLoader "<i>_import_blob_loader</i> method expects a class at <br/> the <b>provider_obm.blob_manager.BlobLoaderAsync</b> module path.<br/>This means that the library with this path should be already installed"


    class BlobLoaderAsync {
        <<abstract>>
        +async upload_storage_location_blob(upload_url: str, blob: bytes)
        +async download_storage_location_blob(download_url: str): bytes
    }


    class SpecificBlobManagerAsync {
        <<specific implementation>>
        +async upload_storage_location_blob(upload_url: str, blob: bytes)
        +async download_storage_location_blob(download_url: str): bytes
    }

    note for SpecificBlobManagerAsync "This is a user created plugin"


    style CimplBlobLoader fill:lightgreen,stroke:#f66,stroke-width:2px
    style BlobLoaderAsync fill:lightgreen,stroke:#f66,stroke-width:2px
    style SpecificBlobManagerAsync fill:lightgreen,stroke:#f66,stroke-width:2px

    %% Definition of dynamic provider selection function
    get_blob_loader ..> IBlobLoader : returns realization of this interface>>
    
    %% Inheritance Structure
    IBlobLoader <|.. AzureBlobLoader
    IBlobLoader <|.. AWSBlobLoader
    IBlobLoader <|.. GoogleBlobLoader
    IBlobLoader <|.. CimplBlobLoader
    BlobLoaderAsync <|.. SpecificBlobManagerAsync

    CimplBlobLoader ..> BlobLoaderAsync: uses

For demonstration purposes, we did the following:

1. Added a new provider with the name "cimpl" (community implementation). MR.
2. Created abstract classes for BlobLoader
3. Now, users can write their own BlobLoader implementations, create a package from it; The class should be at the following path provider_obm.blob_manager.BlobLoaderAsync -- it is a requirement
4. Created the GC implementation where the BlobLoaderAsync is at the provider_obm.blob_manager.BlobLoaderAsync

Rationale

The decision to implement the "Community Implementation" concept for RAFS using a plugin approach is motivated by several key factors:

• Extensibility and Modularity: By adopting a plugin system, the RAFS can become more extensible and modular. This system allows developers to create and integrate their specific implementations as separate entities, which do not interfere with the core functionality.

• Low Impact on Main Repository: A plugin approach minimizes the impact on the core codebase. Developers can contribute new features or integration without altering the existing code structure.

• Simplicity of Integration: The structured approach where plugins must adhere to specific abstract classes simplifies the integration process. The interface requirements ensure that new plugins can seamlessly integrate with the existing system, reducing integration complexities and potential errors.

Consequences

Pros:

• Reducing the risks of dependency conflicts between different implementations
• Flexibility: if anyone wants to add their own implementation, they can do this in their own repositories without affecting the main one
• This approach can be extended to other Python service if it is viable for the RAFS.

Cons:

• Possible dependency conflicts between core code and specific implementation. Python is known for its dependency resolution issues.

When to revisit

Tradeoff Analysis - Input to decision

Alternatives and implications

Decision criteria and tradeoffs

Decision timeline