Community Implementation for EDS Compatibility

Title:
Ensure Compatibility and Standardization of Graduated EDS Components with the Community Implementation

Description:
As a contributor to the EDS community, I want to ensure that all EDS components, including graduated components and those in the sandbox stage (such as the Naturalization DAG), are fully compatible with the community implementation, so that the EDS stack operates seamlessly across diverse environments, remains free from provider-specific dependencies, and is designed for future community adoption.

Acceptance Criteria:

  1. Airflow DAGs Verification and Implementation:
    • EDS Fetch & Ingest DAG (Graduated): Verify compatibility with the community implementation.
    • EDS Scheduler DAG (Graduated): Verify compatibility with the community implementation.
    • Naturalization DAG (Sandbox): Provide a bare-metal implementation for uploading and downloading bulk data and validate its functionality.
  2. EDS Python Code Modules:
    • Ensure all graduated EDS Python code modules are standardized and compatible with the community implementation.
  3. EDS DMS Testing and Refactoring:
    • Test EDS DMS functionality in a bare-metal environment without requiring code refactoring.
    • If above is not successful, make EDS DMS compatible with the CI pipeline.
    • Refactor EDS DMS code to generalize it and remove provider-specific customizations.
    • Implementation avoids provider-specific customizations and follows a standardized approach aligned with community guidelines.
    • Ensure a modular approach (e.g., core-plus module) is considered for scalability and standardization.

Technical Notes:

  1. Tasks will commence after processes for a dedicated CI environment are established to avoid redundant work.
  2. The design should prioritize compatibility and scalability for future enhancements in the EDS community.
  3. Bare-metal implementations of both EDS DMS and Naturalization DAG must be thoroughly tested to ensure functionality without provider-specific dependencies.
  4. Code refactoring efforts for EDS DMS should aim to align with the community's standardized practices.

Related Issues:

Edited by Teo Sheng Pu