Skip to content

EDS Naturalization - Pydantic Datamodel implementation

Description:

As a developer, I would like to implement Pydantic data model for EDS Naturalization so that the code would be cleaner & maintainable and easier to implement other functionalities in the future

Acceptance Criteria

 Task
  1. Implementation of Pydantic data models
    • Create any additional data models (ie. abstract datamodel classes
    • Change the code for EDS Ingest using PyDantic data class model implementation
  • Perform EDS Naturalization code change (Input model)
  • Perform EDS Naturalization code change (Abstract class)
  • Perform EDS Naturalization code change (API input output models)
  • Perform EDS Naturalization code change (Fetch CRE, CSDJ)
  • Perform EDS Naturalization code change (Fetch retrieval instruction)
  • Perform EDS Naturalization code change (Fetch dataset metadata)
  • Perform EDS Naturalization code change (Download and upload to CSP)
  • Perform EDS Naturalization code change (Create upload dataset record)
  • Perform EDS Naturalization code change (Output model)
  • Merge and update the M24 changes to Pydantic
  1. Fix/re-write unit test based on the Pydantic datamodel to ensure code testing coverage and behavior
  • Update the unit test for M24 changes accordingly
  • Setup local debugging with Test Driven Development concepts + troubleshooting on Airflow Mocking
  • Cloud Service Provider (Nisha, TSP)
    • AWS (TSP)
      • File Collection
      • File Generic
    • GC (TSP)
      • File Collection
      • File Generic
    • Azure (TSP)
      • File Collection
      • File Generic
    • Default (Shreya)
      • File Collection
      • File Generic
  • Core Naturalization (Shreya, TSP)
  • Transfer Instruction (TSP)
    • Process (File Collection)
    • Signed URL process (File Generic)
  • RegisterDataset.Create (TSP)
    • AbstractVectorHeaderMapping
    • AbstractFileCollection
    • AbstractFile
  • Dataset Service (TSP)
  1. Perform end-to-end testing / developer testing to ensure that the codes are working and behave as it is before
  • File Collection SEGY without vector header mapping (Nisha)
    • AWS -> GC
    • AWS -> Azure
    • GC -> Azure
    • Azure -> GC
    • AWS (No Dev environment - skip)
  • File Collection SEGY with vector header mapping (Nisha, Shankar)
    • AWS -> GC
    • AWS -> Azure
    • GC -> Azure
    • Azure -> GC
    • AWS (No Dev environment - skip)
  • File Collection Generic (Nisha, Shankar)
    • AWS -> GC
    • AWS -> Azure
    • GC -> Azure
    • Azure -> GC
    • AWS (No Dev environment - skip)
  • File Generic (Nisha)
    • AWS -> GC
  • Testing with direct download (Katalyst) [Nisha]
    • File Generic
    • File Collection
  • Update DAG file for new data model (Shankar)
  • Logging the Xcom in Airflow (Shankar)
  1. Ensure that no breaking changes
  • Verify that all changes in M24 preship is merged in (TSP, JK)
  • M25 dev end-to-end testing (Pavithra, JK, Priyanka)
    • Compile the end-to-end testing scenarios and test data in various CSP (Pavithra)
  1. Document in the readme.md and other relevant locations about the input and output model as well as how to use it
  • Guide on how to debug on local machine using unit tests (TSP)
  • Reviewed existing documentation (Shreya)
  • Updated documentations (Pavithra & Shreya)
Edited by Teo Sheng Pu