Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in / Register
  • H Home
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 61
    • Issues 61
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Open Subsurface Data Universe SoftwareOpen Subsurface Data Universe Software
  • Platform
  • Data Flow
  • Data IngestionData Ingestion
  • External Data Sources
  • Home
  • Issues
  • #121
Closed
Open
Issue created Apr 03, 2021 by Debasis Chatterjee@debasiscOwner2 of 6 checklist items completed2/6 checklist items

Implement incremental fetch-and-ingest

Such as – 1-Apr-2021 – initial fetch-and-ingest brings 4 Master data record from AWS and adds to Azure.

On 3-Apr-2021, @naufalnoori89 adds 2 more records in the same Master entity (ex: Wellbore).

On 4-Apr-2021, @rajesh747 makes another pass of fetch-and-ingest (AWS->Azure). “Happy path” would bring in 2 new records from AWS and add to Azure.

Bonus – Also simulate “update” use case. Such as @naufalnoori89 updating some record in AWS and whether @rajesh747 will bring that updated record across too.

Tasks

  • Create test data at source (Provider) at different timings (coordinate with Developers)
  • Test fetch-and-ingest on day-1
  • Add test data on day-2
  • Repeat test of fetch-and-ingest on day-3
  • Success criteria - check log to see if only incremental data is considered
  • Repeat above test by performing "update" rather than "insert" on day-2

Search API merge request

Edited Aug 06, 2021 by Jacob Rougeau
Assignee
Assign to
Time tracking