Implement incremental fetch-and-ingest
Such as – 1-Apr-2021 – initial fetch-and-ingest brings 4 Master data record from AWS and adds to Azure.
On 3-Apr-2021, @naufalnoori89 adds 2 more records in the same Master entity (ex: Wellbore).
On 4-Apr-2021, @rajesh747 makes another pass of fetch-and-ingest (AWS->Azure). “Happy path” would bring in 2 new records from AWS and add to Azure.
Bonus – Also simulate “update” use case. Such as @naufalnoori89 updating some record in AWS and whether @rajesh747 will bring that updated record across too.
Tasks
- Create test data at source (Provider) at different timings (coordinate with Developers)
- Test fetch-and-ingest on day-1
- Add test data on day-2
- Repeat test of fetch-and-ingest on day-3
- Success criteria - check log to see if only incremental data is considered
- Repeat above test by performing "update" rather than "insert" on day-2