Handle audit trail fields from source data
What are general thoughts about audit trail fields from external source data?
Ex: query response from TGS wrapper API shows creation username as “Jim Burke” and record creation date as “12-Dec-2020”. As you know, this creation date is important for support of incremental query. Ex: what is incremental data since 1-Dec-2020? (candidate for fetch)
However, upon ingestion by Operator BigOil, creation username can be “Service account name” and creation date as “24-Dec-2020”.
Question – Do we plan to preserve TGS-side audit trail fields (record creation user/date such as Jim and 12-Dec-2020)? Or will these be lost after ingesting at BigOil end?