Options for Consumer to manage bulk data files (ex: LAS, DLIS, SegY) from Data Provider
Refer to recent meeting held with @jrougeau and @Rawaa on this subject.
The current approach is to bring meta data of work-product-component (ex: WellLog) permanently across by fetch-and-ingest process. But for actual bulk data, the understanding is that Operator BigOil will bring the file across "on demand" later which will retain the file in "cache" for limited period (ex: 7 days). So, "Proxy delivery" process will have the intelligence to check if it is already in cache, or bring a fresh copy if it is not in cache.
The argument in favour of this is that Operator BigOil will always get a fresh copy of bulk data in case Subscription company made some updates at their end. In addition, this approach is more secured and automatically mitigates the risk of Operator Bigoil retaining subscribed data beyond license expiry.
This approach makes OSDU data in BigOil repository somewhat heterogeneous i.e. some data (loaded by BigOil's own Data Loaders by using standard OSDU ingestion process) appears in one way and some other subscribed data (integrated using EDS' feature of fetch-and-ingest).
We solicit input from other Operators about their expectation of handling of bulk data (ex: LAS, DLIS, SegY files). Copying to @Rawaa and @jyc9999 for their respective inputs.
Also copying to @nhung.nguyenparker for her input from Data Subscriber perspective (i.e. what is real life situation with her company's subscription data).