POC: WPC records index for the search improvements
Context
Several search use cases include a mix of bulk and meta-data, such as a search by WellID, which is not a part of the bulk or sample data. The goal of the POC is to verify that the Redis index can manage the index to retrieve the data performantly and avoid introducing a new infrastructure component (like RDBMS, Spark, Apache Ignite, etc.).
Scope
- Implement a worker that retrieves RAFS DDMS records in the catalog by some schedule; looks up the key properties to extract their references (like FieldName, WellID, Basin, etc.); builds the index in Redis
- Figure out how to mix the index and bulk data in a search request
- Verify the DuckDB support of the required search use cases
Optionally
- Verify if the DuckDB supports reading the parquet files from a bucket in a cloud storage mixing with the index data.