Transformer - Add Support for Filtered Ingestion
As a GCZ user, I would like to configure a query string preset in my Transformer's kind configuration to filter the records ingested in the first stage. This would allow me to specify tags or other qualifying rules to better organize or restrict the records that GCZ ingests.
Acceptance Criteria
- New configuration parameter supported under the
kind
group:query
- New ingestion logic: if
query
property is detected on akind
, then thequery
property is baked into every OSDU Search Request for that kind.
Assumption
-
query
contents will be legal Lucene syntax to abide by OSDU Search Service Text Query requirements.
Gotchas
- Watch out for special characters, quotes, etc. which are expected with Lucene syntax but may be unsupported or otherwise altered by the YAML input