Skip to content

Refactor Transformer for batch `ingest->process` workflow

Levi Remington requested to merge transformer-refactor-memory into master
  • Leverage schema service for authoritative source on property types
    • Combine with inferred types (assumed string) for maximum coverage and compatibility
  • Introduce loadInBatch capability which enables a substantially more performant and memory-conscious workflow for ingestion and processing (stages 1 & 2 of Transformer)
    • batchSize is configurable for fine-tuning the performance to an operator's environment
  • Docs updated to reflect new capabilities
  • Minor bug fixes to Provider

Merge request reports

Loading