Skip to content

Improve Ignite Performance

Levi Remington requested to merge improve-ignite-memory-usage into master

Only decode JSON object when needed, otherwise store as stringified. This reduces memory consumption for most of the ingestion lifecycle, however memory consumption returns to higher level on GeoJSON generation - unavoidable. After implementing this approach, we no longer see Transformer lock up when ingesting higher record counts (~350,000).

Also add logs to communicate record count prior to batch query - helps user understand the wait time for larger datasets.

Added screenshot and rearranged postman examples for accuracy.

Merge request reports