M7 Manifest based ingestion - Load Testing
Definitions
Load Testing - The number of records or manifests that can be processes at a time.
Background
Throughout M7 there have been a number of performance improvements delivered by EPAM, as well as work on improving issues with configuration etc. We expect this has made a significant improvement to the capacity of the manifest based ingestion, but we don't have a specific figure.
The process of load testing should be repeatable, with the expectation it will be applied to the upcoming Airflow 2+ changes.
Requirements
We need test the "5000 manifest test" @debasisc @todaiks to be re-run on the M7 release. The result should a binary pass/fail and the wall-time for executing the job. For completeness (Table 1) we show a set of recommended test cases that we believe should ultimately be automated and runnable through the QA group.
| Test | Issue | AWS | Azure | GCP | IBM |
|---|---|---|---|---|---|
| the "5000 manifest" | Our current baseline | ||||
| 1 Manifest with 5,000 records | |||||
| 1 Manifest with 20,000 records | |||||
| 1 Manifest with 50,000 records | Limit on the size of the request body | ||||
| 50K manifests in multiple requests, not simultaneously | Airflow 1.X doesn’t allow sending multiple requests (Fixed in Airflow 2.0) | ||||
| chunks of 50, 1000 DAG runs | 1. max_active_runs (50) limitation 2. limitation of workflow service: java heap error Issue 64 3. Storage Service has a limitation of storing no more than 500 records/s | ||||
| chunks of 1000 | see above | ||||
| 50 DAG runs | |||||
| Launch several different DAGS simultaneously | |||||
| Ingest the Volve data | to promote adoption | ||||
| Ingest the TNO data | to promote adoption |