Improve the validation when datasets/parquets are corrupted
Context
Improve the validation errors and decide whether the system will fail if some of the datasets/parquets are corrupted, etc. Now, we just log the error and skip the file.
While RAFS DDMS processes datasets (work with parquet files) in runtime, the system must fail each request if some of the requested datasets (files) are corrupted or unavailable. Meantime, the response must contain a list of errors that occurred during the request.
Acceptance Criteria
- Service responds with the 500 error status if some datasets are unavailable or corrupted; the response format is following:
{
errors: [
"code": "<error details>",
...
]
}
- Service responds with status 200 if no error occurs with dataset processing.
Edited by Siarhei Khaletski (EPAM)