Jhallal/dask update and optimization
Dask 2021.07.2 provide memory optimization.
There is also a new parrameter on read_parquet ("aggregate_files") that will help when we receive lots of small chunks
https://docs.dask.org/en/latest/changelog.html
On Sunday, May 30th, we will be performing critical infrastructure maintenance on our Disaster Recovery processes between 20:30 and 00:30 EDT (00:30 and 2:30 UTC). This will necessitate brief outages for Community GitLab during that time. If you are not able to access one of our services or websites, please wait a few minutes and try again. Additional status updates will be available on our status page at https://status.opengroup.org/.
Dask 2021.07.2 provide memory optimization.
There is also a new parrameter on read_parquet ("aggregate_files") that will help when we receive lots of small chunks
https://docs.dask.org/en/latest/changelog.html