The goal is to do some clean up around get bulk data workflow in order to ease further work (especially the move of serialization inside Dask worker). Mainly function/method/variable renaming and type hint/comment addition but there are also the following changes:
-
bulk filter parsing done using a regex instead of manual parsing. It also allows to filter if the column name contains ':' by enclosing it in double quotation marks (e.g.
"Sensor:0":lt:1
"Sensor:180":gt:42
). -
bulk filter regex defined at query param declaration.
-
bulk filter operators defined only once as an enum instead of multiple times.
-
explicit class instead of raw dict.
-
explicit model for describe response instead of raw dict.
renaming/move:
-
GetDataParams
=>GetDataQueryParameters
-
_process_request_v1
=>load_dataframe_storage_V1
+ moved from bulk_routes.py to utils.py beside others bulk existing utils functions. -
process_params
=>filter_dataframe
-
DataFrameRender.df_render
=> split intoDataFrameRender.describe
&DataFrameRender.render
. -
get_size
=>get_row_count
to avoid misleading (see pandas.Dataframe.size <=> number of values) -
filter_test.py
moved from "routers" folder to "model" folder