Skip to content

Draft: read bulk some code clean up + usage of regex

Yannick requested to merge read_bulk_code_clean_up into master

The goal is to do some clean up around get bulk data workflow in order to ease further work (especially the move of serialization inside Dask worker). Mainly function/method/variable renaming and type hint/comment addition but there are also the following changes:

  • bulk filter parsing done using a regex instead of manual parsing. It also allows to filter if the column name contains ':' by enclosing it in double quotation marks (e.g. "Sensor:0":lt:1 "Sensor:180":gt:42).

  • bulk filter regex defined at query param declaration.

  • bulk filter operators defined only once as an enum instead of multiple times.

  • explicit class instead of raw dict.

  • explicit model for describe response instead of raw dict.

renaming/move:

  • GetDataParams => GetDataQueryParameters
  • _process_request_v1 => load_dataframe_storage_V1 + moved from bulk_routes.py to utils.py beside others bulk existing utils functions.
  • process_params => filter_dataframe
  • DataFrameRender.df_render => split into DataFrameRender.describe & DataFrameRender.render.
  • get_size => get_row_count to avoid misleading (see pandas.Dataframe.size <=> number of values)
  • filter_test.py moved from "routers" folder to "model" folder
Edited by Yannick

Merge request reports