Provide a new implementation of pagination with cursor to replace the existing implementation 'query with cursor'
This is revised ADR of the original ADR ADR: Pagination Query API.
There are minor differences between this ADR and the original ADR ADR: Pagination Query API:
- This ADR proposes to provide a new implementation of pagination with cursor to replace the existing implementation of the query with cursor instead of creating a new API with new implementation
- This ADR proposes not to force existing applications to change in order to use the implementation of pagination with cursor via
ElasticSearch's search_after
.
As we know, current query with cursor has limit on max. 500 open cursor. It will cause the applications failed randomly when using it as there is no way to coordinate among different applications to avoid it. According to ElasticSearch
documentation, it does not recommend to use query with cursor (scroll API)
to support pagination with cursor. It recommends to use search_after
, a light-weight solution which is also recommended in the ADR ADR: Pagination Query API.
As OSDU is matured and lots of applications have adopt OSDU search, we try to provide a solution that can switch the two solutions transparently without forcing the applications to change.