Reindex API not working

Reindex API fails in general failing for most of the requests with following error messages:

{
    "code": 415,
    "reason": "Unsupported media type",
    "message": "upstream server responded with unsupported media type: text/html"
}

This happens on larger data set (> 100K) on most of requests and never gets finished.

Reindex API uses different Storage API to retrieve and per log message if it encounters non-json response from storage service, reindex will stop working.

Another insight, reindex API uses Storage get record by Kind using cursor. Cursor returned by this API at times is too long, when we try to make request to this API (tried with Postman rest client) using such cursor we do get non-JSON 400 response.

<!doctype html>
<html lang="en">

<head>
	<title>HTTP Status 400 – Bad Request</title>
	<style type="text/css">
		body {
			font-family: Tahoma, Arial, sans-serif;
		}

		h1,
		h2,
		h3,
		b {
			color: white;
			background-color: #525D76;
		}

		h1 {
			font-size: 22px;
		}

		h2 {
			font-size: 16px;
		}

		h3 {
			font-size: 14px;
		}

		p {
			font-size: 12px;
		}

		a {
			color: black;
		}

		.line {
			height: 1px;
			background-color: #525D76;
			border: none;
		}
	</style>
</head>

<body>
	<h1>HTTP Status 400 – Bad Request</h1>
</body>

</html>

Edited Sep 10, 2021 by Neelesh Thakur