Configurable limit for HTTP APIs and pagination indicator

[nirvana-msu]

Thank you for the valuable tool!

A few suggestions on improvements for HTTP API:

1. Allow user to specify the limit (when we don't need the whole 10k - currently hardcoded limit). It could still be capped at 10k to prevent abuse.
2. Provide indication in the response whether "next page" exists

Both of these would allow to query the service in a more considerate manner, which should also help reduce the load.

If possible, it would also be helpful if total count could be included in any paginated response.

[Cgboal]

Doing a count to see how many rows there are in total can take quite a while on some queries which is why it wasn't included in the response. Doing dynamic limits should be doable though.

[nirvana-msu]

Doing a count to see how many rows there are in total can take quite

Yeah, that's why I added a suggestion to add an indicator of whether "next page" exists - that would at least tell users if they should make another query if they need all results.

Thanks!

[Cgboal]

Try the gRPC API instead, you don't need to bother with pagination there as it streams the results.