Idea: retry label values and names requests in the query-frontend

[dimitarvdimitrov]

What is the problem you are trying to solve?

A transient failure for a label names or values query is directly returned to the user.

Which solution do you envision (roughly)?

Extend the middleware machinery so that we can reuse the retry middleware on labels requests too. Currently it can't be directly reused because middlewares are only working with MetricsQueryRequest.

Maybe we should make retry generic so that it works with LabelsQueryRequest too.

Have you considered any alternatives?

Implement a retry round-tripper and wrap the labels round-tripper. This will duplicate code and will require us to keep the two implementations in sync. I think we were also trying to steer away from round-trippers. See #7536, https://github.com/grafana/mimir-squad/issues/1938

Any additional context to share?

No response

How long do you think this would take to be developed?

Small (<= 1 month dev)

What are the documentation dependencies?

No response

Proposer?

No response