CassandraHealthCheck can have spurious failures.

[zmanji]

I have a dropwizard application which uses dropwizard-cassandra. I have noticed that despite my application queries always succeeding, the provided CassandraHealthCheck is reporting failures and causing my application to report unhealthy.

After some investigation, I realized the root cause of this is that the query used in the health check is not marked idempotent and the Cassandra driver will not retry that query in accordance to the retry policy of the application. The other SELECT queries in my application are marked idempotent, and will be retried. When they fail, it is after the retry policy has been exhausted.

From the documentation on idempotence:

In most cases, you must set that flag manually. The driver does not parse query strings, so it can’t infer it automatically (except for statements coming from the query builder, see below).

I believe to fix these spurious failures is to mark the validation query as idempotent in the health check or allow disabling the health check in the config so users can specify their own health check that has a query marked as idempotent.

To work around this in my application, I am unregistering the health check that is auto registered by CassandraHealthCheck and providing my own health check with a query marked as idempotent.