Occasional Error Emails

[RichieRogers]

Hi,
Everything is working great on my cluster. I have email notifications setup and I occasionally (numerous times a day) get the following:
vault-unseal ran into errors when attempting to check seal status/unseal. here are the errors:

14 Jan 22 11:50 GMT :: error: checking seal status: Error making API request.

URL: GET https://node03:8200/v1/sys/seal-status
Code: 500. Errors:

• Unexpected response code: 500

sent from vault-unseal. version: v0.0.8, compile date: 2021-10-17T19:40:03Z, hostname: consul03

These alerts come from any of the three vault-unseal nodes and also report the issue on any of the Vault nodes - so it's not an issue with any specific node.
Can anything be done to reduce these - some sort of further check, increased timeout?

Thanks,
Richie

[lrstanley]

Increasing timeout wouldn't resolve the issue, as 500 is service issue in vault. could retry and require a minimum number of failures before sending a notification, which isn't a bad idea.

I would recommend investigating why your vault nodes are returning a 500, via the logs. it's potentially possible that raft voting was happening, or maybe something is intermittently wrong with your storage backend.

Will leave it open and look into retries at some point.