Take service readiness into account

[christophermaier]

This is related to #5326 but somewhat distinct. If health checks can be incorporated into ongoing service state dissemination, they could also be used to gate when the Supervisor broadcasts that it's actually running that service. In other words, we would only introduce new service rumors for services that are healthy and running. This can be particularly useful for services with long startup times, and by extension, the services that depend on them via binds.

Kubernetes does this with their readiness probes.

If we do this, it's also worth considering if it's worth adding a distinct readiness hook.

[mwrock]

Would/could this also impact rolling updates? I have a use case where it can take severla seconds for a run hook to actually get a service into a properly running state. Currently that means that rolling updates bring everyone down because while the code may be update one at a time, the rolling update does not wait for each member to be available.

Ideally a rolling update should move to the next member (or a member should only be marked updated) once the updated member is healthy.

[christophermaier]

@mwrock Yes! I think this should affect rolling updates, too.

[themightychris]

It would be amazing if habitat's lifecycle could remove the need to have sleep loops in any hooks waiting for a service to come up, they usually end up running similar code to the health check

[christophermaier]

@themightychris Yup, I think that's the end goal here. The current structure of the code makes that a bit hard, but we've got a pathway to fixing that.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.