Making workflow engine truly stateless

[Charles546]

The workflow engine currently has an in memory data structure that holds all in-fly workflow sessions. We can't safely restart the daemon when there are workflows running. We have an issue to have a way to gracefully restart the daemon #190, but for a busy daemon, it will be challenging to find a window to do upgrade/maintenance etc.

The proposed solution is to persist the data into some kind of database such as consul, redis, memcached, couchdb etc. This way, we can safely/gracefully shutdown the daemon for upgrade or maintenance.

[Charles546]

The issue #226 is the first step in resolving this issue. We still need to change the code in internal/workflow to use the cache, and the communication between the workflow engine and operator needs to be node neutral (currently, we use engine IP to identify the node of the engine like below.)

honeydipper/drivers/cmd/redisqueue/main.go

Lines 100 to 111 in ade16d7

	if msg.Subject == "command" {
	sessionID, ok := msg.Labels["sessionID"]
	if ok && sessionID != "" {
	msg.Labels["engine"] = dipper.GetIP()
	}
	topic = eventbus.CommandTopic
	} else if msg.Subject == "return" {
	returnTo, ok := msg.Labels["engine"]
	if !ok {
	log.Panicf("[%s] return message without receipient", driver.Service)
	}
	topic = eventbus.ReturnTopic + returnTo