Flow control implementation

[zubairov]

When working in the multi-tenant integration environment it's important to obey the API and consumption limits imposed by the platform. This is not only a condition for you integrations to run on the platform (and not begin suspended), but also a good integration practice to sustainably and efficiently use resources.

Imagine a system where one party (published) publishes to the queue and one or more consumers consume from the queue. If publishers are writing to the queue faster than consumers read data from the queue, queue will earlier or later be overfilled. Once one queue of your integration flow will grow to a particular limit, the complete integration flow will be suspended and author will be informed about it. Flow control is a build-in mechanism in the SDK that will help you to prevent the overflow to happen.

There are two types of flow control:

• Static flow control - the hard limit of the events (e.g. messages published to the queue) that can be generated by component. This limit is dictated by your pricing plan and will limit protect the platform from the extensive load.
• Dynamic flow control - the limit that is imposed based on the state of individual queue, more messages are in the queue, slower publisher could write to it.

TODOs

• EventEmitter is async now, emit returns a promise
• Data events are rate limited, with environment variable to configure it
• Tests unit and integration for it
• Switch from regular RabbitMQ Chanel to ConfirmationChannel - dynamic flow control
• Add rate limits for all other events (errors, snapshots, etc.)
• Add environment variables
• Update documentation
• Test on real components

[jhorbulyk]

I'll chat with the team about this. If we are going to redesign sailor APIs, there are other approaches & concerns that we may want to address.