Fails to regulate if needing more than 10 PX workers

[jonnor]

Heroku has a limit of 10 PX workers.
When input rate increases rapidly, the scaling algorithm may request more than 10 workers. When this happens, Heroku gives a 422, leaving the actual number of workers << 10.
On next cycle, the same happens again and again - as long as input queue stays quite full - causing degraded quality-of-service.

Workaround: Configure maximum: 10 when using PX workers.

[jonnor]

Maybe what we should do is try a larger number until we hit limit, for instance binary-search upwards between current and target.

[jonnor]

Same case happens for free workers, but there the limit is 2

[jonnor]

Looks like the error response now contains the limit value, which is nice.

2017-01-12T22:25:23.030277+00:00 app[scale.1]: body:
2017-01-12T22:25:23.030277+00:00 app[scale.1]: { limit: 2,
2017-01-12T22:25:23.030278+00:00 app[scale.1]: size: 11,
2017-01-12T22:25:23.030278+00:00 app[scale.1]: id: 'cannot_run_above_limit',
2017-01-12T22:25:23.030279+00:00 app[scale.1]: message: 'Cannot run more than 2 Free size dynos.',
2017-01-12T22:25:23.030280+00:00 app[scale.1]: url: 'https://devcenter.heroku.com/articles/dyno-size#default-scaling-limits' }

[jonnor]

What is trickier in the Free case is that the limit is for the entire formation - not per individual dyno. And this includes the web role, so in practice there is 0 or 1 background workers.

To scale such an app in general, one would have to round-robin which role gets that one worker. Will be interesting to see if Hobby has similar limitations.