/events URL returns no content in 0.31.0 #1024
Comments
|
That's probably caused by the pubsubstub update, but it's been a while so I don't quite remember what we changed exactly. Ideally, If you can, I recommend to run pubsubstub in it's own process as described here: #496 (comment) (it's really how it's meant to be used, the integrated form is for ease of use but isn't ideal at all). Alternatively, it also has a degraded polling mode you can enable with: I'd be happy to dig into this problem but, I wouldn't know where to start without a way to reproduce the problem, there is just so many moving parts it could be anything. |
|
Thanks, I tried this and it does appear to be working better now |
|
I spoke too soon, I'm still seeing the behaviour where the /events URL misbehaves even when running pubsubstub as a separate process
|
|
That's weird. Cloud you post more logs? I'd like to fix this issue if I can reproduce it. Also I'm interested to know how you run pubsubstub externally. With puma? What version ? How many threads? Any info you can provide is welcome. |
|
Also do you see any |
|
Sure, here's my external proc setup, though the behaviour manifested the same as when it was set up with the defaults. let me know if I can provide anything else:
versions: Procfile: pubsubstub/config.ru: pubsubstub/puma_config.rb output logs: |
|
And do you have anything else between the client and the puma server? A reverse proxy maybe, e.g. nginx? If so how is it configured ? |
|
Yes, we are using nginx. Here's our nginx conf, which is pretty vanilla |
|
I suspect the problem is caused by nginx buffering. There's two way to confirm it:
If that indeed solve the issues we can have |
|
Hum, actually we already handle that: https://github.com/gmalette/pubsubstub/blob/78dfb5605aa0801350066056685f383b9fe8e2ba/lib/pubsubstub/stream_action.rb#L6 Could you also post an HTTP response? I'd like to see the headers etc. |
|
Since I could witness the same behaviour even bypassing nginx (i.e. |
Then it should be easy to reproduce locally no? If you can produce this behavior at will let me know. |
|
@casperisfine we took down the instance with pubsubstub as a separate process since we couldn't get it to work, we're now using |
|
Can't you just run pubusbstub on your local machine? |
|
Here's what I see when from beyond that point it hangs forever, and I never receive a response Here's what I see in the same environment on 0.26.0 In both cases I am bypassing nginx |
|
So I built a debug repo with the informations you gave: https://github.com/casperisfine/pubsubstub-debug I kinda have trouble running ruby 2.5.1, but I ran it with 2.5.3 and it works fine. Can you try this repo to see if it reproduce the issue for you? |
|
@casperisfine I think I have a better understanding of what's happening here. Running pubsubstub alone works fine but it appears to be the connection between shipit and pubsubstub, which is not induced by the standalone repo above shipit is running on port 3000, as is the default. pubsubstub is running on port 8000, because of the line from the linked comment. as in the Procfile:
So what's not clear to me is
|
It's not shipit's role to forward anything. When running pubsubstub externally, you have to do the proper routing in your reverse proxy (nginx). e.g.: |
|
Thanks @byroot, that makes sense. I have the following questions:
At the end of the day, I got my setup working (with |
It should still mostly work, it's just discouraged as longed-lived streaming requests don't mix well with short lived transactional request. So there's probably a bug, (likely some mutex or something) causing pubsubstub to freeze when in process. I'll try to see if I can debug it, but either way I don't recommend running it this way (And I'm thinking about making
Seems like we just had the same idea.
Ideally yes, but in practice since the reverse proxy prevent it from ever be hit, it's not really worth the extra configuration. |
We have just upgraded to 0.31.0 from 0.26.0 to pick up the recent GitHub API change.
The installation runs fine for several hours but then all requests will start timing out
It appears to be because an unbounded number of Pubsubstub connections are being created. I see many repeated events of the form
followed by an ever-growing number of repeated events of the form
This appears to be because the requests to /events?channels%5B%5D=stack.1 hang and return no content. In 0.26.0 the requests return content of the form:
Reverting to 0.26.0 appears to fix the issue
Please let me know if I can provide any additional information to help diagnose the issue
The text was updated successfully, but these errors were encountered: