Errno::ECONNRESET, Connection reset by peer after a few minutes when using apns2

[favrik]

Hello!,

have you seen anything like this before? I've searched through the issues, and the ones that are similar are marked as "Stale".

Updated rpush version to 5.2.0, the updates in latest versions seem to be unrelated

Describe the bug
We recently switched to rpush 5.2.0 in order to use apns2 due to the binary protocol coming to end of life. I'm not sure if this is a bug, but we are getting the following error between successful IOS push notifications:

2021-03-30 03:37:17] [ERROR] Errno::ECONNRESET, Connection reset by peer
/usr/lib/ruby/2.5.0/openssl/buffering.rb:182:in `sysread_nonblock'
/usr/lib/ruby/2.5.0/openssl/buffering.rb:182:in `read_nonblock'
/vendor/bundle/ruby/2.5.0/gems/net-http2-0.18.4/lib/net-http2/client.rb:145:in `block in socket_loop'
/vendor/bundle/ruby/2.5.0/gems/net-http2-0.18.4/lib/net-http2/client.rb:142:in `loop'
/vendor/bundle/ruby/2.5.0/gems/net-http2-0.18.4/lib/net-http2/client.rb:142:in `socket_loop'
/vendor/bundle/ruby/2.5.0/gems/net-http2-0.18.4/lib/net-http2/client.rb:114:in `block (2 levels) in ensure_open'

To Reproduce
Steps to reproduce the behavior:
(note: we start rpush with bundle exec rpush start -f -e <env>)

1. Send a push notification to a IOS device: the notification is delivered.
2. Wait around 20 minutes
3. Attempt to send another IOS push notification: we get the error above, and the notification is not delivered

This is the code we are using to send a push notification:

def send_ios_notification
    pn = Rpush::Apns2::Notification.new
    pn.app = apns_app
    pn.event_id = event_id
    pn.device_id = id
    pn.device_token = push_id
    pn.badge = sendings.where("status='sent'").count if type == :alert_message

    pn.alert = { title: title, body: message }
    pn.data = ios_data_payload(title, message, type, action)
              .merge(headers: { 'apns-topic' => apns_app.bundle_id })
    pn.save
end

def ios_data_payload(title, message, type, action)
    json = { type: type }
    json[:callback_id] = action[:callback_id] if action
    json[:title] = title if title.present?
    json[:body] = message if message.present?
    { json: json }
end

Expected behavior
We don't get a connection reset, or the notification is retried successfully

Logs and other output

System configuration (please complete the following information):

• OS: [Linux]
• OS version: [Ubuntu 20.04.2 LTS]
• Ruby version: [2.5.8p224]
• Rails version: [5.2.3]
• Rpush version: [5.2.0]

Additional context

[aried3r]

Hey @favrik! Sadly I am not a user of the APNs part of the code base, so I'm afraid I cannot directly help out. But since version 5.0.0 that you are using there have been quite some APNs fixes, see the CHANGELOG.md for more details as well as the commits since the last release.

It would help to figure out if this behavior still exists in the latest version.

[favrik]

Thank you @aried3r !

Just remembered we tested up to 5.2.0, but will test with latest version, and see if I'm able to reproduce.

[CR4567]

this is a duplicate of #538
Right now, there is no fix for this problem.

[benlangfeld]

@favrik Are you able to confirm that your experience is compatible with the theories in #538 regarding multiple pushes to the same device?

[favrik]

Hi @CR4567 and @benlangfeld !

the experience has been similar, in the sense that a notification can fail for a device that was successful previously, or is successful afterwards.

However, I'm not sure how the longer stack trace at #538 was produced (I might ask on the issue later).

We could close as duplicate, the main difference I'm seeing is that I'm able to reproduce every time, seems the current magic number is 13 minutes, so I would send a notification to a device, wait 13 minutes, and then submit again and get an Connection Reset error.

I've been testing apnotic, where I get the error as a Connection timed out: once the request is made, it takes 16.5 minutes to time out, a workaround is to set a timeout for client.join, and reload the connection when AsyncRequestTimeout is raised by it.

When reading rpush code, where either the Connection timed out or Connection reset by peer is thrown, it could be rescued and the connection/client be reset, but for the notification retry mechanism to work, the code would need to be changed I think significantly. 🤔

Once I get more details, I'll update this issue, and/or the other one. Thanks!!

[edmondchui]

I experience this issue but it mostly appeared when during the off-peak hours when my Rpush server is idle. During peak time, I'm sending tens of thousands of notifications to Apnd2 over 256 connections. When I increase the number of connections to Apns2 to 512 I saw that happened more. Based on these observations, I suspect Apple is closing the socket on the Apn2 without explicitly telling the developers what is the limit in the Apple Developer documentation.

@favrik and others, what is the number connection specified in your config/initializers/rpush.rb file?

[OskarAtJoint]

This is still a problem, any clue/suggestions on how to resolve it ?!

[taras-viiatyk-yalantis]

+1