Octopus 0.10.0 returns nil instead of column value in some circumstances

[kjvarga]

We have been running Octopus 0.10.0 in production on a Rails 5.1.5 app since around its release. We've noticed that when under heavy load, Octopus is occasionally returning null values from the database, even when that record definitely contains a value for that column. For example, on a NOT NULL column, we are sometimes getting a nil value back from an ActiveRecord find. We also notice that sometimes when creating a record, we get nil created_at updated_at and id values back from the INSERT. But the record is created because we can find it in the database. And it has an id in the database, obviously. But somehow that value does not make it back in the ActiveRecord call. You might be tempted to think that it's not Octopus causing the issue. But when we upgrade to 0.10.2 we see the problem disappear. We have monitoring on occurrences of these nil values, and we see the graphs go to zero under 0.10.2. When we revert back to 0.10.0 we see the issue spike again.

Unfortunately, we can't keep version 0.10.2 because of two other issues 1) our Sidekiq workers eventually freeze and stop processing any jobs #521, and 2) If we try to upgrade to Rails 5.2, we run into the compatibility issue with 0.10.2 and Puma #489, for which there are already issues filed. So we're stuck.

We are using Octopus for replication against an Aurora database which has a single writer node. We don't do any sharding. We run Rails with Puma, and the most affected queries are invoked separate ways. One is via a REST endpoint. The create is in a worker running in Sidekiq.

Rails 5.1.5
Puma
Aurora
Octopus 0.10.0

Regards,
Karl