OOM on 32GB AWS Instance When Hosting 5GB Data - Is There A Way to Manage Cached Memory Use / Priority of DB Connections?

[MrLukey]

I am working on a very large implementation of couchdb across multiple scaled VMs, and have ran into some issues regarding memory use and db connection protocols. As discussed here, we had an issue where instances with 32GB of MEM would quickly run OOM due to view processes caching too much data. A quick fix I found was to set the max_dbs_open value to some % ratio of the total db shards available, which forced connections to close periodically, killing the view processes associated and freeing up memory. The results of this change were dramatic, with memory reduced by ~80% (40-56GB used to serve ~5GB of data -> 8-10GB used), however this has caused knock-on effects for our Software Defined Stores, and hence our largest customers (7k+ stores).

One issue we have is in the closing of continuous connections used for changes feeds. Because couch gives no way to prioritise one connection type above another, we have no way to ensure that all continuous feed connections are maintained, while memory intensive view caching connections are closed. This causes a bunch of issues for us upstream, as we require the changes feed connections to be maintained to allow seamless modification of data without risking a race condition. We have experimented with upping the ratio of max_dbs_open/db_shards in the hope that continuous connections were already prioritised, and that with enough spare they would remain open, but to no avail, as regardless of how high this value is set some continuous connections are still closed.

A possible alternative to fiddling with this value would be to limit the amount of views cached, but to my knowledge there is no defined control of this. Is this the case, or have I missed an important config setting? If not, is there a somewhat hacky way to achieve this which couch would be happy with, e.g. manually closing a connection / killing a process associated with large quantities of views cached?

Alternatively, could we request a change that would allow the prioritisation of continuous connections over temporary, or limit the cache size of views? This is a priority issue for us, so any help you can offer would be hugely appreciated!

Thanks in advance! ^_^

max_dbs_open > db_shards (1200 > 804)

root@zyncouchs1:~$ ls -l /proc/$(pidof beam.smp)/fd | grep \\\.couch | wc -l
804
root@zyncouchs1:~$ ls -l /proc/$(pidof beam.smp)/fd | grep \\\.view | wc -l
16391

max_dbs_open < db_shards (500 < 804)

root@zyncouchs3:~$ sudo ls -l /proc/$(pidof beam.smp)/fd | grep \\\.couch | wc -l
500
root@zyncouchs3:~$ sudo ls -l /proc/$(pidof beam.smp)/fd | grep \\\.view | wc -l
768

[rnewson]

There is no caching of views (except on disk as .mrview files). What I think you are seeing is the memory usage of couchjs processes as they process document updates and transform them to view rows. Once a view is built this no longer happens. It sounds like you've raise max_dbs_open higher than your memory allows, though these numbers seem low to me. c.f cloudant nodes have 1-2x that amount of ram and we open many more dbs at once than your setting. Perhaps your documents are, on average, quite large?

Connections won't be closed by couchdb unless a crash occurs, node is rebooted (or its host is rebooted, obviously), or deliberately asking for this (with fabric:end_changes() in remsh or putting the node into maintenance mode). so I'm not sure what you're seeing there. Continuous changes feeds can and do live for a long time, and this is often a problem. It is, however, not reasonable to expect a permanent _changes response, especially not for application correctness. Even if you do manage a long-term changes response I don't believe it gives you "seamless modification of data without risking a race condition". To correctly process the changes feed you need to follow the recipe (periodically recording a seq value to persistent storage, being able to tolerate seeing a change more than once, etc).

Noting that the config item max_dbs_open predates the clustering introduced in 2.0 and refers to individual .couch files. So a factor of q is in play when understanding that value and its implications on system resources.

Finally, I can confirm that 5GB of data is well within couchdb's abilities, it's tiny. however, I suspect that is primary data and not including indexes. Can you give a ballpark figure for how many databases/documents/design documents/views you have here?

[MrLukey]

We only raised max_dbs_open from the default value of 500 to 1200, and gave it as much resource as it needed to cope with this (64GB MEM / 8 Core instance). Certainly there is work ongoing to reduce the size of design documents etc, but these will require db upgrades across the estate and so will have to be rolled out slowly. My work is focused on the Intelligent Control Plane side, trying to figure out config setting for the scaled couchs that will improve resource use.

Certainly we need to change how we hook into the changes feeds, as our current design freaks out if the feed is interrupted too often, but I was hoping to find a quick fix I could implement which would allow the current build to handle the reduction of db connections allowed.

Our q value is still at the default of 2, so don't think thats a major factor involved here.

Yeah 5GB is absolutely tiny, which is the fact that led me to seriously question the high memory use observed on all scaled VMs. Whatever the issue is, it is certainly related to the number of view processes associated with beam, and forcing db connections to close by limiting max_dbs_open to a value < db_shards appears to be the only mechanism at our disposal to limit these.

[nickva]

Like @rnewson mentioned q sharding factor makes a difference. If you can lower your q value, you might have less total db handle and index files open.

Keeping continuous connections open might not be the best strategy. The replicator for instance, doesn't do that. It sets a timeout=$msec parameter so the changes periodically end with a last_seq value. Then, the client can save that sequence and start another changes feed, a bit later, using that value as since=....

So, for instance, you can make your clients timeout, sleep a bit with some random backoff, and then connect against. That would give the time to for some db handles to close if you also set idle_check_timeout to a lowish value:

[couchdb]
idle_check_timeout = 5000

If it is the couch_js processes which take up the memory and not beam.smp you can try adjusting os_process_limit or os_process_soft_limit perhaps.

[MrLukey]

As I mentioned above our q value is still at default value of 2, so not sure I would want to reduce this any further!

Yeah we are going to change the changes feeds in line with the replicators, as it does seem a bit silly to maintain the connections permanently, but at this stage I am looking for alternative methods to address the memory use while that work is underway.

Both the os_process_limit and os_process_soft_limit are still at default values of 100 (if memory serves), which surprised me as the number of view processes associated with beam is the tens of thousands. Is there a way to specifically limit these view processes?

[rnewson]

I should have added "other than when forcibly closed by the LRU behind max_dbs_open" but I figured it was implied.