More intelligent mmapping by storage_allocator

[ujvl]

On Linux machines the vm.max_map_count limit is hit very quickly (default on an ec2 c4.8xlarge instance is 65536). This needs to be manually increased for archival to be successful (since reflogs have relatively small buckets and each bucket is memory mapped).

Instead the storage_allocator could potentially memory map larger-than-requested chunks of files and store the excess of the available range for future calls. It would also have to map at a different granularity depending on what's being mmapped (data log vs reflog buckets). This could be done by just looking at the size passed to the mmap call.

[anuragkh]

I think we'd discussed this before --- if we keep a single file per reflog, then we can remap whenever we archive a new block to keep contiguous memory locations for all reflog blocks. That way, we perform a single mmap per reflog, and reap benefits of contiguous memory.

This seems cleaner than mmapping larger than requested file-chunks to me.

[ujvl]

You could do remapping, but then you need to redesign swappable_ptr to support more than a single swap, which I found hard to do when I tried to initially. There was more memory overhead for each pointer and I don't even think I covered all the possible race conditions. At first look it seems a lot harder to get that right than pushing the complexity to the allocator, but I'm not totally sure, would need to spend more time on it.