Berkeley DB Java Engine Tuning Tips

Here are some useful tips for the java BDB engine for Voldemort.

1. First look at the mBean voldemort.store.stats.aggregate with jconsole

voldemort.store.stats.aggregate:type=aggregate-perf

This gives some great stats on performance and operations. You’ll want to pay attention to any operations that are taking a long time.

2. If you have a large JVM > 8G, try allocating about 60% to the BDB cache .

Modify parameter bdb.cache.size inside the server.properties.

This will significantly boost performance for operations by keeping more entries in memory. Obviously you will want to tune this depending on your workload.

For smaller JVM sizes, you might be more sensitive to garbage collection so might want to allocate less to the BDB cache.

3. Look underneath the covers at some JBDB cache miss / size statistics with JMX.

See mbean located at:

voldemort.store.bdb:type=bdbStorageConfiguration

Run operation getEnvStatsAsString:

and look at the javadocs for com.sleepycat.je.EnvironmentStats

Now if you are experiencing high read times, pay attention to the nCacheMiss number, this indicates that your data access patterns going to disk instead of hitting memory.

Example Output:

I/O: Log file opens, fsyncs, reads, writes, cache misses.
bufferBytes=3,145,728
endOfLog=0xa3/0×16b8e9b
nBytesReadFromWriteQueue=0
nBytesWrittenFromWriteQueue=0
nCacheMiss=0
nFSyncRequests=0
nFSyncTimeouts=0
nFSyncs=0
nFileOpens=0
nLogBuffers=3
nNotResident=0
nOpenFiles=3
nRandomReadBytes=0
nRandomReads=0
nRandomWriteBytes=0
nRandomWrites=0
nReadsFromWriteQueue=0
nRepeatFaultReads=0
nSequentialReadBytes=0
nSequentialReads=0
nSequentialWriteBytes=0
nSequentialWrites=0
nTempBufferWrites=0
nWriteQueueOverflow=0
nWriteQueueOverflowFailures=0
nWritesFromWriteQueue=0
Cache: Current size, allocations, and eviction activity.
adminBytes=4,743
cacheTotalBytes=3,225,039
dataBytes=74,250
lockBytes=318
nBINsStripped=0
nEvictPasses=0
…….

3. Visit the Oracle Berkeley DB Java Edition Forums:

http://forums.oracle.com/forums/forum.jspa?forumID=273&start=150

4. Estimate your required BDB cache size….

Use the nice utility included in the java berkeley DB jar file.

http://www.oracle.com/technology/documentation/berkeley-db/je/java/com/sleepycat/je/util/DbCacheSize.html

Below tells us that we need 26 GB of memory allocated to the BDB cache if we need to keep 20 million entries in cache.

$ java -jar je-4.0.92.jar DbCacheSize -records 20000000 -key 20 -data 1000

Inputs: records=20000000 keySize=20 dataSize=1000 nodeMax=128 density=80% overhead=10%

Cache Size Btree Size Description

---

-—————- -———— 1,746,091,324 1,571,482,192 Minimum, internal nodes only 2,364,425,475 2,127,982,928 Maximum, internal nodes only

25,390,535,768 22,851,482,192 Minimum, internal nodes and leaf nodes
26,008,869,920 23,407,982,928 Maximum, internal nodes and leaf nodes

Btree levels: 4