Lmdb and Lmdbjava Usage Recommendations

Owned by Griff McClellan (Unlicensed)
Last updated: Oct 27, 2017

This document records the results of an investigation into how memory and storage are used by lmdb and lmdbjava. This document ends with usage recommendations.

Lmdb and Lmdbjava Summary

  • Lmdb is written in C and located at https://symas.com/lmdb
  • Lmdbjava is a Java wrapper for lmdb, which uses JNI and is located at https://github.com/lmdbjava/lmdbjava
  • The performance of lmdbjava compared to lmdb is consistent with the performance of a generic Java program compared to the equivalent C program
  • Lmdbjava does not enhance the functionality of lmdb, so in this document when I discuss the capabilities of lmdbjava I am also referring to the capabilities of lmdb

There is a useful summary in lmdb.h:

LMDB is a Btree-based database management library modeled loosely on the
BerkeleyDB API, but much simplified. The entire database is exposed
in a memory map, and all data fetches return data directly
from the mapped memory, so no malloc's or memcpy's occur during
data fetches. As such, the library is extremely simple because it
requires no page caching layer of its own, and it is extremely high
performance and memory-efficient. It is also fully transactional with
full ACID semantics, and when the memory map is read-only, the
database integrity cannot be corrupted by stray pointer writes from
application code.

The library is fully thread-aware and supports concurrent read/write
access from multiple processes and threads. Data pages use a copy-on-
write strategy so no active data pages are ever overwritten, which
also provides resistance to corruption and eliminates the need of any
special recovery procedures after a system crash. Writes are fully
serialized; only one write transaction may be active at a time, which
guarantees that writers can never deadlock. The database structure is
multi-versioned so readers run with no locks; writers cannot block
readers, and readers don't block writers.

Lmdb gains efficiency by letting the operating system do the paging, as explained in https://symas.com/understanding-lmdb-database-file-sizes-and-memory-utilization:

…the calling program accesses the memory address and the operating system automatically handles paging the data into main memory as needed. It should come as no surprise, then, to learn that LMDB has no cache subsystem of its own. The operating system very efficiently handles all of LMDB’s caching needs. It’s also important to point out that only those portions of the database file that are actually accessed are read into primary memory, so other applications are not starved for memory. This is a concept called Single-Level Store, or SLS.

Existing Utilities, mdb_dump.c and dumpbc.lua, are Inadequate

The lmdb literature makes reference to a utility called “mdb_dump,” which displays the property of a database by examining its database file, data.mdb. This utility is written in C and there is a lua instantiation as well.

This utility provides only superficial information about a database. It is only useful for examining a database file about which you know nothing. The output of the utility resembles:

VERSION=3
format=bytevalue
type=btree
mapsize=3200000
maxreaders=126
db_pagesize=4096
HEADER=END
DATA=END

Maximum and Minimum Database Size

The maximum size that an lmdbjava database can be created with is approximately 17 TB. This creation will succeed regardless of the actual RAM or storage available. The precise maximum size is 17592186040320 bytes (approximately 0x0FFFFFFFF000).

The smallest lmdbjava database that can be created is approximately, 500 KB, precisely 448000 bytes. An attempt to create a database with a size smaller than this will fail.

You will receive a MDB_MAP_FULL error (-30791 defined in lmdb.h) if you try to add data that exceeds the size of the database.

When you create a database with size S then a file called data.mdb will be created. The ls command will report the size of the file as S regardless of the contents of the file. The du command will report the actual size of the file.

Computing Size of the Database and Memory in Use

If a program needs to know how much memory it is using, which is the actual size of the database, use the following code:

    unsigned long memoryInUse(MDB_txn *txn, MDB_dbi *dbi)
    {
        long page_size = sysconf(_SC_PAGE_SIZE);
        MDB_stat stat;
        int rc = mdb_stat(txn, *dbi, &stat);
        return page_size * (stat.ms_branch_pages + stat.ms_leaf_pages + stat.ms_overflow_pages);
    }

Size Limitation on Keys and Values

Keys are limited in size to 511 bytes.

The standard configuration of an lmdb database allows only one value to be associated with each key. The size of each value is limited only by the disk size of the host.

When an lmdb database is created with the MDB_DUPSORT configuration, then multiple values can be assorted with each key. In this configuration, the size of each value associated with a key is limited to 511 bytes.

Lmdb Allows One Writer per Database

Lmdb only allows one writer at a time per database to prevent deadlock. A writer that attempts to write while another writer exists blocks.

Closing the Database

Although there is a call for closing a database, it is unnecessary. It is sufficient to just close the environment that contains the database. Specifically,

    mdb_dbi_close(MDB_env *env, MDB_dbi dbi)

is unnecessary. Just use

    mdb_env_close(MDB_env *env)

As explained in lmdb.h:

Closing a database handle is not necessary, but lets #mdb_dbi_open() reuse the handle value. Usually it's better to set a bigger mdb_env_set_maxdbs(), unless that value would be large.

Transactions

http://www.lmdb.tech/doc/index.html#caveats_sec cautions against the use of one transaction with lots of operations within it:  “Avoid long-lived transactions. Read transactions prevent reuse of pages freed by newer write transactions, thus the database can grow quickly. Write transactions prevent other write transactions, since writes are serialized.”

Avoid aborting a process with an active transaction.

Threads

As explained in lmdb.h:

  • Each transaction belongs to one thread.
  • A thread can only use one transaction at a time, plus any child transactions.
  • A transaction and its cursors must only be used by a single thread, and a thread may only have a single transaction at a time.
  • Environment handles should only be closed by a single thread, and only if no other threads are going to reference the database handle or one of its cursors any further.
  • Do not close an environment handle if an existing transaction has modified its database. Doing so can cause misbehavior from database corruption to errorslike MDB_BAD_VALSIZE (since the DB name is gone).

Recommendations for Use

  • A database can only have one writer at a time
  • du, not ls, will tell you the actual size of the data.mdb file
  • The same thread that opens a database must close it
  • Don’t close a database, close its environment
  • Don’t share transactions between threads
  • Favor shorter transactions over longer ones