Note About Using Transparent Huge Pages

Transparent Huge Pages (THP) is a Linux feature that reduces the overhead of using large amounts of memory. However, THP causes a problem for NuoDB memory management and must be disabled.

The problem is that THP prevents certain pages from being purged from resident memory. Unpurged pages include those that are marked with madvise(...,...,MADV_DONTNEED). Since NuoDB relies on jemalloc, which uses this madvise() system call, the result is a slow but steady increase in Resident Set Size (RSS). This decreases the performance of NuoDB transaction engines and storage managers and the Linux kernel will eventually kill them.

NuoDB installation automatically disables THP on Linux distributions running on kernel versions 2.6.38 or later. However, it is possible to enable THP on any kernel version whether or not NuoDB has been installed. You should ensure that THP is disabled even on a distribution that is running kernel version 2.6.38 or later.

Kernel versions prior to 2.6.38 do not support madvise(...,...,MADV_NOHUGEPAGE). Consequently, you must manually disable THP on these Linux distributions. To disable THP, see the information in the following sections:

NuoDB Behavior If THP is Enabled

If you install NuoDB on a system on which THP is enabled then NuoDB processes (TEs and SMs) do not start and NuoDB sends an error message to its agent log. The error message is something like the following:

2015-01-19T15:18:25.697-0500 [1605] NuoDB will not run with transparent hugepages enabled, please disable or set to [madvise]
2015-01-19T15:18:25.697-0500 [1605] See for details
2015-01-19T15:18:25.697-0500 INFO com.nuodb.agent.event.EventManager.notifyNodeFailure (serv-socket5-thread-2) Node failed peer=[Peer null:48004 (local)] startId=[12,772,642,010,001]: NuoDB will not run with transparent huge pages enabled. See for details
2015-01-19T15:18:25.698-0500 WARNING com.nuodb.agent.description.Enforcer.nodeFailed (serv-socket5-thread-2) Local node failed startId=12772642010001: NuoDB will not run with transparent hugepages enabled.
See for details

Determining If THP is Enabled

Before you install NuoDB, you might want to determine if THP is enabled. (Of course, you can also do this after you install NuoDB.) Run the following command:

$ cat /sys/kernel/mm/transparent_hugepage/enabled

Replace transparent_hugepage with the correct filename for your system. For example, on Red Hat the value is redhat_transparent_hugepage. If the content of the file is either madvise or never then THP is disabled. If the setting is always, as shown below, then THP is enabled and you must disable it in order to run NuoDB.

$ cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
[always] madvise never

Disabling THP Until the Next Restart

To disable THP until the next system restart, run the following commands as root. Replace transparent_hugepage with the value for your system. While NuoDB recommends the madvise setting, a setting of never also lets NuoDB work correctly.

$ echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
$ echo madvise > /sys/kernel/mm/transparent_hugepage/defrag

For example, on Red Hat you would execute the following:

$ echo madvise > /sys/kernel/mm/redhat_transparent_hugepage/enabled
$ echo madvise > /sys/kernel/mm/redhat_transparent_hugepage/defrag

Disabling THP in this manner is temporary. The next time the system is restarted THP will no longer be disabled.

If the response to these commands is something like the following, see Disabling THP on Read-Only File Systems.

cannot create /sys/kernel/mm/transparent_hugepage/enabled: Read-only file system

Disabling THP Permanently

To permanently disable THP, do one of the following:

For either of these changes to take effect, you must restart your system. If it is not convenient to restart now then you can run the commands described in Disabling THP Until the Next Restart.

Editing the Kernel Bootloader

If you are using a bootloader, for example /etc/grub.conf, add the following to the end of the kernel boot line. Consult the documentation for your system before editing your bootloader configuration.


Editing Your System Startup Script

Edit your system startup script by adding the following lines. This is likely to be in the /etc/rc.local file (Red Hat) or an equivalent system startup script.

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
  echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
  echo madvise > /sys/kernel/mm/transparent_hugepage/defrag
if test -f /sys/kernel/mm/redhat_transparent_hugepage/enabled; then
  echo madvise > /sys/kernel/mm/redhat_transparent_hugepage/enabled
if test -f /sys/kernel/mm/redhat_transparent_hugepage/defrag; then
  echo madvise > /sys/kernel/mm/redhat_transparent_hugepage/defrag

To ensure that the startup script is executed when the system reboots, you must run:

chmod +x system_startup_file

Replace system_startup_file with one of the following:

Disabling THP on Read-Only File Systems

On some hosts, the /sys directory is mounted on a a read-only file system. This requires additional steps to disable THP. To determine whether /sys is mounted on a read-only file system, run the following command:

grep sysfs /proc/mounts

You will see a line such as the following. If the first option is ro, as in this example, then the /sys directory is mounted as a read-only file system.

sysfs /sys sysfs ro,nosuid,nodev,noexec,relatime 0 0

To disable THP, do the following:

  1. Remount the /sys directory as read-write:

    $ sudo mount -o rw,remount /sys

  2. As root, change permissions as follows. Be sure to replace transparent_hugepage with the value for your system.

    $ chmod o+w /sys/kernel/mm/transparent_hugepage/enabled

    $ chmod o+w /sys/kernel/mm/transparent_hugepage/defrag

  3. As root, run the following commands. Be sure to replace transparent_hugepage with the value for your system.

    $ echo madvise > /sys/kernel/mm/transparent_hugepage/enabled

    $ echo madvise > /sys/kernel/mm/transparent_hugepage/defrag

    This step disables THP until the next host restart. If that is not what you want, see Disabling THP Permanently.