April 29, 2014

OOM relation to vm.swappiness=0 in new kernel

April 28, 2014 by 5 Comments

I have recently been involved in diagnosing the reasons behind OOM invocation that would kill the MySQL server process. Of course these servers were primarily running MySQL. As such the MySQL server process was the one with the largest amount of memory allocated.

But the strange thing was that in all the cases, there was no swapping activity seen and there were enough pages in the page cache. Ironically all of these servers were CentOS 6.4 running kernel version 2.6.32-358. Another commonality was the fact that vm.swappiness was set to 0. This is a pretty much standard practice and one that is applied on nearly every server that runs MySQL.

Looking into this further I realized that there was a change introduced in kernel 3.5-rc1 that altered the swapping behavior when “vm.swappiness=0″.

Below is the description of the commit that changed “vm.swappiness=0″ behavior, together with the diff:

This change was merged into the RHEL kernel 2.6.32-303:

This obviously changed the way we think about “vm.swappiness=0″. Previously, setting this to 0 was thought to reduce the tendency to swap userland processes but not disable that completely. As such it was expected to see little swapping instead of OOM.

This applies to all RHEL/CentOS kernels > 2.6.32-303 and to other distributions that provide newer kernels such as Debian and Ubuntu. Or any other distribution where this change has been backported as in RHEL.

Let me share with you memory zones related statistics that were logged to the system log from one of the OOM event.

As can be seen the amount of free memory and the amount of memory in the page cache was greater than the high watermark, which prevented any swapping activity. Yet unnecessary memory pressure caused OOM to be invoked which killed the MySQL server process.

MySQL getting OOM’ed is bad for many reasons and can have an undesirable impact such as causing loss of uncommitted transactions or transactions not yet flushed to the log because of innodb_flush_log_at_trx_commit=0, or a much more heavy impact because of cold caches upon restart.

I prefer the old behavior of vm.swappiness and as such I now set it to a value of “1″. Setting vm.swappiness=0 would mean that you will now have to be much more accurate in how you configure the size of various global and session buffers.

Filed Under: Insight for DBAs, MySQL Tagged With: , , , ,
About Ovais Tariq

Ovais Tariq works as a Support Engineer for Percona. Before joining Percona, he worked for different types of companies ranging from a startup to a big media organization. He has worked in different roles, including that of a software engineer, development manager, DBA and consultant.

Comments

  1. Adam Scott says:
    April 28, 2014 at 2:48 pm

    Thank you for the heads-up Ovais. With vm.swappiness=1 does MySQL perform worse under certain loads (and roughly optimal buffer sizes)?

    I set echo -17 > /proc/pidof -s mysqld/oom_adj in the init script to lower mysqld’s priority by OOM. With vm.swappiness=0, does this setting prevent getting OOMed in the newer kernels?

  2. Alen Krmelj says:
    April 28, 2014 at 6:07 pm

    Witnessed similar invocations of OOM killer when servers were under heavy load a while ago. Our entire Hadoop clusters got killed becouse of it. My observations showed that setting it to something small from 1-10 is safe bet with basically any service/app running under heavy load.

  3. Ovais Tariq says:
    April 28, 2014 at 6:44 pm

    Adam,
    With swappiness=1 there can be some swapping activity for example in case the session buffers are not optimally sized, but that was also true for the older “swappiness=0 implementation”.

    Adjusting oom_adj is a recommended practice, however even though it would make it less likely for MySQL to be OOM’ed, it wouldn’t completely prevent that in all possible conditions.

  4. Raghavendra says:
    April 28, 2014 at 11:47 pm

    I think there may be some confusion here.

    The original specifications of swappiness from kernel documentation
    have indicated that it is a preference for LRU reclaim of pages from ANON v/s
    FILE backed pages.

    Setting it to zero implied that the user/admin didn’t want any
    ANON to be reclaimed at the cost of file backed pages.

    Now,

    “”
    But with current reclaim implementation, the kernel may swap out even if
    we set swappiness=0 and there is pagecache in RAM.
    “”

    This was likely the bug that they fixed. ie, when zero, it should
    always try to dirty write / swap out file backed pages (unless
    pinned etc.) whenever possible, and only go to ANON after that.

    Now, about OOM, does it make sense to reclaim pages from ANON when there
    is nothing in page cache and system is running out of memory? Of course
    it does (your system is asphyxiating from lack of memory, you will want
    to do anything to recover from that). That is why following exists:

    That is why following exist: http://lxr.linux.no/#linux+v3.13.5/mm/vmscan.c#L1859

    1859 /*
    1860 * Global reclaim will swap to prevent OOM even with no
    1861 * swappiness, but memcg users want to use this knob to
    1862 * disable swapping for individual groups completely when
    1863 * using the memory controller’s swap limit feature would be
    1864 * too expensive.
    1865 */
    1866 if (!global_reclaim(sc) && !vmscan_swappiness(sc)) {
    1867 scan_balance = SCAN_FILE;
    1868 goto out;
    1869 }
    1870
    1871 /*
    1872 * Do not apply any pressure balancing cleverness when the
    1873 * system is close to OOM, scan both anon and file equally
    1874 * (unless the swappiness setting disagrees with swapping).
    1875 */
    1876 if (!sc->priority && vmscan_swappiness(sc)) {
    1877 scan_balance = SCAN_EQUAL;
    1878 goto out;
    1879 }
    1880

    (Please check if they backported these bits (or equivalent) to centos/rhel or
    not)
    =================================================================

    So, the original specification of swappiness has remained
    unchanged, they have fixed what seems to be an off-by-one kind of
    miscalculation in its calculations. (otherwise, even with 0 it
    scanned some percentage of ANON v/s none).

    The OOM/reclaim shouldn’t be affected by this unless there was
    another bug in between which has been fixed now.

    Setting it to non-zero (>=1) would be mean asking ANON to be
    scanned to that extent.

    =================================

    Now, for mysql the ratio of ANON to FILE is quite high (due to
    Innodb, unless heavy myisam usage and/or very large innodb log
    files), so tuning swappiness is not going to help much (YMMV), it
    may only defer the inevitable.

    If the memory usage of mysqld is very close to system limits, it
    has to die unless kernel itself wants to crash. That is why it is
    recommended to do this either at application level or level of
    cgroups.

    Mysql has to implement per-thread or global limits, since unbounded
    memory like buffers (unlike BP) which scale with connections, so can
    potentially explode the system’s memory. Also, this can be done with
    rlimit where it can be done similar to how open-files-limit is balanced
    with max_connections (there is a formula for this lurking in mysql
    sources).

    Regarding cgroup, it may help to corral threads into a memcg with
    tight bounds on memory. Also, cgroups allow for notifications for
    various stages of memory exhaustion events, mysql can use that to
    adjust connections. (there is an unsubmitted talk by me on
    cgroups and mysql :)).

    Regarding OOM otherwise, it can be due to zone reclaim due to
    unbalanced zones and NUMA, there are whole blog posts on this :)
    (though the numactl fix is not a optimal one but ‘amortizes’ the cost of foreign accesses).

    ===================================

    There is also the vm.vfs_cache_pressure, which can be
    interesting for people with hundreds of thousands of tables and
    running out of memory.

  5. Ovais Tariq says:
    April 29, 2014 at 10:49 am

    Raghu,

    The purpose of this blog post was to share the fact that there is a difference now as to how swappiness=0 is handled by the kernel versus how it had been working for long (and what people had used to be). Of course the current implementation prefers to not swap anonymous pages. However, there is a regression somewhere in the new implementation which causes the kernel to prefer to OOM even though it could remove pages from the page cache. This is shown in the blog post, taken from one of the couple of cases where the kernel preferred to invoke the OOM with the swap usage at 0 and good amount of memory used up by pages in the page cache.

    Swapping is only initiated when nr_free + nr_filebacked high watermark, and hence there was no swapping, yet kernel invoked OOM. That’s exactly what I have tried to portray in the blog post that practicality shows that vm.swappiness=0 is not working as expected.

    I am pretty aware about the swapping issue and its related to NUMA, but that is a wholly different topic. In that particular case default NUMA memory allocation policy causes unnecessary swapping because of imbalance on how the memory allocation is done from the nodes. However, the case that this blog post is talking about is when you have *zero swap* usage and OOM getting invoked, when vm.swappiness=0. Comparing this to the NUMA-swappiness issue is like comparing oranges to apples.

Speak Your Mind

*