Out of Memory Error

If your server hangs / crashes after throwing errors in messages such as follows :-

Jul 18 03:07:00 xxxx kernel:  [<ffffffff810a170a>] ? cpuset_print_task_mems_allowed+0x92/0x9e
Jul 18 03:07:00 xxxx kernel:  [<ffffffff810da742>] oom_kill_process+0x85/0x25b
Jul 18 03:07:00 xxxx kernel:  [<ffffffff810dac18>] ? select_bad_process+0xbc/0x102
Jul 18 03:07:00 xxxx kernel:  [<ffffffff810dac9b>] __out_of_memory+0x3d/0x86
Jul 18 03:07:00 xxxx kernel:  [<ffffffff810daf6b>] out_of_memory+0xfc/0x195

======

Try and find out the following

1. Check memory usage at the time system triggered OOM , check how much is free memory

cache + buffer + free / total memory = If this value is less then 20% or so , your system is definitely under memory pressure.

2. Check if system is swapping or not , this can be checked at cat /proc/meminfo and the swapping data should look like this :-

SwapTotal:      12578852 kB
SwapFree:        4765960 kB

which means system has been swapping, and ideally system should never swap and only under memory pressure does system swap.

3. Check the value of AnonPages in cat /proc/meminfo, basically this value indicates the memory which has been assigned to processes but forgotten to be taken back from then. (malloc) , if this value is significant percentage of your total memory, then you got the root cause of the issue.
================
There are few kernel tweaks which could be done to ease out memory pressure, keep

vm.min_free_kbytes = 512000
vm.swappiness = 100 
vm.zone_reclaim_mode = 1

in /etc/sysctl.conf  and then reload the file via the command sysctl -p . If it is a database server, try and configure hugepages for your setup. get help of google to setup hugepages.