If your server hangs / crashes after throwing errors in messages such as follows :-
Jul 18 03:07:00 xxxx kernel: [<ffffffff810a170a>] ? cpuset_print_task_mems_allowed+0x92/0x9e
Jul 18 03:07:00 xxxx kernel: [<ffffffff810da742>] oom_kill_process+0x85/0x25b
Jul 18 03:07:00 xxxx kernel: [<ffffffff810dac18>] ? select_bad_process+0xbc/0x102
Jul 18 03:07:00 xxxx kernel: [<ffffffff810dac9b>] __out_of_memory+0x3d/0x86
Jul 18 03:07:00 xxxx kernel: [<ffffffff810daf6b>] out_of_memory+0xfc/0x195
======
Try and find out the following
1. Check memory usage at the time system triggered OOM , check how much is free memory
cache + buffer + free / total memory = If this value is less then 20% or so , your system is definitely under memory pressure.
2. Check if system is swapping or not , this can be checked at cat /proc/meminfo and the swapping data should look like this :-
SwapTotal: 12578852 kB
SwapFree: 4765960 kB
which means system has been swapping, and ideally system should never swap and only under memory pressure does system swap.
3. Check the value of AnonPages in cat /proc/meminfo, basically this value indicates the memory which has been assigned to processes but forgotten to be taken back from then. (malloc) , if this value is significant percentage of your total memory, then you got the root cause of the issue.
================
There are few kernel tweaks which could be done to ease out memory pressure, keep
vm.min_free_kbytes = 512000
vm.swappiness = 100
vm.zone_reclaim_mode = 1
in /etc/sysctl.conf and then reload the file via the command sysctl -p . If it is a database server, try and configure hugepages for your setup. get help of google to setup hugepages.
Jul 18 03:07:00 xxxx kernel: [<ffffffff810a170a>] ? cpuset_print_task_mems_allowed+0x92/0x9e
Jul 18 03:07:00 xxxx kernel: [<ffffffff810da742>] oom_kill_process+0x85/0x25b
Jul 18 03:07:00 xxxx kernel: [<ffffffff810dac18>] ? select_bad_process+0xbc/0x102
Jul 18 03:07:00 xxxx kernel: [<ffffffff810dac9b>] __out_of_memory+0x3d/0x86
Jul 18 03:07:00 xxxx kernel: [<ffffffff810daf6b>] out_of_memory+0xfc/0x195
======
Try and find out the following
1. Check memory usage at the time system triggered OOM , check how much is free memory
cache + buffer + free / total memory = If this value is less then 20% or so , your system is definitely under memory pressure.
2. Check if system is swapping or not , this can be checked at cat /proc/meminfo and the swapping data should look like this :-
SwapTotal: 12578852 kB
SwapFree: 4765960 kB
which means system has been swapping, and ideally system should never swap and only under memory pressure does system swap.
3. Check the value of AnonPages in cat /proc/meminfo, basically this value indicates the memory which has been assigned to processes but forgotten to be taken back from then. (malloc) , if this value is significant percentage of your total memory, then you got the root cause of the issue.
================
There are few kernel tweaks which could be done to ease out memory pressure, keep
vm.min_free_kbytes = 512000
vm.swappiness = 100
vm.zone_reclaim_mode = 1
in /etc/sysctl.conf and then reload the file via the command sysctl -p . If it is a database server, try and configure hugepages for your setup. get help of google to setup hugepages.
