Critical Multipath parameters in Linux

# man multipath.conf

fast_io_fail_tmo
Specify the number of seconds the scsi layer will wait after a problem has been detected on a FC remote port before failing IO to devices on that remote port. This should be smaller than dev_loss_tmo. Setting this to off will disable the timeout.

dev_loss_tmo
Specify the number of seconds the scsi layer will wait after a problem has been detected on a FC remote port before removing it from the system. This can be set to "infinity" which sets it to the max value of 2147483647 seconds, or 68 years.
----------------

Regarding fail_if_no_path, please check:
----------------
https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/DM_Multipath/queueifnopath_issues.html

5.4. Issues with queue_if_no_path feature

If features "1 queue_if_no_path" is specified in the /etc/multipath.conf file, then any process that issues I/O will hang until one or more paths are restored. To avoid this, set the no_path_retry N parameter in the /etc/multipath.conf file (where N is the number of times the system should retry a path).

When you set the no_path_retry parameter, remove the features "1 queue_if_no_path" option from the /etc/multipath.conf file as well. If, however, you are using a multipathed device for which the features "1 queue_if_no_path" option is set as a compiled-in default, as it is for many SAN devices, you must explicitly add features "0" to override this default. You can do this by copying the existing devices section for your device from /usr/share/doc/device-mapper-multipath-0.4.7/multipath.conf.defaults into /etc/multipath.conf and editing it to suit your needs.

If you need to use the features "1 queue_if_no_path" option and you experience the issue noted here, use the dmsetup command to edit the policy at runtime for a particular LUN (that is, for which all the paths are unavailable). For example, if you want to change the policy on the multipath device mpath2 from "queue_if_no_path" to "fail_if_no_path", execute the following command.

dmsetup message mpath2 0 "fail_if_no_path"

Note that you must specify the mpathn alias rather than the path. 

Configuring UDEV in Linux

This section is for systems with external or networked storage; that is, Fibre Channel or iSCSI based storage devices. It is recommended that those systems have persistent device names configured for your hosts. This assists live migration as well as providing consistent device names and storage for multiple virtualized systems.

Universally Unique Identifiers(UUIDs) are a standardized method for identifying computers and devices in distributed computing environments. This sections uses UUIDs to identify iSCSI or Fibre Channel LUNs. UUIDs persist after restarts, disconnection and device swaps. The UUID is similar to a label on the device.

Systems which are not running multipath must use single path config. Systems running multipath can use multiple path config.

Single path configuration

This procedure implements LUN device persistence using udev. Only use this procedure for hosts which are not using multipath.

1. 
   Edit the /etc/scsi_id.config file.
   1. 
      Ensure the options=-b is line commented out.
      

      # options=-b
      

   2. 
      Add the following line:

      options=-g
      

      
      This option configures udev to assume all attached SCSI devices return a UUID.
2. 
   To display the UUID for a given device run the scsi_id -g -s /block/sd* command. For example:

   # scsi_id -g -s /block/sd*
   3600a0b800013275100000015427b625e
   

   
   The output may vary from the example above. The output displays the UUID of the device /dev/sdc.
3. 
   Verify the UUID output by the scsi_id -g -s /block/sd* command is identical from computer which accesses the device.
4. 
   Create a rule to name the device. Create a file named 20-names.rules in the /etc/udev/rules.d directory. Add new rules to this file. All rules are added to the same file using the same format. Rules follow this format:

   KERNEL=="sd[a-z]", BUS=="scsi", PROGRAM="/sbin/scsi_id -g -s /block/%k", RESULT="UUID", NAME="devicename"
   

   
   Replace UUID and devicename with the UUID retrieved above, and a name for the device. This is a rule for the example above:

   KERNEL="sd*", BUS="scsi", PROGRAM="/sbin/scsi_id -g -s", RESULT="3600a0b800013275100000015427b625e", NAME="rack4row16"
   

   
   The udev daemon now searches all devices named /dev/sd* for the UUID in the rule. Once a matching device is connected to the system the device is assigned the name from the rule. In the a device with a UUID of 3600a0b800013275100000015427b625e would appear as /dev/rack4row16.
5. 
   Append this line to /etc/rc.local:

   /sbin/start_udev
   

6. 
   Copy the changes in the /etc/scsi_id.config, /etc/udev/rules.d/20-names.rules, and /etc/rc.local files to all relevant hosts.

   /sbin/start_udev
   

Networked storage devices with configured rules now have persistent names on all hosts where the files were updated This means you can migrate guests between hosts using the shared storage and the guests can access the storage devices in their configuration files.

Multiple path configuration

The multipath package is used for systems with more than one physical path from the computer to storage devices. multipath provides fault tolerance, fail-over and enhanced performance for network storage devices attached to Red Hat Enterprise Linux systems.

Implementing LUN persistence in a multipath environment requires defined alias names for your multipath devices. Each storage device has a UUID which acts as a key for the aliased names. Identify a device's UUID using the scsi_id command.

# scsi_id -g -s /block/sdc

The multipath devices will be created in the /dev/mpath directory. In the example below 4 devices are defined in /etc/multipath.conf:

multipaths { 
 multipath { 
 wwid  3600805f30015987000000000768a0019 
 alias  oramp1 
 } 
 multipath { 
 wwid  3600805f30015987000000000d643001a 
 alias  oramp2 
 } 
 mulitpath { 
 wwid  3600805f3001598700000000086fc001b 
 alias  oramp3 
 } 
 mulitpath { 
 wwid  3600805f300159870000000000984001c 
 alias  oramp4 
 } 
}

This configuration will create 4 LUNs named /dev/mpath/oramp1, /dev/mpath/oramp2, /dev/mpath/oramp3 and /dev/mpath/oramp4. Once entered, the mapping of the devices' WWID to their new names are now persistent after rebooting.   

Out of Memory Error

If your server hangs / crashes after throwing errors in messages such as follows :-

Jul 18 03:07:00 xxxx kernel:  [<ffffffff810a170a>] ? cpuset_print_task_mems_allowed+0x92/0x9e
Jul 18 03:07:00 xxxx kernel:  [<ffffffff810da742>] oom_kill_process+0x85/0x25b
Jul 18 03:07:00 xxxx kernel:  [<ffffffff810dac18>] ? select_bad_process+0xbc/0x102
Jul 18 03:07:00 xxxx kernel:  [<ffffffff810dac9b>] __out_of_memory+0x3d/0x86
Jul 18 03:07:00 xxxx kernel:  [<ffffffff810daf6b>] out_of_memory+0xfc/0x195

======

Try and find out the following

1. Check memory usage at the time system triggered OOM , check how much is free memory

cache + buffer + free / total memory = If this value is less then 20% or so , your system is definitely under memory pressure.

2. Check if system is swapping or not , this can be checked at cat /proc/meminfo and the swapping data should look like this :-

SwapTotal:      12578852 kB
SwapFree:        4765960 kB

which means system has been swapping, and ideally system should never swap and only under memory pressure does system swap.

3. Check the value of AnonPages in cat /proc/meminfo, basically this value indicates the memory which has been assigned to processes but forgotten to be taken back from then. (malloc) , if this value is significant percentage of your total memory, then you got the root cause of the issue.
================
There are few kernel tweaks which could be done to ease out memory pressure, keep

vm.min_free_kbytes = 512000
vm.swappiness = 100 
vm.zone_reclaim_mode = 1

in /etc/sysctl.conf  and then reload the file via the command sysctl -p . If it is a database server, try and configure hugepages for your setup. get help of google to setup hugepages.