Critical Multipath parameters in Linux

# man multipath.conf

fast_io_fail_tmo
Specify the number of seconds the scsi layer will wait after a problem has been detected on a FC remote port before failing IO to devices on that remote port. This should be smaller than dev_loss_tmo. Setting this to off will disable the timeout.

dev_loss_tmo
Specify the number of seconds the scsi layer will wait after a problem has been detected on a FC remote port before removing it from the system. This can be set to "infinity" which sets it to the max value of 2147483647 seconds, or 68 years.
----------------

Regarding fail_if_no_path, please check:
----------------
https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/DM_Multipath/queueifnopath_issues.html

5.4. Issues with queue_if_no_path feature

If features "1 queue_if_no_path" is specified in the /etc/multipath.conf file, then any process that issues I/O will hang until one or more paths are restored. To avoid this, set the no_path_retry N parameter in the /etc/multipath.conf file (where N is the number of times the system should retry a path).

When you set the no_path_retry parameter, remove the features "1 queue_if_no_path" option from the /etc/multipath.conf file as well. If, however, you are using a multipathed device for which the features "1 queue_if_no_path" option is set as a compiled-in default, as it is for many SAN devices, you must explicitly add features "0" to override this default. You can do this by copying the existing devices section for your device from /usr/share/doc/device-mapper-multipath-0.4.7/multipath.conf.defaults into /etc/multipath.conf and editing it to suit your needs.

If you need to use the features "1 queue_if_no_path" option and you experience the issue noted here, use the dmsetup command to edit the policy at runtime for a particular LUN (that is, for which all the paths are unavailable). For example, if you want to change the policy on the multipath device mpath2 from "queue_if_no_path" to "fail_if_no_path", execute the following command.

dmsetup message mpath2 0 "fail_if_no_path"

Note that you must specify the mpathn alias rather than the path. 

Configuring UDEV in Linux

This section is for systems with external or networked storage; that is, Fibre Channel or iSCSI based storage devices. It is recommended that those systems have persistent device names configured for your hosts. This assists live migration as well as providing consistent device names and storage for multiple virtualized systems.

Universally Unique Identifiers(UUIDs) are a standardized method for identifying computers and devices in distributed computing environments. This sections uses UUIDs to identify iSCSI or Fibre Channel LUNs. UUIDs persist after restarts, disconnection and device swaps. The UUID is similar to a label on the device.

Systems which are not running multipath must use single path config. Systems running multipath can use multiple path config.

Single path configuration

This procedure implements LUN device persistence using udev. Only use this procedure for hosts which are not using multipath.

1. 
   Edit the /etc/scsi_id.config file.
   1. 
      Ensure the options=-b is line commented out.
      

      # options=-b
      

   2. 
      Add the following line:

      options=-g
      

      
      This option configures udev to assume all attached SCSI devices return a UUID.
2. 
   To display the UUID for a given device run the scsi_id -g -s /block/sd* command. For example:

   # scsi_id -g -s /block/sd*
   3600a0b800013275100000015427b625e
   

   
   The output may vary from the example above. The output displays the UUID of the device /dev/sdc.
3. 
   Verify the UUID output by the scsi_id -g -s /block/sd* command is identical from computer which accesses the device.
4. 
   Create a rule to name the device. Create a file named 20-names.rules in the /etc/udev/rules.d directory. Add new rules to this file. All rules are added to the same file using the same format. Rules follow this format:

   KERNEL=="sd[a-z]", BUS=="scsi", PROGRAM="/sbin/scsi_id -g -s /block/%k", RESULT="UUID", NAME="devicename"
   

   
   Replace UUID and devicename with the UUID retrieved above, and a name for the device. This is a rule for the example above:

   KERNEL="sd*", BUS="scsi", PROGRAM="/sbin/scsi_id -g -s", RESULT="3600a0b800013275100000015427b625e", NAME="rack4row16"
   

   
   The udev daemon now searches all devices named /dev/sd* for the UUID in the rule. Once a matching device is connected to the system the device is assigned the name from the rule. In the a device with a UUID of 3600a0b800013275100000015427b625e would appear as /dev/rack4row16.
5. 
   Append this line to /etc/rc.local:

   /sbin/start_udev
   

6. 
   Copy the changes in the /etc/scsi_id.config, /etc/udev/rules.d/20-names.rules, and /etc/rc.local files to all relevant hosts.

   /sbin/start_udev
   

Networked storage devices with configured rules now have persistent names on all hosts where the files were updated This means you can migrate guests between hosts using the shared storage and the guests can access the storage devices in their configuration files.

Multiple path configuration

The multipath package is used for systems with more than one physical path from the computer to storage devices. multipath provides fault tolerance, fail-over and enhanced performance for network storage devices attached to Red Hat Enterprise Linux systems.

Implementing LUN persistence in a multipath environment requires defined alias names for your multipath devices. Each storage device has a UUID which acts as a key for the aliased names. Identify a device's UUID using the scsi_id command.

# scsi_id -g -s /block/sdc

The multipath devices will be created in the /dev/mpath directory. In the example below 4 devices are defined in /etc/multipath.conf:

multipaths { 
 multipath { 
 wwid  3600805f30015987000000000768a0019 
 alias  oramp1 
 } 
 multipath { 
 wwid  3600805f30015987000000000d643001a 
 alias  oramp2 
 } 
 mulitpath { 
 wwid  3600805f3001598700000000086fc001b 
 alias  oramp3 
 } 
 mulitpath { 
 wwid  3600805f300159870000000000984001c 
 alias  oramp4 
 } 
}

This configuration will create 4 LUNs named /dev/mpath/oramp1, /dev/mpath/oramp2, /dev/mpath/oramp3 and /dev/mpath/oramp4. Once entered, the mapping of the devices' WWID to their new names are now persistent after rebooting.   

Out of Memory Error

If your server hangs / crashes after throwing errors in messages such as follows :-

Jul 18 03:07:00 xxxx kernel:  [<ffffffff810a170a>] ? cpuset_print_task_mems_allowed+0x92/0x9e
Jul 18 03:07:00 xxxx kernel:  [<ffffffff810da742>] oom_kill_process+0x85/0x25b
Jul 18 03:07:00 xxxx kernel:  [<ffffffff810dac18>] ? select_bad_process+0xbc/0x102
Jul 18 03:07:00 xxxx kernel:  [<ffffffff810dac9b>] __out_of_memory+0x3d/0x86
Jul 18 03:07:00 xxxx kernel:  [<ffffffff810daf6b>] out_of_memory+0xfc/0x195

======

Try and find out the following

1. Check memory usage at the time system triggered OOM , check how much is free memory

cache + buffer + free / total memory = If this value is less then 20% or so , your system is definitely under memory pressure.

2. Check if system is swapping or not , this can be checked at cat /proc/meminfo and the swapping data should look like this :-

SwapTotal:      12578852 kB
SwapFree:        4765960 kB

which means system has been swapping, and ideally system should never swap and only under memory pressure does system swap.

3. Check the value of AnonPages in cat /proc/meminfo, basically this value indicates the memory which has been assigned to processes but forgotten to be taken back from then. (malloc) , if this value is significant percentage of your total memory, then you got the root cause of the issue.
================
There are few kernel tweaks which could be done to ease out memory pressure, keep

vm.min_free_kbytes = 512000
vm.swappiness = 100 
vm.zone_reclaim_mode = 1

in /etc/sysctl.conf  and then reload the file via the command sysctl -p . If it is a database server, try and configure hugepages for your setup. get help of google to setup hugepages.

Repairing Corrupted GRUB in Linux

1. Insert the dvd(or 1st cd ) of RHEL5.
2. when your linux cd boot your pc and you have a prompt like this 

boot:_

Write linux rescue command here

boot: linux rescue 

3. Now select the language and other appropriate options

4. Then select continue to mount your partition in read and write mode.

5. After mounting your partition in read and write mode type the command given below when you have sh shell prompt 

chroot /mnt/sysimage 

6. Now type this command 

grub-install /dev/hda (if you have sata hard disk then use sda in place of hda)

7. Type exit 

8. Type exit again 

9. System will Reboot !!! enjoy

Disabling a CPU in running Linux System

• The following command will disable a CPU:

# echo 0 > /sys/devices/system/cpu/cpuX/online

where X is the ID of the CPU as determined from /proc/cpuinfo.

• To re-enable the CPU, run:

# echo 1 > /sys/devices/system/cpu/cpuX/online

verify  by # cat /proc/cpuinfo 

ulimit in Linux Explained.

ulimit

User limits - limit the use of system-wide resources.

Syntax
      ulimit [-acdfHlmnpsStuv] [limit]

Options

   -S   Change and report the soft limit associated with a resource. 
   -H   Change and report the hard limit associated with a resource. 

   -a   All current limits are reported. 
   -c   The maximum size of core files created. 
   -d   The maximum size of a process's data segment. 
   -f   The maximum size of files created by the shell(default option) 
   -l   The maximum size that may be locked into memory. 
   -m   The maximum resident set size. 
   -n   The maximum number of open file descriptors. 
   -p   The pipe buffer size. 
   -s   The maximum stack size. 
   -t   The maximum amount of cpu time in seconds. 
   -u   The maximum number of processes available to a single user. 
   -v   The maximum amount of virtual memory available to the process. 

ulimit provides control over the resources available to the shell and to processes started by it, on systems that allow such control.

The soft limit is the value that the kernel enforces for the corresponding resource. The hard limit acts as a ceiling for the soft limit.

An unprivileged process may only set its soft limit to a value in the range from 0 up to the hard limit, and (irreversibly) lower its hard limit. A privileged process may make arbitrary changes to either limit value.

If limit is given, it is the new value of the specified resource. Otherwise, the current value of the soft limit for the specified resource is printed, unless the `-H' option is supplied.

When setting new limits, if neither `-H' nor `-S' is supplied, both the hard and soft limits are set.

Restricting per user processes ( -u) can be useful for limiting the potential effects of a fork bomb.

Values are in 1024-byte increments, except for `-t', which is in seconds, `-p', which is in units of 512-byte blocks, and `-n' and `-u', which are unscaled values.

The return status is zero unless an invalid option is supplied, a non-numeric argument other than unlimited is supplied as a limit, or an error occurs while setting a new limit.

ulimit is a bash built in command.

ulimit example

# ulimit -a 
core file size          (blocks, -c) 0 
data seg size           (kbytes, -d) unlimited 
file size               (blocks, -f) unlimited 
pending signals                 (-i) 8191 
max locked memory       (kbytes, -l) 32 
max memory size         (kbytes, -m) unlimited 
open files                      (-n) 1024 
pipe size            (512 bytes, -p) 8 
POSIX message queues     (bytes, -q) 819200 
stack size              (kbytes, -s) 8192 
cpu time               (seconds, -t) unlimited 
max user processes              (-u) 8191 
virtual memory          (kbytes, -v) unlimited 
file locks                      (-x) unlimited 

Creating Network Bond in Linux

Linux network Bonding is creation of a single bonded interface by combining 2 or more Ethernet interfaces. This helps in high availability of your network interface and offers performance improvement. Bonding is same as port trunking or teaming.

Bonding allows you to aggregate multiple ports into a single group, effectively combining the bandwidth into a single connection. Bonding also allows you to create multi-gigabit pipes to transport traffic through the highest traffic areas of your network. For example, you can aggregate three megabits ports into a three-megabits trunk port. That is equivalent with having one interface with three megabytes speed

Steps for bonding in Oracle Enterprise Linux and Redhat Enterprise Linux are as follows..

Step 1.

Create the file ifcfg-bond0 with the IP address, netmask and gateway. Shown below is my test bonding config file.

$ cat /etc/sysconfig/network-scripts/ifcfg-bond0

DEVICE=bond0
IPADDR=192.168.1.12
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
USERCTL=no
BOOTPROTO=none
ONBOOT=yes
Step 2.

Modify eth0, eth1 and eth2 configuration as shown below. Comment out, or remove the ip address, netmask, gateway and hardware address from each one of these files, since settings should only come from the ifcfg-bond0 file above. Make sure you add the MASTER and SLAVE configuration in these files.

$ cat /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
# Settings for Bond
MASTER=bond0
SLAVE=yes

$ cat /etc/sysconfig/network-scripts/ifcfg-eth1

DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
# Settings for bonding
MASTER=bond0
SLAVE=yes

$ cat /etc/sysconfig/network-scripts/ifcfg-eth2

DEVICE=eth2
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes

Step 3.

Set the parameters for bond0 bonding kernel module. Select the network bonding mode based on you need. The modes are

mode=0 (Balance Round Robin)
mode=1 (Active backup)
mode=2 (Balance XOR)
mode=3 (Broadcast)
mode=4 (802.3ad)
mode=5 (Balance TLB)
mode=6 (Balance ALB)
Add the following lines to /etc/modprobe.conf # bonding commands
alias bond0 bonding
options bond0 mode=1 miimon=100

Step 4.

Load the bond driver module from the command prompt.

$ modprobe bonding

Step 5.

Restart the network, or restart the computer.

$ service network restart # Or restart computer

When the machine boots up check the proc settings.

$ cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.0.2 (March 23, 2006)

Bonding Mode: adaptive load balancing
Primary Slave: None
Currently Active Slave: eth2
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth2
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:13:72:80: 62:f0

Look at ifconfig -a and check that your bond0 interface is active. You are done!.

To verify whether the failover bonding works..
Do an ifdown eth0 and check /proc/net/bonding/bond0 and check the “Current Active slave”.
Do a continuous ping to the bond0 ipaddress from a different machine and do a ifdown the active interface. The ping should not break.