Tuesday, June 5, 2012

RAID Levels Explained



RAID 0, RAID 1, RAID 5, RAID 10 Explained with Diagrams



RAID stands for Redundant Array of Inexpensive (Independent) Disks.


On most situations you will be using one of the following four levels of RAIDs.


§ RAID 0


§ RAID 1


§ RAID 5


§ RAID 10 (also known as RAID 1+0)





This article explains the main difference between these raid levels along with an easy to understand diagram.

In all the diagrams mentioned below:


§ A, B, C, D, E and F – represents blocks


§ p1, p2, and p3 – represents parity







RAID LEVEL 0



Following are the key points to remember for RAID level 0.



§ Minimum 2 disks.


§ Excellent performance ( as blocks are striped ).


§ No redundancy ( no mirror, no parity ).


§ Don’t use this for any critical system.




RAID LEVEL 1


Following are the key points to remember for RAID level 1.


§ Minimum 2 disks.


§ Good performance ( no striping. no parity ).


§ Excellent redundancy ( as blocks are mirrored ).





RAID LEVEL 5



Following are the key points to remember for RAID level 5.


§ Minimum 3 disks.


§ Good performance ( as blocks are striped ).


§ Good redundancy ( distributed parity ).


§ Best cost effective option providing both performance and redundancy. Use this for DB that is heavily read oriented. Write operations will be slow.







RAID LEVEL 10





Monday, June 4, 2012

Linux Bash Shell Options


Displaying options
Use the -o option to set to display all shell options:
willy:~> set -o
allexport              off
braceexpand            on
emacs                  on
errexit                off
hashall                on
histexpand             on
history                on
ignoreeof              off
interactive-comments   on
keyword                off
monitor                on
noclobber              off
noexec                 off
noglob                 off
nolog                  off
notify                 off
nounset                off
onecmd                 off
physical               off
posix                  off
privileged             off
verbose                off
vi                     off
xtrace                 off
See the Bash Info pages, section Shell Built-in Commands->The Set Built-in for a description of each option. A lot of options have one-character shorthands: the xtrace option, for instance, is equal to specifying set -x.
Changing options

Shell options can either be set different from the default upon calling the shell, or be set during shell operation. They may also be included in the shell resource configuration files.
The following command executes a script in POSIX-compatible mode:
willy:~/scripts> bash --posix script.sh
For changing the current environment temporarily, or for use in a script, we would rather use set. Use - (dash) for enabling an option, + for disabling:
willy:~/test> set -o noclobber

willy:~/test> touch test

willy:~/test> date > test
bash: test: cannot overwrite existing file

willy:~/test> set +o noclobber

willy:~/test> date > test
The above example demonstrates the noclobber option, which prevents existing files from being overwritten by redirection operations. The same goes for one-character options, for instance -u, which will treat unset variables as an error when set, and exits a non-interactive shell upon encountering such errors:
willy:~> echo $VAR


willy:~> set -u

willy:~> echo $VAR
bash: VAR: unbound variable
This option is also useful for detecting incorrect content assignment to variables: the same error will also occur, for instance, when assigning a character string to a variable that was declared explicitly as one holding only integer values.
One last example follows, demonstrating the noglob option, which prevents special characters from being expanded:
willy:~/testdir> set -o noglob

willy:~/testdir> touch *

willy:~/testdir> ls -l *
-rw-rw-r--    1 willy    willy        0 Feb 27 13:37 *

Linux Boot Sequence



6 Stages of Linux Boot Process (Startup Sequence)

Press the power button on your system, and after few moments you see the Linux login prompt.

Have you ever wondered what happens behind the scenes from the time you press the power button until the Linux login prompt appears?

The following are the 6 high level stages of a typical Linux boot process.


1. BIOS
  • BIOS stands for Basic Input/Output System
  • Performs some system integrity checks
  • Searches, loads, and executes the boot loader program.
  • It looks for boot loader in floppy, cd-rom, or hard drive. You can press a key (typically F12 of F2, but it depends on your system) during the BIOS startup to change the boot sequence.
  • Once the boot loader program is detected and loaded into the memory, BIOS gives the control to it.
  • So, in simple terms BIOS loads and executes the MBR boot loader.

2. MBR
  • MBR stands for Master Boot Record.
  • It is located in the 1st sector of the bootable disk. Typically /dev/hda, or /dev/sda
  • MBR is less than 512 bytes in size. This has three components 1) primary boot loader info in 1st 446 bytes 2) partition table info in next 64 bytes 3) mbr validation check in last 2 bytes.
  • It contains information about GRUB (or LILO in old systems).
  • So, in simple terms MBR loads and executes the GRUB boot loader.

3. GRUB
  • GRUB stands for Grand Unified Bootloader.
  • If you have multiple kernel images installed on your system, you can choose which one to be executed.
  • GRUB displays a splash screen, waits for few seconds, if you don’t enter anything, it loads the default kernel image as specified in the grub configuration file.
  • GRUB has the knowledge of the filesystem (the older Linux loader LILO didn’t understand filesystem).
  • Grub configuration file is /boot/grub/grub.conf (/etc/grub.conf is a link to this). The following is sample grub.conf of CentOS.

#boot=/dev/sda

default=0

timeout=5

splashimage=(hd0,0)/boot/grub/splash.xpm.gz

hiddenmenu

title CentOS (2.6.18-194.el5PAE)

          root (hd0,0)

          kernel /boot/vmlinuz-2.6.18-194.el5PAE ro root=LABEL=/

          initrd /boot/initrd-2.6.18-194.el5PAE.img
  • As you notice from the above info, it contains kernel and initrd image.
  • So, in simple terms GRUB just loads and executes Kernel and initrd images.

4. Kernel
  • Mounts the root file system as specified in the “root=” in grub.conf
  • Kernel executes the /sbin/init program
  • Since init was the 1st program to be executed by Linux Kernel, it has the process id (PID) of 1. Do a ‘ps -ef | grep init’ and check the pid.
  • initrd stands for Initial RAM Disk.
  • initrd is used by kernel as temporary root file system until kernel is booted and the real root file system is mounted. It also contains necessary drivers compiled inside, which helps it to access the hard drive partitions, and other hardware.

5. Init
  • Looks at the /etc/inittab file to decide the Linux run level.
  • Following are the available run levels
    • 0 – halt
    • 1 – Single user mode
    • 2 – Multiuser, without NFS
    • 3 – Full multiuser mode
    • 4 – unused
    • 5 – X11
    • 6 – reboot
  • Init identifies the default initlevel from /etc/inittab and uses that to load all appropriate program.
  • Execute ‘grep initdefault /etc/inittab’ on your system to identify the default run level
  • If you want to get into trouble, you can set the default run level to 0 or 6. Since you know what 0 and 6 means, probably you might not do that.
  • Typically you would set the default run level to either 3 or 5.

6. Runlevel programs
  • When the Linux system is booting up, you might see various services getting started. For example, it might say “starting sendmail …. OK”. Those are the runlevel programs, executed from the run level directory as defined by your run level.
  • Depending on your default init level setting, the system will execute the programs from one of the following directories.
    • Run level 0 – /etc/rc.d/rc0.d/
    • Run level 1 – /etc/rc.d/rc1.d/
    • Run level 2 – /etc/rc.d/rc2.d/
    • Run level 3 – /etc/rc.d/rc3.d/
    • Run level 4 – /etc/rc.d/rc4.d/
    • Run level 5 – /etc/rc.d/rc5.d/
    • Run level 6 – /etc/rc.d/rc6.d/
  • Please note that there are also symbolic links available for these directory under /etc directly. So, /etc/rc0.d is linked to /etc/rc.d/rc0.d.
  • Under the /etc/rc.d/rc*.d/ directories, you would see programs that start with S and K.
  • Programs starts with S are used during startup. S for startup.
  • Programs starts with K are used during shutdown. K for kill.
  • There are numbers right next to S and K in the program names. Those are the sequence number in which the programs should be started or killed.
  • For example, S12syslog is to start the syslog deamon, which has the sequence number of 12. S80sendmail is to start the sendmail daemon, which has the sequence number of 80. So, syslog program will be started before sendmail.

There you have it. That is what happens during the Linux boot process.

Sunday, May 27, 2012

Saturday, May 26, 2012

understanding Linux configuration files


Every Linux program is an executable file holding the list of opcodes the CPU executes to accomplish specific operations. For instance, the ls command is provided by the file /bin/ls, which holds the list of machine instructions needed to display the list of files in the current directory onto the screen. The behaviour of almost every program can be customized to your preferences or needs by modifying its configuration files.
In a word, no. Users who are new to Linux (rightly) feel frustrated that each configuration file looks like a new challenge to figure out. In Linux each programmer is free to choose the configuration file format he or she prefers. Format options range from the /etc/shells file, which contains a list of possible shells separated by a newline, to Apache's complex /etc/httpd.conf file.
The kernel itself may be considered a "program." Why does the kernel need configuration files? The kernel needs to know the list of users and groups in the system, and manage file permissions (that is, determine if a file can be opened by a specific user, according to the permissions, UNIX_USERS). Note that these files are not specifically read by programs, but by a function provided by a system library, and used by the kernel. For instance, a program needing the (encrypted) password of a user should not open the /etc/passwd file. Instead, it should call the system library function getpw(). This kind of function is also known as a system call. It is up to the kernel (through the system library) to open the /etc/passwd file and after that, search for the password of the requested user.
Most of the configuration files in the Red Hat Linux system are in the /etc directory unless otherwise specified. The configuration files can be broadly classified into the following categories:
/etc/host.confTells the network domain server how to look up hostnames. (Normally /etc/hosts, then name server; it can be changed through netconf.)
/etc/hostsContains a list of known hosts (in the local network). Can be used if the IP of the system is not dynamically generated. For simple hostname resolution (to dotted notation), /etc/hosts.conf normally tells the resolver to look here before asking the network nameserver, DNS or NIS.
/etc/hosts.allowMan page same as hosts_access. Read by tcpd at least.
/etc/hosts.denyMan page same as hosts_access. Read by tcpd at least.
/etc/issue & /etc/issue.netThese files are read by mingetty (and similar programs) to display a "welcome" string to the user connecting from a terminal (issue) or through a telnet session (issue.net). They include a few lines stating the Red Hat release number, name, and Kernel ID. They are used by rc.local.
/etc/redhat-releaseIncludes one line stating the Red Hat release number and name. Used by rc.local.
/etc/rc.d/rcNormally run for all run levels with level passed as argument. For example, to boot your machine in the Graphics mode (X-Server), run the following command from your command line: init 5. The runlevel 5 is starts the system in graphics mode.
/etc/rc.d/rc.localNot official. May be called from rc, rc.sysinit, or /etc/inittab.
/etc/rc.d/rc.sysinitNormally the first script run for all run levels.
/etc/rc.d/rc/rcX.dScripts run from rc (X stands for any number from 1 to 5). These directories are "run-level" specific directories. When a system starts up, it identifies the run-level to be initiated, and then it calls all the startup scripts present in the specific directory for that run-level. For example, the system usually starts up and the message "entering run-level 3" is shown after the boot messages; this means that all the init scripts in the directory /etc/rc.d/rc3.d/ will be called.
The kernel provides an interface to display some of its data structures that can be useful for determining the system parameters like interrupts used, devices initialised, memory statistics, etc. This interface is provided as a separate but dummy filesystem known as the /proc filesystem. Many system utilities use the values present in this filesystemf or displaying the system statistics. For example, the file /proc/modules lists the currently loaded modules in the system. This information is read by the command lsmod, which then displays it in a human readable format. In the same way, the file mtab specified in the following table reads the /proc/mount file, which contains the currently mounted filesystems.
/etc/mtabThis changes continuously as the file /proc/mount changes. In other words, when filesystems are mounted and unmounted, the change is immediately reflected in this file.
/etc/fstabLists the filesystems currently "mountable" by the computer. This is important because when the computer boots, it runs the command mount -a, which takes care of mounting every file system marked with a "1" in the next-to-last column of fstab.
/etc/mtools.confConfiguration for all the operations (mkdir, copy, format, etc.) on a DOS-type filesystem.
/etc/groupContains the valid group names and the users included in the specified groups. A single user can be present in more than one group if he performs multiple tasks. For example, is a "user" is the administrator as well as a member of the project group "project 1", then his entry in the group file will look like: user: * : group-id : project1
/etc/nologinIf the file /etc/nologin exists, login(1) will allow access only to root. Other users will be shown the contents of this file and their logins refused.
etc/passwdSee "man passwd". Holds some user account info including passwords (when not "shadowed").
/etc/rpmrcrpm command configuration. All the rpm command line options can be set together in this file so that all of the options apply globally when any rpm command is run on that system.
/etc/securettyContains the device names of tty lines (one per line, without leading /dev/) on which root is allowed to login.
/etc/usertty
/etc/shadow
Contains the encrypted password information for users' accounts and optionally the password aging information. Included fields are:
  • Login name
  • Encrypted password
  • Days since Jan 1, 1970 that password was last changed
  • Days before password may be changed
  • Days after which password must be changed
  • Days before password is to expire that user is warned
  • Days after password expires that account is disabled
  • Days since Jan 1, 1970 that account is disabled
/etc/shellsHolds the list of possible "shells" available to the system.
/etc/motdMessage Of The Day; used if an administrator wants to convey some message to all the users of a Linux server.
/etc/gated.confConfiguration for gated. Used only by the gated daemon.
/etc/gated.versionContains the version number of the gated daemon.
/etc/gatewayOptionally used by the routed daemon.
/etc/networksLists names and addresses of networks accessible from the network to which the machine is connected. Used by route command. Allows use of name for network.
/etc/protocolsLists the currently available protocols. See the NAG (Network Administrators Guide) and man page.
C interface is getprotoent. Should never change.
/etc/resolv.confTells the kernel which name server should be queried when a program asks to "resolve" an IP Address.
/etc/rpcContains instructions/rules for RPC, which can be used in NFS calls, remote file system mounting, etc.
/etc/exportsThe file system to be exported (NFS) and permissions for it.
/etc/servicesTranslates network service names to port number/protocol. Read by inetd, telnet, tcpdump, and some other programs. There are C access routines.
/etc/inetd.confConfig file for inetd. See the inetd man page. Holds an entry for each network service for which inetd must control daemons or other servicers. Note that services will be running, but comment them out in /etc/services so they will not be available even if running. Format:
/etc/sendmail.cfThe Mail program sendmail's configuration file. Cryptic to understand.
/etc/sysconfig/networkIndicates NETWORKING=yes or no. Read by rc.sysinit at least.
/etc/sysconfig/network-scripts/if*Red Hat network configuration scripts.
System commands are meant exclusively to control the system, and make everything work properly. All the programs like login (performing the authentication phase of a user on the console) or bash (providing the interaction between a user and the computer) are system commands. The files associated with them are therefore particularly important. This category has the following files of interest to users and administrators.
/etc/lilo.confContains the system's default boot command line parameters and also the different images to boot with. You can see this list by pressing Tab at the LILO prompt.
/etc/logrotate.confMaintains the log files present in the /var/log directory.
/etc/identd.confIdentd is a server that implements the TCP/IP proposed standard IDENT user identification protocol as specified in the RFC 1413 document. identd operates by looking up specific TCP/IP connections and returning the user name of the process owning the connection. It can optionally return other information instead of a user name. See the identd man page.
/etc/ld.so.confConfiguration for the Dynamic Linker.
/etc/inittabThis is chronologically the first configuration file in UNIX. The first program launched after a UNIX machine is switched on is init, which knows what to launch, thanks to inittab. It is read by init at run level changes, and controls the startup of the main process.
/etc/termcapA database containing all of the possible terminal types and their capabilities.
A daemon is a program running in non-interactive mode. Typically, daemon tasks are related to the networking area: they wait for connections, so that they can provide services through them. Many daemons are available for Linux, ranging from Web servers to ftp servers.
/etc/syslogd.confThe configuration file for the syslogd daemon. syslogd is the daemon that takes care of logging (writing to disk) messages coming from other programs to the system. This service, in particular, is used by daemons that would not otherwise have any means of signaling the presence of possible problems or sending messages to users.

/etc/httpd.conf
The configuration file for Apache, the Web server. This file is typically not in /etc. It may be in /usr/local/httpd/conf/ or /etc/httpd/conf/, but to make sure, you need to check the particular Apache installation.
/etc/conf.modules or /etc/modules.confThe configuration file for kerneld. Ironically, it is not the kernel "as a daemon". It is rather a daemon that takes care of loading additional kernel modules "on the fly" when needed.
In Linux (and UNIX in general), there are countless "user" programs. A most common user program config file is /etc/lynx.cfg. This is the configuration file for lynx, the well-known textual browser. Through this file you can define the proxy server, the character set to use, and so on. The following code sample shows a part of the lynx.cfg file that can be modified to change the proxy settings of the Linux system. These settings apply (by default) to all the users running lynx in their respective shells, unless a user overrides the default config file by specifying --cfg = "mylynx.cfg.

Proxy settings in /etc/lynx.cfg 
.h1 proxy
.h2 HTTP_PROXY
.h2 HTTPS_PROXY
.h2 FTP_PROXY
.h2 GOPHER_PROXY
.h2 NEWS_PROXY
.h2 NNTP_PROXY
# Lynx version 2.2 and beyond supports the use of proxy servers that can act as
# firewall gateways and caching servers. They are preferable to the older
# gateway servers. Each protocol used by Lynx can be mapped separately using
# PROTOCOL_proxy environment variables (see Lynx Users Guide). If you have 
# not set them externally, you can set them at run time via this configuration file.
# They will not override external settings. The no_proxy variable can be used
# to inhibit proxying to selected regions of the Web (see below). Note that on
# VMS these proxy variables are set as process logicals rather than symbols, to
# preserve lowercasing, and will outlive the Lynx image.
#
.ex 15
http_proxy:http://proxy3.in.ibm.com:80/
ftp_proxy:http://proxy3.in.ibm.com:80/
#http_proxy:http://penguin.in.ibm.com:8080
#ftp_proxy:http://penguin.in.ibm.com:8080/

.h2 NO_PROXY
# The no_proxy variable can be a comma-separated list of strings defining
# no-proxy zones in the DNS domain name space.  If a tail substring of the
# domain-path for a host matches one of these strings, transactions with that
# node will not be proxied.
.ex
no_proxy:demiurge.in.ibm.com, demiurge


When changing a configuration file, make sure that the program using that configuration is restarted if it's not controlled by the system administrator or the kernel. A normal user doesn't usually have privileges to start or stop system programs and/or daemons.
Changing configuration files in the kernel immediately affects the system. For example, changing the passwd file to add a user immediately enables that user. Also there are some kernel tunable parameters in the /proc/sys directory on any Linux system. The write-access to all these files is given only to the super-user; other users have only readonly access. The files in this directory are classified in the same manner as the Linux kernel source. Every file in this directory represents a kernel data structure that can be dynamically modified to change the system performance.
Note: Before changing any value in any of these files, make sure you know everything about the file to avoid irreparable damage to the system.
Files in the /proc/sys/kernel/ directory
File nameDescription
threads-maxThe maximum number of tasks the kernel can run.
ctrl-alt-delIf 1, then pressing this key sequence cleanly reboots the system.
sysrqIf 1, then Alt-SysRq is active.
osreleaseDisplays the release of the operating system.
ostypeDisplays the type of the operating system.
hostnameThe host name of the system.
domainnameNetwork domain of which the system is a part.
modprobeSpecifies whether modprobe should be automatically run at startup, and load the necessary modules.
A daemon is a program that is always running in background, quietly carrying out its task. Common ones are in.ftpd (ftp server daemon), in.telnetd (telnet server daemon), and syslogd (system logging daemon). Some daemons, while running, keep a close watch on the configuration file and reload it automatically when it changes. But most of the daemons do not reload automatically. We need to "tell" them somehow that the configuration file has changed and that it should be reloaded. This can be achieved (on Red Hat Linux systems) by restarting the services using the service command.
For example, if we have changed the network configuration, we need to issue:
service network restart.
Note: The services are most commonly the scripts present in the /etc/rc.d/init.d/* directory and are started by the init when the system is booted. So, to restart the service you can also do the following:
/etc/rc.d/init.d/ start | stop | status
start, stop, and status are the values that these scripts take as input to perform the action.
A user or system program reads its configuration file every time it is launched. Remember, though, that some system programs are spawned when the computer is turned on, and their behaviour depends on what they read in the configuration files in /etc/. So, the first time a user program is started, the default configuration is read from the files present in the /etc/ directory. Later, the user can customise the programs by using rc and . (dot) files as explained in the next section.
We have seen how programs can be easily configured. But what if someone does not like the way a program has been configured in /etc/? A "normal" user cannot simply go into /etc and change the configuration files; they are owned -- from the filesystem's point of view -- by root! This is why most user programs define two configuration files: the first one at a "system" level, located in /etc/; and the other one, "private" to the user, that can be found in his or her home directory.
For example, in my system I have installed the very useful wget utility. In /etc/ there is an /etc/wgetrc file. In my home directory, there is a file named .wgetrc, which describes my customised configuration (which will be loaded only when I, the user run the wget command). Other users may also have the .wgetrc file in their home directory (/home/other); this file will be read, of course, only when the user runs the wget command. In other words, the /etc/wgetrc file provides "default" values for wget, while the /home/xxx/.wgetrc file lists the "customisations" for a certain user. It is important to understand that this is the "general rule," and is not necessarily true for all cases. A program like pine, for instance, does not have any files in /etc/, but only the custom configuration in the users' home directory, in a file named .pinerc. Other programs may only have a default configuration file in /etc/, and may not let users "customize" them (it's the case with only a few of the config. files in the /etc dir.).

Commonly used rc and . (dot) files
FilenameDescription
~/.bash_loginLook at "man bash". Treated by bash like ~/.bash_profile if that doesn't exist.
~/.bash_logoutLook at "man bash".Sourced by bash login shells at exit.
~/.bash_profileSourced by bash login shells after /etc/profile.
~/.bash_historyThe list of commands executed previously.
~/.bashrcĂ‚ Look at "man bash". Sourced by bash non-login interactive shells (no other files are). Non-interactive shells source nothing unless BASH_ENV or ENV are set.
~/.emacsRead by emacs at startup.
~/.forward If this contains an e-mail address, then all mail to owner of ~ will be forwarded to that e-mail address.
~/.fvwmrc ~/.fvwm2rcConfig files for fvwm and fvwm2 (the basic X Window manager).
~/.hushloginLook at "man login". Causes a "quiet" login (no mail notice, last login info, or MOD).
~/.mail.rcUser init file for mail program.
~/.ncftp/Directory for ncftp program; contains bookmarks, log, macros, preferences, trace. See man ncftp. The purpose of ncftp is to provide a powerful and flexible interface to the Internet standard File Transfer Protocol. It is intended to replace the stock ftp program that comes with the system.
~/.profileLook at "man bash". Treated by bash like ~/.bash_profile if that and ~/.bash_login don't exist, and used by other Bourn-heritage shells too.
~/.pinercPine configuration
~/.muttrcMutt configuration
~/.exrcConfiguration of vi can be controlled by this file.
Example: set ai sm ruler
Writing the above line in this file makes vi set the auto-indentation, matching brackets and displaying line number and rows-columns options.
~/.vimrcDefault "Vim" configuration file. Same as .exrc.
~/.gtkrcGNOME Toolkit.
~/.kdercKDE configuration.
~/.netrcDefault login names and passwords for ftp.
~/.rhostsUsed by the r-tools: rsh, rlogin, etc. Very weak security since host impersonation is easy.
  1. Must be owned by user (owner of ~/) or superuser.
  2. Lists hosts from which users may access this account.
  3. Ignored if it is a symbolic link.
~/.rpmrcSee "man rpm". Read by rpm if /etc/rpmrc is not present.
~/.signatureMessage text that will be appended automatically to the mail sent from this account.
~/.twmrcConfig file for twm (The Window Manager).
~/.xinitrcRead by X at startup (not by xinit script). Mostly starts some progs.
Example: exec /usr/sbin/startkde
If the above line is present in this file, then the KDE Window Manager is started in when the startx command is issued from this account.
~/.xmodmaprcThis file is passed to the xmodmap program, and could be named anything (~/.Xmodmap and ~/.keymap.km, for example).
~/.xserverrcRun by xinit as the X server if it can find X to execute.
~/News/Sent-Message-IDsDefault mail history file for gnus.
~/.XauthorityRead and written by xdm program to handle authorization. See the X, xdm, and xauth man pages.
~/.Xdefaults,
~/.Xdefaults-hostname
Read by X applications during startup on hostname. If the -hostname file can't be found, .Xdefaults is looked for.
~/.XmodmapPoints to .xmodmaprc; Red Hat had (has) .xinitrc using this name.
~/.XresourcesUsually the name for the file passed to xrdb to load the X resources database, to avoid the need for applications to read a long .Xdefaults file. (~/.Xres has been used by some.)

~/mbox

User's old mail.