Welcome to the World of Linux

The for Loop

passion@linux — Fri, 07 Dec 2007 09:14:10 +0000

The for loop is used to process a list of items. The list can be supplied in the script. It could also be taken from the script arguments, from a file, or from the output of a command.

The syntax of the for construct is as follows:

for list
do
command_list
done

All the commands in the list are carried out for each item in the list.

The following script invokes the C compiler cc for all files with the extension of .c held in the current directory.

for filename in $(ls *.4gl) do cc $filename done

A similar script could be written to compile several programs specified on the command line. The user can specify as many programs as he wants.

for filename in $* do cc $filename done

$* expands to all the arguments given by the user. Each argument will count as a single item in the list.

This construct is very useful for working with commands that give more than one line of output. Note, however, that the loop will be carried out for each word of the output, rather than once for each line. Commands whose output consist of several columns separated by spaces will be treated as one list item for each column of each row.

To illustrate this, we will look at a script that takes the output from the ps command, and attempts to process it as a list. The output from ps looks something like this:

PID TTY TIME CMD
3218 pts/2 0:01 ksh
3207 pts/2 0:00 sh
3550 pts/2 0:00 ps

The script is as follows:

for process in $(ps) do echo $process done

and gives the following output:

PID
TTY
TIME
CMD
3218
pts/2
0:01
ksh
3207
pts/2
0:00
sh
3579
pts/2
0:00
ps
3578
pts/2
0:00
ksh

If we wanted to carry out commands on each line rather than on each word, and ignore the column headings, we would have to use some ingenuity. The following script displays the PID and command from each line output from the ps command.

компютри

count1=0 count2=0 for process in $(ps) do count1=$(($count1 + 1)) if test $count1 -lt 5 then continue fi count2=$(($count2 + 1))

if test $count2 -eq 1 then pid=$process fi if test $count2 -eq 4 then cmd=$process echo $pid $cmd count2=0 fi done

The output would look like this:

3218 ksh
3207 sh
3609 ps
3608 ksh

The script uses two counters. The first, count1, is used to skip out the headings. The first four items in the list are headings, so if the counter is less than 5, it uses the continue command to skip all further commands for that item. The second counter is used to establish which column of the ps output is being processed. When it is column 1, it stores the PID in a variable. For column 4, it stores the command in a variable, echoes both variables to the user, and resets the count to zero.

The While Loop

passion@linux — Fri, 07 Dec 2007 09:13:08 +0000

In common with most programming languages, Unix shell scripting caters for instructions to be carried out repetitively.

The while loop will repeat a list of commands as long as a given condition is true. All of the conditional statements used in the if…fi construct can be used here, including string and numeric comparisons, file checking, null checking and the exit status of a command. The syntax is

while condition
do
list of commands
done

As an example, let’s write a script named table that will take a number as a command line argument, and print its ‘times table’. The sample below shows the output we would expect from the script:

$ table 5 1 times 5 is 5 2 times 5 is 10 3 times 5 is 15 4 times 5 is 20 5 times 5 is 25 6 times 5 is 30 7 times 5 is 35 8 times 5 is 40 9 times 5 is 45 10 times 5 is 50 11 times 5 is 55 12 times 5 is 60

To achieve this, we will need a variable that will act as a counter. This will begin at 1 and be incremented within the loop. The loop will keep repeating while the counter is less than 13. Within the loop, we will multiply the command line argument $1 by the counter, display the result of this sum, and increment the counter.

The table script is shown below.

counter=1 while [ $counter -lt 13 ] do calc=$(( $counter * $1 )) echo $counter times $1 is $calc counter=$(( $counter + 1 )) done

There are two additional commands that can be used within loops. The break command breaks out of the loop, even though the terminating condition has not yet been met. The continue command goes back to the beginning of the loop without executing any subsequent commands. If the condition controlling the loop is still true, the loop will be repeated.

A loop can be coded to repeat itself infinitely, using the construct while true. This can be used for repeated checking of some aspect of the system. Generally, in this kind of loop, the user is given the option to quit the loop, although he can simply terminate it using the interrupt key.

The following script will display the number of processes which the current user has running, giving him an option to list the processes. He can also opt to quit the script. A delay of 10 seconds is introduced into the loop by using the sleep command that pauses for a given number of seconds.

while true do sleep 10 set $(ps -u $LOGNAME | wc) echo You have $1 processes running echo Enter l to list them, q to quit or c to continue read u_reply if [ $u_reply = q ] then break fi if [ $u_reply = c ] then continue fi ps -u $LOGNAME done

The command set $(ps –u $LOGNAME | wc) lists processes belonging to the current login name and pipes the output to the wc command. This returns the number of words, lines and bytes contained in the answer. This output is used to set the script arguments $1 etc. The first of these arguments, the number of lines returned by the ps command, will be the number of processes this user has running. This is displayed to the user.

If he chooses to continue without listing the processes, the continue command returns to the beginning of the loop. If he chooses to quit, the break command breaks out of the loop. Otherwise, the script will carry out he second ps command that lists all the processes.

Using the set Command

passion@linux — Tue, 04 Dec 2007 19:15:22 +0000

The set command can be used to store a new set of values in the script arguments ($1, $2 etc.). This can be useful for separating the output from a command into separate components that can be worked with individually. The set command requires a string of arguments separated by spaces.

Suppose we have a script that runs at the end of each day, carrying out various tasks. On Fridays only, we want to remove a file named weekhist and create an empty one in its place.

The date command will tell us the day of the week, but it also gives a lot of other information that is not relevant to this task. The output from the date command looks like this:

Mon Sep 3 18:38:21 GMT 2007

We are only interested in the first column, which tells us the day of the week.

Placing the output from this command into the script arguments allows us to separate the output so that we can access the bit we want.

The command:

set $(date)

will have the following effect:

$1 becomes Mon
$2 becomes Sep
$3 becomes 3
$4 becomes 18:38:21
$5 becomes GMT
$6 becomes 2007

We can then check the value of $1 to see if it is a Friday. The relevant section of the script would be:

set $(date) if [ $1 = Fri ] then rm weekhist touch weekhist fi

We could use other portions of the output from date for other purposes. Suppose that rather than deleting the weekhist file, we want instead to move it to a file named weekhist suffixed by today’s date.

We could change the script as follows:

set $(date) if [ $1 = Fri ] then mv weekhist weekhist_$2_$3_$6 touch weekhist fi

If run on Friday September 4 2007, this would have the effect of renaming the old weekhist file to weekhist_Sep_4_2007.

This technique can be used with any command that gives a single line of output consisting of columns separated by spaces. In a later article, we will look at how to deal with the output from commands that give many lines of output.

Providing Input to Commands Within a Script

passion@linux — Tue, 04 Dec 2007 19:13:39 +0000

Scripts should run with as little user intervention as possible. There are several reasons for this. Firstly, the whole point of having a script is to reduce the amount of typing needed to run a job. Secondly, scripts should be able to be run by users with little or no technical knowledge. Thirdly, they are designed to make sure that nothing is left out or mistyped, thus reducing operator errors.

Many Unix commands and programs require input from the user. In an automated script, you would often want to provide this information from the script itself. We have seen in an earlier article that the standard input and output can be diverted to a file instead of the terminal by using output redirection (>), input redirection (<) and error redirection (2>).

Input can be redirected to come from the script itself by using the symbols <<. The text to be taken as input must be enclosed in exclamation marks (!). The following example illustrates how this is done.

cat > textfile < This is some sample text
!

Again, this is a feature which can be very useful in installation scripts, which may need to set up initial files, customized scripts for running jobs etc.

Let’s look at an installation script that needs to schedule a day end job to be run at 6 PM daily. The cron utility is used for job scheduling. cron can take details of the job from a file, or they can be entered from the terminal. In the example below, the information is supplied within the installation script.

cron < 00 18 * * * /usr/progs/dayend

The Case command

passion@linux — Tue, 04 Dec 2007 19:12:27 +0000

In a previous article, we looked at making commands within the script execute only if a given condition is true, using the if….else…fi construct. However, this construct can often become very clumsy when there are several possible conditions to be accounted for. Take the menu script example below:

clear echo User Menu for $LOGNAME echo Accounts......................1 echo Star Office...................2 echo Wages .........................3 echo Exit .............................4 echo echo Please enter your choice 1,2,3 or 4 read menu_choice if [ $menu_choice = 1 ] then accounts elif [ $menu_choice = 2] then star elif [ $menu_choice = 3 ] then wages else exit fi $0

Here we have four different conditions, all depending on the single variable $menu_choice. If we wanted to add more choices to the menu, it would become even more complex.

There is a better way of doing this, provided that all the conditions depend on the value of a single variable. This is to use the case construct.

clear echo User Menu for $LOGNAME echo Accounts......................1 echo Star Office...................2 echo Wages .........................3 echo Exit .............................4 echo echo Please enter your choice 1,2,3 or 4 read menu_choice case $menu_choice in 1) accounts;; 2) star;; 3) wages;; 4) exit;; *) echo $menu_choice is not a valid option;; esac $0

This new script does exactly the same as the old one, but it is much neater, and much easier to expand if more menu choices are added.

As with the if…fi construct, the syntax of the case…esac construct is very strict. It must begin with the word case followed by the name of the variable whose contents are to be examined. The word in must be placed on a separate line. The expected values must then be listed, each on a separate line followed immediately by the close bracket symbol. After the bracket, there is a space followed by one or more commands to be executed if the variable has this value. A double semi-colon terminates the list of commands.

The values to be matched may contain wild cards. Thus the *) matches to any value of menu_choice. Any value of menu_choice (e.g. 5 or 6) not previously matched will be seen as a match here. This then causes any value that is not between 1 and 4 to display the error message.

Command Substitution

passion@linux — Fri, 30 Nov 2007 08:32:15 +0000

Normally, when you run a command from a script, the output from the command goes to the standard output: usually the user’s terminal. However, you may want to do further work with the output from the command, or use it to make decisions within the script.

Lets take a simple example. Suppose that you want to display the message, “Your current working directory is “ followed by the working directory. This can be done by inserting the output from the pwd command into the echo command, using a technique known as command substitution. There are two ways of doing this. The first is to enclose the name of the command in back quotes thus: `pwd`. The second is to enclose it in brackets and prefix it with the $ sign as follows: $(pwd).

So the above example could be entered into a script as follows:

echo Your current working directory is $(pwd)

This facility can be very useful in installation scripts, which may need to install different resources depending on system information such as the type of processor or operating system.

The uname command displays information about the current operating environment. Various switches can be used with it:

-a  Display all information
-s  Display the name of the operating system
-p  Display the type of processor
-r  Display the release number of the operating system.

The following is part of a script that uses the name of the operating system to decide which files to install.

ins_opsys=$(uname –s) if [ $ins_opsys = SunOS ] then tar xvf sunfiles.tar elif [ $ins_opsys = UnixWare ] then tar xvf unixwarefiles.tar elif [ $ins_opsys = AIX ] then tar xvf aixfiles.tar else echo Operating system $ins_opsys not yet supported fi

Command substitution becomes even more useful when combined with techniques covered in the next few articles. We will learn how to split the output from the command so that we can use selected parts of it, or reformat it. We will also learn how to deal with multi-line output.

Metacharacters

passion@linux — Fri, 30 Nov 2007 08:22:27 +0000

Certain characters have special meaning to the shell. These are known as metacharacters. They are listed below.

Pattern Matching:

This set of metacharacters is used in pattern matching, also known as wild cards.

*    Match zero or more characters
?    match exactly one character
[]   Match a list or range of characters eg [AB], [A-N]
[!]  Match anything except the list or range of characters e.g. [!AB]

Redirection:

>    Redirect away from the standard output file
>>   Redirect away from the standard output and append to the target file
2>   Redirect away from the standard error file
<    redirect away from the standard input file
<<   Start of a Here document
|    Pipe the output of one command to the input of a second command

Miscellaneous

&    Run the command in the background
()   Execute command(s) in sub-shell
$    Substitute the value of a variable or argument
#    ignore text to end of line
;    New line
:    Null command

Literal

\    take the literal of the following character – this is used to include an actual metacharacter in a string, rather than have the shell try to interpret it e.g.\$
‘’   Interpret all characters enclosed in single quotes as a literal
“”   Interpret all characters enclosed in double quotes as a literal, except the dollar sign.

Literals are important, since without them, the shell will first interpret any metacharacters before working with the string. For example, the following command would produce an error:

echo Enter your name>

since the > would be interpreted as a redirection symbol. This can be avoided by placing the text in quotes:

echo ‘Enter your name>’

Understanding RAID

passion@linux — Sat, 17 Nov 2007 16:40:48 +0000

The term RAID stands for Redundant Array of Independent Disks. RAID technology allows you to group together the storage capacities of two or more disks, and manage them as a single storage unit. Users are unaware that this array is not a single disk. A RAID array is known as a virtual device, since the operating system views it as a single device when storing data.

RAID can either be controlled at the software or the hardware level. If it is hardware controlled, a RAID controller will be supplied. This is a plug-in card that is attached to the motherboard, and the disks are plugged directly into this controller. It controls all RAID activities, so that the motherboard, and the operating system, are not aware that it is anything other than a single disk device. This is a good option for large RAID arrays, since it tends to give better performance.

When using this option, RAID devices are configured into the operating system in exactly the same way as single-unit devices. There are no special configuration considerations.

If RAID is configured at the software level, partitions are grouped together into a virtual device and appear to higher layers of the operating system as a single unit. This gives good performance under Linux, and is a less expensive option.

Depending on how it is used, RAID can have several advantages, including increased performance, better fault tolerance and data integrity, and more efficient storage. RAID levels are a way of defining the actual type of RAID configuration.

Setting User Limits

passion@linux — Sun, 11 Nov 2007 04:26:20 +0000

In a large installation, it is sometimes difficult to keep control of user activities. Some user programs can use excessive resources, to the detriment of other users. This is particularly true in universities and colleges, where student’s activities can leave few resources for programs run by staff. Also some services automatically grab as many resources as they find available at run time, leaving few resources for subsequent programs. Resources include disk space, memory, CPU time and I/O requests.

It can be useful, therefore, to set limits for each user or group of users. The /etc/security directory holds files that can be used to limit user activities. The most important of these files is /etc/security/limits.conf. It holds entries in the following format:

where
   is the name of the user or group (groups must be prefixed with @)
     is either ‘hard’ or ‘soft’
     is one of the items listed below
    is the value required

Items include:
core - limits the core file size (KB)
data - maximum data size (KB)
fsize - maximum filesize (KB)
memlock - maximum locked-in-memory address space (KB)
nofile - number of open files
rss - max resident memory set size (KB)
stack - stack size (KB)
cpu - maximum CPU time
nproc - maximum number of processes
as - address space limit
maxlogins - maximum number of logins for this user
priority - the priority used for user processes
locks - maximum number of file locks

When setting these limits, you should check the documentation of any software packages that you are running, to ensure that they are able to run within the limits that you have set.

Another useful file in this directory is /etc/security/access.conf, which is used to control which users are allowed to log in from which sources. For example, it is possible to restrict a user’s access to one particular terminal, or to prevent users from logging in via the network.

This file holds entries in the format:

::
where
can be + or -
A user name, group name, or ALL, or ALL except user or group name
A TTY number, host name or domain name

Further uses of the Find command

passion@linux — Thu, 08 Nov 2007 16:42:20 +0000

In another article, we briefly looked at the find command, which searches for specified files. We will now look at more uses of this very powerful command.

It can be used to search for files using various criteria, including name, owner and time last changed. Additionally, it can execute a command or script for each of the matching files. It could therefore be used for many purposes, for example, to remove files not accessed since a given date, to recompile all programs modified since the time of the last compile, or to backup all the files belonging to a specific user.

The syntax of the command is:

find
where is the pathname to the directory where the search is to begin and
and specify the search criteria and any actions to be taken, as detailed below:

-name Find files whose name matches the given pattern. The pattern should be in quotes.
-print Display the path of matching files
-user Find files belonging to a specific user
-exec command {} \; Execute Unix/Linux command for each matching file.
-atime (+t,-t,t) Find files accessed more that +t days ago, less than -t or precisely t days ago.
-ctime (+t,-t,t) Find files changed more that +t days ago, less than -t or precisely t days ago.

-perm Find files with given permissions.
-type Locate files of a given type:
c=character device b=blocked device d=directory
p=pipe s=socket f=normal file l=symbolic link
-size nx Find files with size larger than n where x can be b=bytes;
c=characters; k=kilobytes w=2 byte word:

Examples:

find . –name myfile
Search for myfile under the current directory

find /usr/sam/cprogs –ctime –1 –exec cc {} \;

Compile all programs that have changed less than one day ago.