Skip navigation.

Unix Performance Tuning

On Windows 2000, it's easy to look at statisics with the Performance Monitor. If you're into watching pretty graphs, it can even get pretty addictive. Unix provides some decent tools to do the analysis you need to tune your system too. You just need to learn how to read the hieroglyphics.

I had to learn to diagnose performance bottlenecks quickly for a client because we had an Oracle data migration that we were doing for a client on a Sun server. Based on our initial benchmarks, the server performed so slowly that we figured we would be a month behind schedule. So we got deep into the system and managed to figure out the bottlenecks. Turned out one of the hard disks was over-utilized and was going crazy with the workload. Spreading the load made everything run just fine. This document will try to show you some of the tools you can use to tune you system.

First the tools:

ps provides process information

vmstat which provides paging and CPU utilization info. It also provides disk utilization data for 4 devices, but if you have many hard disks on your system, I prefer iostat.

iostat provides disk i/o info.

netstat provides network utilization data.

ab is apachebench which simulates multiple web browsers. A good networking and application server test.

PS

On Solaris, try /usr/ucb/ps uax. Similarly on Linux, use ps uax. This command gives you the percentage CPU and Memory used.
USER       PID %CPU %MEM   SZ  RSS TT       S    START  TIME COMMAND
root     16755  0.1  1.0 1448 1208 pts/0    O 17:33:35  0:00 /usr/ucb/ps uax
root         3  0.1  0.0    0    0 ?        S   May 24  6:19 fsflush
root         1  0.1  0.6 2232  680 ?        S   May 24  3:10 /etc/init -
root       167  0.1  1.3 3288 1536 ?        S   May 24  1:04 /usr/sbin/syslogd
root         0  0.0  0.0    0    0 ?        T   May 24  0:16 sched
root         2  0.0  0.0    0    0 ?        S   May 24  0:00 pageout
gdm      14485  0.0  0.9 1424 1088 pts/0    S 16:17:57  0:00 -csh

VMSTAT

Run vmstat to get the following on a SunOS server:
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 s0 s1 s2   in   sy   cs us sy id
 0 1 0 2011344 50640 23   1 381 192 194 0 9 0  2 18  3  339   52  403  5  2 93
and on Linux:
   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 0  0  0      0   5220 103956 752848   0   0     0     0  139    68   0   0 100
The key figures for CPU utilization are (us sy id), with "id" being the percentage CPU that is idle. A value below 10% is great. If you have multiple CPUs, this is an average.

On SunOS, the pi and po are the pages swapped in and out (in KB/s), while on Linux we have si and so for pages swapped in and out per second. High po values indicate that there is not enough real RAM, and memory is being paged out - this is bad of course. In this example on SunOS, our server is swapping out 192 pages a second; assuming 4K pages, this means about 0.5 Mb of memory is being swapped out a second, which is still ok. The Sun server has 2 Gb of memory, of which 50 M is still free, the rest has been pre-allocated for Oracle. If the free memory was higher, we could allocate more of it to Oracle.

You can run vmstat every n seconds by entering vmstat [n] on the command line, where [n] is a number. Vmstat only gives you an average for all processors. If you need to analyze per processor, use mpstat on Solaris. On Linux, you can also use the really excellent command top d 1 which provides a graphical UI which refreshes every second, showing processes and memory usage and CPU usage on a per processor basis.

IOSTAT

To understand disk i/o, first a introduction to Sun hard disk conventions: On SunOS, to list all hard disk partitions, use df -k.
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0t0d0s0     192790   61959  111552    36%    /
/dev/dsk/c0t0d0s6    1191020  905172  226297    80%    /usr
/proc                      0       0       0     0%    /proc
fd                         0       0       0     0%    /dev/fd
mnttab                     0       0       0     0%    /etc/mnttab
/dev/dsk/c0t0d0s4     674047   61569  551814    11%    /var
swap                 1948096       8 1948088     1%    /var/run
/dev/dsk/c0t1d0s3    6049124 5816355  172278    98%    /u01
/dev/dsk/c0t1d0s4    6049124 1557316 4431317    27%    /u02
/dev/dsk/c0t0d0s7   19110978 7748692 1071177    62%    /home0
/dev/dsk/c0t1d0s5    6049124 5122531  866102    86%    /u03
/dev/dsk/c0t0d0s5    1688242    2132 1635463     1%    /opt
/dev/dsk/c0t1d0s6    6049124      13 5988620     1%    /u04
/dev/dsk/c0t3d0s6   16516485 9229999 7121322    57%    /u08
/dev/dsk/c0t1d0s7   10812598 8196075 2508398    77%    u05
/dev/dsk/c0t2d0s6   16516485 11013540 5337781   68%    /u06
This will give you a listing like the above. Each hard disk is identified by a 6 letter name cNtNdN where N is a digit, e.g.: c0t2d0. A hard disk can be partitioned and each formatted hard disk will have at least one partition, s0 (fully c0t2d0s0 in the above example).

On SunOS, use iostat -xn to get an i/o listing by hard disk. If you want to get a listing by partition, use iostat -xnp. On Linux, iostat -x is sufficient.

The following are some SunOS statistics. The r/s and w/s are the number of reads and writes per second. A measure of throughput is the kr/s and kw/s which stands for the Kb read/s and Kb writes/s.

bash-2.03$ iostat -xn
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 fd0
    0.3    2.0   14.6   18.5  0.0  0.0    0.0    7.9   0   1 c0t0d0
    1.2   16.5   66.5  135.1  0.0  0.2    0.0   10.8   0  12 c0t1d0
    0.3    3.0   26.6   24.5  0.0  0.0    0.0    7.0   0   2 c0t2d0
    0.2    2.7   19.0   22.8  0.0  0.0    0.0    8.0   0   2 c0t3d0
    0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c1t6d0
    1.9    6.5   43.9   58.3  0.0  0.2    0.0   29.3   0   7 c2t0d0
    7.9   24.2  211.0  197.9 0.0  0.8    0.0   25.4   0  25 c2t1d0

Perhaps the most two most important columns in the above display are the %w and %b columns. %w is the percentage of time spent waiting for transactions to complete, and is a measure of how many jobs are contending to use the hard disk (eg. high is bad), while %b is the percent of time the hard disk is busy (eg. one high device relative to the other devices is bad).

From the above figures, we can see that hard disk c2t1d0 is the most highly utilized for some reason. Although this is not serious (only 25% utilization), perhaps this should probably be investigated further based on what applications or databases are using that hard disk.

Network Analysis

The problem with netstat is there is a staggering amount of information available. Try netstat -s for network statistics.

RAWIP
        rawipInDatagrams    =     0     rawipInErrors       =     0
        rawipInCksumErrs    =     0     rawipOutDatagrams   =     0
        rawipOutErrors      =     0

UDP udpInDatagrams =17227479 udpInErrors = 0 udpOutDatagrams =17210476 udpOutErrors = 0

TCP tcpRtoAlgorithm = 4 tcpRtoMin = 400 tcpRtoMax = 60000 tcpMaxConn = -1 tcpActiveOpens =121449 tcpPassiveOpens =123593 tcpAttemptFails = 1198 tcpEstabResets = 469 tcpCurrEstab = 89 tcpOutSegs =99321141 tcpOutDataSegs =94129579 tcpOutDataBytes =1737821124 tcpRetransSegs = 6771 tcpRetransBytes =4047173 tcpOutAck =5190950 tcpOutAckDelayed =2633452 tcpOutUrg = 134 tcpOutWinUpdate = 12831 tcpOutWinProbe = 26 tcpOutControl =491560 tcpOutRsts = 2614 tcpOutFastRetrans = 352 tcpInSegs =106046901 tcpInAckSegs =94153690 tcpInAckBytes =1737971355 tcpInDupAck =363643 tcpInAckUnsent = 0 tcpInInorderSegs =100156017 tcpInInorderBytes =278299936 tcpInUnorderSegs = 2077 tcpInUnorderBytes =2975624 tcpInDupSegs = 16862 tcpInDupBytes =1618160 tcpInPartDupSegs = 6 tcpInPartDupBytes = 3844 tcpInPastWinSegs = 8 tcpInPastWinBytes = 47840 tcpInWinProbe = 884 tcpInWinUpdate = 26 tcpInClosed = 20 tcpRttNoUpdate = 2988 tcpRttUpdate =93909335 tcpTimRetrans = 5873 tcpTimRetransDrop = 22 tcpTimKeepalive = 47189 tcpTimKeepaliveProbe= 15890 tcpTimKeepaliveDrop = 31 tcpListenDrop = 0 tcpListenDropQ0 = 0 tcpHalfOpenDrop = 0 tcpOutSackRetrans = 497

IPv4 ipForwarding = 2 ipDefaultTTL = 255 ipInReceives =101796067 ipInHdrErrors = 0 ipInAddrErrors = 0 ipInCksumErrs = 0 ipForwDatagrams = 0 ipForwProhibits = 0 ipInUnknownProtos = 0 ipInDiscards = 0 ipInDelivers =123153971 ipOutRequests =94246439 ipOutDiscards = 0 ipOutNoRoutes = 0 ipReasmTimeout = 60 ipReasmReqds = 0 ipReasmOKs = 0 ipReasmFails = 0 ipReasmDuplicates = 0 ipReasmPartDups = 0 ipFragOKs = 0 ipFragFails = 0 ipFragCreates = 0 ipRoutingDiscards = 0 tcpInErrs = 1 udpNoPorts =1343350 udpInCksumErrs = 0 udpInOverflows = 80 rawipInOverflows = 0 ipsecInSucceeded = 0 ipsecInFailed = 0 ipInIPv6 = 0 ipOutIPv6 = 0 ipOutSwitchIPv6 = 3360

The statistics shown here are a small portion of first few screens of data! I am not an expert in all the parameters, but a quick look at the error parameters such as tcpInErrs gives me an idea of the overall health of the network.

You can also use netstat -a to examine for each port the number of bytes still waiting in the queue for transmission and number of received bytes not copied to the application process by your server:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 *:netbios-ssn           *:*                     LISTEN
tcp        0      0 *:webcache              *:*                     LISTEN
tcp        0      0 *:x11                   *:*                     LISTEN
tcp        0      0 *:http                  *:*                     LISTEN
tcp        0      0 *:ssh                   *:*                     LISTEN
tcp        0      0 S34KLJ142:smtp          *:*                     LISTEN
tcp        0      0 *:https                 *:*                     LISTEN
tcp        0      0 200.7.1.142:netbios-ssn 200.7.1.26:1352         ESTABLISHED
tcp        0      0 200.7.1.142:33568       200.1.34.117:1521       
tcp        0    180 200.7.1.142:ssh         200.7.1.25:1404         ESTABLISHED

Note that ssh still has 180 bytes of data to transmit in its queue. Not a problem, but this could be a problem if the value is very large (meaning that the ssh data cannot be transmited fast enough, so it has to be queued).

I also find it quite useful to use simple real-world benchmarks to analyze network performance, such as the time required to transfer a 10 megabyte file (should be 10-15 seconds on a 10 Mbit network), or using the apachebench ab -c1000 -n10 [url] command to simulate 10 clients sending a total of 1000 http requests to a server.

Current versus Historical Statistics

If you run "vmstat 1" you will get the latest statistics once a second. If you run "vmstat" alone, you will get the historical record since the machine booted up. Similarly, "iostat -xn 1" will give you the current machine performance, once a second, while "iostat -xn" gives you the load since the machine booted up.

Further References:
tri: Tuning PHP and Apache on Unix
tri: UNIX and Web Performance by Jaqui Lynch. An updated PDF version.
tri: Sample chapter on Memory Tuning from the O'Reilly book: System Performance Tuning, 2nd Edition.

Oracle Tuning:
tri: Linux Journal has some detailed OS tuning hints. Also see Tuning without Cache-Hit Ratios (Word doc).