Using iotop to check I/O and swap

The iotop utility provides an easy-to-use interface for monitoring swap and disk I/O on a per-process basis.

Watching the disk

Sometimes you get more disk activity on your system than you would like. Maybe you saw the problem by running iostat, or you've noticed that the system is swapping heavily, or you got an email from support pointing out the problem. In cases like that it's useful to know which process is causing all the hubbub.

Enter iotop. It works like the traditional top command but it tracks device I/O (input/output) on a per-process basis. That can help you zero in quickly on which process is responsible for all the I/O and troubleshoot from there.

Prerequisites

The iotop command requires a kernel version 2.6.20 or higher and python 2.5 or higher.

CentOS and Red Hat Enterprise Linux 5.5 (current at the time of this writing) use an older version of python and so cannot run iotop on a standard installation. They can, however, run dstat, which performs a similar function to iotop.

Check your kernel version by running "uname -r":

$ uname -r
2.6.35.1-rscloud

And check your python version by running "python -V":

$ python -V
Python 2.5.2

Install

You can install iotop through your distribution's package manager (if it's available in the repository) or you can download the source package from the project's website.

From package manager

To install iotop using your package manager pick the appropriate command from the list below (aptitude for Ubuntu/Debian, yum for Fedora, etc.):

sudo aptitude install iotop
sudo yum install iotop
sudo emerge iotop
sudo pacman -Sy iotop

If there is no iotop package in your repository but your system meets the prerequisites, you can install from source instead.

From source

The latest version of iotop is available from the project's web site. Look for a link to the latest source package for iotop and download it with wget:

wget http://guichaz.free.fr/iotop/files/iotop-0.4.1.tar.bz2

Once you've downloaded the tarball, unpack it:

tar -xjvf iotop-0.4.1.tar.bz2

From inside that directory you can either run iotop where it is (by running "./iotop.py") or you can run the installer to put iotop in /usr/bin:

sudo ./setup.py install

Basic usage

At its simplest you can run iotop with no arguments:

iotop

You should see a list of processes along with details about their current I/O use:

The column titles should be pretty self-explanatory, but there are two particular stats to note.

IO

The "IO" column lists total I/O for each process (which includes disk use and swap). Results are sorted by I/O by default when iotop is launched.

SwapIn

The "SwapIn" column lists swap activity for each process.

Runtime commands

Using the right and left arrow keys while viewing iotop will change the column used to sort the results.

If you want the processes sorted in reverse order hit "r" while viewing iotop.

Hitting "o" will filter the list so it will only show processes that are actively doing I/O. Typing "o" again will return to a list of all processes.

Script usage

You can use iotop in "batch mode" to have it print just one set of results instead of launching a constantly-updating interface. Batch mode can be useful when you want a script that logs regular I/O usage to a file.

Sending the "-b" flag to iotop will tell it to run in batch mode. You'll usually want to combine that with some other flags:

The "-o" flag causes iotop to report only process with active I/O.

The "-t" flag will add a timestamp to the results.

The "--iter=#" option will limit the number of samples iotop returns when run.

The "-q" flag will prevent iotop from displaying column headers after its first run. You can prevent headers from being printed at all with "-qq". To prevent both headers and I/O summaries from being printed use "-qqq".

For example, the result of:

iotop -bto --iter=1

Would look like:

Total DISK READ: 0.00 B/s | Total DISK WRITE: 27.64 K/s
    TIME  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
17:08:25   177 be/4 root        0.00 B/s    7.90 K/s  0.00 %  0.05 % [kjournald]

A sample cron script

You can use cron task scheduling to get iotop to run every minute and log any I/O activity it detects. Open this file for editing (with sudo):

/etc/cron.d/iotop

And put the following into that file:

#  Run iotop and log any detected activity.
* * * * * root iotop -botqqq --iter=3 >> /var/log/iotop

Save the file. That cron.d entry will cause iotop to run every minute, logging any I/O activity it finds to "/var/log/iotop". It takes three samples when it runs (five seconds apart), doesn't log headers or summary information, and only logs processes with measured I/O activity.

After a minute take a look in /var/log. If the script ran as expected there should now be an "iostat" log there (though it may be an empty file if it didn't find any I/O to log).

Tweak the command to your liking. If you want a different number of samples per minute change the "--iter" value. If you want I/O summaries printed when the script runs change "qqq" to "qq".

Rotate the created log

If you plan to leave the iotop logger running (and aren't just going to run it for a day or so to check on a burst of disk activity) you'll want to rotate its log occasionally to keep it from getting too big.

To that end, create a file for editing at "/etc/logrotate.d/iotop" (again using sudo).

Put the following into the file:

/var/log/iotop {
  rotate 5
  weekly
  compress
  missingok
  notifempty
}

For more details on what that logrotate config will do you can look through our articles on logrotate. For example, you might change the frequency of rotations from "weekly" to "daily" if you find you're logging a lot of disk I/O. You could then increase the number of archived logs it keeps accordingly.

Similar commands

Another program that can be used to check I/O is dstat. While its presentation isn't necessarily as attractive as iotop's, it allows for more flexibility in what can be written to a log when used in a script. It also runs on older versions of python than iotop so it will run on more distributions.

The "sysstat" package includes several commands useful for gathering system usage statistics, and newer versions of the package include a program named "pidstat". When run with the "-d" option pidstat will display disk I/O information on a per-process basis.

Summary

As "top" is to processor and memory use, so "iotop" is to disk I/O and swap use. The ability to see I/O on a per-process basis is relatively new to Linux (thus the requirement for a newer kernel to run iotop), but it was a very welcome addition. Iotop can be very useful when trying to track down which process is using swap memory or is causing an excessive amount of disk activity.

• -- Jered