thelinuxvault guide

Essential Linux Commands for I/O and Storage Monitoring

In the world of Linux systems administration, monitoring storage capacity and I/O (Input/Output) performance is critical for maintaining system reliability, troubleshooting bottlenecks, and ensuring optimal performance. Whether you’re a developer, sysadmin, or DevOps engineer, understanding how to track disk usage, identify storage-hungry processes, and analyze I/O activity can mean the difference between proactive problem-solving and costly downtime. Linux offers a rich ecosystem of built-in tools to monitor storage and I/O. These commands provide granular insights into disk space, partition layouts, file system health, and real-time I/O activity—empowering you to answer questions like: *“Is the disk full?”*, *“Which process is hogging I/O?”*, or *“Why is the server slow?”* In this blog, we’ll explore the most essential Linux commands for storage and I/O monitoring. Each command is explained with its purpose, key options, practical examples, and insights to help you interpret the output. By the end, you’ll have a toolkit to diagnose storage issues and optimize I/O performance like a pro.

Table of Contents

1. Storage Monitoring Commands

Storage monitoring focuses on tracking disk capacity, partition layouts, and file system health. These commands help you answer: “How much space is left?”, “What’s using all the space?”, or “What disks/partitions are attached?”

df: Check File System Disk Space

Purpose: The df (disk free) command reports the total, used, and free disk space for mounted file systems. It’s the first tool to use when investigating “disk full” errors.

Syntax:

df [OPTIONS] [FILESYSTEM...]

Key Options:

  • -h: Human-readable output (e.g., GB, MB instead of raw blocks).
  • -i: Show inode usage (critical for systems where inodes, not space, might be exhausted).
  • -T: Display the file system type (e.g., ext4, xfs).
  • -x TYPE: Exclude file systems of type TYPE (e.g., -x tmpfs to ignore temporary filesystems).

Examples:

  1. Check all mounted file systems (human-readable):

    df -h

    Output:

    Filesystem      Size  Used Avail Use% Mounted on  
/dev/sda1       20G   12G  7.2G  62% /  
tmpfs           3.9G     0  3.9G   0% /dev/shm  
/dev/sdb1       100G   45G   55G  45% /data
    • Size: Total capacity of the file system.
    • Used/Avail: Space used and available.
    • Use%: Percentage of space used (alert if >85%).
    • Mounted on: Path where the file system is mounted.

  2. Check inode usage (common cause of “no space left” errors even with free disk space):

    df -i

    Output snippet:

    Filesystem     Inodes  IUsed  IFree IUse% Mounted on  
/dev/sda1      5242880 345000 4897880    7% /

    If IUse% is 100%, you can’t create new files even if there’s free space.

  3. Check a specific file system (e.g., /data):

    df -h /data

du: Measure Directory/File Disk Usage

Purpose: The du (disk usage) command calculates the space used by individual files or directories. Unlike df, which reports on entire file systems, du drills down into specific files/folders.

Syntax:

du [OPTIONS] [FILE/DIRECTORY...]

Key Options:

  • -h: Human-readable output.
  • -s: Summary (total size of the directory, not subdirectories).
  • -a: Include all files (not just directories).
  • -c: Show a grand total at the end.
  • --max-depth=N: Limit recursion to N levels (e.g., --max-depth=1 for top-level subdirectories).

Examples:

  1. Get total size of a directory (summary view):

    du -sh /var/log

    Output:

    4.2G    /var/log

    -s gives the total, -h makes it readable.

  2. List sizes of subdirectories (1 level deep):

    du -h --max-depth=1 /home

    Output:

    2.1G    /home/user1  
5.3G    /home/user2  
7.4G    /home

  3. Find largest files in a directory (combine with sort):

    du -ah /tmp | sort -rh | head -5
    • -a includes files, sort -rh sorts by size (reverse human-readable), head -5 shows top 5 largest.

lsblk: List Block Devices

Purpose: lsblk (list block devices) displays information about all block devices (hard drives, SSDs, USB drives, partitions) attached to the system. It’s ideal for visualizing disk layouts.

Syntax:

lsblk [OPTIONS]

Key Options:

  • -f: Show file system type and UUID (useful for mounting).
  • -m: Show permissions and ownership.
  • -o NAME,SIZE,TYPE,MOUNTPOINT: Customize output columns.

Examples:

  1. Basic block device list:

    lsblk

    Output snippet:

    NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT  
sda      8:0    0   200G  0 disk  
├─sda1   8:1    0    20G  0 part /  
├─sda2   8:2    0     2G  0 part [SWAP]  
└─sda3   8:3    0   178G  0 part /data  
sdb      8:16   0   500G  0 disk  
└─sdb1   8:17   0   500G  0 part /backup
    • NAME: Device name (e.g., sda is the first disk, sda1 its first partition).
    • SIZE: Capacity of the device/partition.
    • MOUNTPOINT: Where the partition is mounted (if applicable).

  2. Show file system types and UUIDs:

    lsblk -f

    Output snippet:

    NAME   FSTYPE  LABEL UUID                                 MOUNTPOINT  
sda1   ext4    root  123e4567-e89b-12d3-a456-426614174000 /  
sda2   swap          98765432-1234-5678-90ab-cdef01234567 [SWAP]  
sdb1   xfs     backup 5f4dcc3b-5aa7-40b3-9dfe-000000000000 /backup

blkid: Identify Block Device Attributes

Purpose: blkid (block ID) retrieves attributes like UUIDs, file system types, and labels for block devices. It’s often used in scripts or /etc/fstab to reference devices by UUID (more reliable than device names like /dev/sda1).

Syntax:

blkid [OPTIONS] [DEVICE...]

Examples:

  1. List all block devices with attributes:

    blkid

    Output snippet:

    /dev/sda1: UUID="123e4567-e89b-12d3-a456-426614174000" TYPE="ext4" PARTUUID="00000000-01"  
/dev/sda2: UUID="98765432-1234-5678-90ab-cdef01234567" TYPE="swap" PARTUUID="00000000-02"  
/dev/sdb1: UUID="5f4dcc3b-5aa7-40b3-9dfe-000000000000" TYPE="xfs" LABEL="backup"

  2. Get UUID of a specific device:

    blkid /dev/sdb1

mount/umount: Manage File System Mounts

Purpose: mount attaches a file system to a directory (mount point), and umount detaches it. These commands are essential for managing storage devices like USB drives or network shares.

Syntax (mount):

mount [OPTIONS] DEVICE MOUNT_POINT

Syntax (umount):

umount [OPTIONS] MOUNT_POINT/DEVICE

Key Options (mount):

  • -t TYPE: Specify file system type (e.g., ext4, xfs, ntfs).
  • -o OPTIONS: Mount options (e.g., ro for read-only, rw for read-write, noatime to disable access time logging).

Examples:

  1. Mount a USB drive (e.g., /dev/sdc1 to /mnt/usb):

    sudo mount /dev/sdc1 /mnt/usb

  2. Mount a file system read-only:

    sudo mount -o ro /dev/sdb1 /backup

  3. Unmount a device (by mount point or device):

    sudo umount /mnt/usb  
# OR  
sudo umount /dev/sdc1

    Note: If the device is “busy” (e.g., a process is using it), use lsof /mnt/usb to find the process and kill it first.

2. I/O Monitoring Commands

I/O monitoring focuses on tracking activity: how fast data is read/written, which processes are causing I/O, and whether I/O bottlenecks are slowing the system. These commands answer: “Is the disk I/O saturated?”, “Which process is writing 100MB/s?”, or “What’s the read latency?”

iostat: Report CPU and I/O Statistics

Purpose: iostat (I/O statistics) provides detailed metrics on CPU usage and I/O performance for storage devices. It’s ideal for identifying I/O bottlenecks (e.g., high latency, saturated throughput).

Syntax:

iostat [OPTIONS] [INTERVAL] [COUNT]
  • INTERVAL: Time (seconds) between reports.
  • COUNT: Number of reports to generate (omit for continuous output).

Key Options:

  • -x: Show extended I/O statistics (detailed metrics like latency, queue length).
  • -d: Focus on device I/O (exclude CPU stats).
  • -k/-m: Report in KB/MB per second (instead of blocks).

Examples:

  1. Basic I/O and CPU stats (run once):

    iostat

    Output snippet:

    avg-cpu:  %user   %nice %system %iowait  %steal   %idle  
           2.30    0.10    1.50    0.80    0.00   95.30  

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn  
sda               5.20        120.00        80.00     120000      80000  
sdb               1.10         10.00         5.00      10000       5000
    • tps: Transactions per second (reads + writes).
    • kB_read/s/kB_wrtn/s: Read/write throughput.
    • %iowait (CPU section): Percentage of time CPU is idle waiting for I/O (high values indicate I/O bottlenecks).

  2. Extended I/O stats (every 5 seconds):

    iostat -x 5

    Output snippet (focus on sda):

    Device            r/s     w/s     rkB/s     wkB/s   avgrq-sz  avgqu-sz     await     r_await     w_await  svctm  %util  
sda              3.20    2.00     128.00     80.00     65.00      0.05      9.50       8.00       12.00   2.10   1.09

    Key metrics:

    • await: Average time (ms) for I/O requests to complete (includes queueing + service time).
    • avgqu-sz: Average number of I/O requests in the queue (high = congestion).
    • %util: Percentage of time the device is busy ( >70% may indicate saturation).

vmstat: Virtual Memory and I/O Activity

Purpose: vmstat (virtual memory statistics) reports on memory, swap, processes, and I/O activity. It’s a quick way to check overall system health, including I/O.

Syntax:

vmstat [OPTIONS] [INTERVAL] [COUNT]

Key Columns (I/O section):

  • bi: Blocks received from disk (read, KB/s).
  • bo: Blocks sent to disk (write, KB/s).
  • wa: % of time CPU is waiting for I/O (similar to iostat’s %iowait).

Example:

vmstat 2 3  # Report every 2s, 3 times

Output snippet:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----  
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st  
 1  0      0 2048000  51200 1024000    0    0   120    80  500 1000  2  1 95  2  0  
 0  0      0 2047800  51200 1024000    0    0   110    75  490  980  2  1 96  1  0
  • bi=120: Reading 120 KB/s from disk.
  • bo=80: Writing 80 KB/s to disk.
  • wa=2: 2% of CPU time is spent waiting for I/O.

sar: Historical System Activity (Including I/O)

Purpose: sar (system activity reporter) collects and displays historical system data, including I/O, CPU, memory, and network stats. It requires the sysstat package (install with sudo apt install sysstat or sudo yum install sysstat).

Syntax:

sar [OPTIONS] [INTERVAL] [COUNT]

Key Options:

  • -b: Report I/O and transfer rate statistics.
  • -d: Report block device statistics (similar to iostat -x).
  • -f FILE: Read data from a saved file (e.g., /var/log/sysstat/sa24 for the 24th of the month).

Examples:

  1. Real-time I/O transfer stats (every 5s):

    sar -b 5

    Output snippet:

    14:30:00          tps      rtps      wtps   bread/s   bwrtn/s  
14:30:05          5.20      3.20      2.00     128.00      80.00
    • tps: Transactions per second.
    • rtps/wtps: Read/write transactions per second.
    • bread/s/bwrtn/s: Read/write throughput (blocks/s).

  2. Historical I/O data (e.g., from 2 days ago):

    sar -d -f /var/log/sysstat/sa22

iotop: Real-Time I/O Usage by Process

Purpose: iotop (I/O top) shows per-process I/O activity, making it easy to identify which process is hogging disk I/O. It’s like top but for I/O.

Syntax:

sudo iotop [OPTIONS]

Key Options:

  • -o: Show only processes actively doing I/O.
  • -b: Batch mode (non-interactive, for scripting).
  • -P: Show total I/O per process (not per thread).

Example:
Run sudo iotop -o to see active I/O processes:

Total DISK READ:         0.00 B/s | Total DISK WRITE:      102.40 K/s  
Current DISK READ:       0.00 B/s | Current DISK WRITE:     51.20 K/s  
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND  
 1234  be/4  user1      0.00 B/s   51.20 K/s  0.00 %  0.00 %  dd if=/dev/zero of=/tmp/test bs=1M count=100
  • DISK WRITE: The dd process is writing 51.2 KB/s.
  • IO>: Percentage of time the process is waiting for I/O.

lsof: List Open Files and I/O Connections

Purpose: lsof (list open files) shows all files currently open by processes, including regular files, network sockets, and pipes. It’s invaluable for finding processes holding onto files (e.g., preventing unmounting or freeing disk space).

Syntax:

lsof [OPTIONS] [FILE/PROCESS...]

Examples:

  1. Find which process is using a file (e.g., /var/log/syslog):

    lsof /var/log/syslog

    Output snippet:

    COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME  
rsyslogd  456 root    5w   REG    8,1    12345  67890 /var/log/syslog

    Here, rsyslogd (PID 456) has the file open for writing (45w = file descriptor 5, write mode).

  2. Find processes holding deleted files (causing disk space to not free up):

    lsof | grep deleted

    Output snippet:

    java      789 user1  123w   REG    8,1  1000000  12345 /tmp/bigfile (deleted)

    The java process is still writing to a deleted file. Restarting the process will free the space.

dd: Benchmark Disk I/O (With Caution!)

Purpose: dd (data duplicator) is primarily for copying data, but it can also benchmark disk read/write speeds. Caution: dd can overwrite data if misused—always double-check the if (input file) and of (output file) parameters!

Examples:

  1. Test write speed (write 1GB of zeros to a test file, bypassing cache with oflag=direct):

    dd if=/dev/zero of=/tmp/test bs=1G count=1 oflag=direct

    Output:

    1+0 records in  
1+0 records out  
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.12345 s, 506 MB/s
    • 506 MB/s is the write speed.

  2. Test read speed (read the test file, bypass cache with iflag=direct):

    dd if=/tmp/test of=/dev/null bs=1G count=1 iflag=direct

    Output:

    1+0 records in  
1+0 records out  
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.87654 s, 1.2 GB/s
    • 1.2 GB/s is the read speed.

Cleanup: Delete the test file afterward:

rm /tmp/test

Conclusion

Mastering Linux I/O and storage monitoring commands is foundational for maintaining healthy, performant systems. From checking disk space with df to diagnosing I/O hogs with iotop, these tools empower you to proactively identify issues before they cause downtime.

Start with the basics (df, du, iostat) and gradually explore advanced tools like sar or lsof. Remember: context matters—combine metrics (e.g., high %iowait in iostat + iotop to find the culprit process) for a complete picture.

References