A Comprehensive Guide to rdiff-backup: Efficient Incremental Backups in Linux

Table of Contents#

What is rdiff-backup?
Key Features and Advantages
Installation
Basic Syntax and Structure
Common Use Cases with Examples
Advanced Features
Restoring Data
Best Practices
Troubleshooting Common Issues
Reference

What is rdiff-backup?#

rdiff-backup is a Python-based backup utility that maintains a mirror of a file or directory on another directory, typically on a different filesystem or machine. What sets it apart is its ability to store historical reverse diffs, allowing you to recover files as they existed at various points in time. It uses the librsync library to efficiently compute and store differences between file versions.

Key Features and Advantages#

Incremental backups with full restore capability: Only changed data is transferred after the initial backup
Network transparency: Can backup to/from remote systems using SSH
Preservation of file metadata: Maintains permissions, ownership, timestamps, and extended attributes
Efficient storage: Reverse diffs are typically much smaller than full file copies
Point-in-time recovery: Restore files as they existed at any backup interval
No proprietary formats: Backup data is stored in regular filesystem format

Installation#

Ubuntu/Debian#

sudo apt update
sudo apt install rdiff-backup

CentOS/RHEL/Fedora#

# CentOS/RHEL (EPEL repository required)
sudo yum install epel-release
sudo yum install rdiff-backup
 
# Fedora
sudo dnf install rdiff-backup

From Source#

# Install dependencies
sudo apt install librsync-dev python3-dev
 
# Download and compile
wget https://github.com/rdiff-backup/rdiff-backup/releases/download/v2.0.5/rdiff-backup-2.0.5.tar.gz
tar -xzf rdiff-backup-2.0.5.tar.gz
cd rdiff-backup-2.0.5
python setup.py install

Basic Syntax and Structure#

The basic syntax of rdiff-backup is:

rdiff-backup [options] source_directory destination_directory

Common options:

--verbosity LEVEL: Set output verbosity (0-8)
--terminal-verbosity LEVEL: Set terminal output level
--exclude PATTERN: Exclude files matching pattern
--include PATTERN: Include files matching pattern
--exclude-filelist FILE: Read exclude patterns from file
--remote-schema CMD: Specify remote command schema
--force: Force operation despite warnings
--version: Display version information

Common Use Cases with Examples#

1. Local Backup#

# Basic local backup
rdiff-backup /home/user/documents /backup/documents
 
# With increased verbosity
rdiff-backup --verbosity 3 /home/user/documents /backup/documents
 
# Excluding specific patterns
rdiff-backup --exclude "*.tmp" --exclude "cache/*" /home/user /backup/home

2. Remote Backup over SSH#

# Backup local directory to remote server
rdiff-backup /home/user server.example.com::/backup/user-home
 
# Backup remote directory to local storage
rdiff-backup server.example.com::/var/www /backup/website
 
# Using custom SSH options
rdiff-backup --remote-schema "ssh -p 2222 -i /home/user/.ssh/backup_key %s rdiff-backup --server" \
    /home/user [email protected]::/backup/user

3. Automated Backups with Cron#

Create a backup script (/usr/local/bin/backup-script.sh):

#!/bin/bash
# Daily backup script
 
LOG_FILE="/var/log/backup.log"
SOURCE_DIR="/home"
DEST_DIR="/backup/home"
REMOTE_DEST="backup-server::/backup/company-home"
 
echo "$(date): Starting backup" >> $LOG_FILE
 
# Local backup
rdiff-backup --exclude-filelist /etc/backup-excludes $SOURCE_DIR $DEST_DIR 2>> $LOG_FILE
 
# Remote backup
rdiff-backup --exclude-filelist /etc/backup-excludes $SOURCE_DIR $REMOTE_DEST 2>> $LOG_FILE
 
# Remove backups older than 30 days
rdiff-backup --remove-older-than 30D --force $DEST_DIR 2>> $LOG_FILE
 
echo "$(date): Backup completed" >> $LOG_FILE

Add to crontab:

# Run daily at 2 AM
0 2 * * * /usr/local/bin/backup-script.sh

4. Including and Excluding Patterns#

Create an exclude file (/etc/backup-excludes):

# Exclude temporary files
*.tmp
*.swp
*.cache

# Exclude specific directories
home/*/downloads/
home/*/.cache/
home/*/.local/share/Trash/

# But include important hidden files
+ home/*/.bashrc
+ home/*/.ssh/

Use the exclude file:

rdiff-backup --exclude-filelist /etc/backup-excludes /home /backup/home

Advanced Features#

1. Setting Backup Intervals#

# Backup with specific intervals
rdiff-backup --print-statistics /home /backup/home
 
# Force a specific backup time
rdiff-backup --current-time 20240115120000 /home /backup/home

2. Backup Verification#

# Compare source and backup
rdiff-backup --verify /home /backup/home
 
# Compare at a specific time
rdiff-backup --verify-at 20240115T120000 /backup/home /home

3. Bandwidth Limiting#

# Limit bandwidth to 1Mbps during backup
rdiff-backup --remote-schema "ssh -o ConnectTimeout=60 -o BatchMode=yes \
    -o StrictHostKeyChecking=yes %s 'rdiff-backup --bandwidth-limit 1000'" \
    /home user@backup-server::/backup/home

Restoring Data#

1. Restore Latest Version#

# Restore entire directory
rdiff-backup -r now /backup/home/documents /home/user/documents
 
# Restore single file
rdiff-backup -r now /backup/home/documents/file.txt /home/user/documents/file.txt

2. Restore from Specific Date#

# Restore from specific time
rdiff-backup -r 2024-01-15T12:00:00 /backup/home/documents /home/user/documents_old
 
# Restore from 3 days ago
rdiff-backup -r 3D /backup/home/documents /home/user/documents_3days_ago

3. List Available Backups#

# List backup increments
rdiff-backup --list-increments /backup/home
 
# List increments in JSON format (version 2.x)
rdiff-backup --list-increments --format json /backup/home

4. Partial Restore#

# Restore only specific files
rdiff-backup -r 7D --include "/home/user/documents/important/*" /backup/home /restore/location

Best Practices#

1. Backup Strategy#

#!/bin/bash
# Comprehensive backup strategy
 
# Daily incremental backups
rdiff-backup /data /backup/daily
 
# Weekly full verification
if [ $(date +%u) -eq 7 ]; then  # Sunday
    rdiff-backup --verify /backup/daily /data
fi
 
# Monthly archive maintenance
if [ $(date +%d) -eq 1 ]; then  # First day of month
    # Keep 6 monthly backups, 8 weekly, 31 daily
    rdiff-backup --remove-older-than 6M --force /backup/daily
    rdiff-backup --remove-older-than 8W --force /backup/daily
    rdiff-backup --remove-older-than 31D --force /backup/daily
fi

2. Security Considerations#

# Use dedicated backup user with restricted permissions
# Create backup user on remote system
sudo useradd -r -s /bin/bash -d /backup backupuser
sudo passwd backupuser
 
# Set up SSH key authentication
ssh-keygen -t ed25519 -f ~/.ssh/backup_key
ssh-copy-id -i ~/.ssh/backup_key.pub backupuser@backup-server
 
# Secure SSH configuration on client (~/.ssh/config)
Host backup-server
    HostName backup.example.com
    User backupuser
    IdentityFile ~/.ssh/backup_key
    Port 2222
    ServerAliveInterval 60
    ServerAliveCountMax 3

3. Monitoring and Logging#

#!/bin/bash
# Backup script with comprehensive logging
 
LOG_FILE="/var/log/backup-$(date +%Y%m%d).log"
ERROR_LOG="/var/log/backup-errors-$(date +%Y%m%d).log"
 
{
    echo "=== Starting backup at $(date) ==="
    
    # Test connection
    if ! ssh -o ConnectTimeout=10 backup-server true; then
        echo "ERROR: Cannot connect to backup server"
        exit 1
    fi
    
    # Perform backup
    rdiff-backup --verbosity 3 \
        --exclude-filelist /etc/backup-excludes \
        /data backup-server::/backup/data
    
    BACKUP_EXIT=$?
    
    if [ $BACKUP_EXIT -eq 0 ]; then
        echo "Backup completed successfully"
        # Send success notification
        echo "Backup successful: $(date)" | mail -s "Backup Status" [email protected]
    else
        echo "Backup failed with exit code: $BACKUP_EXIT"
        echo "Backup failed: $(date)" | mail -s "Backup FAILED" [email protected]
        exit $BACKUP_EXIT
    fi
    
    echo "=== Backup finished at $(date) ==="
} >> $LOG_FILE 2>> $ERROR_LOG

4. Performance Optimization#

# Use these options for large backups
rdiff-backup \
    --no-compression \           # If source is already compressed
    --parsable-output \          # Machine-readable output
    --terminal-verbosity 3 \     # Balanced output
    /large-dataset backup-server::/backup/large-data
 
# For many small files, consider increasing file opening limits
ulimit -n 10000
rdiff-backup /home /backup/home

Troubleshooting Common Issues#

1. Permission Problems#

# Check and fix backup directory permissions
sudo chown -R backupuser:backupuser /backup
sudo chmod -R 700 /backup
 
# Ensure read access to source files
sudo rdiff-backup --exclude '/home/*/.gvfs' /home /backup/home

2. SSH Connection Issues#

# Test SSH connection separately
ssh -v -i /path/to/key backup-user@server
 
# Use specific SSH options in rdiff-backup
rdiff-backup --remote-schema "ssh -o ConnectTimeout=30 -o ServerAliveInterval=60 %s rdiff-backup-server" \
    /source backup-user@server::/backup

3. Disk Space Management#

# Check backup size
du -sh /backup/home
 
# List and manage increments
rdiff-backup --list-increments /backup/home
 
# Remove old increments strategically
# Keep: 7 daily, 4 weekly, 6 monthly
rdiff-backup --remove-older-than 6M --force /backup/home

4. Corrupted Backups#

# Check backup integrity
rdiff-backup --verify /backup/home /home
 
# Recover by starting fresh (as last resort)
mv /backup/home /backup/home.corrupted.$(date +%Y%m%d)
rdiff-backup /home /backup/home

Reference#

Common Time Formats for Restore#

now - Current time (latest backup)
2024-01-15T12:00:00 - Specific timestamp
7D - 7 days ago
2W - 2 weeks ago
1M - 1 month ago
1B - 1 backup session ago

Exit Codes#

0 - Success
1 - Syntax or usage error
2 - File I/O error
8 - Memory error
16 - Network error

Useful Companion Commands#

# Check disk space
df -h /backup
 
# Monitor backup progress
watch -n 5 'du -sh /backup/home'
 
# Check backup integrity periodically
rdiff-backup --verify /backup/home /home
 
# Generate backup reports
rdiff-backup --list-increment-sizes /backup/home

Official Resources#

By mastering rdiff-backup, you'll have a robust, efficient backup solution that provides both immediate access to current files and comprehensive version history. Its combination of simplicity and power makes it an excellent choice for both personal and enterprise backup needs.