An In-Depth Look at ZFS Backups on Linux

Understanding ZFS Fundamentals for Backups
- 1.1 What is ZFS?
- 1.2 Key Concepts: Datasets, Snapshots, and Clones
- 1.3 Why ZFS Simplifies Backups
ZFS Backup Strategies
- 2.1 Local Snapshots: Point-in-Time Recovery
- 2.2 Remote Replication: Offsite Redundancy
- 2.3 Incremental Backups: Efficiency with zfs send/zfs receive
Essential ZFS Backup Tools
- 3.1 Native Tools: zfs snapshot, zfs send, zfs receive
- 3.2 sanoid/syncoid: Automated Snapshots and Replication
- 3.3 zrepl: Advanced Replication for Enterprise
- 3.4 zfsnap: Lightweight Snapshot Management
Step-by-Step Guides
- 4.1 Creating and Managing Local Snapshots
- 4.2 Basic Remote Replication with zfs send/zfs receive
- 4.3 Automating Backups with sanoid/syncoid
Best Practices for ZFS Backups
- 5.1 Follow the 3-2-1 Rule
- 5.2 Test Restores Regularly
- 5.3 Define Snapshot Retention Policies
- 5.4 Encrypt Backups
- 5.5 Monitor Backup Health
Troubleshooting Common ZFS Backup Issues
- 6.1 Broken Incremental Streams
- 6.2 Insufficient Disk Space
- 6.3 Network Failures During Replication
Advanced Topics
- 7.1 Compression and Deduplication in Backups
- 7.2 Cloud Integration: Sending ZFS Snapshots to S3/Cloud Storage
- 7.3 Hybrid Backups: Combining ZFS with rsync or Borg
Conclusion
References

1. Understanding ZFS Fundamentals for Backups

1.1 What is ZFS?

ZFS is a combined file system and volume manager. Unlike traditional Linux file systems (e.g., ext4 or XFS), ZFS manages storage at the “pool” level: you create a zpool from physical disks, then carve out datasets (file systems) or volumes (block devices) from the pool. ZFS ensures data integrity with checksums, supports RAID-like configurations (RAID-Z1/2/3, mirroring), and includes advanced features like snapshots, clones, and deduplication.

1.2 Key Concepts: Datasets, Snapshots, and Clones

To master ZFS backups, you need to understand three core concepts:

Datasets: The basic unit of storage in ZFS. A dataset is a file system (e.g., tank/docs or pool/media) with its own properties (compression, encryption, etc.).
Snapshots: Read-only point-in-time copies of a dataset. ZFS snapshots are “copy-on-write” (CoW), meaning they only store changes made after the snapshot was created. This makes them space-efficient and fast to create.
Clones: Writable copies of a snapshot. Clones are useful for testing restores without modifying the original dataset.

1.3 Why ZFS Simplifies Backups

ZFS’s design makes backups more efficient than traditional tools (e.g., rsync or tar):

Atomic Snapshots: Snapshots capture the dataset state instantly, even if files are being written to. No more “inconsistent” backups.
Incremental Sends: zfs send can generate incremental streams (only changes between two snapshots), reducing bandwidth and storage usage.
Integrity Checks: Backups inherit ZFS’s checksumming, ensuring corrupted data is detected during replication.

2. ZFS Backup Strategies

2.1 Local Snapshots: Point-in-Time Recovery

Use Case: Quick recovery from accidental deletions or file corruption.
How it works: Create regular snapshots of datasets on the same zpool. For example:

zfs snapshot tank/docs@20240520  # Snapshot "docs" dataset with timestamp

Pros: Fast, no extra hardware needed.
Cons: Vulnerable to zpool failure (e.g., if the drive dies, snapshots are lost).

2.2 Remote Replication: Offsite Redundancy

Use Case: Protecting against disasters (theft, fire) or zpool failures.
How it works: Send snapshots to a remote ZFS system (e.g., a secondary server or external drive) using zfs send/zfs receive.
Example:

# Send a full snapshot to remote server "backup-server"
zfs send tank/docs@20240520 | ssh backup-server "zfs receive backup-pool/docs"

Pros: Offsite protection, isolated from the primary zpool.
Cons: Requires network access or external storage.

2.3 Incremental Backups: Efficiency

Use Case: Reducing backup time/bandwidth for large datasets.
How it works: Send only changes between two snapshots (e.g., from @20240520 to @20240521).
Example:

# Send incremental changes between two snapshots
zfs send -i tank/docs@20240520 tank/docs@20240521 | ssh backup-server "zfs receive backup-pool/docs"

Pros: Saves space/bandwidth vs. full backups.
Cons: Requires a “base” snapshot on the remote system.

3. Essential ZFS Backup Tools

3.1 Native Tools: `zfs snapshot`, `zfs send`, `zfs receive`

ZFS includes built-in commands for backups:

zfs snapshot: Create a snapshot.

zfs snapshot -r tank@daily  # -r = recursive (snapshot all child datasets)

zfs send: Generate a stream of snapshot data (full or incremental).

zfs send tank/docs@20240520 > /backup/docs_20240520.zfs  # Save to file

zfs receive: Restore a snapshot stream to a dataset.

zfs receive backup-pool/docs < /backup/docs_20240520.zfs  # Restore from file

3.2 sanoid/syncoid: Automated Snapshots and Replication

sanoid is a popular tool for automating ZFS snapshots and managing retention policies. Its companion, syncoid, handles replication to remote systems.

Setup:

Install sanoid (Debian/Ubuntu):

apt install sanoid  # Or compile from GitHub for latest version

Configure /etc/sanoid/sanoid.conf to define snapshot schedules:

[tank/docs]
use_template = production  # Reuse a predefined template

[template_production]
hourly = 6  # Keep 6 hourly snapshots
daily = 30  # Keep 30 daily snapshots
weekly = 52  # Keep 52 weekly snapshots
auto_prune = yes  # Delete old snapshots automatically

Run sanoid manually or via cron:

sanoid --cron  # Creates/prunes snapshots per config

Replication with syncoid:

# Sync "tank/docs" to remote server "backup-server"
syncoid tank/docs backup-server:backup-pool/docs --compress=zstd

3.3 zrepl: Advanced Replication for Enterprise

zrepl is a modern replication tool designed for large-scale deployments. It supports:

Scheduled replication (via daemon).
Resumable transfers (no need to restart failed sends).
Bandwidth limiting and compression.

Example Config (/etc/zrepl/zrepl.yml):

jobs:
- name: backup_docs
  type: push
  source:
    type: zfs
    dataset: tank/docs
  target:
    type: ssh
    host: backup-server
    user: root
    dataset: backup-pool/docs
  snapshotting:
    type: periodic
    prefix: zrepl_
    interval: 1h

3.4 zfsnap: Lightweight Snapshot Management

zfsnap is a minimal tool for creating timestamped snapshots and pruning old ones. It’s ideal for users who want simplicity over advanced features.

Example:

# Create a snapshot with 30-day retention
zfsnap snapshot -a 30d tank/docs

# Prune snapshots older than their retention period
zfsnap destroy -r tank  # -r = recursive

4. Step-by-Step Guides

4.1 Creating and Managing Local Snapshots

Goal: Automate daily snapshots of tank/docs with 7-day retention.

Create a bash script (/usr/local/bin/snapdocs.sh):

#!/bin/bash
TIMESTAMP=$(date +%Y%m%d)
zfs snapshot tank/docs@$TIMESTAMP
# Prune snapshots older than 7 days
zfs list -t snapshot -o name -s creation | grep tank/docs@ | head -n -7 | xargs -I {} zfs destroy {}

Make it executable:
```
chmod +x /usr/local/bin/snapdocs.sh
```

Add to cron (daily at 2 AM):

crontab -e
# Add line: 0 2 * * * /usr/local/bin/snapdocs.sh

4.2 Basic Remote Replication with `zfs send`/`zfs receive`

Goal: Replicate tank/docs to a remote server over SSH.

First, send a full snapshot (run on the source server):

# Create initial snapshot
zfs snapshot tank/docs@initial

# Send to remote (ensure SSH key auth is set up)
zfs send tank/docs@initial | ssh backup-server "zfs receive backup-pool/docs"

Subsequent incremental sends:

# Create a new snapshot
zfs snapshot tank/docs@20240521

# Send only changes since "initial" snapshot
zfs send -i tank/docs@initial tank/docs@20240521 | ssh backup-server "zfs receive backup-pool/docs"

4.3 Automating Backups with sanoid/syncoid

Goal: Set up hourly snapshots and real-time replication of tank/media to a USB drive.

Configure sanoid to snapshot tank/media hourly:
Edit /etc/sanoid/sanoid.conf:

[tank/media]
use_template = hourly

[template_hourly]
hourly = 24  # Keep 24 hourly snapshots
daily = 0    # No daily snapshots
auto_prune = yes

Replicate to USB drive (mounted at /mnt/backup-drive):

# Add to cron (run hourly)
0 * * * * syncoid tank/media /mnt/backup-drive/backup-pool/media --compress=lz4

5. Best Practices for ZFS Backups

5.1 Follow the 3-2-1 Rule

3 copies of data: Original + 2 backups.
2 different media: e.g., local zpool + remote server + cloud.
1 offsite copy: Protect against physical disasters.

5.2 Test Restores Regularly

A backup is useless if you can’t restore from it! Test monthly:

# Clone a snapshot to test
zfs clone backup-pool/docs@20240520 tank/test_restore
# Verify files, then clean up
zfs destroy tank/test_restore

5.3 Define Snapshot Retention Policies

Avoid filling your zpool with old snapshots. Use tools like sanoid or zfsnap to auto-prune:

Hourly snapshots: Keep 6–24 (for same-day recovery).
Daily snapshots: Keep 30–90 (for weekly/monthly recovery).
Monthly snapshots: Keep 12–24 (for long-term retention).

5.4 Encrypt Backups

Encrypt sensitive datasets at rest using ZFS native encryption:

# Create an encrypted dataset
zfs create -o encryption=on -o keyformat=passphrase tank/encrypted_docs
# Backups will inherit encryption, so ensure keys are stored securely!

5.5 Monitor Backup Health

Use zpool status to check for errors in backup zpools.
Log replication jobs (e.g., syncoid output to /var/log/syncoid.log).
Set up alerts (e.g., with Prometheus + Grafana) for failed backups.

6. Troubleshooting Common ZFS Backup Issues

6.1 Broken Incremental Streams

Issue: zfs receive fails with invalid backup stream.
Fix: Ensure the remote dataset has the base snapshot. If not, send a full snapshot:

# Force a full send (overwrites remote dataset!)
zfs send -R tank/docs@latest | ssh backup-server "zfs receive -F backup-pool/docs"

6.2 Insufficient Disk Space

Issue: zfs receive runs out of space.
Fix:

Prune old snapshots on the remote zpool: zfs list -t snapshot backup-pool/docs → zfs destroy <snapshot>.
Enable compression on the remote dataset: zfs set compression=zstd backup-pool/docs.

6.3 Network Failures During Replication

Issue: syncoid or zfs send stalls due to network drops.
Fix:

Use syncoid --resume (if supported) or zrepl (resumable transfers).
Add --compress=zstd to reduce bandwidth usage.

7. Advanced Topics

7.1 Compression and Deduplication in Backups

Compression: Always enable compression on backup datasets (e.g., zstd or lz4) to reduce storage usage:
```
zfs set compression=zstd backup-pool/docs
```
Deduplication: Use with caution! Deduplication requires significant RAM (1GB per 1TB of storage) and may slow down writes. Enable only if you have many duplicate files:
```
zfs set dedup=on backup-pool  # Apply to entire pool
```

7.2 Cloud Integration: Sending ZFS Snapshots to S3/Cloud Storage

Tools like zfs-cloud-backup or zfs-s3 let you send snapshots to S3-compatible storage:

# Example with zfs-s3 (send snapshot to S3)
zfs-s3 send tank/docs@20240520 --bucket my-backups --region us-west-2

7.3 Hybrid Backups: Combining ZFS with rsync or Borg

For non-ZFS remote systems, use zfs send to a file, then sync with rsync or Borg:

# Send snapshot to a file, then rsync to remote
zfs send tank/docs@20240520 > /tmp/snap.zfs
rsync -av /tmp/snap.zfs backup-server:/backups/

8. Conclusion

ZFS transforms backups from a chore into a streamlined, reliable process. With features like atomic snapshots, incremental sends, and tools like sanoid/syncoid, you can protect your data against everything from accidental deletions to disasters. By following best practices (3-2-1 rule, testing restores, encryption) and leveraging advanced features (compression, cloud integration), you’ll ensure your data is safe, accessible, and future-proof.

Table of Contents