thelinuxvault guide

From Basics to Advanced: A Comprehensive Bash Automation Tutorial

Bash (Bourne Again SHell) is more than just a command-line interface—it’s a powerful scripting language that enables users to automate repetitive tasks, manage systems, and streamline workflows. Whether you’re a developer, system administrator, or DevOps engineer, mastering Bash scripting can significantly boost your productivity and problem-solving skills. This tutorial takes you on a journey from Bash fundamentals to advanced automation techniques, with practical examples and real-world projects. By the end, you’ll be able to write robust, efficient scripts to handle everything from simple file management to complex system monitoring.

Table of Contents

  1. What is Bash?
  2. Why Automate with Bash?
  3. Bash Basics
  4. Intermediate Bash Scripting
  5. Advanced Bash Techniques
  6. Real-World Automation Projects
  7. Conclusion
  8. References

What is Bash?

Bash is a Unix shell and command language developed by Brian Fox in 1989 as a free replacement for the Bourne Shell (sh). It’s the default shell on most Linux distributions, macOS (until macOS Catalina, which switched to Zsh, but Bash remains available), and BSD systems.

  • Interactive Mode: When you open a terminal, Bash runs in interactive mode, allowing you to type commands and see results immediately (e.g., ls, cd, echo).
  • Script Mode: Bash can execute scripts—text files containing a sequence of commands—enabling automation of multi-step tasks.

Why Automate with Bash?

Automation with Bash offers several advantages:

  • Efficiency: Replace manual, repetitive tasks (e.g., backups, log rotation) with scripts that run in seconds.
  • Consistency: Eliminate human error by standardizing workflows (e.g., deploying code the same way every time).
  • Scalability: Manage hundreds of systems or files with a single script (e.g., bulk user creation).
  • Accessibility: Bash is pre-installed on most Unix-like systems, requiring no additional setup.

Bash Basics

Variables and Data Types

Variables store data for later use. Bash is dynamically typed (no need to declare types), and variable names are case-sensitive.

Declaring Variables

Use = to assign values (no spaces around =!):

name="Alice"  # String variable
age=30        # Numeric variable
is_active=true  # Boolean (treated as string in Bash)

Accessing Variables

Prefix the variable name with $:

echo "Name: $name"  # Output: Name: Alice
echo "Age: $age"    # Output: Age: 30

Environment Variables

Global variables available to all processes (e.g., PATH, HOME). Use export to make a local variable global:

export PATH="$PATH:/new/directory"  # Add a directory to PATH
echo "Home directory: $HOME"        # Output: /home/alice (varies by user)

Command Substitution

Capture the output of a command and store it in a variable using $(command) or backticks `command` (preferred: $() for readability).

Example: Get the current date and store it in a variable:

current_date=$(date +"%Y-%m-%d")  # Format: YYYY-MM-DD
echo "Today is $current_date"     # Output: Today is 2024-05-20

Example: Count files in the current directory:

file_count=$(ls | wc -l)
echo "Number of files: $file_count"

Control Structures

Control structures (conditionals, loops) dictate the flow of execution.

Conditional Statements (if-else)

Check conditions (e.g., file existence, numeric comparisons) with if, elif, and else.

Syntax:

if [ condition ]; then
  # Code to run if condition is true
elif [ another_condition ]; then
  # Code if first condition fails, second succeeds
else
  # Code if all conditions fail
fi

Common Conditions:

  • -f file: True if file exists and is a regular file.
  • -d dir: True if dir exists and is a directory.
  • -z string: True if string is empty.
  • $a -eq $b: Numeric equality (use -ne for not equal, -gt for greater than, etc.).

Example: Check if a file exists:

filename="data.txt"
if [ -f "$filename" ]; then
  echo "$filename exists."
else
  echo "$filename not found. Creating it..."
  touch "$filename"  # Create the file if missing
fi

Case Statements

Use case for multi-condition checks (cleaner than nested if-else):

read -p "Enter a number (1-3): " num

case $num in
  1) echo "One" ;;
  2) echo "Two" ;;
  3) echo "Three" ;;
  *) echo "Invalid number" ;;  # Default case
esac

Loops

Repeat commands until a condition is met.

For Loops: Iterate over a list (files, numbers, strings):

# Iterate over strings
for fruit in apple banana cherry; do
  echo "Fruit: $fruit"
done

# Iterate over numbers (1 to 5)
for i in {1..5}; do
  echo "Count: $i"
done

# Iterate over files in a directory
for file in /home/alice/documents/*.txt; do
  echo "Processing $file"
done

While Loops: Run until a condition is false:

count=1
while [ $count -le 5 ]; do
  echo "Count: $count"
  count=$((count + 1))  # Increment count (Bash arithmetic)
done

Until Loops: Run until a condition is true (opposite of while):

count=5
until [ $count -lt 1 ]; do
  echo "Countdown: $count"
  count=$((count - 1))
done

Functions

Functions group reusable code into modular blocks.

Defining Functions

greet() {
  local name=$1  # Local variable (only accessible in the function)
  echo "Hello, $name!"
}
  • $1, $2, etc., are positional parameters (arguments passed to the function).
  • local restricts the variable to the function (avoids global scope pollution).

Calling Functions

greet "Bob"  # Output: Hello, Bob!

Return Values

Bash functions return exit codes (0 = success, non-zero = error), not values. Use return to set the exit code:

add() {
  local sum=$(( $1 + $2 ))
  echo $sum  # "Return" value via output
}

result=$(add 5 3)  # Capture output with command substitution
echo "5 + 3 = $result"  # Output: 5 + 3 = 8

Intermediate Bash Scripting

Input/Output Redirection

Control where command input comes from and output goes (files, pipes, etc.).

Redirect Output (>, >>)

  • >: Overwrite a file with output.
  • >>: Append output to a file.

Example: Save ls output to files.txt:

ls -l > files.txt  # Overwrite existing content
echo "New line" >> files.txt  # Append to files.txt

Redirect Input (<)

Read input from a file instead of the terminal:

sort < unsorted.txt > sorted.txt  # Sort unsorted.txt and save to sorted.txt

Pipes (|)

Send the output of one command as input to another:

ps aux | grep "bash"  # List processes and filter for "bash"
cat access.log | grep "ERROR" | wc -l  # Count ERROR entries in a log

Process Management

Control running processes (background jobs, killing tasks, etc.).

Run in Background (&)

Add & to run a command in the background (free up the terminal):

long_running_script.sh &  # Run script in background

Manage Jobs

  • jobs: List background jobs.
  • fg %1: Bring job 1 to the foreground.
  • bg %1: Resume job 1 in the background.
  • kill %1: Terminate job 1.

Check and Kill Processes

Use ps to list processes and kill to terminate them:

ps aux | grep "python"  # Find Python processes
kill 1234  # Kill process with ID 1234 (use -9 for force kill: kill -9 1234)

Error Handling

Prevent scripts from failing silently with error-checking tools.

set Options

Add these at the top of scripts to enforce strictness:

set -e  # Exit on any command failure
set -u  # Treat unset variables as errors
set -o pipefail  # Exit if any command in a pipe fails

trap for Cleanup

Run commands on script exit (e.g., delete temporary files):

temp_file=$(mktemp)  # Create a temporary file

# Clean up temp_file when script exits (success or failure)
trap 'rm -f "$temp_file"' EXIT

# ... script logic that uses $temp_file ...

Advanced Bash Techniques

Regular Expressions

Bash works with tools like grep, sed, and awk to match patterns in text.

grep: Search for Patterns

Use -E for extended regex (supports +, ?, ()):

# Find lines with "ERROR" in a log file
grep "ERROR" app.log

# Find valid email addresses (simplified regex)
grep -E '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}$' emails.txt

sed: Stream Editor

Modify text in-place (use -i for editing files):

# Replace "old" with "new" in a file
sed -i 's/old/new/g' document.txt  # g = global replace (all occurrences)

Arrays

Store multiple values in a single variable (Bash 4+ supports associative arrays for key-value pairs).

Indexed Arrays

fruits=("apple" "banana" "cherry")  # Declare array

echo ${fruits[0]}  # Access first element (0-based index): apple
echo ${fruits[@]}  # All elements: apple banana cherry
echo ${#fruits[@]}  # Number of elements: 3

Loop Through Arrays

for fruit in "${fruits[@]}"; do
  echo "I like $fruit"
done

Debugging Bash Scripts

Debugging tools help identify issues in scripts:

  • set -x: Print commands and their arguments as they execute (trace mode).
  • set -v: Print input lines as they are read (verbose mode).
  • echo: Add debug messages to track variables: 
    name="Alice"
echo "Debug: name is $name"  # Output: Debug: name is Alice

Real-World Automation Projects

Project 1: Automated Backup Script

A script to back up a directory to a compressed file (e.g., tar.gz).

Step 1: Define Variables

source_dir="/home/alice/documents"
backup_dir="/mnt/backup"
timestamp=$(date +"%Y%m%d_%H%M%S")  # Unique filename
backup_file="$backup_dir/backup_$timestamp.tar.gz"

Step 2: Check Dependencies

Ensure tar is installed and backup_dir exists:

if ! command -v tar &> /dev/null; then
  echo "Error: tar is not installed."
  exit 1
fi

if [ ! -d "$backup_dir" ]; then
  echo "Creating backup directory: $backup_dir"
  mkdir -p "$backup_dir"
fi

Step 3: Run Backup

echo "Backing up $source_dir to $backup_file..."
tar -czf "$backup_file" "$source_dir"

# Check if backup succeeded
if [ $? -eq 0 ]; then  # $? = exit code of last command (0 = success)
  echo "Backup completed successfully."
else
  echo "Backup failed!"
  exit 1
fi

Project 2: Log File Parser

Parse a web server log (e.g., Nginx) to count 404 errors and list top IPs.

Sample Log Format

192.168.1.1 - - [20/May/2024:12:00:00 +0000] "GET /page HTTP/1.1" 404 123
10.0.0.2 - - [20/May/2024:12:01:00 +0000] "GET /home HTTP/1.1" 200 456

Script Code

log_file="/var/log/nginx/access.log"

# Count 404 errors
echo "404 Errors: $(grep " 404 " "$log_file" | wc -l)"

# Top 5 IPs with most requests
echo -e "\nTop 5 IPs:"
awk '{print $1}' "$log_file" | sort | uniq -c | sort -nr | head -5

Project 3: System Monitoring Tool

Monitor CPU, memory, and disk usage; alert if thresholds are exceeded.

Script Code

#!/bin/bash
set -euo pipefail

# Thresholds (adjust as needed)
CPU_THRESHOLD=80  # %
MEM_THRESHOLD=80  # %
DISK_THRESHOLD=85  # %

# Check CPU usage
cpu_usage=$(top -bn1 | grep "Cpu(s)" | awk '{print $2 + $4}')  # User + System CPU
if (( $(echo "$cpu_usage > $CPU_THRESHOLD" | bc -l) )); then
  echo "ALERT: High CPU usage - $cpu_usage%"
fi

# Check memory usage
mem_usage=$(free | grep Mem | awk '{print $3/$2 * 100.0}')
if (( $(echo "$mem_usage > $MEM_THRESHOLD" | bc -l) )); then
  echo "ALERT: High Memory usage - $mem_usage%"
fi

# Check disk usage
disk_usage=$(df -h / | grep / | awk '{print $5}' | sed 's/%//')
if [ "$disk_usage" -gt "$DISK_THRESHOLD" ]; then
  echo "ALERT: High Disk usage - $disk_usage%"
fi

Conclusion

From variables and loops to advanced regex and system monitoring, this tutorial has covered the full spectrum of Bash automation. The key to mastery is practice: experiment with scripts, break them, and fix them. Start small (e.g., a file organizer) and gradually tackle complex projects (e.g., deployment pipelines).

Bash is a timeless tool—investing in it will pay dividends in efficiency and versatility across your technical career.

References