thelinuxvault blog

How to Scrape Web Pages from the Command Line Using htmlq

In an era where data is abundant, web scraping has become an essential skill for extracting information from websites. While tools like Python’s BeautifulSoup or Scrapy are popular for complex scraping tasks, sometimes you need a lightweight, no-frills solution that works directly from the command line. Enter htmlq—a fast, simple command-line tool built in Rust that lets you parse HTML using CSS selectors. Think of it as jq (the JSON parser) but for HTML.

Whether you need to extract article titles, links, images, or prices from a webpage, htmlq simplifies the process with minimal setup. It’s ideal for quick scripts, automation, or when you don’t want to write a full-fledged Python/Ruby program. In this guide, we’ll cover everything from installing htmlq to advanced scraping techniques, with real-world examples to get you started.

2025-12

Table of Contents#

  1. Prerequisites
  2. Installing htmlq
  3. Basic Usage: Getting Started
  4. Advanced Techniques
  5. Real-World Examples
  6. Tips and Best Practices
  7. Conclusion
  8. References

Prerequisites#

Before diving in, ensure you have the following tools and knowledge:

  • A terminal: macOS (Terminal/iTerm), Linux (GNOME Terminal, Konsole), or Windows (WSL, Command Prompt with Rust installed).
  • curl or wget: To fetch web pages (preinstalled on most systems; install via brew install curl on macOS or sudo apt install curl on Linux if missing).
  • Basic CSS selector knowledge: Understand how to target elements (e.g., div, .class, #id, a). If you’re new to CSS selectors, refer to MDN’s CSS Selector Guide.
  • Rust and Cargo (optional but recommended): Required for installing htmlq via cargo (Rust’s package manager).

Installing htmlq#

htmlq is a Rust program, so the easiest way to install it is via cargo. If you don’t have Rust installed, first set it up with Rustup:

# Install Rust (includes Cargo)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Follow the prompts, then restart your terminal or run source $HOME/.cargo/env to load Cargo into your PATH.

Now install htmlq:

cargo install htmlq

Verify the installation:

htmlq --version  # Should output something like "htmlq 0.4.0"

Alternative Installation Methods:

  • Arch Linux: Use the AUR package: yay -S htmlq.
  • macOS (Homebrew): Install with brew install htmlq.
  • Windows: Use WSL and follow the Linux steps, or install Rust via Rustup and run cargo install htmlq.

Basic Usage: Getting Started#

htmlq works by taking HTML input (from a file or piped from curl) and querying it with CSS selectors. Let’s start with the basics.

Fetching a Web Page#

To scrape a remote webpage, first fetch its HTML with curl and pipe it to htmlq. For example, to scrape the homepage of example.com:

curl -s https://example.com | htmlq "selector"

The -s flag in curl silences progress bars, making output cleaner.

Extracting Elements with CSS Selectors#

Use CSS selectors to target elements. For example, to extract all <h1> tags from example.com:

curl -s https://example.com | htmlq "h1"

Output (simplified):

<h1>Example Domain</h1>

To target elements by class, use .classname. For example, extract all elements with class intro:

curl -s https://example.com | htmlq ".intro"

To target by ID, use #idname:

curl -s https://example.com | htmlq "#main-content"

Web scraping often requires extracting attributes like href (links) or src (images). Use the --attribute (or -a) flag to get an attribute’s value.

Example 1: Extract all links (<a> tags’ href)

curl -s https://example.com | htmlq "a" --attribute href

Output:

https://www.iana.org/domains/example

Example 2: Extract image URLs (<img> tags’ src)

curl -s https://example.com | htmlq "img" --attribute src

Extracting Text Content#

By default, htmlq returns the full HTML of matched elements. To get only the text content (stripping tags), use the --text (or -t) flag.

Example: Get text from <p> tags

curl -s https://example.com | htmlq "p" --text

Output:

This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.

Advanced Techniques#

Once you’re comfortable with basics, try these advanced workflows.

Combining Selectors#

Chain selectors to target nested elements. For example, extract paragraphs inside a <div> with class content:

curl -s https://example.com | htmlq "div.content p" --text

Or target the first <li> in an unordered list with ID menu:

curl -s https://example.com | htmlq "ul#menu li:first-child" --text

Using Pseudo-Classes and Filters#

htmlq supports CSS pseudo-classes like :first-child, :last-child, :nth-child(), and attribute filters.

Example 1: Get the first 3 list items

curl -s https://example.com | htmlq "ul li:nth-child(-n+3)" --text

Example 2: Target elements with a specific attribute
Extract all <input> tags with type="email":

curl -s https://example.com | htmlq 'input[type="email"]' --attribute name

Limiting Results#

Use --limit N to return only the first N matches. For example, get the top 5 article titles from a blog:

curl -s https://blog.example.com | htmlq ".article-title" --text --limit 5

Parsing Local HTML Files#

htmlq can parse local files with the --file (or -f) flag. This is useful for testing or scraping saved pages:

# Save a webpage locally first
curl -s https://example.com -o example.html
 
# Parse the local file
htmlq "title" --text -f example.html

Output:

Example Domain

Real-World Examples#

Let’s put htmlq to work with practical scenarios.

Suppose you want to extract titles and URLs of recent articles from a blog (e.g., https://blog.example.com). Assume articles are in <h2 class="post-title"> with links inside <a> tags.

Step 1: Extract titles (text)

curl -s https://blog.example.com | htmlq "h2.post-title a" --text

Step 2: Extract URLs (href attributes)

curl -s https://blog.example.com | htmlq "h2.post-title a" --attribute href

Combine into a script (save as scrape_blog.sh):

#!/bin/bash
URL="https://blog.example.com"
echo "Recent Articles:"
echo "================="
 
titles=$(curl -s $URL | htmlq "h2.post-title a" --text)
urls=$(curl -s $URL | htmlq "h2.post-title a" --attribute href)
 
while IFS= read -r title && IFS= read -r url <&3; do
  echo "- $title: $url"
done <<< "$titles" 3<<< "$urls"

Example 2: Extracting Product Prices#

Scrape prices from an e-commerce site (e.g., https://store.example.com), where prices are in <span class="price"> tags.

curl -s https://store.example.com/products | htmlq "span.price" --text

Output:

$29.99
$49.99
$15.50

Example 3: Parsing Weather Data#

Some weather websites (e.g., https://weather.example.com) display current temp in <div class="current-temp">. Extract it:

curl -s https://weather.example.com | htmlq "div.current-temp" --text

Output:

72°F

Tips and Best Practices#

To avoid issues and ensure ethical scraping, follow these guidelines:

  1. Respect robots.txt: Check https://example.com/robots.txt to see if scraping is allowed. For example, Disallow: / means no scraping.
  2. Add Delays: If scraping multiple pages, use sleep to avoid overwhelming the server: 
    for url in $(cat urls.txt); do
  curl -s $url | htmlq "title" --text
  sleep 2  # Wait 2 seconds between requests
done
  3. Use a Custom User-Agent: Some sites block default curl user-agents. Spoof a browser: 
    curl -s -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36" https://example.com | htmlq "title"
  4. Handle Errors: Check if curl succeeds before parsing: 
    if curl -s https://example.com -o temp.html; then
  htmlq "title" -f temp.html --text
else
  echo "Failed to fetch page"
fi
  5. Legal Compliance: Scraping may violate a site’s Terms of Service (ToS). Always review the ToS before scraping.

Conclusion#

htmlq is a powerful, lightweight tool for command-line web scraping. With its CSS selector support and simple syntax, it eliminates the need for complex scripts when you need quick data extraction. Whether you’re scraping links, prices, or text, htmlq gets the job done efficiently.

Start small: experiment with curl and basic selectors, then move to advanced workflows like combining attributes and text. Remember to scrape ethically and respect website policies.

For more details, check the htmlq documentation or run htmlq --help to explore all options.

References#