thelinuxvault blog

Gau (GetAllUrls) Review - Tool For Discovering URL in Kali Linux

In the world of penetration testing and web application security, discovering all the URLs associated with a target domain is a crucial step. Gau (GetAllUrls) is a powerful tool that simplifies this process. It is designed to quickly and efficiently gather URLs from various historical archive sources such as Wayback Machine and CommonCrawl. In this blog post, we will explore what Gau is, how it works, its features, common and best practices for using it, and provide example usage.

2026-05

Table of Content#

  1. What is Gau?
  2. Installation in Kali Linux
  3. How Gau Works
  4. Features of Gau
  5. Common Practices
  6. Best Practices
  7. Example Usage
  8. References

What is Gau?#

Gau is a command-line tool written in Go. Its main purpose is to scrape and collect URLs related to a given domain. It works by querying multiple historical archive sources and public data sources (such as Wayback Machine and CommonCrawl) to gather URLs. It then outputs a list of these URLs, which can be further processed by other tools for tasks like vulnerability scanning, content analysis, etc.

Installation in Kali Linux#

Gau is written in Go and can be installed using the Go package manager. Open the terminal and run the following command:

go install github.com/lc/gau@latest

This will download and install the Gau tool.

How Gau Works#

  • Historical Archive Sources: Gau queries multiple historical archive sources (such as Wayback Machine, CommonCrawl, and URLScan) to discover URLs associated with a target domain. It retrieves cached versions of web pages and extracts URLs from these archives.
  • Output: The collected URLs are then printed to the console or can be redirected to a file for later use.

Features of Gau#

  • Speed: Gau is optimized to quickly gather a large number of URLs. It can handle multiple requests in parallel (depending on the system resources).
  • Customization: You can specify options like the number of results to fetch and delay between requests using flags like -t for timeout and -d for delay to avoid being blocked by archive sources.
  • Output Formats: It can output the URLs in a simple text format, which is easy to process with other command-line tools like grep, awk, etc.

Common Practices#

  • Basic Domain Enumeration: When starting a web application security assessment, use Gau to quickly get an initial list of URLs. For example, if you are testing example.com, run gau example.com to see what pages and subdomains exist.
  • Combining with Other Tools: After getting the list of URLs from Gau, pipe the output to tools like ffuf (for directory and file brute-forcing). For instance:
gau example.com | ffuf -w - -u FUZZ -mc 200

Here, ffuf will take each URL from Gau's output (the -w - option reads from stdin) and check if the page exists (looking for a 200 HTTP status code).

Best Practices#

  • Respecting Search Engine Policies: Don't overuse Gau with aggressive settings (like very short timeouts and no delays) when querying search engines. This can lead to your IP being blocked. Use reasonable values for timeout (-t 10 for 10 seconds) and delay (-d 2 for 2 seconds between requests).
  • Filtering Output: Use tools like grep to filter out unwanted URLs. For example, if you only want URLs that contain a specific string (like login), you can do:
gau example.com | grep login
  • Storing Results: Always save the output of Gau to a file. You can use redirection:
gau example.com > urls.txt

This way, you can refer back to the list later and also use it as input for other scripts or tools.

Example Usage#

Example 1: Basic Usage#

To get all URLs for example.com:

gau example.com

This will start querying search engines and performing DNS lookups. The output will be a list of URLs like:

http://example.com/about
http://subdomain.example.com/contact
https://example.com/blog/post1

Example 2: With Customization#

If you want to limit the number of search results from each search engine (say, 50 results) and add a 3-second delay between requests:

gau -t 50 -d 3 example.com

Example 3: Combining with grep#

Suppose you are only interested in URLs that have admin in them:

gau example.com | grep admin

References#

By following the above guide, you can effectively use Gau in Kali Linux to discover URLs for your web application security testing and other related tasks.