Screaming Frog SEO Spider is one of the most powerful and widely used tools for a site audit. It reveals technical issues, content glitches, link problems, and more. Here’s how to use it effectively—even if you’re just getting started.
1. Install and Configure It Properly
Start by downloading Screaming Frog and installing it (the free version crawls up to 500 URLs).
Before crawling, configure your settings to ensure a comprehensive audit:
- Switch to Database Storage mode (ideal for SSDs)—it enables larger crawls and auto-saves progress .
- Allocate sufficient RAM, leaving about 2 GB free so your system stays responsive .
These steps help avoid interruptions and maximize crawl depth.
2. Tailor Spider Settings
Go to Configuration → Spider and customize the crawl:
- Enable: pagination, hreflang, AMP, and JavaScript rendering if appropriate for your site .
- Check “Crawl All Subdomains” and follow “nofollow” links (internal and external) for full coverage .
In Configuration → XML Sitemaps, allow crawling of sitemaps by all available options to detect orphaned pages .
3. Set Exclusions & Crawl Limits
Exclude non-relevant URLs—for example, admin pages or URL parameters like ?sort=price—via Configuration → Exclude . This keeps the crawl focused and relevant.
Manage crawl depth (5–7 clicks) via Spider → Limits, ensuring higher priority pages get crawled first .
4. Start the Crawl
Enter your site’s URL and click Start. Monitor the progress panel to track key metrics—total URLs crawled, response sizes, load times .
5. Analyze Results by Tab
Screaming Frog organizes issues into intuitive tabs:
- Response Codes: identify broken links (404), redirect chains, and server errors .
- Page Titles & Meta Descriptions: find missing, duplicate, or overly long tags .
- Headers (H1/H2): check for missing or misused tags .
- Images: spot missing alt text or large images slowing your site .
- Content: filter by word count to find thin pages (<500 words) and detect duplicates .
- Directives: check for noindex, canonical, and robots-blocked pages .
6. Leverage Crawl Analysis
Under Crawl Analysis → Configure, integrate with Google Search Console or Analytics. This provides data on sitemap coverage, orphan pages, and indexing issues .
Use “Indexed but not in sitemap” filters to spot bloat, then decide whether to noindex, redirect, or remove those pages .
7. Export Issues and Assign Priorities
Use Bulk Export or manually download filtered tables (e.g., broken links, missing metadata, thin content). Organize them in a spreadsheet with columns like:
- URL
- Issue type
- Priority (High/Medium/Low)
- Proposed fix
Prioritize based on frequency and impact: fix broken links and missing metadata first, then address content and crawlability issues.
8. Implement Fixes and Validate
- Fix broken links, update titles, descriptions, headers, and alt text.
- Address directives: fix indexing or crawling restrictions.
- Reduce thin pages by adding content or consolidating.
- Resolve redirect chains to improve site speed.
Once implemented, re-crawl to ensure problems are resolved and track progress over time.
9. Advanced Features to Explore
As you grow in expertise, use these features:
- Custom Extraction for specific data via CSS, XPath, or regex .
- Structured Data validation via integrated checks .
- Integrate API data (GSC, GA, PageSpeed) to prioritize issues .
- Crawl visualisations to map site architecture .
10. Make Audits Routine
Set up monthly or quarterly crawls—even automated—to catch emerging issues early .
Save your configuration (File → Configuration → Save As) so each audit starts with consistent settings .
Final Takeaway
A Screaming Frog audit is one of the smartest moves you can make for site health. By configuring your crawl properly, analyzing key technical and content issues, exporting your findings, fixing systematically, and repeating regularly, you establish a data-driven routine that boosts SEO performance and user experience.
