Step 1: Set Up & Configure Screaming Frog
A precise audit starts before crawling:
- Install Screaming Frog (free up to 500 URLs; paid removes limits).
- Use Database Storage mode if possible—this enables larger crawls and auto-save support.
- Allocate maximum RAM (leaving ~2 GB free) for performance.
- Configure Spider settings via Configuration → Spider:
- Enable crawl of JavaScript (if needed), pagination tags, hreflang, AMP, subdomains, and external nofollow/Follows.
- Set limits and exclude irrelevant URLs (like admin, search query, affiliate or staging URLs).
- Enable extraction features under Configuration > Extraction: HTTP headers, structured data formats, schema validation, storing HTML (raw and rendered).
- Save the configuration as default for consistency.
These settings ensure your crawl is complete, accurate, and tailored to a technical audit.
Step 2: Run the Crawl
- Enter your target domain or URL list in List Mode if running focused audits.
- Click Start, then monitor the Overview tab for key metrics: URL counts, depth, status codes, and crawl speed.
Let the crawl complete fully. If paused or interrupted, database mode preserves progress.
Step 3: Use Crawl Analysis & Filter Insights
Post-crawl, leverage Crawl Analysis (Setup under Crawl Analysis > Configure) to layer in Google Analytics, Search Console, or sitemap data:
- Flag orphan pages, URLs in sitemap but not crawl, index bloat, and content duplicates.
- Identify pages with >90% duplicate content; adjust threshold depending on site size.
This helps focus your audit on what matters most.
Step 4: Audit Key Technical Areas
4.1 Response Codes & Redirects
- In Response Codes, filter 4xx/5xx (broken links) and 3xx (redirects).
- Detect and break redirect chains via Reports > Redirect Chains. Simplify to single-step redirects.
4.2 Metadata & Headers
- Use Page Titles, Meta Description, H1, H2 tabs: identify missing, duplicate, too-long, or keyword-poor elements.
- Ensure one H1 per page and logical header structure.
4.3 Content Quality
- Leverage the Content tab with duplicate filters to flag thin/duplicate pages.
- Image audit (Images tab): find missing alt text and oversized files.
4.4 Internal & External Links
- Internal tab: spot orphan pages and review anchor text quality. External tab: flag broken external links.
4.5 Indexation & Directives
- In Directives tab: check canonical tags, meta robots, noindex issues. Ensure no valuable pages are blocked.
- Confirm sitemap coverage against crawl; identify pages missing or misaligned.
4.6 JavaScript & Rendering
- If JS-heavy, enable rendering and compare raw vs rendered HTML in Internal tab HTML view.
4.7 Speed & Security
- Use Download Time column to highlight slow pages; follow up with PageSpeed Insights API integration.
- Validate SSL, mixed content, HTTPS redirect compliance.
4.8 Structured Data
- Check schema via Structured Data extraction tab; validate against Google standards.
Step 5: Export and Prioritize Findings
- Export relevant reports: Broken links, duplicate metadata, redirects, missing alt tags, security issues, etc.
- Use spreadsheets or project tools to classify issues by impact and difficulty:
- High priority: 404s, 5xx errors, missing metadata, slow pages
- Medium: Duplicate content, weak header structure, missing alt text
- Low: Minor tag or URL formatting issues
Step 6: Implement Fixes and Monitor
- Fix broken links, update or optimize metadata, correct header structure.
- Adjust redirect chains and enforce HTTPS/security best practices.
- Optimize images and speed issues flagged earlier.
- Re-crawl after changes to ensure fixes have taken effect.
Step 7: Ongoing Audit & Process Integration
- Schedule regular crawls (monthly for small sites, weekly for larger) to catch new issues.
- Enrich crawls with API data from GA/GSC to prioritize based on traffic/backlink value.
- Save and version configurations to maintain consistency in recurring audits.
Final Thoughts
A full technical SEO audit using Screaming Frog is remarkably effective when paired with thoughtful configuration, thorough analysis, and prioritized implementation. By following these seven steps—setup, crawl, analyze, export, fix, re-test, and repeat—you become proactive in maintaining a healthy, crawlable, and search-optimized website.
