16 tips to improve your SEO with Screaming Frog

Screaming Frog’s SEO Spider is a crawler that most people working in SEO know well. Here are 16 tips that can make a real and fairly quick difference to your SEO.

If you would like help with your SEO, take a look at our SEO services or start with a free search audit.

• Hack: Improve your internal links and anchor text by finding pages with few internal links or poor anchor text.
• How to do it:
  1. Run a crawl and go to the Internal tab, then the Inlinks column.
  2. Sort pages by number of inlinks to find those with few internal links.
  3. Click the page you want to analyse and study its anchor text in the Inlinks tab of the lower window.
  4. Add relevant internal links from other pages and make sure the anchor text matches the search intent of the page it points to.
• Why it matters: Internal links spread link equity where it should go, make navigation easier for visitors and help search engines find and understand your content. Anchor text reinforces what the target page is about.

• Hack: Find pages with duplicate content on your site.
• How to do it:
  1. Go to the Content tab in Screaming Frog.
  2. Filter by Exact Duplicates or Near Duplicates.
  3. Decide what to do page by page. If you are unsure, read the article on duplicate content linked above.
• Why it matters: Removing duplicate content prevents keyword cannibalisation and stops search engines being unsure which page is most relevant, which holds back your rankings.

If no results appear, check three things:

1. That Enable Near Duplicates is ticked under Configuration > Content > Duplicates, as shown below. Adjust the similarity threshold (90% by default) to suit the site. Two location pages, say for Leeds and Bradford, can reasonably share a fairly high percentage of content, whereas two different blog posts should have almost none in common.

2. Make sure Near Duplicates is ticked under Crawl Analysis in the configuration.

3. Click Crawl Analysis > Start. You only need to do this if Auto Analyse At End of Crawl isn’t ticked.

• Hack: Get a visual overview of your own (or a competitor’s) internal links.
• How to do it:
  1. Go to Visualisations > Crawl Tree Graph in Screaming Frog.
  2. Analyse the map of internal links on your site.
• Why it matters: We covered internal links above. Seeing them visually makes it easier to understand how everything fits together, and it works as a summary you can act on.

• Hack: Find redirect chains and redirect loops across the whole site.
• How to do it:
  1. Go to the Response Codes tab.
  2. Filter by Internal Redirect Chain and Internal Redirect Loop.
• Why it matters: Fixing redirect problems helps your crawl budget and speeds up the site. In some cases a broken redirect also stops visitors reaching the page they wanted, which can cost you customers.

• Hack: Combine data from Google Analytics and Search Console with your crawl for a fuller picture.
• How to do it:
  1. Go to Configuration > API Access.
  2. Connect your Google Analytics 4 and Google Search Console accounts.
  3. Use the combined data in the relevant tabs to spot new insights and refine your SEO strategy.
• Why it matters: A page with no traffic is worth questioning. Improve it, merge it into a stronger page, or remove it. Google’s guidance on core updates says that deleting unhelpful content can help the good content on a site perform better, and on large sites, low-value URLs also take up crawl budget. You can see the traffic data directly in Google Analytics and Search Console too, but the crawl puts it next to everything else.

• Hack: Find spelling and grammar mistakes on your website.
• How to do it:
  1. Enable the checks under Configuration > Content > Spelling & Grammar in Screaming Frog.
  2. Run a crawl.
  3. Review the Spelling Errors and Grammar Errors filters in the Content tab, and the Spelling & Grammar Details tab in the lower window, to find and correct mistakes.
• Why it matters: Visitors and Google both prefer a site without spelling and grammar errors.

• Hack: See every H1, meta title and meta description in one view.
• How to do it:
  1. Go to the Internal tab.
  2. Move the Title 1, Meta Description 1 and H1-1 columns next to each other so you can see them all at once.
  3. Update them on your site one by one.
• Why it matters: The H1 and meta title help explain to Google what the page is about. The meta title and meta description matter for click-through rate.

• Hack: Find thin pages that may struggle to rank because they have too little content, and expand them.
• How to do it:
  1. Go to the Content tab.
  2. Use the Low Content Pages filter, or sort by the Word Count column.
• Why it matters: There is no magic rule for how many words a page should have. Where you think a page is too thin, expand the text, and compare it with the competitors ranking above you.

• Hack: Find images that have no alt text.
• How to do it:
  1. Go to the Images tab in Screaming Frog.
  2. Filter by Missing Alt Text to find images without a descriptive alt attribute.
• Why it matters: Alt text for image SEO improves accessibility for people with visual impairments and helps search engines understand what the image shows.

• Hack: Get a quick overview of your structured data (schema): which types you use and whether there are any errors.
• How to do it:
  1. Before crawling, enable JSON-LD (and Microdata or RDFa if you use them) plus Schema.org and Google Rich Result Feature validation under Configuration > Spider > Extraction, then use the Structured Data tab in Screaming Frog.
  2. Review any errors or warnings.
  3. Check which types you use and that the pages that should carry structured data actually have it.
• Why it matters: Correct structured data can make your pages eligible for rich results, which improves how they appear in search.

• Hack: Use Screaming Frog’s AI integration with OpenAI (ChatGPT) to get AI-generated suggestions for improving your content.
• How to do it:
  1. Go to Configuration > API Access > AI and choose OpenAI (Gemini, Anthropic and Ollama are also available)
  2. Enter your API key on the Account Information tab and click Connect
  3. On the Prompt Configuration tab, click Add from Library and choose what you want to do, or write your own prompt
  4. If the prompt uses page text or HTML, enable Store HTML under Configuration > Spider > Extraction
  5. Run the crawl and read the results in the AI tab
• Why it matters: Speeding up work that used to take a long time helps you get more done in less time than your competitors.

Bear in mind that you pay per API request. If you are analysing a lot of URLs, keep an eye on the cost.

• Hack: Use Screaming Frog’s integration with Google PageSpeed Insights to get a better overview of which pages are slow.
• How to do it:
  1. Go to Configuration > API Access > PageSpeed Insights.
  2. Enter your Secret Key.
  3. Run a crawl and check the PageSpeed tab.
• Why it matters: A fast site gives a better user experience, and Google recommends good Core Web Vitals for success in Search. You can also check this in Search Console, but Screaming Frog puts every metric for every URL in one place.

• Hack: Audit staging environments and password-protected sites before they go live, so you can find and fix SEO problems in good time.
• How to do it:
  1. If the site blocks crawlers with a disallow rule, go to Configuration > robots.txt and choose Ignore robots.txt. Do this before you crawl.
  2. If the site is password-protected with a browser login, Screaming Frog will ask for the username and password. For a login form, add it under Configuration > Authentication > Forms Based.
• Why it matters: The time you need Screaming Frog most is usually while a site is being built. Read our SEO checklist for launching a new website.

• Hack: Search for outdated text or code with Screaming Frog’s custom search. A site: search on Google, or similar methods, only finds pages that are indexed.
• How to do it:
  1. Go to Configuration > Custom > Search in Screaming Frog.
  2. Enter the text, code or pattern you want to find, for example an old year.
• Why it matters: Above all it saves time, so you get more done. Neither visitors nor Google like out-of-date information.

• Hack: N-grams give you a visual overview of the most common words and phrases on a page. You can use them, for example, to understand search intent.
• How to do it:
  1. Before you crawl, enable the feature
     • Configuration > Spider > Extraction, tick Store HTML and Store Rendered HTML
  2. Crawl your site in Screaming Frog.
  3. When it has finished, click the page you want to analyse in the Internal tab.
  4. Open the N-grams tab in the lower window and choose how many words to analyse (2-gram, 3-gram and so on, up to 6-gram).
• Why it matters: N-grams show you the word combinations that appear most often on your site, as search engines see them.

• Hack: Screaming Frog added this in version 22.0, released on 10 June 2025. It uses embeddings from an AI model to measure how semantically similar pages are, so two pages with entirely different wording can still turn out to be near-identical in meaning.
• How to do it:
  1. Go to Configuration > Content > Embeddings in Screaming Frog, enable the embedding functionality and tick “Semantic Similarity” (and “Low Relevance”).
  2. Connect an AI provider (OpenAI, Gemini or Ollama) via Configuration > API Access > AI.
  3. On the Prompt Configuration tab, use Add from Library to add the preset “Extract Semantic Embeddings from Page”, and make sure “Store HTML” is ticked under Configuration > Spider > Extraction.
  4. Run a new crawl once you have changed the settings above, then run crawl analysis. You will then find these filters in the Content tab:
     • Semantically Similar: shows pages with high similarity (on a 0–1 scale; above 0.95 by default)
     • Low Relevance Content: flags pages that sit outside the site’s main semantic focus.
  5. For the visualisation shown above, choose Visualisations > Content Cluster Diagram.
• Why it matters: Clearing out pages that compete for the same intent can make a clear difference to your rankings. Read more about duplicate content and SEO.