Screaming Frog SEO Spider is a crawler you install on your computer (Windows, macOS or Linux). It visits a site the way a search engine would and lists every URL it finds, with status code, title, meta description, canonical, directives and links. It costs nothing up to 500 URLs per crawl; the licence costs £199 a year per user and unlocks 24 of the 31 features listed on the official price list.
That "24 of 31" is the part nobody spells out.
On 10 October 2026 we read the top results on Google UK for "screaming frog" (5,400 searches a month in the UK) and "screaming frog seo spider" (720) on a smartphone: Screaming Frog's own pages, video tutorials, a TechRadar review, a definition, a software directory listing, Reddit threads. The ones we opened mention the 500-URL limit. None lists everything that stays locked in the free version, or says whether 500 URLs is enough for a typical business website.
Isn't the page limit the only difference?
No. Without a licence the configuration is locked: you can't exclude folders or parameters, you can't change the user-agent, you can't render JavaScript, you can't connect Search Console and you can't save a crawl to reopen it tomorrow.
We see it on the sites we work on. Sunscape, an Italian solar farm developer, has 46 URLs in its sitemap: the free version crawls all of it, with room to spare. Macropix, an LED screen manufacturer near Milan that went from position 88 to 2 for the Italian keyword "monitor pubblicitario" (advertising monitor), has 688: there, the free version stops before the end. (Why that jump took five years is in our guide to how long SEO takes.)
Same tool, two different answers.
In this guide you'll find:
- what the free version does and what the licence unlocks, feature by feature;
- our test on 19 business websites: how many of them fit in 500 URLs;
- how we use it in audits, with a case of 500 URLs returning 404;
- what it costs in pounds, with the price history since 2010 and VAT for UK businesses;
- what changes with version 24, the one that connects to Claude.
Let's start.
What Screaming Frog does (and what it doesn't: it's not a keyword suite)
Screaming Frog does one thing: it crawls your site and tells you what it looks like to a crawler. It starts from the homepage, follows the links in the HTML and, for every URL, records the server response, redirects, indexability, metadata and inbound and outbound links. According to the official SEO Spider page it flags more than 300 issues, warnings and opportunities.
It doesn't tell you which keywords you rank for, how much traffic you get or who links to you. That's the job of Search Console and of suites like Semrush or Ahrefs, which we compare in our guide to the best SEO tools. How a crawler works in general, Googlebot included, is in our guide to the web crawler.
There are two main crawl modes:
- Spider: you enter the homepage and the tool follows links on the same subdomain;
- List: you upload or paste a list of URLs and it checks only those.
And if I don't know what to look at?
There's the Issues tab, on the right. Each issue has a type (issue, warning, opportunity) and an estimated priority, and according to the official getting started guide the explanation of each issue is available in English, German, Spanish, French and Italian.
Be careful with that priority, though. It's estimated from the type of issue, not from your site. On a company website we audited in January 2026, the crawl gave a 40% pass rate on the on-page checks: on some pages the title tag sat outside the head. An error like that matters more than a hundred images without alt text, but in the list they appear side by side.
Free version and licence: 7 features out of 31 (the configuration stays locked)
The free version of Screaming Frog includes 7 of the 31 features listed in the official comparison, plus the limit of 500 URLs per crawl. The other 24 need a licence.
| Feature | Free | With licence |
|---|---|---|
| Broken links, errors and redirects | yes | yes |
| Page titles and meta descriptions | yes | yes |
| Meta robots and directives | yes | yes |
| Hreflang | yes | yes |
| Exact duplicate pages | yes | yes |
| XML sitemap generation | yes | yes |
| Site visualisations | yes | yes |
| Crawl configuration (exclusions, user-agent, limits) | no | yes |
| Saving and reopening crawls | no | yes |
| JavaScript rendering | no | yes |
| Structured data and validation | no | yes |
| Near duplicate content | no | yes |
| Search Console, Analytics and PageSpeed Insights integration | no | yes |
| Custom extraction (XPath, CSS, regex) | no | yes |
| Crawling with OpenAI and Gemini | no | yes |
| Crawl comparison and scheduled crawls | no | yes |
| Accessibility auditing | no | yes |
Three rows of this table change the way you work.
The locked configuration is the most underrated. The official FAQ says it plainly: without a licence the configuration options are unavailable. So no excluding filter parameters, no depth limit, no Googlebot user-agent.
JavaScript rendering is paid. The free version reads only the HTML the server returns, as many AI model crawlers do, which we cover in our guide to AI crawlers.
Structured data isn't validated without a licence. If your goal is to check Product or FAQ markup across every page, you need the paid version or Google's Rich Results Test, one page at a time.
So is the free version useless?
Not at all. Broken links, redirects, titles, canonicals and hreflang are the checks we run first on every site, and they're all there. The real question is a different one: does your site fit inside it?
The test: is 500 URLs enough for a business website? (12 sites out of 22 say yes)
To answer with numbers, on 10 October 2026 we took 22 business websites: the 17 company sites shown on page one of Google UK on a smartphone for "led wall" and "lease land for solar farm" (organic results, local pack and the "Find results on" box, excluding marketplaces, forums, social networks and video sites), plus 5 of our own clients, all Italian, with a published case. For each one we counted the URLs listed in the XML sitemap, following the indexes, and requested the homepage once as Chrome and once with the user-agent "Screaming Frog SEO Spider/24.3".
| Site | Source | URLs in the sitemap | Is 500 URLs enough? |
|---|---|---|---|
| sunenergygroup.net | SERP "lease land for solar farm" | 35 | yes |
| sunscape.it | Visilay client (Italy) | 46 | yes |
| ledvideowall-hire.co.uk | SERP "led wall" | 117 | yes |
| viridire.com | SERP "lease land for solar farm" | 143 | yes |
| digitalsignbox.co.uk | SERP "led wall" | 161 | yes |
| scovaventi.it | Visilay client (Italy) | 175 | yes |
| saturnvisual.com | SERP "led wall" | 182 | yes |
| scanlite.co.uk | SERP "led wall" | 201 | yes |
| lumifyenergy.com | SERP "lease land for solar farm" | 235 | yes |
| pioneerled.uk | SERP "led wall" | 369 | yes |
| planar.com | SERP "led wall" | 467 | yes |
| dynamo-led-displays.co.uk | SERP "led wall" (local pack) | 497 | yes |
| macropix.it | Visilay client (Italy) | 688 | no |
| tsplegal.com | SERP "lease land for solar farm" | 1,115 | no |
| valoresolare.it | Visilay client (Italy) | 1,138 | no |
| ac-et.com | SERP "led wall" | 1,637 | no |
| bosatta.it | Visilay client (Italy) | 3,600 | no |
| tvc.uk.com | SERP "led wall" | 3,858 | no |
| visunext.co.uk | SERP "led wall" | 6,191 | no |
| savills.co.uk | SERP "lease land for solar farm" | 7,813 | no |
| onedirect.co.uk | SERP "led wall" | 8,485 | no |
| sunvest.com | SERP "lease land for solar farm" | site blocked our requests | to be checked |
The result in three lines:
- 12 sites out of 22 are under 500 URLs in their sitemap (10 of the 17 from Google UK), and for them the free version covers the whole site;
- 9 out of 22 are over, and the biggest are catalogues or libraries: Visunext, Onedirect and AC-ET are online shops, Savills is a property consultancy with a large research archive, and our client Bosatta sells Vespa spare parts in four countries;
- 1 out of 22 couldn't be checked: Sunvest answered every request from our server with a 403, robots.txt included, and there the only way to know how many URLs it has is to crawl it from another connection.
Two more findings, sharper than we expected.
One site blocks Screaming Frog by name. Savills' homepage answered 200 to Chrome and 403 to the crawler's user-agent, twice in a row, and its robots.txt doesn't mention it: the block sits at server or firewall level, the case the official FAQ describes. On the other 20 sites we could reach, the homepage answered 200 to both.
All 21 homepages we could read have their text in the HTML, without running JavaScript: from 431 words (Pioneer LED) to 3,299 (Onedirect). To read the content of these sites you don't need paid JavaScript rendering.
What about Dynamo LED Displays, with 497 URLs?
That's the one to watch. A 497-URL sitemap leaves three URLs of headroom: blog tags, paginated pages and URLs with parameters are often missing from the sitemap, but the crawler finds them by following links. Planar, at 467, is in the same position. Above 300 URLs in the sitemap it's worth running a first crawl on the free version and seeing whether it stops at 500.
How we use it in audits (the case of the 500 URLs returning 404)
In our audits Screaming Frog is there to answer specific questions, not to produce a list of errors. The order we read the results in is the one in our guide to the SEO audit; here are the three checks we repeat on every site.
1. Are removed URLs still linked?
In March 2026 a clothing brand that sells in several countries sent us a file of about 500 URLs returning 404, flagged by Search Console: language and country combinations created in the past and then deleted. The question was whether they were still linked from inside the site.
List mode isn't enough here: it tells you those URLs return 404, which you already know. You need a full crawl of the site in Spider mode and then the Inlinks tab on each 404, which shows which page still contains the link. When a 404 is no longer linked, it's often the right response, as we explain in our guide to the 404 error.
2. How many redirect chains are there?
After a migration it's the first check. The redirect chains report shows every intermediate hop, and every hop is one more request for Googlebot. How to set them up properly is in our guide to the 301 redirect.
3. Do canonicals and internal links say the same thing?
A page with a canonical pointing to another page, but with dozens of internal links pointing to it, sends two opposite signals. Screaming Frog puts the two columns side by side in the same view. The rules for choosing the canonical are in our guide to the canonical URL.
And the other 300 checks?
We read them, but later. On a site of a few hundred pages we start from these three; missing alt text and long meta descriptions come afterwards.
500 URLs returning 404 and nobody knows where they come from?
You've just seen how a list of errors turns into a list of pages to fix. We crawl your site, read it alongside Search Console and hand you the fixes in order of impact.
Book a strategy consultationWhat it costs in pounds (and why it costs more than before 2023)
A Screaming Frog licence costs £199 a year per user, $279 or €245 if you pay in dollars or euros. The sterling price hasn't changed since January 2023; the dollar and euro prices were adjusted in May 2025 for the exchange rate.
Screaming Frog says so itself: its FAQ publishes the full price history:
| Date | Pounds | Dollars | Euros | What changed |
|---|---|---|---|---|
| November 2010 | £99 | not sold | not sold | launch, sold in pounds only |
| August 2016 | £149 | $209 | €185 | first increase |
| January 2023 | £199 | $259 | €239 | second increase |
| May 2025 | £199 | $279 | €245 | currency adjustment |
Why does the history matter?
Because several pages still quote an old price. On page one of Google UK for "screaming frog", Software Advice UK still showed US$259 in its search snippet on 10 October 2026: that's the 2023 price list. A software download directory on page one for "screaming frog seo spider" showed $209, the 2016 price. In fifteen years there have been two real increases, and for UK buyers paying in pounds the price hasn't moved since 2023.
Four details that matter for a UK business:
- the licence is per person: two people using it need two licences, but the same person can use it on several computers;
- from the fifth licence there's a discount: £189 for 5 to 9 licences, £179 for 10 to 19, £169 for 20 or more (pricing page);
- the VAT exemption is for EU businesses only: Screaming Frog is a UK company, and its FAQ lets you buy without VAT only if you're in the EU with a valid VAT number. A UK business pays UK VAT and, if it's VAT-registered, reclaims it like any other business expense. The pricing page doesn't say whether the £199 includes VAT, so check the total at checkout;
- it doesn't renew automatically, unless you chose that at purchase. When it expires you go back to the free version and you can no longer open your saved crawls.
How much the licence weighs in a tools budget we worked out in our guide to low-cost SEO tools.
Large sites: memory, SSDs and when you don't need to crawl everything
For sites beyond tens of thousands of URLs the limit isn't the licence but the computer. The official guide to crawling large websites gives precise numbers:
- with 4 GB of RAM allocated you can crawl about 2 million URLs, with 8 GB about 5 million, in database storage mode on an SSD;
- in memory storage mode a machine with 8 GB of RAM generally gets to a couple of hundred thousand URLs;
- a 1 million page site at 5 URLs per second takes over two days to crawl.
The default speed is 5 threads, according to the configuration guide, precisely so as not to overload the server. The same guide warns that raising it increases the number of requests and can hurt the site's response times: on shared hosting, you slow the site down while you're analysing it.
Do you really need to crawl everything?
Often not, and the same guide says so: websites are built on templates, and a sample of each page type is enough to make decisions. On Bosatta, 3,600 URLs in the sitemap across four languages, what we check after every change is languages and hreflang, not the number of pages. For catalogues with filters and variants, where URL counts really explode, the discussion is in our guide to ecommerce SEO tools.
Version 24: MCP and AI inside the crawler (what has changed since 19 May 2026)
Version 24 of Screaming Frog, released on 19 May 2026, lets you run crawls, analyse and export data from Claude, LM Studio and other AI assistants through an MCP server. The launch post is by Dan Sharp, and it comes after three major releases in eighteen months.
From the release history:
- 12 November 2024, version 21: direct integration with the OpenAI, Gemini and Ollama APIs;
- 10 June 2025, version 22: semantic similarity analysis and content clusters;
- 20 October 2025, version 23: PageSpeed audits updated to Lighthouse Insights;
- 19 May 2026, version 24: MCP server and automatic comparison of scheduled crawls.
In practice, with MCP you ask in plain language for a crawl to be run and its issues summarised, and the assistant uses Screaming Frog to do it; according to the same post, not every feature is available through MCP yet. What the MCP protocol is, and why it's reaching websites too, we explain in our guide to WebMCP and MCP.
Two words of caution, both written by Screaming Frog itself.
The first: according to the FAQ crawl data stays on your computer, but features that use external APIs such as OpenAI or Gemini may send crawl data to those services. On a client site covered by a confidentiality agreement, ask first.
The second is in the launch post: "This is not a replacement for an experienced SEO professional, obviously."
Alternatives: when another crawler makes sense
BeamUsUp is a crawler that costs nothing and, according to its own site, has no page limit; it runs on Windows, Mac and Linux. Sitebulb, a paid tool, comes with more discursive explanations of the data. Semrush and Ahrefs have cloud-based audits, with pages counted against your plan.
Our rule is a practical one: if the site is under 500 URLs and doesn't use JavaScript for its content, the free version of Screaming Frog is enough; if it's over, the £199 licence costs less than two months of Semrush's entry SEO plan ($139 a month, about £105 at the Bank of England rate of 8 October 2026; price checked on 10 October 2026).
Frequently asked questions
Yes. According to the official FAQ the software contains no spyware, malware or adware (as verified by Softpedia for Windows and macOS) and crawl data stays on your computer. The only exception is features that use external APIs such as OpenAI or Gemini, which may send crawl data to those services.
It depends on the number of URLs and on what you need to check. In our test on 10 October 2026, 12 business websites out of 22 had fewer than 500 URLs in their sitemap. If you need JavaScript rendering, structured data validation or the Search Console integration, you need a licence anyway.
Yes, but by one person only. The licence is individual: the same person can use it on several devices, while two people using the software need two licences. From the fifth licence the price drops to £189 a year each.
Yes. The SEO Spider is available for Windows, macOS (both Apple Silicon and Intel) and Linux, with packages for Ubuntu and Fedora. Version 24.3, released on 29 June 2026, was the latest available on 10 October 2026.
It depends on the part of the software. According to the official getting started guide, the explanations of issues in the Issues tab are available in English, German, Spanish, French and Italian. The documentation and guides on the official site are in English.
The software goes back to the free version: the configuration is locked, the limit goes back to 500 URLs and saved crawls can no longer be opened. Licences don't renew automatically, unless auto renewal was chosen at the time of purchase.
What to take away (the licence is the easy part)
In short:
the free version covers 7 features out of 31, and the checks that matter most are among those 7;
12 business websites out of 22 are under 500 URLs in their sitemap;
the licence costs £199 a year, a price that hasn't changed in pounds since 2023;
since version 24 the crawler can also be driven by an AI assistant.
There's a consequence of that last point the industry doesn't talk about much. If an assistant can launch the crawl, export the data and chart it, the work left for a human is deciding which of the 300 flagged issues to fix first. That's why an audit should be judged by the order of its fixes, not by the length of its list: with MCP the list takes even less time, the order doesn't. The method we use for business websites is in our guide to technical SEO.
If you'd like us to read the crawl for you and tell you what to fix and in what order, that's the work of our SEO service.
See you next time!
