The traffic spike nobody ordered
Check your server logs and you will almost certainly find them: GPTBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, Meta's crawler, Amazonbot, ByteDance's Bytespider and CCBot from Common Crawl, all requesting page after page. For a small UK business on shared hosting, AI crawler traffic can now rival or exceed human visits, and unlike humans these bots never buy anything while they are there.
That has a real cost. Crawlers hit uncached and dynamic pages, chew through bandwidth allowances, and on a busy day can slow the site for actual customers. Hosting plans with visitor or resource caps feel it first. The question in 2026 is not whether AI bots visit, but which ones you let in and on what terms.
Work out who is actually crawling you
Before making any policy decision, get the facts from your own infrastructure:
- Pull your raw access logs (Hostinger, cPanel and most hosts expose them) and search for user agents such as GPTBot, ClaudeBot, PerplexityBot, Bytespider, CCBot, Google-Extended and Meta-ExternalAgent.
- Count requests per bot per day and note which pages they hit hardest.
- If you use Cloudflare, the free dashboard breaks down bot traffic for you, and Cloudflare Radar publishes wider crawler trends for context.
- Verify anything suspicious: the big providers publish their crawler IP ranges, and spammers routinely fake respectable user agents.
Distinguish two behaviours while you are in there. Training crawlers bulk-download content to build models. Retrieval fetchers, such as ChatGPT-User or Perplexity-User, grab a page on demand because a real person just asked a question that involves you. The second kind is much closer to a human visit, and that difference should shape your policy.
Need a hand with this?
Our team delivers Hosting & Support for UK businesses — with a free initial consultation, transparent fixed quotes and no lock-in contracts. Tell us what you're working on →
What robots.txt can and cannot do
robots.txt is a request, not a lock. The major labs, including OpenAI, Anthropic and Google, state that their crawlers respect it, and in practice they largely do. Others, Bytespider most notoriously, have been widely reported ignoring it. So treat robots.txt as your published policy and pair it with enforcement for the bots that will not listen.
A typical selective setup disallows training bots while leaving search untouched:
- User-agent: GPTBot, Disallow: / — blocks OpenAI's training crawler.
- User-agent: Google-Extended, Disallow: / — opts out of Gemini training without affecting Google Search rankings at all.
- User-agent: CCBot, Disallow: / — Common Crawl's archive feeds many training datasets.
- Leave Googlebot and Bingbot fully allowed; they still drive your organic traffic and revenue.
Cloudflare, blocking and pay-per-crawl
If you want enforcement rather than polite requests, put the site behind Cloudflare. Its free tier includes a one-click rule to block known AI crawlers, and since 2025 new Cloudflare zones block them by default. Bots that fake their user agent are caught by behavioural fingerprinting rather than name-matching, which is what makes this stronger than robots.txt alone.
Cloudflare has also been trialling a pay-per-crawl model, in which AI companies pay per request to access participating sites, built on the long-dormant HTTP 402 "Payment Required" status code. For a typical small-business site the sums involved will be pocket money, but the direction matters: crawl access is becoming something publishers price rather than give away. If content is your product (original guides, data, journalism), watch this space closely.
The case for letting some bots in
Blocking everything feels tidy but carries a cost of its own. AI assistants are now a genuine discovery channel: people ask ChatGPT, Claude, Perplexity or Google's AI results for "accountants near West Bromwich" or "best CRM for a small trades firm", and the answers cite and link their sources. If a retrieval fetcher cannot reach your site, you are invisible in that conversation while a competitor gets quoted.
Referral clicks from AI answers are still modest compared with classic search, but they arrive warm: the assistant has effectively pre-sold the recommendation. For most service businesses, being findable in AI answers is worth more than the bandwidth those fetchers consume.
Key Takeaway
Check your server logs before deciding anything: identify which AI crawlers visit and how often. Then split the decision in two. Retrieval bots that fetch pages to answer a live question can send you customers, so most small businesses should allow them. Bulk training crawlers offer little in return, so blocking them via robots.txt and Cloudflare's AI controls is reasonable, especially if content is your product. Revisit the policy quarterly, because this area is moving fast.
A sensible default policy
- Allow search crawlers (Googlebot, Bingbot) unconditionally.
- Allow retrieval and search-facing AI bots (OAI-SearchBot, ChatGPT-User, PerplexityBot) so you appear and get cited in AI answers.
- Block bulk training crawlers in robots.txt, and enforce with Cloudflare if your logs show they persist anyway.
- Rate-limit or block anything unverified that hammers the server, whatever it calls itself.
- Review the logs quarterly; bot names, behaviour and the monetisation options are all changing quickly.
None of this needs more than an afternoon to set up, and the visibility upside usually outweighs the server cost once the worst offenders are filtered out. If you would rather someone audited your logs and configured the rules for you, our team can help.