llms.txt Explained: Should Your Website Speak to AI Readers?

A plain-text welcome mat for AI

llms.txt is a proposed convention: a plain markdown file placed at your site's root (yoursite.co.uk/llms.txt) that gives AI systems a curated summary of what your site is and where its most useful content lives. It was proposed in September 2024 by Jeremy Howard, co-founder of Answer.AI and fast.ai, and has since spread widely across developer documentation sites.

The logic is straightforward. When an AI assistant wants to use your site to answer a question, it has a limited context window and no patience for navigation menus, cookie banners and repeated boilerplate. A concise, curated index in markdown, a format language models handle exceptionally well, lets the model find your best material without scraping half the site to work out what matters.

Not robots.txt, not a sitemap

It is easy to lump these root files together, but they do different jobs:

• robots.txt sets permissions: which bots may fetch which paths. It says nothing about what the content means.
• sitemap.xml lists every indexable URL for search engines. It is exhaustive by design and useless as a summary.
• llms.txt is editorial: a short, human-written guide to your most valuable pages, aimed at machine readers.
• llms-full.txt is a companion convention that inlines the full text of your key pages or documentation into one large file.

A related convention appends .md to individual page URLs to serve clean markdown versions of those pages, which some documentation platforms now generate automatically alongside the HTML.

What one looks like in practice

The format is deliberately simple: an H1 with the site name, a blockquote summary, then headed sections of links with one-line descriptions. For a small agency site it might read:

• # Thind Global Services
• > A West Bromwich digital agency helping UK small businesses with web design, ecommerce and search marketing.
• ## Services
• - [Web design](https://example.com/web-design): custom sites for small businesses, from brochure sites to ecommerce
• ## Guides
• - [Local SEO guide](https://example.com/guides/local-seo): step-by-step ranking help for UK service businesses
• ## Optional
• - [About the company](https://example.com/about): background material, safe to skip

The "Optional" heading has a defined meaning in the proposal: content an AI can skip when its context budget is tight. Writing the file takes half an hour. The hardest part is being honest about which ten pages actually matter.

The honest adoption picture

Here is the part most write-ups skip. Adoption on the publishing side is real: documentation platforms such as Mintlify generate llms.txt automatically, AI companies including Anthropic publish the file for their own docs, and directories list thousands of sites carrying one.

Consumption is another matter. As of mid-2026, no major AI provider has confirmed that its crawlers or assistants systematically fetch and use llms.txt, and Google's John Mueller has publicly compared it to the old keywords meta tag: a self-declaration with no verified consumer. Check your own server logs and you will likely find few or no requests for the file. It remains a proposal with momentum, not a standard with guarantees.

So should your business bother?

Treat it as a low-cost bet. The file costs perhaps thirty minutes, carries no SEO risk (search engines simply ignore it) and cannot hurt you. If the convention wins, early adopters are ready; if it fades, you have lost one tea break.

It makes most sense if your site is documentation-heavy, publishes substantial guides, or serves developers, because those are the audiences whose tools already touch these files. A five-page brochure site gains little. Either way, add it after the fundamentals are in place, not instead of them.

AI readability is bigger than one file

Whatever happens to llms.txt, the underlying goal is sound: make your content easy for machine readers to parse, because AI assistants increasingly mediate how customers discover businesses. The reliable wins are:

• Server-rendered HTML: many AI fetchers execute little or no JavaScript, so content that only appears client-side may be invisible to them.
• A clean heading hierarchy and semantic markup, so extracted text preserves your structure.
• Structured data (Organization, Product and FAQ schema) that states facts unambiguously.
• Concise, self-contained answers to real customer questions within your copy.
• Consistent name, address and phone details across the site for local relevance.

Those improvements help search engines and human visitors too, which is why they are worth doing regardless of how the llms.txt story ends. If you want an AI-readability review alongside conventional SEO, our team can take a look.