Do I need an llms.txt file?

The evidence says AI systems are barely reading them: in Ahrefs’ data, 97% of the llms.txt files it found got no requests in a month, and Google has said it doesn’t support the file. It costs nothing to publish one. It isn’t a ranking factor, and we don’t score it as one.

Should I block AI crawlers?

That’s your call, and it’s mostly about how your content may be used. Worth knowing first: blocking OAI-SearchBot keeps you out of ChatGPT search, OpenAI’s crawlers are set independently of each other, and Google-Extended doesn’t affect Google Search but does decide whether Gemini can use your content.

Site Audit

Can AI crawl, read,
and quote you?

Three questions, in that order. If the crawlers can’t reach you, nothing else matters. We check all three, and tell you which findings rest on evidence and which are our judgment.

Book a demo See the method

30-day free trial · no credit card

One line in a file
can make you invisible.

A blocked agent, a 403, or a page that only renders in JavaScript doesn’t lower your visibility. It takes you out of the answer entirely.

Two things broke retrieval.
Four popular fixes changed nothing.

Vercel and the ora.ai research lab ran 190 controlled fetch probes and observed 1,033 agent runs across 25 sites. Here’s what actually moved.

Stopped retrieval

Returning a 403
An answer that only appears with JavaScript, as in an empty app shell

Measured with two fetch clients looking for an answer planted on the page.

Changed nothing, when the answer was already in the HTML

Adding llms.txt
Adding JSON-LD
Adding a sitemap
Adding Markdown mirrors

Source · Vercel and ora.ai, “Make your site readable by AI agents”, Aug 2026

“There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary.”

Google Search Central, AI features and your website, which adds: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.”

Crawl, read, quote

The audit asks the three questions in the order they fail.

AI agents5 allowed1 challenged1 blocked

GPTBotAllowed
OAI-SearchBotAllowed
ChatGPT-UserAllowed
ClaudeBotBlocked · robots.txt
Claude-UserAllowed
PerplexityBot403 · CDN challenge
Google-ExtendedAllowed

01Crawl

Can the engines reach you at all?

Every AI agent checked by name against your robots.txt, your response codes and your CDN, so “we’re blocking AI” becomes a fact rather than a suspicion.

OpenAI’s crawlers are set separately. Blocking GPTBot doesn’t block OAI-SearchBot, and a robots.txt change takes about a day to reach ChatGPT search.
Perplexity’s user fetcher works differently. When a person asks, Perplexity-User generally ignores robots.txt.
Google-Extended isn’t about Search. It doesn’t affect Google Search or rankings. It does decide whether Gemini can use your content, including to ground its answers.

Pages, as an agent reads them24 crawled

/Readable · 200
/products/rep-seamless-leggingsReadable · 200
/pages/size-guideJavaScript only
/blogs/training/squat-formnoindex
/collections/leggingsReadable · 200

02Read

Once they arrive, is the content there?

Agents that don’t run JavaScript see only what the server sends. We read your pages the way they do.

Server-rendered contentResponse codesCanonical tagsnoindex on key pagesHeading hierarchyContent freshnessStructured content

A clear value proposition
Consistent brand naming
Consistent contact detailsFix · Use one phone number on all 3 pages
An About page
An FAQ sectionFix · Add an FAQ to product pages
Case studies

03Quote

Is there anything worth lifting into an answer?

Answers are built from clear, consistent statements. We check the ones engines lean on to describe a brand, and attach a fix to every miss.

A clear value propositionConsistent naming and contact detailsAbout, FAQ and legal pagesCase studiesExternal profilesLocal relevance

What you get to look at

One crawl, read the way the engines read it, and every gap with its fix.

**A grade and six categories**Technical, content, entity, trust, local and AI-answer readiness, scored, with the worst gaps first.

**A fix on every finding**Each gap comes with the specific change and, where it applies, a snippet you can paste.

**Findings you can close**Mark an issue fixed, and see it reopen if the next crawl finds it again.

**Every agent, by name**GPTBot, ClaudeBot, PerplexityBot and the rest, each checked on its own, because each follows its own rules.

Evidence or judgment,
on every finding

Every check in the report carries one of two tags, so you know which fixes the research backs and which are our advice.

Evidence

Measured in a study we cite, or documented by the engine itself.

For exampleA crawler blocked in robots.txt. A 403. A page that only renders in JavaScript.

Judgment

Our recommendation, from how answers get assembled. Not a measured ranking factor.

For exampleAn FAQ section. A clearer value proposition. Consistent contact details.

97%of llms.txt files got no requests in a month

We check whether you have an llms.txt. You should also know what the evidence says: in Ahrefs’ data across 137,210 domains, almost nothing is reading them, and Google has said it doesn’t support the file. It costs nothing to have one. It isn’t a ranking factor, and we won’t score you as if it were.

Ahrefs, data from May 2026 · Gary Illyes, Google, July 2025

Read how we measure

Isn’t this just an SEO audit with “AI” written on it?

Partly, and we’d rather say so. The content checks overlap with good SEO, because an answer engine and a search engine both need to find the sentence that answers the question.

The crawl checks are different: other agents, other rules, and a CDN that may be turning them away without you knowing.