Don’t expect llms.txt to earn AI citations. SE Ranking’s study of about 300,000 domains found no measurable link between having the file and being cited, and removing it from their model made the predictions better. About 10.13% of the domains had one. Google says Search ignores the file. It may still give AI agents a useful map of your site, so a file you already have is fine to keep.
Contents
- What the 300,000-Domain Study Found
- Do AI Crawlers Even Read the File?
- Why It Doesn’t Earn Citations
- What Correlates With AI Citation Instead
- The WordPress Plugin Problem
- Google’s Guidance: AI Search Is Still Search
- When llms.txt Is Worth Having
- What to Do With the Time You’d Spend on llms.txt
- FAQ
What the 300,000-Domain Study Found
SE Ranking (vendor data) compared llms.txt presence with how often each domain appeared as a source in sampled LLM responses. It used Spearman correlation, an XGBoost regression and SHAP analysis. Its article doesn’t name the engines it sampled.
The key result: “when we removed the LLMs.txt factor, the model’s predictions actually improved.” The authors read this as the variable adding noise rather than value.
Adoption was similar in the three traffic bands SE Ranking reports, which also argues against the file being a mark of successful sites. It was 9.88% for sites with 0 to 100 visits, 10.54% for 1,001 to 5,000 visits and 8.27% for sites above 100,000 visits.
The limits matter. This is an observational study of one dataset, and the authors call their findings “context-dependent.” It shows no measurable link to citations. It can’t prove the file never helps any site, and it says nothing about how agents use the file once they arrive.
Do AI Crawlers Even Read the File?
SE Ranking measured the outcome. Ahrefs’ June 2026 analysis (vendor data) measured the input: who requests the file. Across 137,210 domains in its Web Analytics product in May 2026, 28% published an llms.txt, and 97% of those files received no requests that month.
The roughly 22,000 requests that did arrive went to about 1,100 domains. AI retrieval bots, the kind that fetch pages to answer a question, made 233 of them. Training crawlers asked for the file nearly five times as often. GPTBot accounted for 4.51% of requests against 0.74% for OAI-SearchBot, and Slackbot fetched it more often than PerplexityBot.
Read those as ceilings, as Ahrefs does, since a fetch doesn’t prove a bot used the file. Ahrefs also says its customers skew technical and SEO-aware, which makes its 28% an upper bound. That is the likely reason it sits so far above SE Ranking’s 10.13%.
Adoption counts also include a lot of defaults. Common Crawl’s July 2026 analysis of 584,107 files found about 68% were template-generated, with Wix alone at 41.34%. Only 49.90% had the full structure of title, summary and link sections, and 22.56% contained no links at all. A rising count mostly tells you what platforms switched on.
Why It Doesn’t Earn Citations
llms.txt was never designed as a citation signal. Jeremy Howard’s September 2024 proposal describes a file to “help agents use a website,” one that is “used on demand, when an agent needs information about a topic while assisting a user.”
The explanation below is our reading, since the study didn’t test why. OpenAI and Perplexity describe search crawlers that surface pages and user-triggered fetchers that visit them. Citations point to pages that answered a query. A separate file listing your best pages doesn’t change which pages a search retrieves, and we found no study showing otherwise.
The Ahrefs request data fits that reading: if retrieval bots rarely open the file, it has little chance of affecting what they cite.
What Correlates With AI Citation Instead
None of these is proven to cause citations. They are the better-supported associations, and each comes from a study with its own scope.
Brand mentions across the web. Ahrefs’ May 2025 study of 75,000 brands (vendor data) found a 0.664 Spearman correlation between branded web mentions and AI Overview visibility, against 0.218 for backlinks. It covers Google AI Overviews only.
Recent content. TJ Digital’s October 2026 study (agency data, 78 client companies) found 88% higher odds of citation for pages published or updated in the past 12 months, after adjusting for domain rating and page type. The author says plainly that these are associations.
Earned coverage. Muck Rack’s analysis of more than 25 million links (vendor data) puts earned media at about 84% of citations in ChatGPT, Claude and Gemini. That describes where citations go. It doesn’t show that any one placement will earn you one, and Muck Rack’s page doesn’t define its earned category.
Structured data, with less confidence. The evidence is mixed and depends on schema type and platform. TJ Digital found 13% higher odds for pages with schema, Rankability (vendor data) found that no schema type consistently improved citation position, and Google says no special markup is needed.
The WordPress Plugin Problem
A search for “llms.txt” in the WordPress plugin directory returned 727 results on 10 November 2026. Not all of them generate the file, but the volume shows how easy this feature is to sell.
Don’t pick a plugin because it generates llms.txt. If you already use one that does, leave it on, since the file does no harm in Google Search. Judge the plugin on what else it does for your site.
Google’s Guidance: AI Search Is Still Search
Google’s AI optimization guide says: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search.” On llms.txt specifically, it says that creating the file for other services is fine, and that “doing so will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.”
The guide also says “you don’t need to write in a specific way just for generative AI search” and that “there’s no requirement to break your content into tiny pieces.” It warns that creating separate content for every search variation primarily to manipulate rankings or AI responses violates Google’s policies.
For AI Overviews and AI Mode, a page must be indexed and eligible to show a snippet, and Googlebot access is the control. Measure the result with the Generative AI performance report in Search Console.
This covers Google only. It says nothing about how ChatGPT, Claude or Perplexity treat the file.
A note on agents. SE Ranking measured citations. llms.txt may still give an AI agent a useful map of your site, which is a separate question and closer to what the proposal was for.
When llms.txt Is Worth Having
Citations aren’t the only use. The proposal was written for agents helping a user, and Ahrefs found that Claude Code fetched the file more often than every AI retrieval bot and assistant in its data. Developer tools are the clearest case, and the rest of this table is our judgment.
| Your site | What to do | Why |
|---|---|---|
| Developer docs or an API | Build one by hand and link to clean Markdown pages | Closest to the proposal’s purpose, and coding agents are among the few real readers |
| CMS or plugin offers it | Leave it on and don’t maintain it by hand | It costs nothing, and Ahrefs gives the same advice |
| B2B or marketing site without one | Skip it for now | No citation evidence, and the work below is worth more |
| Large ecommerce catalog | Skip it | A curated list can’t cover the catalog, and crawlers reach products through pages and feeds |
| Any site that has one | Review it like code | Ahrefs advises this for security, since an agent may act on what the file says |
If you do build one, the format is small. Only an H1 with the site name is required. A blockquote summary and H2 sections of links are optional, and an H2 named “Optional” marks links an agent can skip when it needs less context.
Check your own logs before deciding. Client-side analytics won’t show this, so use server logs. In the common combined log format, this counts requests by user agent:
grep "GET /llms.txt" access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rnIf the list is SEO tools and nothing else, leave the file alone. If a coding agent or a user-triggered fetcher such as ChatGPT-User keeps appearing, confirm that every link in the file returns a 200 and points at a current page. That is all the upkeep it needs.
What to Do With the Time You’d Spend on llms.txt
Four things to do this week instead:
- Check that AI search crawlers can reach you. In robots.txt, confirm that Googlebot, OAI-SearchBot, Claude-SearchBot and PerplexityBot aren’t caught by a broad disallow, then check your CDN rules too. A blocked crawler is the one problem here with a certain effect.
- Refresh your five most important pages. Update what has actually changed and show the revision date. Pages updated within the past year had better odds in TJ Digital’s data.
- Find one source that AI engines already cite in your category. Run your target prompts, note the third-party sites that keep appearing, and pitch one of them something specific.
- Add Organization schema with
sameAslinks if you don’t have it. It removes ambiguity about who you are, even though its effect on citations is unproven.
LLMagnet measures AI visibility across seven engines, and its agent prepares fixes for your approval. The free audit shows where you stand today.
Gemini
30-day free trial · no credit card
