Updated: March 2026
TL;DR: llms.txt is a lightweight Markdown file placed at the root of your website that guides AI systems toward your most relevant, authoritative content. It was proposed in September 2024 by Jeremy Howard of Answer.AI and is not an official web standard. It does not replace robots.txt or sitemap.xml, and it does not guarantee AI citations or rankings. What it does is reduce the ambiguity AI systems face when trying to extract useful content from your site in real time. For businesses whose most important pages are buried in complex navigation, JavaScript-heavy layouts, or inconsistent internal structures, llms.txt acts as a clear guide. This post explains what it is, how it works, what it cannot do, and how to build one for your site today.
There is a version of your website that your customers never see.
It is what AI systems encounter when they arrive at your domain looking for content to cite. No images. No navigation. No hero section. Just raw code, navigation elements, tracking scripts, and somewhere inside all of that, the actual information the AI was trying to find.
AI systems like ChatGPT, Perplexity, and Google's AI Overviews do not index your site the way Google's traditional crawler does. They often retrieve content in real time, at the moment a user asks a question. They land somewhere on your site, attempt to extract relevant information, and move on. If your most important pages are hard to find from that entry point, they may not be found at all.
This is the problem llms.txt was designed to address.
Where llms.txt Came From
The llms.txt proposal was published in September 2024 by Jeremy Howard, co-founder of Answer.AI. The concept was straightforward. Just as robots.txt tells crawlers which pages they can and cannot access, and sitemap.xml enumerates a site's URLs for indexing, llms.txt would provide AI systems with a curated, human-readable guide to a site's most valuable content.
The format uses plain text with Markdown structure. It typically includes a short description of what the site covers, followed by organized sections linking to priority pages, each with a brief note on what that page contains.
Howard's stated motivation, documented in the original proposal, was practical. Large language models have limited context windows. They cannot process an entire website at once. Converting complex HTML pages filled with navigation, ads, and JavaScript into clean, readable text is computationally expensive and imprecise. A structured text file pointing directly to the right content reduces that friction.
The idea gained traction quickly. By early 2026, major technology companies including Vercel, Stripe, Cloudflare, and Coinbase had implemented llms.txt files. Mintlify documented that Anthropic, the company behind Claude, specifically requested that Mintlify implement both llms.txt and llms-full.txt for their documentation. Profound, a company that tracks AI crawler behavior, confirmed that models from Microsoft, OpenAI, and others are actively crawling and indexing both file variants.
It is worth noting clearly: llms.txt is a community proposal, not an official web standard. No governing body oversees it. Implementation varies across AI providers. Google's John Mueller stated in mid-2025 that Google's AI Overviews do not currently use llms.txt to determine citations. This does not mean the file is useless. It means the expectations around it need to be accurate.
How llms.txt Differs From robots.txt and sitemap.xml
Understanding what llms.txt is requires understanding what it is not.
robots.txt controls access. It tells crawlers which pages or directories they are permitted to retrieve. It is an instruction file.
sitemap.xml maps discovery. It lists URLs so search engines can find and index the pages on your site. It helps crawlers know what exists.
llms.txt guides retrieval. It tells AI systems which pages are most authoritative, relevant, and worth referencing. It is a curation file, not a control mechanism.
These three files serve complementary purposes. None replaces the others. A site with all three is better equipped for both traditional search and AI-assisted discovery than a site with only one or two.
One distinction worth understanding clearly: llms.txt does not grant or deny access. It does not prevent AI systems from crawling pages you exclude from it. If you want to restrict AI training crawlers from accessing your content, that belongs in robots.txt using directives like GPTBot or ClaudeBot. llms.txt only influences how AI systems find and prioritize content, not whether they can access it.
What llms.txt Actually Does (and Does Not Do)
The honest answer about what llms.txt delivers depends on which AI system you are talking about.
For AI agents that browse the web at the moment a user asks a question, such as ChatGPT's browsing mode, Perplexity, and Gemini, llms.txt can serve as an efficient entry point. Instead of landing on a page, attempting to follow internal links, and hoping to find the right content, an AI agent can read llms.txt first, understand the site's structure, and navigate directly to the most relevant pages. Mintlify's data showed that LLMs are accessing llms-full.txt even more frequently than the base llms.txt file, suggesting active use of these files during retrieval.
For Google's AI Overviews specifically, the evidence is more limited. Google's AI Overviews use retrieval-augmented generation that pulls from the existing search index, not from real-time file reads. Adding llms.txt does not change how Google's crawler indexes your content, and it does not directly influence which sources appear in AI Overviews. That remains driven by the same factors that influence traditional search rankings, including content quality, authority, structure, and relevance.
Vercel has reported that 10 percent of their signups now come from ChatGPT, a result they attribute to deliberate GEO strategy that includes llms.txt as one component. This is meaningful evidence, but it reflects a company with extensive technical documentation, a developer audience that uses AI assistants heavily, and a comprehensive AI visibility strategy. The same result cannot be assumed for every business category.
The honest framing is this: llms.txt is low effort, low risk, and potentially useful, especially for businesses whose audience uses AI assistants to research purchasing decisions. It is not a ranking signal. It is not a guarantee of citation. It is a structural improvement that reduces the ambiguity AI systems encounter when they arrive on your site.
Who Benefits Most From llms.txt
Not every business needs to prioritize this equally.
The businesses that benefit most from llms.txt share a common characteristic: their customers use AI assistants to research, compare, or evaluate their category before making a decision.
If someone is likely to ask ChatGPT, Perplexity, or Gemini a question that your business should be the answer to, then making your site easier for those systems to parse is a legitimate investment.
Service businesses with multiple offerings, location pages, and pillar content are well positioned to benefit. A clearly structured llms.txt that points AI systems toward your most authoritative service pages, your most credible case studies, and your clearest explanations of what you do can meaningfully reduce the chance that an AI system misrepresents or overlooks what you offer.
Businesses with documentation-heavy websites, technical products, or complex service structures benefit from the clarity llms.txt imposes on their content architecture. The act of deciding which pages belong in your llms.txt often reveals gaps in your content strategy that are worth addressing regardless of AI visibility.
E-commerce brands with product catalogs, buyer's guides, and comparison content are positioned to benefit as AI systems increasingly assist users in making product decisions. Yotpo's analysis noted that for e-commerce brands, the effort to implement llms.txt is often low while the potential benefit, ensuring products are accurately represented in AI-generated answers, is significant.
How to Create a llms.txt File
The format is intentionally simple. It lives at the root of your domain at yourdomain.com/llms.txt and uses Markdown formatting.
A well-structured llms.txt includes three elements.
A header that briefly describes who you are and what your site covers. This gives the AI system immediate context before it follows any links.
Organized sections that group related pages by topic or function, with a short descriptive note for each URL. The description should explain what the page contains, not just what it is called.
A focused list of your highest-priority pages. The goal is curation, not completeness. A llms.txt file pointing to 20 to 50 of your most important pages is more useful than one attempting to replicate your sitemap.
Here is an example of the basic structure:
# Rakos Media Group
> Digital marketing agency specializing in SEO, GEO, paid acquisition, and conversion-focused web design for service businesses in Buffalo NY and the Florida Gulf Coast.
## Services
- [SEO Services](https://rakosmediagroup.com/seo-service-company/): Organic search strategy including technical SEO, local SEO, and AI search visibility.
- [Google and Social Ads](https://rakosmediagroup.com/google-and-social-media-ads/): Paid acquisition strategy across Google Search, Google Display, and Meta platforms.
- [Conversion-First Web Design](https://rakosmediagroup.com/website-design/): Website design built around conversion psychology and Core Web Vitals performance.
- [AI Consulting](https://rakosmediagroup.com/ai-consultant-buffalo-rochester/): AI strategy, ChatGPT training, and AI workflow integration for service businesses.
## Frameworks
- [Momentum Framework](https://rakosmediagroup.com/momentum-framework/): Rakos Media Group's diagnostic approach to identifying growth friction before execution.
- [Search Everywhere Framework](https://rakosmediagroup.com/seo-for-ai-search/): The Rakos framework for visibility across traditional search, AI search, and local discovery.
## Educational Resources
- [How To Do Keyword Research](https://rakosmediagroup.com/how-to-do-keyword-research/): Step-by-step guide to keyword research using Google Trends, Keyword Planner, AnswerThePublic, and Ubersuggest.
- [Buffalo SEO Expert](https://rakosmediagroup.com/buffalo-seo-expert-who-builds-revenue-not-just-rankings/): How SEO connects to revenue strategy in competitive local markets.
- [Why Service Businesses With Good Reviews Still Don't Show Up in AI Search](https://rakosmediagroup.com/why-service-businesses-with-good-reviews-still-dont-show-up-in-ai-search/): Why traditional credibility signals do not automatically transfer to AI search visibility.
## About
- [Our Purpose](https://rakosmediagroup.com/purpose-driven-businesses-always-succceed/): The founders and values behind Rakos Media Group.
- [Case Studies](https://rakosmediagroup.com/case-studies/): Client results across healthcare, hospitality, and service businesses.The URLs, section labels, and descriptions should reflect your actual site. This example is provided as a structural reference. Every business's llms.txt will look different based on what their site contains and what their customers most often search for.
How to Add llms.txt to Your Site
WordPress Sites Using Yoast SEO
Yoast SEO has added llms.txt generation to its plugin functionality. Once enabled, Yoast automatically creates and maintains the file, selects relevant URLs from your published content, and refreshes the file weekly using WordPress cron jobs. You can preview the generated file before it goes live.
To enable it, update Yoast SEO to the current version, navigate to the Yoast settings panel, and look for the llms.txt option. The plugin handles the file placement and formatting automatically.
This is the lowest-friction implementation path for most WordPress sites. The limitation is that automated generation will not apply the editorial judgment that a manually curated file would. After enabling it, review the output and adjust which pages are included based on what you actually want AI systems to prioritize.
Manual Implementation
If you are not using Yoast or are on a different platform, the manual process is straightforward.
Create a plain text file named llms.txt. Write your header section, organize your pages into logical sections, and add a brief description for each URL. The file should use Markdown formatting.
Place the completed file in the root directory of your site. For most hosting environments, this means uploading it via FTP, SFTP, or your hosting control panel's file manager so it appears at yourdomain.com/llms.txt.
Test the file by visiting yourdomain.com/llms.txt in a browser. It should load as plain text. If it downloads as a file or returns an error, check your hosting configuration for how it handles text file requests from root.
Other WordPress Plugins
If you are not using Yoast, plugins such as Website LLMs.txt, which has accumulated over 30,000 installs, provide similar automated generation and can track whether AI crawlers like GPTBot or ClaudeBot are accessing the file. This crawler data is useful for monitoring whether AI systems are finding and reading your file.
Shopify and Other Platforms
Shopify sites can add llms.txt by placing the file in the theme's root directory and ensuring it is accessible without redirects. Other CMS platforms vary in their approach. The core requirement is that the file is accessible at the root path of your domain, loads as plain text, and uses correct Markdown formatting.
What to Include and What to Leave Out
The most important editorial decision in building your llms.txt is what not to include.
Include pages that represent your strongest, most authoritative content. Service pages. Evergreen educational content. Case studies. Foundational explanations of what you do and how you work.
Leave out pages that are transactional without context, such as checkout pages or contact forms with no surrounding explanation. Leave out thin or outdated content you would not want an AI system to cite as representative of your expertise. Leave out pages that are already gated or require a login, since AI systems cannot access them regardless.
Search Engine Land's analysis framed llms.txt well: it is not about letting AI systems wander your site blindly. It is about giving them a direct path to the content that reflects your actual expertise. The value of that framing is that it forces a useful question: if an AI system could only read twenty pages on your site, which twenty would best represent what you do and why someone should trust you?
Answering that question honestly is worth doing regardless of whether you publish a llms.txt file.
Maintaining Your llms.txt Over Time
A llms.txt file that points to outdated, restructured, or deleted pages does more harm than an absent one. It either misrepresents your current offerings or directs AI systems to broken or irrelevant content, which can produce inaccurate AI-generated answers about your business.
Review your llms.txt whenever you publish major new content, restructure your navigation, or remove or redirect pages. A quarterly review is sufficient for most businesses. If you are using an automated plugin, the generation handles URL currency, but you should still review the overall structure periodically to ensure the editorial curation reflects your current priorities.
If you add llms-full.txt, which is an expanded version that includes the full Markdown content of each referenced page rather than just the URL, you take on additional maintenance responsibility. Every page in llms-full.txt needs to stay current. The benefit is that AI systems can retrieve your complete content without needing to follow links, which reduces retrieval friction further. The tradeoff is that maintaining accurate full-content files requires more deliberate upkeep.
How llms.txt Fits Into a Broader AI Visibility Strategy
llms.txt is one layer of a multi-layer approach to AI search visibility. It is not a substitute for the work that actually determines whether AI systems cite you.
The content AI systems select to cite is driven by the same factors that have always determined authority: clarity of information, specificity of answers, structured presentation, credibility signals, and consistency of entity data across the web. A site with a polished llms.txt file and thin, poorly structured content will not be cited more frequently. A site with deeply useful, well-structured content and no llms.txt will still be cited.
What llms.txt does is reduce the friction between AI systems and your best existing content. It improves the likelihood that when an AI system arrives on your domain, it finds the right pages rather than the wrong ones. That is a meaningful operational improvement, particularly for sites with complex architectures or significant content depth.
The businesses that will earn consistent AI citation are those building genuine topical authority through research-backed content, clear entity signals, structured data, and consistent messaging across every channel where customers and AI systems might encounter them. llms.txt supports that strategy. It does not replace it.
What You Can Do This Week
If you want to implement llms.txt for your site this week, the process is practical.
- Start by identifying the ten to twenty pages on your site that best represent what you do, how you work, and why a customer should trust you. These are your foundation.
- Create your llms.txt file using the structure described above. Write a genuine one-sentence description for each page that explains what the page contains from the perspective of someone trying to understand your business.
- Upload the file to your root directory and verify it loads correctly at yourdomain.com/llms.txt.
- If you are on WordPress, enable Yoast's llms.txt feature or install a dedicated plugin and review the auto-generated output against your manual selection.
- Set a calendar reminder to review the file quarterly.
- Then return to the work that actually builds AI citation over time: publishing clear, specific, research-backed content that answers the real questions your customers ask.
Frequently Asked Questions
Does llms.txt guarantee that AI systems will cite my site?
No. llms.txt is a guidance mechanism, not a ranking signal. It reduces the ambiguity AI systems face when navigating your site, but citation decisions are made based on content quality, relevance, authority, and the specific query. Google's AI Overviews do not currently use llms.txt to determine citations. Other AI systems, including ChatGPT's browsing mode and Perplexity, may use it to navigate to relevant pages more efficiently, but inclusion is not guaranteed.
Will llms.txt affect my traditional Google search rankings?
No. llms.txt is not read by Google's traditional crawler and has no effect on organic search rankings. It complements robots.txt and sitemap.xml for AI retrieval purposes but operates entirely separately from conventional SEO signals.
Is llms.txt an official web standard?
Not yet. It is a community proposal introduced by Jeremy Howard of Answer.AI in September 2024. It has no formal governing body and adoption varies across AI providers. The proposal is open source and has an active development community, but implementation behavior differs between AI systems.
How is llms.txt different from robots.txt?
robots.txt controls access. It tells crawlers which pages they can and cannot retrieve, and includes directives for specific bots including AI training crawlers. llms.txt guides curation. It tells AI systems which of your pages are most authoritative and relevant, but it does not grant or restrict access. Both files serve different purposes and should coexist.
Do I need technical skills to create a llms.txt file?
No. The file uses plain text with basic Markdown formatting. WordPress users can generate it automatically through Yoast SEO or a dedicated plugin. Manual creation requires only a text editor and the ability to upload a file to your root directory via your hosting control panel or FTP.
How often should I update my llms.txt?
Review it quarterly at minimum, and immediately whenever you restructure your navigation, publish major new content, or remove or redirect important pages. Stale llms.txt files that reference outdated or deleted pages can produce inaccurate AI-generated answers about your business.
Does llms.txt prevent AI from using my content for training?
No. llms.txt does not control training data access. If you want to limit AI training crawlers from accessing your content, use robots.txt directives for specific bots such as GPTBot (OpenAI) and ClaudeBot (Anthropic). llms.txt only influences how AI systems navigate and prioritize content at inference time.
Sources
Jeremy Howard. llms.txt Proposal. Answer.AI. September 2024.
Mintlify. The Value of llms.txt: Hype or Real? May 2025.
Search Engine Land. llms.txt Isn't robots.txt: It's a Treasure Map for AI. July 2025.
Search Engine Land. Meet llms.txt, a Proposed Standard for AI Website Content Crawling. July 2025.
Yoast. What is llms.txt? How the New AI Standard Works. 2026.
Yotpo. What Is LLMs.txt? The Guide To AI Search and GEO. 2026.
Tacmind. llms.txt Explained: The LLM-Friendly File That Boosts AI Search Visibility. December 2025.
Search Signal. llms.txt in 2026: What It Does and Doesn't Do. January 2026.
