The /llms.txt Standard: Optimizing Web Content for AI Agents & LLMs

Optimizing Web Content for AI Agents & LLMs

Large language models (LLMs) are increasingly reliant on accessing information across the web, yet they face a critical limitation: context windows are generally too small to handle most websites in their entirety. Furthermore, converting complex HTML pages, which often include navigation, ads, and JavaScript, into concise, LLM-friendly plain text is both difficult and imprecise.

To bridge this gap, a proposal standardizes the use of the /llms.txt file. This markdown file is designed to provide specific, LLM-friendly content, helping models utilize website information efficiently at inference time.

Concise, Expert-Level Content

While websites cater to both human readers and LLMs, the latter benefit greatly from concise, expert-level information gathered in a single, accessible location. This is particularly useful in environments like development, where LLMs need quick access to API or programming documentation.

The /llms.txt proposal outlines two main aspects:

1. The /llms.txt File: A file located at the root path of a website (or optionally in a subpath). It uses Markdown, currently the most widely and easily understood format for language models. This file provides brief background, guidance, and links to more detailed markdown files.
2. Clean Markdown Pages: Websites are proposed to offer a clean markdown version of pages that might be useful for LLMs to read. This is achieved by accessing the same URL as the original page, but with .md appended (or index.html.md for URLs without a file name). For example, the FastHTML project follows both of these proposals for its documentation.

The llms.txt markdown is human-readable and LLM-readable, but crucially, it is also in a precise format that allows classical programming techniques, such as parsers and regex, to process it effectively.

Versatility Across Use Cases

The versatility of the llms.txt file means it can serve a wide variety of purposes:

• Software Development: Helping developers navigate complex software documentation.
• Business: Giving companies a way to outline their structure.
• Legislation: Breaking down complex legislation for stakeholders.
• E-commerce: Explaining products and policies.
• Academia: Providing quick access to course information and resources for schools and universities.

Understanding the Structured Format

Though the /llms.txt file uses Markdown rather than a classic structured format like XML, the information follows a specific, programmable format. Providing this basic structure is key to helping a language model find the information it needs.

A file following the specification, located at /llms.txt, must contain sections in this specific order:

1. H1 Header: The name of the project or site. This is the only required section.
2. Blockquote: A short summary of the project, containing key information necessary for understanding the rest of the file.
3. Optional Markdown Sections: Zero or more sections (paragraphs, lists, etc., but not headings) containing more detailed information or guidance on how to interpret the provided files.
4. H2 Delimited File Lists: Zero or more sections delimited by H2 headers, containing "file lists" of URLs where further detail is available.

Each item in a "file list" is a markdown list containing a required markdown hyperlink [name](url), optionally followed by a colon and notes about the file.

A notable feature is the special "Optional" H2 section. If this section is included, the URLs provided there can be skipped when a shorter context is needed, making them suitable for secondary information.

For example, the FastHTML project’s llms.txt includes detailed documentation links under an "H2 Docs" header and links to code examples under an "H2 Examples" header.

LLM Inference vs. Search Indexing

The llms.txt proposal is designed to coexist seamlessly with current web standards.

Standard

Purpose

Relationship with llms.txt

/robots.txt

Lets automated tools (like search index bots) know what access is acceptable.

llms.txt complements robots.txt by providing context for content that is allowed.

/sitemap.xml

Lists all indexable human-readable information on a site.

llms.txt is not a substitute for sitemap.xml. It offers a curated overview for LLMs, includes LLM-readable versions of pages, and may link to external sites that are helpful for context, unlike sitemaps.

The approach of standardizing on a path follows the pattern of /robots.txt and /sitemap.xml. However, the core purpose is different: llms.txt is mainly useful for inference—the moment a user is seeking assistance (e.g., asking a chatbot for information or including a coding library’s documentation in a project).

Getting Started

To create effective llms.txt files, remember these guidelines:

• Use concise, clear language.
• Avoid ambiguous terms or unexplained jargon.
• Include brief, informative descriptions when linking to resources.
• Test your file using a tool that expands the content into an LLM context file to see if models can answer questions about the content.

Several tools and integrations are available to help standardize this process, including llms_txt2ctx (a CLI and Python module for parsing and context generation), and plugins for popular documentation frameworks like VitePress and Docusaurus.

The specification is currently open for community input, with a GitHub repository hosting the overview and a community discord channel available for sharing experiences and best practices. You can find directories listing available llms.txt files at sites such as llmstxt.site and directory.llmstxt.cloud.