llms.txt, robots.txt and the 20-minute agent readiness checklist

There are two different questions hiding inside “are we visible to AI”. One is whether the engines can fetch your pages and make sense of them. The other is whether, having done so, they name you when a buyer asks. This checklist is about the first question only. It is the cheaper one, it is entirely under your control, and it is the one most sites still fail.

1. Let the right bots in

Start with robots.txt, because a block there makes everything else moot. The engines send three kinds of crawler, and the trade-offs differ.

The decision is a policy one, not a technical one. Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended) is a legitimate choice about whether your content trains models, and it does not remove you from answers. Blocking search and user-fetch crawlers does remove you, silently, from the answers your buyers read. Many sites block everything with an AI-sounding name and then ask why they are absent. Decide the policy, write it down, and make robots.txt match it.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

Sitemap: https://www.example.com/sitemap.xml

The free crawler access check reads your live robots.txt and shows, bot by bot, who is allowed. Run it after every deploy that touches the file.

2. Server-render the text that matters

Assume the fetcher does not run JavaScript. Some may; none publish exactly what they render, and you should not build on a guess. Open a product page with JavaScript disabled, or fetch it with curl, and read what is left. Product name, price, availability, key specifications, the answers to the obvious questions: all of it should be in the HTML the server sends.

Common failures: specifications inside an accordion that only populates on click, prices injected by a pricing script, FAQ content loaded from a widget, and single-page apps whose HTML is an empty shell with a spinner. Each is invisible to a fetcher that reads the HTML and moves on.

3. One H1, a canonical, a declared language

Small signals, cheap to fix, often wrong. One H1 per page that says what the page is. A canonical link so duplicates collapse to one URL. A lang attribute on the html element, set per market, so a German page is read as German. A title long enough to say what the page is and short enough to survive truncation, and a meta description that says something specific.

4. JSON-LD: Organization, WebSite, Product

Structured data gives a fetcher facts without parsing prose. Three types cover most sites. Organization on the home page: legal name, logo, sameAs links to your official profiles. WebSite with the site name and, if you have one, the search action. Product on every product page: name, brand, image, offers with price, currency and availability, and aggregateRating only if it is real and visible on the page.

Keep it consistent with what a human sees. Structured data that contradicts the page is worse than none: an engine that notices the mismatch has a reason to trust neither.

5. Publish /llms.txt

An llms.txt is a plain Markdown file at the root of your domain: a short description of what you offer and a curated list of the pages an agent should read first, with a line on each. It is a community convention, not a standard, and no engine has documented that it changes answers. We still recommend it, for three reasons: it costs twenty minutes, it is harmless, and writing it forces a team to decide which pages actually explain the business.

# Nordkamm

> Alpine hardshells and mid-layers, designed and tested in the Alps. Sold direct and through specialist retailers in CH, DE, AT and the US.

## Products
- [Gratwand Pro jacket](https://www.nordkamm.example/gratwand-pro): 3-layer waterproof hardshell for alpine climbing, 470 g, EUR 649
- [Firn mid-layer](https://www.nordkamm.example/firn): breathable insulation for high-output days

## Buying
- [Size guide](https://www.nordkamm.example/size-guide): fit notes by product
- [Where to buy](https://www.nordkamm.example/retailers): retailer list by country

## Company
- [About](https://www.nordkamm.example/about): who we are, where we test
- [Repairs and warranty](https://www.nordkamm.example/repairs)

6. A reachable sitemap and Open Graph tags

The sitemap tells crawlers what exists and when it changed. Reference it from robots.txt, keep it within the size limits, and make sure it lists the pages you want read rather than every faceted filter URL. Open Graph title and image matter less for answers than for previews, but the engines’ user-facing surfaces do render link cards, and an empty card is a missed chance.

Check it in 20 seconds, then measure separately

The free visibility report checks exactly these signals from a domain: crawler access per engine from robots.txt, server-rendered text, H1 and canonical, JSON-LD, llms.txt, sitemap and Open Graph. It returns a readiness score out of 100 with each check labelled pass, warn or fail. Every result is read from public signals; nothing is estimated.

Once the checklist is green, the remaining question is what the engines say. That is where measurement begins, and where the research notes pick up.