Markdown for AI Agents: What Accept: text/markdown Does and Whether to Turn It On

Short answer: When an AI agent sends Accept: text/markdown, a website can return the same page as markdown instead of HTML. This is content negotiation, a long-standing part of HTTP. Cloudflare turned it into a toggle in February 2026, and Claude Code already sends the header. On the six pages we measured, the markdown was 7 to 99 times smaller than the HTML. It does not help Google rankings; it makes your pages cheaper and cleaner for agents to read.

What is markdown content negotiation?

Every HTTP request carries an Accept header that lists the formats the client wants. Browsers send text/html. An agent that prefers markdown sends text/markdown, usually with text/html as a fallback. The server reads the header and picks a format; this has been part of HTTP (RFC 9110) for a long time. A correct markdown response carries two headers:

Content-Type: text/markdown; charset=utf-8
Vary: Accept

See Markdown content negotiation for the concept. It is one of the 9 core checks of Agent Readiness.

Why do AI agents want markdown?

Agents pay per token and have a context limit. An HTML page carries navigation, scripts, styles and CSS classes the agent does not need. Cloudflare's example: the heading ## About Us costs about 3 tokens, while the HTML version with a class and an id costs 12 to 15. We measured six pages on 19 October 2026, once as a browser and once as an agent:

Page	HTML (~tokens)	Markdown (~tokens)	Smaller by
vercel.com/docs	223,044	2,259	99×
docs.stripe.com/payments	110,265	1,289	86×
docs.claude.com (Claude Code overview)	139,316	4,129	34×
blog.cloudflare.com/markdown-for-agents	79,155	4,523	18×
developers.cloudflare.com (Markdown for Agents)	44,016	4,089	11×
citestat.com/en/blog/llms-txt-guide (after our fix)	14,657	2,075	7×

Source: CiteStat, scripts/growth/markdown-negotiation-check.ts. Tokens are estimated as characters divided by 4, a rough rule, not a real tokenizer. The ratio depends on what each site puts in its markdown; pages with heavy navigation and scripts shrink the most. None of the responses we received carried an x-markdown-tokens header.

The mistake we made

The CiteStat site had "supported" markdown since June: a middleware returned llms.txt whenever the Accept header contained text/markdown. Our own scanner marked it as passing, because it only asked the home page then (fixed on 21 October 2026). When we tested page by page, we found two bugs:

• An agent that asked for the blog post /en/blog/llms-txt-guide got llms.txt, a summary of the whole site, not the post.
• A header that lists text/html as a fallback (for example text/markdown, text/html, */*) was ignored, so the agent got the full HTML.

The fix: the home page still returns llms.txt (that really is the home page's markdown), each blog post returns itself as markdown, and other pages stay HTML. Every response carries Vary: Accept, and each post has a .md copy with a Link header pointing to it. The lesson: don't ask "does it return markdown?", ask "does it return this page?"

How to turn on markdown negotiation

Option 1: Cloudflare

If your site is on Cloudflare Pro or higher, enable Markdown for Agents in AI Crawl Control. Cloudflare converts the HTML at the edge, up to 6 MiB of HTML per page, adds Vary: Accept, drops ETag and Last-Modified on the markdown version, and adds a default Content-Signal header if your origin does not set one (Cloudflare docs). The Free plan does not include it.

Option 2: on your own server

1. Read the Accept header. Choose markdown when text/markdown is listed and not ranked below text/html (compare the q-values).
2. Build the markdown from the page's own content: title, main text, tables, absolute links. Leave out navigation, footer and scripts.
3. Return Content-Type: text/markdown; charset=utf-8 and Vary: Accept. Set Vary: Accept on the HTML version too.
4. Optional: a .md copy at the page URL plus .md, announced with Link: </post.md>; rel="alternate"; type="text/markdown" and listed in llms.txt.

Test with two commands: curl -sI -H 'Accept: text/markdown' https://your-site/page should show a markdown Content-Type and Vary: Accept; curl -s -H 'Accept: text/markdown' https://your-site/page | head should start with that page's own title.

Five common mistakes

1. Returning a site summary for every URL (our mistake). The agent asks for one page and gets another.
2. No Vary: Accept. A CDN can cache the markdown and serve it to browser visitors.
3. Matching the Accept header too strictly. Accepting only an exact text/markdown misses agents that also list text/html.
4. Markdown without the main content, because JavaScript builds it. A converter sees only the HTML the server sends, just like AI crawlers that do not run JavaScript.
5. Treating it as a ranking signal. Google's AI features guide says no special files or markup are needed for AI Overviews.

What should come first?

Markdown negotiation is a last step, not a first one. Before it: server-rendered HTML, a robots.txt that does not block the AI crawlers you want, and the title, description and main text in the first HTML response. A small shop can stop there. Documentation, API references and pricing pages, which coding assistants and agents read most, are worth turning it on for.

How does CiteStat check this?

When you scan a URL, CiteStat sends Accept: text/markdown, text/html, */* to that URL itself and passes it only when the response is markdown, its title starts the page's H1 or title (not a site summary), and it carries Vary: Accept. The result sits on the Agent Readiness card next to 8 other checks such as Link headers, MCP Server Card and API Catalog. To test other pages, use the two curl commands above.

The free plan includes 10 scans a day. If you manage several sites or need the full AI render report, see pricing: Pro is 500.000₫/month (excl. VAT) for 100 scans a day.

FAQ

Does Accept: text/markdown help Google rankings?

No. Google reads normal HTML, and Google's AI features guide says no special files or markup are needed to appear in AI Overviews or AI Mode. Markdown negotiation makes your pages cheaper to read for AI agents and coding assistants; it is not a ranking signal.

Which agents send Accept: text/markdown?

Cloudflare names Claude Code and OpenCode in its Markdown for Agents launch post (12 February 2026). Other agents may send it without saying so; your server logs are the reliable source.

Is returning different content to bots and people cloaking?

Not if the markdown carries the same content as the HTML page. Cloaking means showing bots different content. Here only the format changes, and the choice follows the Accept header the client sends, not its user agent.

Do I need a Vary: Accept header?

Yes. Vary: Accept tells CDNs and caches that one URL has two versions depending on the Accept header. Without it, a cache can hand the markdown to a browser or the HTML to an agent.

Should I use negotiation or separate .md files?

You can do both. Negotiation serves agents that ask for markdown at the normal URL. A .md copy (the page URL plus .md) is what the llms.txt spec suggests, and a Link rel="alternate" type="text/markdown" header helps agents find it.