Content Signals — Content-Signal: search, ai-input, ai-train in robots.txt
TL;DR
Content Signals are a robots.txt line, Content-Signal: search=yes, ai-input=yes, ai-train=no, that says how content may be used after it is fetched: for a search index, as input to AI answers, or to train AI models. robots.txt Allow/Disallow says who may fetch; Content-Signal says what they may do with it.
Explain Simply
Allow and Disallow are the gate: who may come in. Content-Signal is the sign behind the gate: you may photograph for the guidebook (search), you may quote us when someone asks (ai-input), you may not copy the whole house to build your own (ai-train).
The syntax: one line inside a User-agent group of robots.txt, next to Allow and Disallow:
User-Agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /Each key takes yes or no, separated by commas. Cloudflare's Markdown for Agents also sends the same list as a Content-Signal HTTP response header.
The three uses (Cloudflare Content Signals Policy, CC0):
- search — building a search index and showing links and short excerpts. It does not include AI-generated summaries.
- ai-input — putting the content into an AI model to answer a question: retrieval-augmented generation, grounding, AI search answers.
- ai-train — training or fine-tuning an AI model.
A key you leave out neither grants nor restricts that use — it is not a "no".
Who started it: Cloudflare published the policy on 24 September 2025 and added Content-Signal: search=yes, ai-train=no to the managed robots.txt of its customers (over 3.8 million domains). Free zones without their own robots.txt get only the policy text as comments, with no signal. The labels are written up as an individual IETF draft (draft-romm-aipref-contentsignals-00, expired April 2026); the IETF AI Preferences (aipref) working group drafts a rival syntax, Content-Usage: train-ai=n (labels search, ai-use, train-ai; values y or n; a robots.txt rule with an optional path, or an HTTP header — draft-ietf-aipref-attach-05 and vocab-08, not yet a standard).
What it is not: meta robots and X-Robots-Tag decide indexing and snippets on a page; they say nothing about AI training. A signal in a # comment is ignored — crawlers skip comments.
⚡Why It Matters
Content-Signal is a preference, not a lock: a crawler must choose to read and follow it — nothing in robots.txt can enforce it. It still costs one line, it records your choice in a machine-readable place, and under the policy text it is an express reservation of rights under Article 4 of EU Directive 2019/790. To actually stop a crawler, use Disallow for its user-agent (see AI crawlers).
🔍How CiteStat Checks
CiteStat reads the site's robots.txt and the page's response headers. It passes when a Content-Signal line (or header) has at least one readable pair for search, ai-input or ai-train — or an IETF Content-Usage rule (or header) has one for search, ai-use or train-ai — and lists what each group declares. A line with no readable pair, signals written only in # comments, or no signal at all are low findings. meta robots and X-Robots-Tag no longer count as content signals.
CiteStat Score Impact
One of the 9 core checks on the Agent Readiness card (score = core checks passed / 9 × 10). It does not change the headline SEO score.
Fix Difficulty
Editor: Easy | Developer: Easy | AI Agent: Easy