Content Signals in robots.txt: search, ai-input, ai-train and the Wrong-Group Mistake
Content-Signal tells bots if your content may feed search, AI answers or AI training. What the signals mean, where the line goes, and what 8 sites got wrong.
By — CTO, ERPFit
Short answer: Content Signals are a Content-Signal: line in robots.txt, for example Content-Signal: search=yes, ai-input=yes, ai-train=no. The line tells bots which of three uses you agree to: search, AI answers, or AI training. It must sit inside a User-agent group, usually User-agent: *. It does not block crawling, and it is a preference, not a command: a bot can ignore it.
What the three signals mean
Cloudflare published the Content Signals Policy on 24 September 2025 (updated 15 July 2026) under CC0, so anyone may use it. The policy's definitions:
| Signal | Meaning (per the policy) | Example |
|---|---|---|
search | Building a search index and providing search results: links and short excerpts. It does not include AI-generated search summaries. | A list of ten links |
ai-input | Putting content into an AI model when it answers: RAG, grounding, real-time use of content for AI search answers. | An AI answer with cited sources |
ai-train | Training or fine-tuning AI models. | Model training data |
yes means content may be collected for that use; no means it may not. A missing signal "conveys no meaning": you neither grant nor restrict. So the line search=yes, ai-train=no says nothing about AI answers.
Where the line goes: the wrong-group mistake
robots.txt is split into groups. Each group starts with one or more User-agent: lines and runs until the next User-agent: line (RFC 9309). Blank lines and # comments do not end a group. The IETF draft that attaches AI preferences to robots.txt also defines its rule as part of a group. So a Content-Signal: line at the end of the file belongs to the group just above it.
On 21 October 2026 we read the robots.txt of 8 sites with the code the CiteStat scan uses (packages/core/content-signals.ts):
| Site | Content-Signal line | Applies to |
|---|---|---|
| blog.cloudflare.com, developers.cloudflare.com, docs.stripe.com | search=yes, ai-input=yes, ai-train=yes | Every bot (User-agent: *) |
| citestat.com | search=yes, ai-input=yes, ai-train=no | Every bot (User-agent: *) |
| www.cloudflare.com | ai-train=yes, search=yes, ai-input=yes | Cohere-ai only |
| vercel.com, docs.claude.com | None | - |
| erpfit.com | Only the policy text, in # comments | No bot can read it |
www.cloudflare.com is the clearest lesson. The file lists 8 groups for individual AI bots (GPTBot, the old Claude-Web, PerplexityBot…), the last one is User-agent: Cohere-ai, then a comment, then the Content-Signal: line. By RFC 9309 grouping, that line is a preference addressed to Cohere only. The intent was clearly every bot, but a parser that follows the standard will not read it that way.
erpfit.com shows the second common mistake: the full policy text (a long comment block) copied in, but no real Content-Signal: line. Comments are for human readers. citestat.com's own robots.txt had this mistake until 21 October 2026.
How to write it
Put the line right after User-agent: *, before the Allow/Disallow lines:
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
Sitemap: https://your-site/sitemap.xmlIf you have a separate group for one bot (for example User-agent: GPTBot), that bot reads only its own group and ignores the * group. If it should get the signals too, add the Content-Signal: line to that group as well. How to write a group per bot: AI crawlers and robots.txt.
Which values? A site that wants to be cited by AI should keep ai-input=yes: that is the signal for AI answers with sources. ai-train is a separate choice; no does not make an answer less likely to cite you. Google Search Console may report "Syntax not understood" for the line; Cloudflare's documentation (updated 8 October 2026) says it has seen no effect on crawl rate or SEO.
Content-Signal and the IETF's Content-Usage
Cloudflare's IETF draft (draft-romm-aipref-contentsignals-00) is an individual draft and has expired. The IETF's official working group, AI Preferences (aipref), is drafting a different syntax:
| Cloudflare Content Signals | IETF aipref (draft) | |
|---|---|---|
| Line name | Content-Signal: | Content-Usage: |
| Labels | search, ai-input, ai-train | search, ai-use, train-ai |
| Values | yes / no | y / n |
| Where | robots.txt | robots.txt (per path) and an HTTP header |
| Status | In use; IETF draft expired | vocab-08 (14 Sep 2026), attach-05 (19 Aug 2026) |
The two share the concepts but not the names, so a reader that knows one syntax will not see the other. That is why the CiteStat scan reads both, including the Content-Usage header. Today Content-Signal is the syntax found on real sites; we will update this post when the IETF drafts become a standard.
Is Content-Signal binding?
No. Cloudflare writes that signals "express preferences; they are not technical countermeasures against scraping", and that some companies might ignore them. The policy text says that restrictions expressed through Content Signals are reservations of rights under Article 4 of EU Directive 2019/790 on text and data mining; what that means legally depends on the country. In the documentation we read, neither Google nor OpenAI has said it follows the line. To actually block a bot, use a Disallow for it (see AI crawlers). To control what Google's AI Overviews quote, use nosnippet and max-snippet.
Check your own site in one minute
Run curl -s https://your-site/robots.txt and find the Content-Signal: line. Check three things: the line does not start with #; the nearest User-agent: line above it is * (or the bot you mean); each value is yes or no.
What does CiteStat check?
In its Agent Readiness section, the CiteStat scan reads robots.txt by group as above, plus the page's Content-Signal header. It passes when at least one readable declaration exists, and it reports signals that exist only in comments or have unreadable values. The free plan includes 10 scans a day. If you manage several sites, see pricing: Pro is 500.000₫/month (excl. VAT) for 100 scans a day.
FAQ
Does Content-Signal block AI crawlers?
No. Content-Signal only says how you want content used after it is collected. Cloudflare states that signals express preferences, are not technical countermeasures, and that some companies might ignore them. To stop collection, use a Disallow for each bot in robots.txt or block it at the firewall.
Where does the Content-Signal line go in robots.txt?
Inside a User-agent group, usually User-agent: *. Under RFC 9309 a group runs until the next User-agent line, so a line placed at the end of the file, after a specific bot's group, belongs to that bot only.
What does leaving a signal out mean?
Nothing. Under Cloudflare's policy, when a signal is missing the site operator neither grants nor restricts permission for that use via content signals.
How is Content-Signal different from Content-Usage?
Content-Signal is Cloudflare's syntax (search, ai-input, ai-train with yes/no). Content-Usage is the syntax in the IETF AI Preferences working group drafts (search, ai-use, train-ai with y/n), as a robots.txt rule and an HTTP header. Neither is a final standard: Cloudflare's IETF draft has expired, and the aipref drafts are still in progress.