Web Bot Auth — signed requests for bots and agents
TL;DR
Web Bot Auth lets a bot prove who it is. The bot signs each request with HTTP Message Signatures (RFC 9421) and names, in a Signature-Agent header, where its public keys are. Its operator publishes those keys at /.well-known/http-message-signatures-directory. It is an IETF webbotauth working-group draft (draft-ietf-webbotauth-httpsig-protocol-00, 1 September 2026) by authors from Cloudflare and Google; ChatGPT's agent already signs this way (see /blog/web-bot-auth).
Explain Simply
A user-agent string is a name tag anyone can write. Web Bot Auth is an ID card: the bot shows a signature, and the site checks it against a public list that only the bot's real owner can publish on its own domain.
Two sides, two jobs:
- The bot operator (an AI company, a crawler, a monitoring service) signs every request and publishes its public keys.
- The website verifies. It, or its CDN, reads the signature, fetches the bot's key directory once, caches it, and decides what to allow. A website that sends no bots publishes nothing.
A signed request (draft-ietf-webbotauth-httpsig-protocol-00, 2026-09-01):
GET /page HTTP/1.1
Host: example.com
Signature-Agent: sig="https://bot.example"
Signature-Input: sig=("@authority" "signature-agent";key="sig");created=1700000000;expires=1700011111;keyid="<JWK thumbprint>";tag="web-bot-auth"
Signature: sig=:<signature>:The key directory, on the bot's origin:
GET /.well-known/http-message-signatures-directory
Content-Type: application/http-message-signatures-directory+json
{"keys": [{"kty": "OKP", "crv": "Ed25519",
"x": "<public key>", "kid": "<JWK SHA-256 thumbprint>"}]}The rules that matter: the tag MUST be web-bot-auth; the signature covers @authority (or @target-uri) and the Signature-Agent member; created, expires and keyid are required, keyid being the base64url JWK SHA-256 thumbprint (RFC 7638); the directory MUST be HTTPS, answer 200 (no redirects) and use the media type above; a kid in the directory MUST equal the thumbprint.
Not Web Bot Auth: /.well-known/web-bot-auth.json. No draft defines it, and no verifier reads it.
⚡Why It Matters
User-agent strings and IP lists are easy to fake or go stale. A signature lets a site tell the real ChatGPT agent from a scraper that copies its name, and let it through a bot wall that would block the copy. On 25 October 2026 CiteStat checked 7 origins: chatgpt.com and Cloudflare's research example publish a valid directory (Cloudflare's sends nbf in milliseconds, a JWK date is seconds); github.com, vercel.com, www.cloudflare.com, developers.cloudflare.com and citestat.com publish none, which is correct for sites that send no signed bots.
🔍How CiteStat Checks
CiteStat fetches /.well-known/http-message-signatures-directory on the scanned origin. A JWK Set served as application/http-message-signatures-directory+json with at least one usable key (OKP, EC or RSA with its public members, kid equal to the thumbprint, inside nbf/exp, no private d) passes. JSON that is not a JWK Set, or no usable key, is a medium finding; the wrong media type is low. No directory means the site sends no signed bots: the check is reported as not applicable. The scan does not test whether your server verifies signatures.
CiteStat Score Impact
A core check on the Agent Readiness card only when the origin publishes a directory; otherwise it is left out of the score (score = core checks passed / core checks that apply × 10). It does not change the headline SEO score.
Fix Difficulty
Editor: Not relevant | Developer: Medium | AI Agent: Easy