Markdown for AI agents: which agents ask for it, and what turning it on signals

Oct 25, 2026 · 13 min read

Markdown for AI agents is a plain-text copy of a web page, served to an AI agent or other software that asks for one instead of the HTML. On Cloudflare it is a switch called Markdown for Agents: when a request's `Accept` header includes `text/markdown`, the network fetches the page from the site's server and converts it on the way out, while people using a browser keep getting HTML. Two things come with the switch that its name does not tell you. In what we read, the clients documented as asking for the copy are coding agents, not the crawlers behind AI answers. And unless your server says otherwise, every converted response carries a stated preference about how AI may use your content.

Cloudflare's documentation, updated on 9 October 2026, spells it out: "When the origin response does not include a content-signal header, Markdown for Agents adds a default Content-Signal: ai-train=yes, search=yes, ai-input=yes, signaling that the content can be used for AI Training, Search results, and AI Input, which includes agentic use."

We have an interest here. Our product tracks the pages AI answers name as their sources, so whether serving machines a different format changes those answers is close to our own business. We have not tested that, and nothing on this page says a plain-text copy helps or hurts. The only thing we checked ourselves is which headers three pages send back, described below.

To write this, we read the company's documentation and posts, the crawler pages of OpenAI, Anthropic, Perplexity and Google, Google's guidance, two published tests, and the first nine Google results for this phrase in the United States, listed at the end.

What does the switch actually do?

Cloudflare's switch converts a page's HTML to Markdown at the network edge, and only for requests that ask for it. The HTML page itself is unchanged, and human visitors still get it. This is ordinary HTTP content negotiation: the client lists the formats it accepts, and the response adds `Accept` to its `Vary` field so caches keep a page in a different format apart from the original. The feature is available "to Pro, Business and Enterprise plans, and SSL for SaaS customers at no cost", and is turned on in the dashboard's AI Crawl Control section or, for chosen subdomains or paths, with a configuration rule. It converts HTML only, up to 6 MiB of decompressed HTML.

The converted page is not a raw dump of the original. A short frontmatter block carries the title, description and image from the page's meta tags; navigation, headers, footers, scripts and styles are stripped; and any JSON-LD structured data is kept, as a fenced block at the end. That last detail separates it from the extraction pipelines discussed on our schema page, which are described there as discarding script tags.

The case for the feature is cost. The company's own launch post, published on 12 February 2026, runs to 16,180 tokens as HTML and 3,150 as Markdown, by its count. That is the vendor's count of its own page, not a rate for the web.

Some site owners already have the copy switched on without pressing anything. Read the Docs serves it on every documentation domain it hosts, "with no configuration required", and says the feature "is powered by Cloudflare". The example response on its page also carries the same default value.

Which agents ask for Markdown?

Coding agents are the clients documented as asking for Markdown in the pages we read. None of the four AI answer companies whose crawler pages we read says its crawlers ask. The launch post names the coding tools: "some of the most popular coding agents today – like Claude Code and OpenCode – send these accept headers with their requests for content."

Three of seven coding agents asked for Markdown in a February 2026 test of their fetch tools by Checkly, a maker of monitoring software. The three were Claude Code, Cursor and OpenCode. The test had each agent fetch a page that shows the request headers it receives. OpenAI's Codex, whose fetch identified itself as ChatGPT-User, did not ask, and neither did Gemini CLI, GitHub Copilot or Windsurf. Those were specific releases tested in February, and a coding agent's request can change with its next one.

The answer engines' own documentation is silent on the question. OpenAI's crawler page describes GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic's describes ClaudeBot, Claude-User and Claude-SearchBot; Perplexity's describes PerplexityBot and Perplexity-User; and Google's crawler and user-triggered fetcher pages describe its own agents. None of those pages, as we read them on 12 October 2026, says a crawler requests Markdown or reads a content signal. We also found no Anthropic page documenting what Claude Code sends; the evidence for it is Cloudflare's and Checkly's.

So turning the switch on serves the tools that ask, and the ones documented as asking, in what we read, are tools a developer runs. Whether the crawler behind an AI answer ever asks is something its owner does not document, and an AI agent that never asks never sees it.

What does turning it on signal?

The switch attaches a stated yes to AI training, search and AI input to each plain-text response, unless the site's server sends a content-signal header of its own. Content signals are the company's framework for stating, in the words of its documentation, "preferences for how their content can be used after it has been accessed". Its policy defines three uses. Search means building an index and returning links and short excerpts, and "does not include providing AI-generated search summaries". AI input means feeding content into a model as it answers, for example to ground a generated search answer. AI training means "training or fine-tuning AI models".

The company is plain about what such a statement is not: "content signals express preferences; they are not technical countermeasures against scraping." And the default gives way to the site: if the origin already sends the field, the documentation says, "your origin's policy is authoritative."

We checked where the statement appears. On 12 October 2026 we requested three pages twice each, once asking for Markdown and once for HTML: the launch post, the documentation page, and a Read the Docs page. We repeated the requests on 13 October with the same result. Every Markdown response carried `content-signal: ai-train=yes, search=yes, ai-input=yes`. None of the HTML responses from the same addresses carried a content-signal line at all. The robots.txt files of the company's two sites state the same yes to all three uses; the Read the Docs site's robots.txt states no such preference, so on that site the statement appears only on the converted version. That is three pages on two days, a check you can repeat on your own site rather than a survey.

None of the four companies' crawler pages we read commits to reading content signals, let alone honouring them.

Can it say something different from your robots.txt?

A Markdown copy can state a different AI-use preference from the site's robots.txt. The documentation mentions no override for its default except the origin's own header, and does not mention robots.txt anywhere. Cloudflare's managed robots.txt writes a Content-signal line that includes `search=yes, ai-train=no` for a site using the managed setting to disallow training, the preference behind the Disallow AI Training setting covered on our Pay Per Use page. That line does a different job from the file's familiar Allow and Disallow rules. Its documentation says the file tells crawlers what they should or should not access, and that the line it adds belongs to directives that "categorize how crawlers may use your content".

The counter-case deserves its full weight. The default applies only where the server sends no header, and a site can also set its own. Robots.txt's Allow and Disallow rules are the long-standing control over crawling, and the crawler pages above are written around them; Anthropic's says its bots respect do-not-crawl signals "by honoring industry standard directives in robots.txt". A header changes nothing for a client that never reads it.

Still, we think a site whose robots.txt states `ai-train=no`, and which then turns this on without setting a header, is publishing two different preferences in two places. A Disallow rule is not part of that clash: a crawler that obeys it does not fetch at all, in either format. In practice a reader of that site would find three answers: the HTML page states no preference, the robots.txt says no to training, and the converted version says yes, each read by different software. Neither of the company's pages we read says which statement a crawler should follow.

What has Google said about Markdown copies?

Google Search does not use Markdown copies, and Google calls making them for other systems fine. Its guide to generative AI features lists Markdown among the files you do not need for Google Search, which our page on the tactics Google says you can ignore quotes in full. A few lines later, the same guide, last updated on 10 July 2026, calls it "completely fine" to build and keep such files for other services that read them.

The sharper remark came from John Mueller, a Search Advocate at Google. On Bluesky on 3 February 2026 he wrote: "Converting pages to markdown is such a stupid idea." He was replying to a consultant who had argued that flattening pages strips out their meaning, nine days before the switch launched. It is a personal post about converting pages, not Google documentation and not a comment on this feature.

Is serving Markdown to AI crawlers cloaking?

A converted copy is not cloaking by itself under Google's own definition. The risk starts where the copy says something the page does not. Google's spam policies define cloaking as "presenting different content to users and search engines with the intent to manipulate search rankings and mislead users", and give as an example inserting text "only when the user agent that is requesting the page is a search engine, not a human visitor".

The switch converts the same HTML for any client that asks, strips layout, and adds only a frontmatter block taken from the page's own meta tags. Our read is that this keeps it a change of format rather than of content, since the added block comes from the page itself. Google has published nothing about this switch that we found, so that is a reading of its general rule, not a ruling.

A forked copy is a different matter. The Register reported in early August 2026, after replicating a developer's finding, that Time was serving Markdown versions of some pages with sponsored text, flagged as sponsored, to selected AI crawlers picked out by user agent, but not to browsers. Our page on LLM seeding examines that kind of placement against the platforms' rules.

Does AI prefer Markdown or HTML?

The evidence on whether AI prefers Markdown or HTML stops at token counts and crawler visits. No test we found measured citations. The token count of a page is lower in the Markdown format, as the vendor's example above shows; that is a saving for whoever reads the page, not evidence about which page gets cited.

The one controlled test we found came from Profound, a company that sells AI-visibility software, as we do. Profound's randomised test, published on 17 February 2026, split 381 pages across six sites: 192 served Markdown to detected bots, 189 served the usual HTML, with bot visits measured from 19 January to 8 February 2026. The pages in the Markdown group saw about one extra median visit over the three weeks, a difference its own tests found "consistent with random noise". It measured visits, not citations, and it chose which visitors got it by detecting bots rather than waiting for a request.

We have not tested citation ourselves, and nothing here says the plain-text version helps or hurts it.

What should you check before turning it on?

Four checks on your own site settle most of the decision.

• Request one of your pages with `Accept: text/markdown`, using any tool that can set a request header, and read the content-signal line that comes back. That line is what you are stating about your content.
• Read your robots.txt and any managed setting for what they already say about training and AI input. If they say something different, set a custom content-signal header at the server.
• Compare the converted copy with the page a human reader sees. Nothing should appear in one that is missing from the other.
• If your logs record the `Accept` header, look at who actually asks for it before deciding it matters.

These are our checks, and they lean towards what we track; weigh them against your own. Whether to turn the switch on is your call, against who visits your site and what you want said about how it is used. When these pages change, we will read them again and report what is different.

Sources

• Cloudflare docs — Markdown for Agents — the trigger, availability, the output format, the 6 MiB limit and the default content signal. Last updated 9 October 2026; checked 12 October 2026.
• Cloudflare — Introducing Markdown for Agents — Celso Martinho and Will Allen; the named coding agents and the token example. Published 12 February 2026, modified 15 July 2026; checked 12 October 2026.
• Cloudflare — Giving users choice with Cloudflare's new Content Signals Policy — Will Allen; the three signals and "preferences, not countermeasures". Published 24 September 2025; checked 12 October 2026.
• Cloudflare docs — Managed robots.txt — the managed `search=yes, ai-train=no` signal. Last updated 8 October 2026; checked 12 October 2026.
• Read the Docs — Markdown for AI agents — automatic on hosted documentation domains, powered by Cloudflare. No date shown; checked 12 October 2026.
• Checkly — The Current State of Content Negotiation for AI Agents (Feb 2026) — Stefan Judis; seven coding agents, three asking for Markdown. Published 19 February 2026; checked 12 October 2026.
• OpenAI — Overview of OpenAI Crawlers — no date shown; checked 12 October 2026.
• Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler? — dated 7 April 2026; checked 12 October 2026.
• Perplexity — Perplexity Crawlers — modified 29 January 2026; checked 12 October 2026.
• Google — Overview of Google crawlers and fetchers — last updated 12 June 2026; checked 12 October 2026.
• Google — Google's user-triggered fetchers — last updated 19 August 2026; checked 12 October 2026.
• Google — Optimizing for generative AI features in Google Search — last updated 10 July 2026; checked 12 October 2026.
• Google — Spam policies for Google web search — the cloaking definition. Last updated 28 August 2026; checked 12 October 2026.
• John Mueller on Bluesky — posted 3 February 2026; checked 12 October 2026.
• The Register — Time Magazine has a separate version of its website with ads only AI can see — Brandon Vigliarolo. Published 5 August 2026; checked 12 October 2026.
• Profound — We ran a controlled experiment on markdown vs. HTML for AI bots — Brandon Punturo; 381 pages, six sites, bot visits. Published 17 February 2026; checked 12 October 2026.
• Our header check — three pages (the Cloudflare launch post, the Cloudflare documentation page above and https://docs.readthedocs.com/platform/stable/), each requested once with `Accept: text/markdown` and once with `Accept: text/html`, on 12 October 2026, repeated on 13 October 2026 together with a read of each site's robots.txt.
• LiamVi — Generative engine optimization strategies for AI visibility — checked 12 October 2026.
• LiamVi — Cloudflare Pay Per Use — checked 12 October 2026.
• LiamVi — Does schema markup affect AI citations — checked 12 October 2026.
• LiamVi — LLM seeding — our own page, published on 19 October 2026, a week after the checks above; no fact on this page rests on it.

The ranking set we read

These are the nine pages Google returned for "markdown for ai agents" in the United States when we read them on 12 October 2026. Two of them could not be read, and each says why.

• Cloudflare blog — Introducing Markdown for Agents (12 February 2026) — https://blog.cloudflare.com/markdown-for-agents/
• Cloudflare docs — Markdown for Agents (last updated 9 October 2026) — https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/
• Digiday — WTF is Markdown for AI agents? (16 February 2026) — https://digiday.com/media/wtf-is-markdown-for-ai-agents/
• Reddit, r/AI_Agents — Is markdown the programming language for agents now? (opened, but the page showed us no text from the thread) — https://www.reddit.com/r/AI_Agents/comments/1sxrypb/is_markdown_the_programming_language_for_agents/
• Read the Docs — Markdown for AI agents (no date shown) — https://docs.readthedocs.com/platform/latest/reference/markdown-for-agents.html
• AgentMail — HTML vs Markdown for AI agents (25 May 2026, updated 18 September 2026) — https://www.agentmail.to/blog/html-vs-markdown-for-ai-agents
• Hartley Brody — Markdown is the new source code: building software in an age of AI agents (9 March 2026) — https://blog.hartleybrody.com/markdown-research-planning/
• Pronovix — How to Serve Markdown to AI Agents Without Breaking Your SEO (no date found) — https://pronovix.com/articles/how-serve-markdown-ai-agents-without-breaking-your-seo
• Medium — Your Agent Is Just a Markdown File. We Can Do Better. (not read: Medium refused us the page on 12 October) — https://medium.com/@mohsenny/your-agent-is-just-a-markdown-file-we-can-do-better-0a681cb78739