Your Next Visitor Will Not Be Human

On 3 June 2026, Matthew Prince, co-founder and chief executive of Cloudflare, announced (opens in a new tab) that bots had just overtaken humans on his network, which serves roughly one website in five. He had expected that crossover at the end of 2027. He attributes it to the growth of agentic traffic, meaning requests sent by programs that browse the web on a person's behalf. According to Cloudflare Radar, bots accounted at that point for just over 57% of web page requests (opens in a new tab), against under 43% for humans.

That figure adds up visitors that have nothing in common. A training bot copies pages by the million to feed a model and never sends anyone to the site. A search bot reads pages to feed a search engine. An AI agent opens a page because a person asked it to find an osteopath with an opening on Thursday or to compare three quotes. That last category was the smallest in 2025 and it is the one growing fastest. These agents do not read a page the way a person does and they do not see the same things in it.

In its August 2025 analysis (opens in a new tab), Cloudflare sorts the bots run by AI companies by what they do with the pages. In July 2025, model training accounted for 79% of their requests, search for 17% and action on behalf of a user for 3.2%. Those proportions hide the trend. Cloudflare Radar's 2025 year in review (opens in a new tab) measures that requests from agents acting for a user were multiplied by more than fifteen over the year.

Training traffic brings nothing to the site. The same review compares the number of pages an AI company fetches with the number of visitors it then sends to those sites. For Anthropic, that ratio reached 500,000 pages fetched for a single visitor sent. That traffic fills in no form and buys nothing. For a site, the question is therefore not whether to accept or refuse bots as a whole. It is how to tell the bot that copies from the bot that comes to fetch information for a person.

HUMAN Security (opens in a new tab), whose platform processed over a quadrillion interactions in 2025, measures on its side a 7,851% year-over-year growth in traffic from agents and agentic browsers, meaning browsers in which an AI navigates in place of the user. That automated traffic grew eight times faster than human traffic. More than 95% of it concentrates on three sectors, retail, media and travel.

These agents can be identified. In its May 2026 reading (opens in a new tab), HUMAN attributes 47% of agentic traffic to Perplexity's Comet browser, 20% to OpenAI's Atlas and 18.6% to the Claude extension for Chrome. OpenAI has since shut Atlas down (opens in a new tab) on 9 August 2026 and folded its browsing features into ChatGPT. The reading also shows which pages these agents visit. Product and search pages receive 76% of their activity, checkout pages 2.4%. An agent compares a lot and still buys little.

Prince gives the order of magnitude in a statement quoted by NBC News (opens in a new tab). A person preparing a purchase looks at five sites. An agent preparing the same purchase queries five thousand. A site the agent cannot read is simply missing from that comparison and no visit statistic signals the absence.

Adobe Analytics measures, across more than a trillion visits, the traffic that AI assistants send to US retail sites. In the first quarter of 2026, that traffic grew 393% year over year (opens in a new tab). In March 2026, it converted 42% better than traffic from other sources, whereas a year earlier it had converted 38% worse. In July 2026 (opens in a new tab), year-over-year growth stood at 62%, conversion was 60% higher than the rest of the traffic and revenue per visit 53% higher.

These visits come from people, not agents. The assistant compared the options and then sent the person to the site it retained. The site therefore only comes in after the comparison, at the moment the decision is made, which partly explains that conversion rate. It also clarifies the role left to the site, convincing a person already ready to buy in a few seconds.

Adobe also measured the share of content that language models manage to read on those same sites. Its tool calculates, for a given page, the percentage of the content a model can read. The homepages of large US retail sites obtain an average score of 75% (opens in a new tab) and product pages 66%. The least readable sites fall to 54% on their homepage. Between a quarter and a half of what these sites display therefore does not exist for the machine that recommends.

An agent does not see a page the way a screen displays it. It receives a transformed version of it. There are three ways of transforming it.

Raw HTML, without JavaScript

The first version is the source code as the server sends it, before the browser interprets it. The bots run by OpenAI, Anthropic, Meta and Perplexity do not execute JavaScript (opens in a new tab) according to Vercel's measurements from December 2024. Only Gemini is the exception, because it relies on Google's rendering infrastructure. Any content that JavaScript builds in the browser is therefore invisible to them. The point was detailed in the article on being cited by AI (opens in a new tab).

The accessibility tree, the page as a screen reader receives it

The second version is the one received by agents that drive a browser, such as Comet, the Claude extension or the ChatGPT agent. They read the accessibility tree, meaning the structure the browser computes from the HTML for screen readers. Every element in it is described by a role, button, link, field, heading or landmark, by a name and by a state, checked, expanded or disabled. Layout, colours and motion are not in it. A button without an accessible name appears there as an empty button, which the agent can neither identify nor choose.

A study presented at CHI 2026 (opens in a new tab) by researchers from the University of Michigan and the University of California, Berkeley measures how much the agent depends on that structure. The researchers had an agent built on Claude Sonnet 4.5 carry out sixty everyday tasks. Under normal conditions, the agent completes 78% of tasks. When it is restricted to navigating by keyboard only, as a person using a screen reader does, it completes no more than 42% of tasks. When its viewport is reduced the way a screen magnifier would, it completes no more than 28%. The obstacles that block a blind person also block the machine, because it navigates through the same structures.

Negotiated markdown, the page reduced to its structure

The third version is more recent. Since February 2026, Cloudflare can convert a page to markdown (opens in a new tab) when an agent asks for it by sending the Accept: text/markdown header. Markdown is a text format that keeps only a page's structure, its headings, lists, links, tables and the alternative text of its images. In Cloudflare's example, a blog post goes from 16,180 tokens in HTML to 3,150 in markdown, 80% less for the model to read. The site has to switch the feature on, it is not active by default.

The three versions have one thing in common. They keep the document, meaning its hierarchy, the names of its elements and its data. They lose everything visual, the layout, the colour and the motion. Content that exists only as pixels is absent from all three, such as text drawn in a canvas, the browser's free drawing area, a 3D scene or an image without alternative text. The HTML-in-Canvas (opens in a new tab) proposal allows real HTML elements to be drawn inside a canvas so that what the machine reads and what the screen displays become the same thing again. For now it exists only behind a Chromium flag.

The five layers that make a site readable by a machine and the agents that read them in September 2026

Semantic HTML, the layer the law already requires

The first layer is accessibility. A site where every button has a name, every field a label and every section a landmark gives the agent exactly what it gives a screen reader. That layer has been a legal obligation (opens in a new tab) since the European Accessibility Act came into application in June 2025. Google has made it an audit criterion for agents. Since Chrome 150, Lighthouse, the audit tool built into Chrome, includes an "agentic browsing" category (opens in a new tab) that checks four things, accessibility from an agent's point of view, layout stability, WebMCP integration and the presence of an llms.txt file.

Structured data, the layer assistants read

The second layer is JSON-LD, a block of data invisible to the visitor, placed in the page's code, which describes without ambiguity what the page contains using the shared schema.org vocabulary. It can be a business, its address and its opening hours. It can also be a product, its price and its reviews. In 2024, 41% of the pages (opens in a new tab) analysed by the Web Almanac contained one, up from 34% two years earlier. Fabrice Canel, principal product manager at Microsoft Bing, confirmed in March 2025 (opens in a new tab) that this markup helps Microsoft's models, Copilot included, understand content. Google writes (opens in a new tab) that it is not required for its generative features and that no AI-specific markup exists, while recommending keeping it for rich results. For a service business, which has neither a catalogue nor a cart, this layer represents most of the work to be done. An agent looking for a practice open on Saturday reads the opening time directly in that data, instead of having to guess it from a paragraph.

Feeds and commerce protocols, the merchants' layer

The next two layers only concern sites that sell products. OpenAI publishes the product feed format (opens in a new tab), meaning the catalogue file a merchant sends it. It has nine required fields, identifier, title, description, URL, brand, seller, image, availability and price. It also accepts files already prepared for Google Merchant Center. An enable_search field allows ChatGPT to suggest the product. An enable_checkout field allows direct purchase and stays off by default.

The purchase itself goes through two competing protocols. The Agentic Commerce Protocol, ACP for short, published by Stripe and OpenAI (opens in a new tab) in September 2025, allows buying directly inside ChatGPT. Its specification (opens in a new tab) is published under the open Apache 2.0 licence, in a version dated 17 April 2026 and still marked beta. The Universal Commerce Protocol, UCP, presented by Google (opens in a new tab) on 11 January 2026 with Shopify, Etsy, Wayfair, Target and Walmart, allows buying in Gemini and in Google Search's AI Mode. A site signals that it supports it through a description file placed at /.well-known/ucp. The protocol is restricted to selected merchants (opens in a new tab) in the United States, Canada and Australia. In both cases the merchant remains the seller of record and the agent never opens the site's pages, it talks to the site through the protocol. For a French small business, these protocols are something to watch in 2026, not yet something to install.

WebMCP, the layer where the page exposes its own actions

The last layer reverses the relationship between the page and the agent. Instead of letting the agent guess how to fill in a form, the page tells it which actions it offers. WebMCP, introduced by Chrome (opens in a new tab) on 10 February 2026, lets a page declare an action, for example booking a slot, with its name, its description and the information it expects. The agent calls that action directly, without handling the form. The name comes from the Model Context Protocol, the format through which AI assistants describe the tools they can call. The specification (opens in a new tab), written by engineers from Microsoft and Google within the W3C's Web Machine Learning group, is a draft dated 17 September 2026. The test phase (opens in a new tab), reserved for sites that sign up, started with Chrome 149. Expedia, Booking.com, Shopify, Etsy, Instacart and Target are experimenting with it. Firefox and Safari have announced nothing. WebMCP is therefore a layer to follow, not yet a layer to put into production.

The file nobody reads

One item appears in every list of advice and is missing from this table. The llms.txt file, a text file meant to summarise the site for language models, is read by no system in production. Google writes (opens in a new tab) that no text file intended for AI is needed. John Mueller, from the Google Search team, called it (opens in a new tab) "purely speculative for now", noting that the file has existed for years without any AI system using it. Lighthouse checks for its presence in anticipation, no real use has been observed.

Once the site is readable, what remains is deciding which bots are allowed to read it. The robots.txt file allows or forbids a bot by its name, not according to what it does with the page. The same bot, GPTBot for instance, can serve to train a model or to answer a question a user has just asked. In September 2025, Cloudflare proposed a complement (opens in a new tab), Content Signals. The site expresses three preferences in the same file, search for indexing by a search engine, ai-input for use of the page in an assistant's answer and ai-train for training. More than 3.8 million domains were then using the robots.txt managed by Cloudflare to refuse training. The default setting allows search, refuses training and says nothing about use in answers.

That third signal is the one that matters for a service business. Refusing training protects the content. Refusing ai-input removes the site from the answers an assistant gives a client looking for a provider. The trap of the button that blocks all AI at once was described in the article on being cited by AI (opens in a new tab).

What remains is knowing whether the bot that shows up is really who it claims to be. An agent can sign its requests with a cryptographic key under RFC 9421 (opens in a new tab), an Internet standard. The site then recognises a declared agent and treats the rest as anonymous traffic. The IETF, the body that publishes these standards, has formed a working group (opens in a new tab) named Web Bot Auth, whose charter explicitly names AI agents retrieving or interacting with content on behalf of end users. Cloudflare has been verifying these signatures (opens in a new tab) since August 2025. For a site hosted behind a network like Cloudflare, accepting identified agents and blocking the others is a dashboard setting, not a development.

Serving these two readers does not call for two sites. The layer the machine reads is the document itself, its structure, the names of its elements and its data. The layer the person perceives is everything added on top of it, the rhythm, the typography, the motion and the way a button responds to hover. That is the principle of progressive enhancement (opens in a new tab), a complete document served to every visitor and then enriched for those who can display that enrichment. Immersive sites are no exception. A 3D scene or a scroll-driven experience are enhancements that have to sit on a document the agent can read. A site whose content exists only inside the scene is missing from the comparison and the person it would have convinced is never sent to it.

The two readers do not judge the same thing. The agent compares facts, a price, an opening time, an availability or a rating. It does not perceive what makes a site singular. The person arrives afterwards, on the visit the assistant brought about. That person judges in fifty milliseconds what the machine does not measure, as described in the article on websites looking alike (opens in a new tab). Adobe's data shows those visits convert 60% better than the others. The layer the machine can read therefore determines whether the site enters the comparison. The layer the person perceives determines what comes out of it. Both are built on the same page.