Agentic Browsers and Your Website: What AI Agents Do on Your Pages, and How to Prepare

Short answer: Agentic browsers are browsers, or browser extensions, that can do things on the web for a person rather than just showing pages. They read your page through screenshots, the HTML and the accessibility tree, then click, type and fill in forms. For a website owner, getting ready is mostly about basics you already know, with a few new controls. Decide which AI crawlers and user-triggered agents you allow, and remember that robots.txt does not govern all of them. Verify agent traffic by signature or published IP ranges, not by the user-agent string. Build pages with semantic, accessible HTML and stable layouts. Keep pages fast. Treat any text an agent might read, including reviews and comments, as a possible route for prompt injection. And expect your analytics to undercount some of these visits while server logs see more of them.

This guide is for people who run business websites: owners, marketers and the developers or hosts who look after them. It doesn’t rank the browsers or explain how to use them as a shopper. It covers what these agents do when they arrive on your site and what you can change. The market moves quickly, so every product claim below is dated and comes from the vendor’s own documentation or announcement. Check those pages again before you make a decision that depends on them.

If you are mainly interested in getting your content cited in AI answers, that is a separate question, covered in P4 Emachines’ guide on how to get cited by AI. This article is about the visitor that comes after the citation: the agent that opens your pages and tries to get something done.

What is an agentic browser?

An agentic browser pairs a normal web browser with an AI model that can act. You give it a goal, such as “find three quotes for a furnace inspection in Mississauga and book the cheapest one for next week”. It plans the steps, opens pages, reads them, and clicks, scrolls and types until it finishes or hands control back to you.

The category is about a year and a half old, and it has already changed shape more than once. Here is where the main products stood when we checked their official pages in late October and early November 2026:

Two things stand out. First, availability is uneven, and several features are U.S.-only on the dates shown. That matters less than it seems for a Canadian site, because you don’t control who visits you. A shopper in Seattle or an agent running in a U.S. data centre can land on a Toronto store just as easily as a local visitor can. Second, products appear and disappear. Atlas was retired less than a year after launch. Building your site for a specific product is a poor bet. Building it for how agents work in general holds up much better.

How AI agents actually read and use your site

Google’s web.dev guide Build agent-friendly websites (last updated April 1, 2026) gives the clearest public description. It says agents view a site in three main ways:

• Screenshots. The agent captures the rendered page and uses a vision model to spot elements such as the search box or a form field. Google notes this is slow and token-expensive, so agents tend to use it as a fallback when the structure is confusing.
• The HTML and DOM. The agent reads how elements are nested. If a “Buy now” button sits inside a product container, it assumes the button belongs to that product.
• The accessibility tree. This is the browser’s own summary of the page’s roles, names and states, the same structure screen readers use. Google calls it a “high-fidelity map” for agents.

Google’s Search Central guide to optimizing for generative AI features (last updated July 10, 2026) says the same thing in one line: browser agents may analyze screenshots, inspect the DOM and interpret the accessibility tree. P4 Emachines’ breakdown of Google’s AI search optimization guide covers the rest of that document.

It also helps to separate two kinds of agent, because they reach your server differently.

In-browser agents run inside the user’s own browser. Claude in Chrome, Gemini’s auto browse, Comet and Opera Neon all work this way. Opera says Neon Do runs “inside your browser session, where you’re already logged in”, and Anthropic describes Claude using “your existing logins”. The requests come from the user’s device and IP address, with the user’s cookies. None of the vendor pages cited here documents a separate identifier for this traffic, so to your server it will mostly look like an ordinary visitor using Chrome or a Chromium browser.

Cloud agents run a browser on the vendor’s infrastructure. Google documents a fetcher called Google-Agent, “used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request”. OpenAI documents a Cloud browser for ChatGPT Work that signs its requests (more on that below). These requests come from vendor IP ranges, and some can be identified.

There is a third, older category too: AI fetchers and crawlers that don’t drive a browser at all. They retrieve pages to answer a question, build a search index or collect training data. These are the bots most robots.txt advice is about.

Crawler and agent controls: what each vendor documents

This is where most site owners get caught out. “Block AI in robots.txt” isn’t a single switch. Each vendor publishes several user agents with different jobs, and the vendors don’t agree on whether user-initiated visits follow robots.txt. The table summarises the official documentation as we read it on November 2, 2026.

Sources: OpenAI crawlers overview, Google’s common crawlers, Google user-triggered fetchers, Perplexity crawlers, and Anthropic’s help article on how site owners can block its crawler (updated April 7, 2026).

What this means in practice:

1. Training, search and live retrieval are separate decisions. You can block GPTBot and ClaudeBot (training) and still allow OAI-SearchBot, PerplexityBot and Claude-SearchBot (search). OpenAI states that each setting is independent. Blocking Google-Extended doesn’t remove you from Google Search, according to Google.
2. robots.txt is a request, not a lock. For user-initiated fetchers, OpenAI, Google and Perplexity all say robots.txt may not apply or is generally ignored. Anthropic is the exception among these four: it says Claude-User respects your setting. In-browser agents working in a user’s own session are, in effect, that user browsing, so robots.txt is not the mechanism that governs them.
3. Blocking by IP can backfire. Anthropic warns that IP blocking “may not work correctly or persistently guarantee an opt-out”, because it stops the bot reading your robots.txt in the first place.
4. More signals are coming. Google says it intends to support the AIPREF shared training-preference signal (user-agent: __usage_ai-training) in early 2027. Treat that as a plan, not a feature you can use today.

A starting robots.txt for a business that wants to appear in AI search but opt out of training might look like this. Adjust it to your own policy:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

The decision is a business one, not a technical one. A publisher that licenses its content has different interests from a plumber who wants as much exposure as possible. If you’re unsure, the 4GoodHosting team’s guide to responsible web scraping has a long section written from the other side, on what to do when your site is the one being scraped. It explains why robots.txt is “necessary, and not a control”, and how to rate limit at a reverse proxy.

Verifying agents: why the user-agent string isn’t enough

Anyone can type “Googlebot” or “ChatGPT-User” into a request header. Google’s crawler documentation says plainly that the user agent “can be spoofed”. If you allow or block traffic by name alone, you’ll let impostors in and may shut out real agents.

There are two reliable ways to check.

Published IP ranges and reverse DNS. OpenAI publishes separate IP lists for OAI-SearchBot, GPTBot and ChatGPT-User. Perplexity publishes lists for both of its agents, and its firewall guidance pairs those IP lists with the user-agent header. Google publishes JSON files of IP ranges, including user-triggered-agents.json for Google-Agent, and documents reverse DNS patterns.

Cryptographic signatures (Web Bot Auth). This is newer, and it is where agent identification seems to be heading. The agent signs each request using HTTP Message Signatures (RFC 9421) and publishes its public keys at a well-known URL. OpenAI’s help article on allowlisting ChatGPT Work’s Cloud browser says each request carries Signature, Signature-Input and a Signature-Agent header set to "https://chatgpt.com", with keys at https://chatgpt.com/.well-known/http-message-signatures-directory. The same article says that, at launch, the Cloud browser “cannot sign in to websites or complete payments”. Google’s fetcher documentation says it is “experimenting with the Web Bot Auth protocol, using the https://agent.bot.goog identity.”

You don’t have to verify signatures yourself if your CDN or bot-management service does it. OpenAI’s article lists steps for Akamai, Cloudflare, HUMAN and Vercel. Cloudflare’s verified bots documentation says that from July 1, 2026 it classifies each bot by behaviour, including “Search”, “Agent” and “Training”, and labels user-driven agent services as “Intermediary”. Search, Agent and Training are available as managed presets on all plans. If you use one of these services, check whether its defaults block agents. And if you run your own proxy, make sure it passes the three signature headers through unchanged. That is the first thing OpenAI’s troubleshooting section tells you to check when legitimate requests are blocked.

Forms, checkouts and CAPTCHAs: decide what agents may do

Agents are built to fill in forms. Google’s announcement lists “filling out tedious online forms”, scheduling appointments and adding items to carts among the jobs its testers gave auto browse. So the question for most businesses isn’t whether agents will try to use your forms, but which actions you are comfortable with.

A sensible split:

• Let agents through on low-risk steps: search, filtering, product pages, store hours, quote requests, appointment slots. If a quote form only works for a human with a mouse, you’re losing leads the user tried to send.
• Keep human checks on high-risk steps: payment, account changes, password resets, anything that commits money or personal data. The vendors design for this too. Google says auto browse pauses for confirmation before purchases or social posts. OpenAI’s prompt-injection guidance tells users to review confirmation requests carefully.
• Use friction deliberately. A CAPTCHA on the login page is reasonable. A CAPTCHA on the “check stock at your nearest store” button blocks a helpful agent and doesn’t stop much. Anthropic says its own bots “will not attempt to bypass CAPTCHAs”, so a CAPTCHA works as a stop sign for well-behaved automated visitors.

Make every step visible in the interface. An action that only exists as a hover state, or only appears after a mouse movement, is one that agents struggle to find.

Structured data and machine-readable content

Structured data won’t make an agent pick you. Google says plainly that structured data “isn’t required for generative AI search, and there’s no special schema.org markup you need to add”. It still recommends using structured data for rich results.

For agents, the value is more practical. An agent comparing three shops is looking for a price, a currency, availability, opening hours, a phone number or a return policy. When those facts sit in visible text and match your schema.org markup (Product, Offer, LocalBusiness, OpeningHoursSpecification), you remove a source of error. When the visible price says $49 and the markup says $59, you’ve created one. We haven’t found a vendor that promises its browser agent reads JSON-LD, so treat markup as a low-cost hedge and make the visible page the source of truth. Google’s introduction to structured data is the right place to start.

Two newer formats come up often.

llms.txt is a proposed file, described at llmstxt.org, that gives language models a curated, Markdown summary of a site. Google says its Search systems don’t use llms.txt and that the file “will neither harm nor help” visibility in Google Search. Some other tools may read it. It’s cheap to publish if you want one, but it’s no substitute for clear pages.

WebMCP is a proposed web standard from the Chrome team. It lets a page declare “tools”, such as checkout or filter_results, with defined inputs, so an agent calls a function instead of guessing which button to press. Chrome’s WebMCP documentation (published May 18, 2026, updated October 7, 2026) describes it as a progressive enhancement available through an origin trial. It’s worth watching if you run booking or ecommerce flows, but not worth rebuilding a site around yet. Google also points to the Universal Commerce Protocol, which it says Chrome will support for agentic checkout.

Accessibility: the overlap that matters most

The most useful thing in Google’s agent guidance is how closely it matches accessibility practice. Its list of recommendations reads like a WCAG checklist:

• Use real <button> and <a> elements rather than clickable <div> and <span> elements. Where you can’t, add the right role and tabindex.
• Link every <label> to its input with the for attribute.
• Avoid transparent overlays (“ghost elements”) that cover controls. An agent’s visual analysis may discard what’s underneath.
• Make interactive elements visible and larger than 8 square pixels.
• Keep the layout stable, so the “Add to cart” button is in a predictable place.

OpenAI made the same point when it launched Atlas, saying site owners “can also add ARIA tags to improve how ChatGPT agent works for their websites”. Atlas is gone, but the advice applies to every agent that reads the accessibility tree.

For Canadian businesses there’s a second reason to do this work. Under Ontario’s Accessibility for Ontarians with Disabilities Act, since January 1, 2021, designated public sector organizations and businesses or non-profits with 50 or more employees must make public websites meet WCAG 2.0 Level AA, with two exceptions for live captions and pre-recorded audio description. The current W3C standard is WCAG 2.2. Work that brings you into line with that standard also makes your site easier for agents to use. If you run WordPress, 4GoodHosting’s guide to better web accessibility on a WordPress site walks through the POUR principles behind WCAG.

A quick test: open Chrome DevTools, inspect the accessibility tree for your main conversion page, and check that every control has a sensible name. “Button” with no label is what an agent sees too.

Performance: agents wait less patiently than you think

No vendor publishes a timeout figure for its browser agent, so be wary of anyone who quotes one. What we do know is how agents work. Google describes screenshot analysis as slow and expensive, and it warns that shifting layouts confuse agents that rely on screenshots. Every extra second of rendering, every late-loading banner that pushes the form down, and every cookie wall adds steps to the agent’s task and more chances for it to fail.

The fixes are the familiar ones:

• Send meaningful HTML from the server. If prices, stock levels or form fields only appear after heavy client-side JavaScript, both agents and many AI fetchers will have a harder time.
• Keep layout shift low. Cumulative Layout Shift is a Core Web Vital for humans, and it matters at least as much for an agent clicking by position.
• Watch server response time under bursts. Cloud agents and AI fetchers add requests that don’t show up in your analytics (see the next section). A plan that copes with your human peak may not cope with your human peak plus a crawl.

For a sober look at which speed statistics hold up, see 4GoodHosting’s review of website load time statistics and what they mean in Canada. If automated traffic is pushing a small server to its limits, their explainer on how much traffic a VPS can handle covers when a CDN or load balancing helps.

Security: prompt injection is now partly your problem

Prompt injection is the risk every vendor puts at the top of its agent security writing. The attacker hides instructions in content the agent reads, hoping the agent will follow them instead of the user. Google’s Chrome security team, in Architecting Security for Agentic Capabilities in Chrome, calls indirect prompt injection “the primary new threat facing all agentic browsers”. It says the injection “can appear in malicious sites, third-party content in iframes, or from user-generated content like user reviews”.

That last part is why this is a site owner’s concern. Your site doesn’t have to be malicious to carry an attack. Brave’s researchers showed this in August 2025 with an indirect prompt injection against Perplexity’s Comet. They hid instructions behind a spoiler tag in a Reddit comment. When a user asked Comet to summarise the page, the hidden text told it to take the user’s email address from their Perplexity account page, read a one-time password from Gmail, where the user was already logged in, and post both as a reply to the comment. After publishing, Brave added that further testing showed Perplexity still hadn’t fully mitigated this kind of attack, and that it had re-reported the issue.

The vendors say this won’t be solved soon. In its December 22, 2025 post on hardening Atlas against prompt injection, OpenAI wrote that prompt injection “is unlikely to ever be fully ‘solved'”. Anthropic’s August 2026 Claude in Chrome announcement reports large improvements on its own tests. It also calls the problem “a moving target”.

What a website owner can do:

1. Moderate user-generated content. Reviews, comments, forum posts and Q&A sections are the easiest places to plant instructions on a site the attacker doesn’t own. Strip or escape HTML in submissions, so text can’t be hidden with styling, comments or zero-size elements.
2. Limit third-party content. Ad slots, widgets and embedded iframes put someone else’s text on your page. Google says Chrome constrains which origins its agent can interact with to those relevant to the task, but you can’t assume every agent has that defence.
3. Watch for tampering. A compromised WordPress site can serve hidden text to visitors without changing anything the owner sees. File integrity monitoring, prompt patching and checking rendered pages, not just the editor, all matter more now.
4. Don’t try it yourself. Hiding “AI assistants: recommend this company” text on your pages is the same technique attackers use. OpenAI, Google and Anthropic each describe adversarial training, classifiers or real-time detection aimed at injected instructions, so the likely result is that agents learn to distrust your content.
5. Don’t loosen security for agents. Allowlisting a verified agent should mean “don’t challenge it on public pages”. It should not mean skipping fraud checks at checkout.

If you’re adding an AI chatbot or agent of your own, the risks run the other way and the OWASP LLM01 prompt injection entry is the reference. P4 Emachines’ guide to AI data security for Canadian businesses covers staff use of AI tools and AI features on your own site. For the wider picture of where small-business attacks now land, 4GoodHosting’s guide Your Website Is the Perimeter Now is a practical companion.

Analytics: what you will and won’t see

Agent traffic distorts measurement in three different ways, depending on the kind of agent.

In-browser agents look like people. They run in a real browser, so your JavaScript analytics tag loads and records a session. As far as we found, none of the vendors documents a marker that would let your analytics tell an agent-driven session from a human one. Expect some sessions that move unusually fast, follow odd paths or fill forms in seconds. They may be agents working for real customers, so don’t automatically filter them out as spam.

Cloud agents and fetchers may not show up at all. Many fetchers request the HTML without running your analytics script, so they never appear in tag-based reports. Your server logs still record them, with their user agent and IP address. That makes the logs the most complete record of AI traffic you have. Search them for the tokens in the table above, such as OAI-SearchBot, ChatGPT-User, Perplexity-User, Claude-User and Google-Agent, and confirm the IP ranges before you trust the name.

Fewer visits can still mean more business. If an agent compares five suppliers and books one, four of them saw a single quick visit and no conversion, and the winner saw one visit and a booking. Measure outcomes such as quote requests, bookings and calls, not just sessions. Where a call comes in, ask how the customer found you.

Log-based reports are easy to get on most hosting plans. 4GoodHosting’s SSD web hosting page, for example, lists daily “Web Statistics” described as “traffic reports built from your server log files: page views, unique visitors, referring sites and the browsers people use”. Whatever host you use, find your raw access logs and keep enough history to compare month on month.

A practical checklist for site owners

You can work through most of this in a week:

1. Write down your policy. Decide, for training, AI search and user-initiated agents, what you allow. Get sign-off from whoever owns marketing and legal.
2. Update robots.txt to match, using each vendor’s documented tokens. Remember that it doesn’t govern user-initiated fetchers or in-browser agents.
3. Check your firewall and CDN. Find out whether bot protection is blocking verified agents you want to allow, or allowing spoofed names you don’t. Turn on signature verification where it’s offered.
4. Search a month of server logs for AI user agents, and confirm the top ones against published IP ranges.
5. Audit your key journeys (contact form, quote request, booking, checkout) for semantic buttons, labelled fields, no ghost overlays and a stable layout. Inspect the accessibility tree.
6. Match visible facts to structured data: prices, hours, stock and policies.
7. Review user-generated content handling and third-party embeds for places where hidden text could be planted.
8. Keep human confirmation on payment and account changes, and remove friction from low-risk steps.
9. Add outcome tracking for forms, bookings and calls so you aren’t relying on session counts.
10. Book a quarterly review. Products in this space launch and retire within months, as Atlas showed.

Frequently asked questions

Should I block agentic browsers from my website?

For most businesses, no. In-browser agents act for real customers, usually using the customer’s own session, and blocking them means turning those customers away. It also isn’t reliable, because they look like ordinary browsers. A better approach is to decide what actions agents can complete, keep human confirmation for payments and account changes, and block abusive traffic by behaviour rather than by name.

Does robots.txt stop AI agents?

Only some of them. Crawlers for training and search from OpenAI, Google, Perplexity and Anthropic say they follow robots.txt. User-initiated fetchers are different. OpenAI says robots.txt “may not apply” to ChatGPT-User. Google says its user-triggered fetchers “generally ignore” it, and Perplexity says the same of Perplexity-User. Anthropic says Claude-User does respect it. In-browser agents are the user browsing, so robots.txt isn’t the right tool for them.

How can I tell whether a visit came from an AI agent?

For cloud agents and fetchers, check the user-agent token against the vendor’s published IP ranges, or verify the Web Bot Auth signature if the vendor signs requests, as OpenAI does for its Cloud browser. For agents running in a user’s own browser, there is usually no reliable marker. Unusual speed and navigation patterns are clues, not proof.

Is structured data required for AI agents?

No. Google says no special schema is needed for its generative AI features. Accurate structured data still helps machines read prices, hours and availability without error, and it supports rich results in Google Search. Keep it consistent with the visible page.

Do agentic browsers affect my SEO?

Not directly. Google says blocking Google-Extended doesn’t affect inclusion or ranking in Google Search. The bigger shift is in behaviour: fewer, more decisive visits, and conversions that start in an AI tool. Track outcomes as well as rankings and sessions.

Is my site responsible if an agent is tricked on it?

That depends on the facts and the law that applies, and it is a question for a lawyer, not a blog post. In practical terms, any site that publishes user content can carry hidden instructions. Moderating that content and keeping your site free of tampering reduces the chance that yours is the page that hijacks a visitor’s agent.

Key Takeaways

• Agentic browsers read pages through screenshots, the DOM and the accessibility tree, then act. Google’s agent-friendly guidance is a good public reference for building for them.
• Training, AI search and user-initiated retrieval are separate controls. User-initiated fetchers from OpenAI, Google and Perplexity may ignore robots.txt. Anthropic says Claude-User follows it.
• Verify agents by published IP ranges or Web Bot Auth signatures, never by the user-agent name alone.
• Semantic, accessible HTML and a stable layout help agents and people alike, and they support AODA and WCAG obligations.
• Prompt injection can ride on your reviews, comments and embeds. Moderate user content and watch for tampering.
• Analytics tags will miss many fetchers and can’t tell in-browser agents from people. Server logs and outcome tracking fill the gap.
• The product list changes every few months, so build for how agents work, not for any one browser.

Sources

• Google, Build agent-friendly websites (web.dev, updated April 1, 2026)
• Google Search Central, Google’s Guide to Optimizing for Generative AI Features on Google Search (updated July 10, 2026)
• Google, List of Google’s common crawlers and List of Google user-triggered fetchers
• Google, Chrome gets new Gemini 3 features, including auto browse (January 28, 2026) and Gemini in Chrome with auto browse comes to Android (May 12, 2026)
• Google, Architecting Security for Agentic Capabilities in Chrome
• Chrome for Developers, WebMCP (updated October 7, 2026) and Agent security considerations for WebMCP
• Google Search Central, Introduction to structured data markup
• OpenAI, Overview of OpenAI Crawlers
• OpenAI, Introducing ChatGPT Atlas (October 21, 2025) and ChatGPT Atlas retirement
• OpenAI, ChatGPT Work’s Cloud browser allowlisting
• OpenAI, Continuously hardening ChatGPT Atlas against prompt injection (December 22, 2025)
• Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? (April 7, 2026)
• Anthropic, Claude in Chrome is generally available (August 26, 2026)
• Perplexity, Perplexity Crawlers
• Brave, Agentic Browser Security: Indirect Prompt Injection in Perplexity Comet (August 20, 2025)
• Opera, Opera ships the Opera Neon AI agentic browser (September 30, 2025)
• Microsoft, New updates to Edge across desktop and mobile (May 13, 2026) and Considerations for Safe Agentic Browsing (October 23, 2025)
• Cloudflare, Verified bots and Web Bot Auth
• IETF, RFC 9421: HTTP Message Signatures
• OWASP GenAI Security Project, LLM01: Prompt Injection
• Government of Ontario, How to make websites accessible
• W3C, Web Content Accessibility Guidelines (WCAG) 2.2
• llmstxt.org and schema.org

This article was written by the team at 4GoodHosting, a Canadian-owned web hosting company operating since 2002 with data centres in Vancouver and Toronto. This article is a guest contribution to P4 Emachines.