The new visitor to the web has just been given an identity

On March 20, 2026, Google quietly added a new entry to its official list of web fetchers. No crawler. Not a training bot. An agent.

Google Agent is the user agent string for AI systems running on Google infrastructure that crawl websites on behalf of users. When someone asks an AI assistant to research a product, fill out a form, or compare options on different websites, Google Agent is the one who actually visits the page. Project Mariner, Google’s experimental AI browsing tool, is the first product to use it.

This is not Googlebot. Googlebot continually crawls the web and indexes pages for search. Google Agent only appears when a human requests it. This distinction changes everything about how it works.

Robots.txt does not apply

Google classifies Google Agent as a user-controlled retriever. The category includes tools such as Google Read Aloud (text-to-speech), NotebookLM (document analysis) and Feedfetcher (RSS). They all have one thing in common: a human initiated the request. Google’s position is that user-triggered retrieval functions “generally ignore robots.txt rules” because the retrieval was requested by a person.

The logic: When you enter a URL in Chrome, the browser will fetch the page regardless of what robots.txt says. Google Agent works on the same principle. The agent is the user’s proxy, not an autonomous crawler.

This is a significant departure from the way OpenAI and Anthropic handle similar traffic. ChatGPT-User and Claude-User both act as user-triggered polling functions, but respect the robots.txt instructions. If you block ChatGPT users in robots.txt, ChatGPT will not fetch your page when a user asks to browse it. Google made a different call.

Website owners who relied on robots.txt as a universal access control mechanism now have a loophole. If you need to restrict Google Agent access, you will need server-side authentication or access controls. The same tools you would use to block a human visitor.

Cryptographic Identity: Web Bot Auth

The more significant development is buried in a single line of Google’s documentation: Google Agent is experimenting with the web-bot-auth Protocol using identity https://agent.bot.goog.

Web Bot Auth is an IETF draft standard that works like a digital passport for bots. Each agent has a private key, publishes its public key in a directory, and cryptographically signs every HTTP request. The website verifies the signature and knows with cryptographic security that the visitor is who it claims to be.

User agent strings can be forged by anyone. Web Bot Auth cannot. Google’s adoption of this protocol, albeit experimentally, signals where the evolution of agent identity is headed. Akamai, Cloudflare and Amazon (AgentCore Browser) already support it. Google brings the critical mass.

This is important because the web will soon have an identity problem. As agent traffic increases, websites must differentiate between legitimate AI agents acting on behalf of real users and scrapers pretending to be agents. IP verification helps, but cryptographic signatures scale better and are harder to forge.

What this means for your website

Google Agent creates a three-tier visitor model for the web:

Human visitors search directly.
Crawlers Indexing of content for search and training (Googlebot, GPTBot, Google-Extended).
agents Act on behalf of specific people in real time (Google agent, ChatGPT user, Claude user).

Each tier has different access rules, different intentions, and different expectations. A crawler wants to index your content. An agent wants to complete a task. This could be reading a product page, comparing prices, filling out a contact form or booking an appointment.

Here’s what to do now:

Monitor your logs. Google Agent identifies itself with a User Agent string that contains: compatible; Google-Agent. Google publishes IP ranges for review. Start by tracking how often agents visit your website, what pages they view, and what they try to do.

Check your CDN and firewall rules. If your security tools aggressively block non-browser traffic, Google Agent may be rejected before it reaches your server. Make sure the IP ranges published by Google are allowed.

Test your forms and processes. Google Agent can submit forms and navigate multi-step processes. If your checkout, booking, or contact forms rely on JavaScript patterns that confuse automated systems, agent visitors will silently fail. Semantic HTML and clear labels remain the foundation.

Accept that robots.txt is no longer a complete access control tool. For content that you really need to restrict, use authentication. robots.txt was developed for crawlers. The age of agents needs different boundaries.

The hybrid web is not coming. It is recorded

A year ago, the idea that AI agents would crawl websites alongside humans was a prediction for a conference talk. Today it has a user agent string, published IP ranges, a cryptographic identity protocol, and an entry in Google’s official documentation.

The web has not split into humans and machines. It merged. Every page you publish now serves both audiences at the same time, and Google has just made it possible to see exactly when the non-human audience is showing up.

Additional resources:

This post was originally published on No Hacks.

Featured Image: Summit Art Creations/Shutterstock

WPAP (907)