On March 20, 2026, Google quietly added a new item to its official list of web downloaders. Not a browser. It is not a training bot. The agent.
Google-Agent is a user agent string of AI programs running on Google’s infrastructure that browse websites on behalf of users. When someone asks an AI assistant to research a product, fill out a form, or compare options across websites, Google-Agent is what visits the page. Project Mariner, Google’s AI browsing tool, is the first product to use it.
This is not a Googlebot. Googlebot crawls the web continuously, indexing pages for search. Google-Agent only appears when someone requests it. This difference changes everything about how it works.
Robots.txt Not Working
Google classifies Google-Agent as a user-activated agent. The category includes tools such as Google Read Aloud (text-to-speech), NotebookLM (document analysis), and Feedfetcher (RSS). They all share one property: the person who initiated the request. Google’s position is that user-activated followers “generally ignore robots.txt rules” because the download was requested by a human.
Concept: When you type a URL in Chrome, the browser loads the page regardless of what robots.txt says. Google-Agent works on the same principle. An agent is a proxy for the user, not an independent browser.
This is a logical departure from how OpenAI and Anthropic handle the same traffic. ChatGPT-User and Claude-User both act as user-configured followers, but respect robots.txt instructions. If you block ChatGPT-User in robots.txt, ChatGPT will not fetch your page when the user requests it to browse. Google made a different call.
Website owners who used to rely on robots.txt as a universal access control method now have a loophole. If you need to restrict access to Google-Agent, you will need server-side authentication or access controls. The same tools you can use to block a human visitor.
Cryptographic Identity: Web Bot Auth
The most important development is buried in one line of Google’s documentation: Google-Agent tries this web-bot-auth a protocol that uses identity https://agent.bot.goog.
Web Bot Auth is a draft IETF standard that acts as a digital passport for bots. Each agent holds a private key, publishes its public key to the directory, and cryptographically signs all HTTP requests. The website verifies the signature and knows, with cryptographic certainty, that the visitor is who they say they are.
User agent threads can be done by anyone. Web Bot Auth cannot. Google accepts this protocol, or through testing, it shows where the agent’s identity is headed. Akamai, Cloudflare, and Amazon (AgentCore Browser) already support it. Google delivers a serious bundle.
This is important because the web is about to have an identity problem. As agent traffic grows, websites need to differentiate between legitimate AI agents serving real users and scrapers pretending to be agents. IP verification helps, but cryptographic signatures are getting better and harder to forge.
What This Means for Your Website
Google-Agent creates a three-tier web visitor model:
- Human visitors direct browsing.
- They don’t crawl search content index and training (Googlebot, GPTBot, Google-Extended).
- Agents representing certain people in real time (Google-Agent, ChatGPT-User, Claude-User).
Each category has different access rules, different goals, and different expectations. The search engine wants to index your content. The agent wants to complete the task. It could be reading a product page, comparing prices, filling out a contact form, or booking an appointment.
Here’s what you need to do now:
Monitor your logs. Google-Agent identifies itself with the user-agent string it contains compatible; Google-Agent. Google publishes IP ranges for verification. Start tracking how often agents visit, what pages they hit, and what they’re trying to do.
Check your CDN and firewall rules. If your security tools strongly block non-browser traffic, Google-Agent may be rejected before it reaches your server. Make sure the Google published IP range is enabled.
Check your forms and flow. Google-Agent can submit forms and navigate through multi-step processes. If your checkout, booking, or contact forms rely on JavaScript patterns that confuse automated systems, the agent’s visitors will silently fail. Semantic HTML and clear tags are always fundamental.
Accept that robots.txt is no longer a perfect access control tool. For content you really need to restrict, use authentication. robots.txt is designed for crawlers. Agent time requires different parameters.
The Hybrid Web Is Not Coming. Included
Last year, the idea that AI agents would browse websites alongside humans was a foreshadowing of a conference speech. Today, it has a user agent chain, a published IP range, a cryptographic identity protocol, and is included in Google’s official documentation.
The web has never been separated between man and machine. Included. Every page you publish now serves two audiences at the same time, and Google has recently made it possible to see exactly where non-human audiences are coming from.
Additional resources:
This post was originally published on No Hacks.
Featured image: Summit Art Creations/Shutterstock
