Google is testing Web Bot Auth, a testing protocol designed to help websites verify that automated traffic is actually coming from the bot or service it claims to represent. The new protocol can provide site owners with a reliable way to distinguish legitimate automated traffic from bots that hide or misrepresent their identities.
A new developer support page has been published providing details on how to authenticate requests with the Web Bot Auth protocol, which is currently in the testing phase.
That’s what Google’s Web Bot Auth is based on
The new protocol is technically called the HTTP Message Signature Directory. It is a proposed technology standard designed to automate trust between web services. It helps websites realize automated authenticated services without requiring each side to exchange authentication keys in advance.
The basic idea is to provide automated guaranteed services with a standardized way to present information. Instead of relying solely on passwords, user agent strings, or privacy settings between companies, the protocol gives websites a repeatable way to check whether an automated request can be authenticated. That’s important because many bots can pretend to be something they’re not. Web Bot Auth doesn’t determine whether a bot is good or bad, but it can give site owners a strong signal about whether a bot is really the service it says it is.
A Reliable Method for Identifying Bots
The cryptographic component is important because it makes the identity difficult to falsify. Today, a rogue bot can impersonate a legitimate browser by copying the name or string of a user agent. Web Bot Auth is designed to go beyond that type of authentication by giving websites a way to verify that an automated request matches the service’s cryptographic credentials.
Under this protocol, a bot would need more than a label to identify itself. It will need to prove that identity in a way that the website can verify. That would give site owners a secure basis for enabling automated authenticated services while preventing bots that can’t prove their identity. The protocol does not automatically decide which bots to allow or block, but it can give websites a more reliable signal in making that decision.
Cryptographic authentication is what makes Web Bot Auth better than current bot authentication methods. Instead of relying on signals that can be misrepresented, it gives websites a way to validate automated requests. That means recognition is based less on what the bot says and more on whether its identity can be verified with cryptographic credentials.
Caveat: It’s in the Experimental Phase
The proposed protocol will make it possible to distinguish between rogue bots masquerading as trustworthy crawlers from real bots from trusted services. This protocol is like a whitelist of what is allowed which makes it easy to distinguish untrustworthy browsers.
However, because this is an experimental phase, the “whitelist” currently only applies to a subset of traffic, such as Google-Agent. Google doesn’t “sign every request,” so a missing signature doesn’t mean the bot is malicious. Site owners are advised to continue to use IP addresses and reverse DNS around the protocol to avoid accidentally blocking legitimate traffic that has not yet moved.
What it does
The new standard replaces the manual setup between websites and bots, crawlers, and other automated services with a three-step discovery process:
- Important Default Files:
The keys are stored in a common format, JSON Web Key Set (JWKS), which can be read by all servers. - Known Addresses:
It defines a specific “home” on the website (/.well-known/) where these keys are always stored. - Identity requests:
Adds a new header, Signature Agent, to HTTP requests that acts like a digital business card, pointing the recipient directly to the sender’s key directory.
Benefits of Automated Services and Websites
Web Bot Auth can make bot authentication easier to scale by reducing the need for manual setup between each website and an automated service. It also gives automated services a more consistent way to stay visible when their security credentials change, which can help avoid broken authentication later.
Web Bot Auth is an experiment
Google insists that users should continue to use existing standards such as IP user agent-based bot authentication, stressing that the standard itself is a proposal that may change.
The new documentation provides the following warning:
“Test mode means:
Not all Google users use Web Bot Auth.
Google has not yet signed all agent requests using the protocol.
We recommend that in addition to Web Bot Auth you continue to rely on IP addresses, reverse DNS, and user agent strings as we gradually phase out signed traffic.
If you are a developer or system administrator who wants to authorize our AI agents to test, you can use authentication with the Web Bot Auth protocol:
- Using a product or service that supports Web Bot Auth
- You verify requests yourself”
However, the standard aims to make it easier to identify bots and control bot traffic by using a cryptographic protocol that a malicious agent can’t fool, provide information on how bots interact with your traffic, and create a better way to manage the out-of-control situation with bot crawling.
Google encourages users interested in the protocol to contact their web hosting provider to see if they intend to support the testing protocol, stay up-to-date with the latest changes published by the Web Bot Auth Working Group and submit feedback via the official Google Web Bot Auth feedback form.
Read the new Google docs:
Authenticate requests with Web Bot Auth (test)
Featured image by Shutterstock/Efkaysim
