AI bots are increasingly affecting website performance, analytics, infrastructure costs, and content visibility. New research and infrastructure data suggest that the challenge is no longer just scraping, but controlling how automated traffic interacts with websites and the businesses that depend on them.
Scratching is the least of the problems
Many discussions between SEOs and site owners focus on the mention of AI bots. It is a valid concern that AI systems favor LLM training content with almost zero annotation when the content is recombined into AI feedback.
- Landlords are concerned about intellectual property.
- Search marketers worry about how AI systems use their content.
But infrastructure teams are increasingly seeing different and equally consequential problems.
The Banality of Bots Losing and Scraping Things
The problem is that many bots create unnecessary load, consume resources, and sometimes get stuck in idle loops.
According to the report, one recurring pattern involved Meta’s meta-externalagent crawler tracking URL variations for days on end before mitigation systems caught on.
This type of behavior is not cruel. It is an automatic operation with poor coding practices or insufficient guard strokes.
Cloudflare’s David Belson demonstrated the ban on lost resources for bots:
“Someone didn’t know what they were doing yesterday, but they wrote a bot today and released it. They don’t even bother to check robots.txt.”
That realization holds an important truth. Today’s infrastructure problems now stem from poorly designed automation at scale.
Bots Consume Resources Without Creating Value
The result of this behavior is that websites use services that provide automated traffic that may provide little or no value in return.
This is a big problem for ecommerce sites. Unlike static page requests, cart-related requests typically bypass caching and require the server to use resources. Depending on the site’s architecture, those requests can initiate PHP execution, database queries, session management, and other resource-intensive processes.
As can be seen from this, scraping is a minor website problem. A browser that repeatedly triggers an expensive application and consumes server resources degrades the performance of legitimate visitors.
The economic impact should not be overlooked. According to the report, about 80% of AI crawling work is associated with model training, overlay search or user-driven crawling.
For most businesses, the question is: Is there any value returned by that traffic to justify the resources used?
Businesses Stuck Between Visibility and Cost
If the solution was simply to block the bots, the problem would be solved. Unfortunately, many automated services that use resources are also tied to availability and visibility.
Some bots help search engines find content. Others may contribute to AI citations and appear in AI-generated responses. Others may use the content and resources without producing directly measurable business benefits.
Businesses are asked to contain the cost of automated traffic while at the same time evaluating whether that traffic is contributing enough visibility to justify that cost.
Question Now: Which Bots Are Worth Paying?
The report says site owners should ask this question:
Which bots, which parts of my site, under what conditions?
Bot management affects visibility, infrastructure costs, and site performance. The goal is to align automated traffic with business goals.
Traffic Numbers May Be Affected
Automated traffic also affects website statistics. According to the report, AI bot traffic has increased by 300% in the last year. By the end of 2025, approximately one in 31 visits to the TollBit network came from an AI bot.
As automated traffic grows, traffic volume alone becomes a less reliable indicator of audience growth.
A site may show increased visit rates while not experiencing a corresponding increase in customers, subscribers, conversions, or revenue. In some cases, additional traffic may occur automatically.
The report argues that the most important signals come from metrics that match actual business results, including demand for branded searches, direct traffic, quality of engagement, and revenue.
Since automated systems make up a large portion of overall traffic, raw visit statistics become less useful as an independent measure of success.
Mitigation Solutions and Strategies
The report advocates a more specific approach to bot management.
The first step is visibility.
Before making changes, site owners should understand what automated traffic actually does. The goal is not to identify every single bot but to identify patterns such as repeated requests, loops, and activity focused on dynamic endpoints.
The second step is to protect the most expensive site operations.
Cart URLs, checkout methods, internal search pages, filtered product pages, and parameter-heavy URLs often use more resources than standard content pages. Limiting unnecessary search engine access to those areas can reduce waste without interfering with valuable content.
The report also recommends separating search engines from AI engines.
Not all bots offer the same value. Search engines contribute directly to discovery and deserve a wider reach than AI-trained crawlers or anonymous scrapers.
A single policy applied to all automated systems will no longer be justified as the ecosystem grows more complex. That’s why the report advocates targeted reforms rather than broad restrictions.
The goal is not to remove automated traffic. The goal is to manage it in a way that supports business goals while reducing unnecessary costs. Another way is to decide which bots can access certain parts of the site and under what conditions.
What is taken
Bot traffic is no longer a problem to scratch. Data suggests that it has become an infrastructure, visibility, analysis, and business management problem.
The biggest challenge is that many bots consume resources, become expensive to operate, inflate traffic metrics, and create costs for site owners to bear.
Bot management is not about preventing more bots. It’s about managing bots according to what the site is working for by distinguishing between valuable and wasteful automated traffic.
Read Kinsta’s data report:
Real-time AI and bot traffic testing
Featured image by Shutterstock/DC Studio
