AI Bot Traffic: Which Bots to Allow or Block

The internet is changing fast.

For years, the main goal of search was simple: to help users find links. A user searched, reviewed results, clicked a website, and consumed the content directly from the source.

But AI is changing that model. Increasingly, users ask AI assistants for answers instead of searching for websites. They seek summaries, recommendations, comparisons, troubleshooting advice and product guidance, often receiving a complete answer without ever visiting the original source site. This creates a new reality for website owners: your content may still influence the customer journey, even when the customer never reaches your website.

This shift is why AI bot traffic matters.

AI bots scan, fetch, and process web content so AI systems can answer questions, summarize pages, recommend vendors, and understand what your business does. Some of this traffic creates value. Some of it can create risk. The challenge is knowing the difference and building the right policy.

Quick answer: Not all AI bots should be treated the same. AI traffic falls into four main types: AI search bots that help your brand appear in AI answers, AI training bots that collect content to train models, AI fetch bots that retrieve a page on behalf of a user, and agentic AI systems that take actions such as logging in, querying inventory, or completing transactions. The right policy is not “allow all” or “block all”; it is to classify each AI bot, decide which to allow on which parts of your site, and inspect every request for malicious behavior. Imperva Advanced Bot Protection classifies AI traffic, while Imperva Web Application Firewall and DDoS Protection inspect and rate-limit it, so you stay visible to trusted AI while blocking abuse.

The new tension: visibility vs. protection

As the risk from AI-driven bots increases, many organizations are now facing a difficult question:

Should we allow AI bots to scan our website, or should we block them?

Blocking all AI bots may seem like the safest option, but it can also reduce your visibility in AI-generated experiences. If your public content is not accessible to the right AI systems, your brand may appear less often when users ask AI tools for recommendations or information.

On the other hand, allowing every AI bot creates business and security risks. AI bots may collect valuable content, increase traffic costs, overload servers, or access areas of the website that were never meant to be used by automated systems.

So, the right question is not “allow or block AI bots?”

The better question is:

Which AI bots should we allow, for what purpose, on which parts of the website, and under what protection?Not all AI bots are the same

The Rise of Agentic AI

The next wave of AI traffic will not come from crawlers alone. It will come from autonomous AI agents acting on behalf of users.

Agentic AI systems can browse websites, compare products, gather information, interact with APIs, complete forms, and execute multi-step workflows with limited human intervention.

Unlike traditional search crawlers, these agents actively interact with business logic. They can create accounts, query inventory, test workflows, retrieve pricing, and trigger application functions.

This creates a new challenge for security teams. The question is no longer simply whether an AI bot is legitimate. The question is whether the action being performed should be allowed.

As AI agents become more capable, organizations need controls that evaluate identity, intent, behavior, and access permissions simultaneously.

What Most Security Teams Cannot See

Imperva Advanced Bot Protection provides one of the industry’s most comprehensive AI traffic classification capabilities, enabling organizations to identify AI search bots, AI training bots, AI fetch bots, and emerging agentic AI systems. Security teams can then apply granular policies to allow, restrict, rate-limit, or block AI traffic based on business risk and application context.

As AI traffic becomes more diverse, organizations need visibility not only into which AI tools are accessing their applications, but also what those tools are doing, which business functions they are interacting with, and whether that activity aligns with organizational policy.

Effective control starts with understanding the different types of AI traffic and applying policies accordingly.

To create a good policy, you first need to understand the different types of AI bots.

1. AI search bots

AI search bots scan public websites so AI-powered search engines and answer engines can understand what content exists online.

These bots may help your business stay visible when users ask AI tools questions like “What is the best solution for…?” or “Which vendor supports…?”

You may want to allow these bots on public marketing pages, blogs, product pages, and documentation — especially content you want customers to discover.

2. AI training bots

Training bots collect public web content that may be used to train or improve AI models.

This is where the discussion becomes more nuanced. You may want your brand to appear in AI answers, but you may not want your full content, research, product information, or intellectual property used to train external models without control or compensation.

For many organizations, training bots require a stricter policy than search bots.

3. AI fetch bots

AI fetch bots usually act in response to a user request. For example, a user asks an AI assistant to summarize a specific URL, check a product page, or compare information from your site.

Because these requests are typically initiated on behalf of a user, they often represent legitimate business value. However, they can also be abused. An attacker may try to manipulate an AI assistant into sending suspicious payloads, scanning pages, or probing your application while hiding behind a legitimate AI fetcher.

That means fetch bots should not automatically be trusted. They should be allowed only with the same security inspection applied to any other request.

AI Is Not Only Changing Search. It Is Changing Attacks

AI is enabling attackers to create bots that learn from mitigation controls, adapt their tactics, and persistently probe applications in ways that were previously impractical at scale.

According to Imperva’s 2026 Bad Bot Report, the average number of AI-driven bot attacks has increased more than tenfold (12.5x) in 2025 compared to the previous year. This growth reflects a broader trend: the same technologies helping AI assistants understand websites can also help malicious actors understand application defenses, automate reconnaissance, and accelerate attack development.

Beyond increasing the scale and sophistication of bot attacks, AI is also lowering the barrier to entry for attackers. With the right prompts, common AI assistants can be weaponized to help identify weaknesses in web applications, generate attack payloads, automate vulnerability discovery, and assist with exploiting business logic flaws. While these tools have safeguards, they can still ramp up activities that traditionally required specialized security knowledge.

AI is also enabling attackers to coordinate large-scale campaigns more efficiently, combining reconnaissance, automation, and exploitation in ways that were previously more resource-intensive and time-consuming.

As a result, organizations must consider not only how AI systems access their applications, but also how AI may be used to automate, scale, and weaponize traditional web attacks.

What we see in real traffic

Thales’s threat research data shows that legitimate AI bots currently account for around 2% of total session traffic. The Bad Bot Report shows that within that AI bot traffic, 85% comes from AI crawlers and 15% comes from AI fetch bots. While AI traffic remains a relatively small proportion of overall web traffic today, its growth rate and increasing sophistication suggest it will become a much larger consideration for application owners over the coming years.

This matters because even legitimate AI bots can introduce operational, security, and business risks.

The main risks are:

Data scraping: AI bots may continuously collect proprietary content, product data, documentation, or intellectual property.

Unintentional overload:: AI bots can be aggressive. High-speed crawling may consume bandwidth, slow the application for real users, and increase infrastructure costs.

Abuse through AI fetch bots: because fetch bots act on user prompts, attackers may try to manipulate them into sending malicious requests or scanning for weaknesses while appearing to come from a legitimate AI source.

The good news is that classic application attacks from legitimate AI bots remain relatively low. Imperva’s findings show low levels of traditional attack vectors such as SQL injection, cross-site scripting, illegal resource access, backdoor attempts, and remote file inclusion. However, there are still real examples where AI fetch bots were manipulated into sending malicious payloads, including SQL injection-style input.

This is exactly why AI bot policy should never be based only on identity. Even a legitimate AI bot must still be inspected for malicious behavior.

Visibility Must Be Paired with Control

Visibility is only the first step. As AI traffic continues to grow, organizations need the ability to distinguish between beneficial AI activity and behavior that introduces operational, security, or business risk.

AI search bots, AI training bots, AI fetch bots, and agentic AI systems all require different treatment. The challenge is not simply identifying them. It is applying the right controls to the right AI traffic at the right time.

This is where advanced bot management becomes critical.

A safe AI bot policy should not be “allow everything”

In our blog, Why AI Bot Protection and Control are Essential for Application Security,  we said, “The ability to control which parts of your application functionality are accessible to AI tools is critical to your AI Security Strategy”.

The safest approach is a layered policy.

Start by separating the website into zones:

Public content you want AI systems to understand
Examples: blogs, product pages, public documentation, company information, support articles, and thought leadership.

Content you may want to restrict
Examples: premium content, customer-only portals, pricing logic, internal search, account pages, sensitive documentation, APIs, and dynamic application flows.

Content that should never be accessed by bots
Examples: admin paths, login actions, checkout abuse paths, personal data, private files, and application endpoints that can trigger expensive actions.

Then decide which AI bot categories should be allowed in each zone.

A practical policy could look like this:

Bot type	Recommended approach
AI search bots	Allow on public content where visibility matters
AI training bots	Review carefully; block or limit where IP protection matters
AI fetch bots	Allow only with full security inspection and rate controls
Unknown AI bots	Treat as suspicious until verified
Aggressive crawlers	Rate-limit or block if they impact performance

How Thales helps you stay visible and protected

The goal is not to trust or distrust AI traffic. The goal is to understand it, classify it, and apply the appropriate controls. Thales helps organizations strike the right balance between AI visibility and application security by combining Imperva Advanced Bot Protection, Imperva Web Application Firewall, and Imperva DDoS Protection. Thales offers customers a number of different options for managing AI bot traffic without choosing between full exposure and full blocking.

Firstly, Thales’ core security protections, delivered through Imperva Web Application Firewall, inspect the request itself. Rules for SQL injection, cross-site scripting, illegal resource access, backdoors, and remote file inclusion do not depend only on who the client claims to be. If an AI bot is manipulated into sending a malicious request, Imperva WAF can still block that request out of the box.

Secondly, Imperva DDoS protection helps reduce the risk of aggressive AI crawling overwhelming the application or creating performance issues. This is important because even legitimate AI bots can generate high request volumes.

Thirdly, customers can choose their own policy. If they want to allow a specific AI bot, using Imperva Advanced Bot Protection, they can create an exception based on the bot’s Client App ID. If they want to block a specific AI bot completely, they can add it to a bad bots list in the Imperva portal.

Most importantly, even when a specific AI bot is excluded from bot protection rules, malicious requests from that bot are still inspected by Imperva’s core security rules. In other words, allowing an AI bot for visibility does not mean you are leaving your application unprotected.

Frequently Asked Questions

Should you block all AI bots?

Not usually. Blocking every AI bot can reduce your visibility in AI-generated answers and recommendations, while allowing all of them exposes you to scraping, infrastructure overload, and abuse. The safer approach is to classify AI bots and allow them selectively, by website zone, with security inspection applied to every request.

What are the main types of AI bots?

There are four: AI search bots (so AI answer engines can understand your public content), AI training bots (which collect content to train AI models), AI fetch bots (which retrieve a specific page on behalf of a user), and agentic AI systems (which take actions such as logging in, querying inventory, or completing transactions).

What is agentic AI, and why is it a security risk?

Agentic AI systems act on behalf of users, browsing sites, filling forms, querying APIs, and running multi-step workflows with limited human input. Because they interact directly with business logic, the key question shifts from “is this bot legitimate?” to “should this action be allowed?”

How much web traffic comes from bots?

According to Imperva’s 2026 Bad Bot Report, automated traffic accounted for more than 53% of all web traffic in 2025. Legitimate AI bots are a small share of that today, around 2% of session traffic in Imperva’s threat-research data, but their volume and sophistication are growing quickly.

Can AI fetch bots be trusted?

Usually, but not blindly. Fetch bots typically act on a real user’s request, so they often carry legitimate business value. However, they can be manipulated into sending malicious payloads, so they should be allowed only with the same security inspection applied to any other request.

To learn more visit