Fastly Threat Insights Report Reveals Organizations are Fighting Back Against the Rise of the Bots

Key Terms

api technical

An API, or Application Programming Interface, is a set of rules that allows different software programs to communicate and work together smoothly, much like a waiter translating your order into the kitchen and then bringing your meal back. For investors, APIs are important because they enable real-time access to financial data, trading systems, and other digital services, making it easier to make informed decisions quickly and efficiently.

ai crawlers technical

AI crawlers are automated programs that surf and read websites, documents, and public data using artificial intelligence to extract, summarize, or classify information. Like a smart robot librarian that not only collects books but also understands and tags their contents, they matter to investors because they can surface or expose valuable data (or sensitive information), affect a company’s online visibility and reputation, and influence competitive intelligence, compliance and the timing of market-relevant disclosures.

headless bots technical

Headless bots are automated computer programs that run tasks without a visible interface or a person watching them, often operating on servers to fetch information, submit orders, or post content. Like an unseen worker that cleans or checks things quietly, they matter to investors because they can speed up legitimate data flows or create misleading activity—such as inflating web traffic, scraping press releases, or executing rapid trades—which can distort market signals and affect liquidity.

generative ai technical

Generative AI is a type of computer technology that can create new content, like text, images, or music, on its own. It’s important because it can produce realistic and useful material quickly, which could change how we create art, write stories, or even develop new products. Think of it as a smart robot that can invent and produce things almost like a human.

From AI Crawlers to Headless Bots, Organizations Face New Challenges in Managing Automated Traffic

SAN FRANCISCO--(BUSINESS WIRE)-- Fastly, Inc. (NASDAQ: FSLY), a leader in global edge cloud platforms, today released its latest Threat Insights Report, offering a detailed analysis of the evolving bot landscape and its implications for businesses. Drawing from trillions of application and API requests across Fastly’s global network, the report highlights how automated traffic is reshaping industries, forcing organizations to adapt their strategies to manage both the risks and opportunities bots present.

The findings reveal that bots now account for nearly one third (29%) of all web traffic, with the majority of this bot traffic (about 25% of all web traffic) classified as unwanted. Additionally, bot requests increased by 2% compared to the prior quarter. This increase represents billions of requests and is driving organizations to respond, particularly in industries like Media and Entertainment, Commerce, and Financial Services.

Key findings from the report:

• Wanted bots act as a double-edged sword: Organizations are increasingly scrutinizing the value of wanted bots, like AI crawlers and fetchers. While these bots play a critical role in indexing and optimizing content, billions of their requests were blocked (4%). This trend reflects growing concerns about data usage and the impact on revenue, particularly in industries like Media and Entertainment, where AI fetchers are reducing referral traffic to source websites.
• Headless bots become a threat to transaction-heavy industries: Headless bots, which mimic human behavior at machine speed, are targeting transaction-heavy industries like Financial Services and Commerce. 89% of headless bot traffic in Q3 targeted these two industries, reinforcing the need for businesses to implement advanced detection strategies that can distinguish between nefarious headless traffic and intended headless usage so as to avoid disrupting software delivery.
• AI crawlers and fetchers dominate the digital landscape: Meta’s AI crawler and OpenAI’s ChatGPT fetcher emerged as the dominant players, accounting for 60% and 68% of their respective traffic categories. This dominance highlights the growing influence of AI-driven automation on web traffic and the critical need for organizations to balance innovation with control over content.

“Fastly’s latest Threat Insights Report highlights the urgency with which organizations should take a proactive approach to bot management as it’s increasingly becoming a critical business issue,” said Marshall Erwin, Chief Information Security Officer at Fastly. “By understanding the nuances of bot traffic, businesses can safeguard their infrastructure, optimize operations, increase brand visibility, and drive innovation.”

The report sheds light on bot industry preferences, and their impact across sectors as organizations reconsider how to remain competitive in the AI era. Findings highlight Commerce (88%) and Public Sector (96%) receiving the highest distribution of crawler traffic, compared to Education (77%) with the highest fetcher volume, suggesting an indication of crawler prioritization or ‘fresh’ yet to be crawled content.

The report also highlights how businesses must adapt with the rise of AI. Media and Entertainment publishers are struggling as AI fetchers scrape their content for generative AI models, leading to diminished referral traffic – only 1% of users click through to source websites that have an AI summary. Commerce saw the highest crawler traffic, driven by demand for real-time inventory and pricing data, underscoring the need to balance automation with competitive protection.

As organizations consider how to effectively operate amid rising bot volume, the latest report provides macro insights to fuel decisions. For more information and to access the full Fastly Threat Insights Report, visit www.fastly.com.

About Fastly:

Fastly’s powerful and programmable edge cloud platform helps the world’s top brands deliver online experiences that are fast, safe, and engaging through edge compute, delivery, security, and observability offerings that improve site performance, enhance security, and empower innovation at global scale. Compared to other providers, Fastly’s powerful, high-performance, and modern platform architecture empowers developers to deliver secure websites and apps with rapid time-to-market and demonstrated, industry-leading cost savings. Organizations around the world trust Fastly to help them upgrade the internet experience, including Reddit, Neiman Marcus, Universal Music Group, and SeatGeek. Learn more about Fastly at https://www.fastly.com, and follow us @fastly.

Source: Fastly, Inc.

View source version on businesswire.com: https://www.businesswire.com/news/home/20251218074042/en/

Media Contact

Stacey Hurwitz

press@fastly.com

Investor Contact

Vernon Essi, Jr.

ir@fastly.com

Source: Fastly, Inc.