The information in this blog post is for those clients of ours who use Cloudflare for DNS management, security, performance, or content delivery. A recent change made by Cloudflare is shaking things up for websites, including eCommerce stores. As of July 1, 2025, Cloudflare has implemented a policy to block AI crawlers for all its customers by default, including those on free plans, and introduced a “Pay Per Crawl” program. This change could significantly impact how your online store’s content is accessed and used by AI systems.
Store owners will now need to allow AI systems to scrape your website’s content, such as product descriptions, images, or customer reviews. For eCommerce businesses, this could protect your intellectual property but may also limit your content’s exposure if you rely on AI-driven platforms to drive traffic. Additionally, Cloudflare’s Pay Per Crawl program, currently only in beta, allows large content publishers to charge AI companies for scraping content, potentially creating a new revenue stream.
The key take-away here is, you’ll need to decide whether to maintain the default block or allow specific crawlers based on your store’s goals.
What Is AI Crawling?
AI crawling refers to automated bots, or web crawlers, that scan websites to collect data for training artificial intelligence models, powering AI search tools, or generating content. Unlike traditional search engine crawlers (like Googlebot), which index content to drive traffic back to your site, AI crawlers often extract data—such as product listings or blog posts—to feed large language models or AI-driven search results without necessarily directing users to your store.
For example, an AI crawler might scrape your product descriptions to train a chatbot or generate AI answers that bypass your website entirely. While traditional crawlers have long supported eCommerce by boosting SEO, AI crawlers can strain server resources and use your content without compensation or attribution, which has sparked many a debate.
Pros and Cons of Blocking AI Crawlers
-
Protect Intellectual Property: Your product descriptions, images, and customer reviews are valuable assets. Blocking AI crawlers prevents unauthorized use in AI models, safeguarding your brand’s unique content.
-
Reduce Server Strain: AI crawlers can mimic DDoS attacks by overwhelming servers with frequent requests. Blocking them can improve site performance, ensuring a smooth shopping experience for customers.
-
Control Over Content Use: You decide which AI crawlers (if any) can access your site and for what purpose—training, search, or inference—giving you greater control over your digital presence.
-
Reduced Discoverability: Some AI-driven platforms, like AI search engines, may rely on crawlers to index your products. Blocking them could limit your store’s visibility in emerging AI-powered search ecosystems.
-
Missed Partnerships: Allowing certain crawlers could lead to collaborations with AI companies if you’re a large brand, such as licensing deals, which might benefit your brand’s reach or revenue.
-
Complexity in Management: Deciding which crawlers to allow or block requires ongoing monitoring and adjustments, adding to your workload.
How to Manage AI Crawler Settings in Cloudflare
Cloudflare makes it easy to control AI crawler access through its dashboard, giving you the flexibility to align settings with your strategy. More information on how to enable or disable AI crawling, allow access to only certain crawlers, or learn more about the upcoming Pay Per Crawl program can be found in Cloudflare’s AI Audit documentation.
Our Take On It
We were a little taken back to learn that blocking would be turned on by default. But at the same time, can appreciate the fact that many website and eCommerce store owners have mixed reactions to AI crawlers scraping site content to use for non-traffic driving means. However, it’s no secret that AI is rapidly changing the face of how people search for content, shop for products, and look for recommendations. It’s important to consider what impacts either blocking or allowing AI crawlers will have on your business before flipping the switch.
Scott Sanfilippo began his eCommerce journey in 1994 by co-founding one of the Internet’s first online retailers, TheFerretStore.com, which was acquired by PetCo in 2006. In 2001, he co-founded the eCommerce design and marketing firm Solid Cactus, which was acquired by web.com in 2009. Today, Scott is the General Manager of Your Store Wizards and lives in Delray Beach, FL. Scott can be contacted at scott@yourstorewizards.com.