ChatGPT's search feature can now pull from a much larger slice of the web, after a pilot program between Cloudflare and OpenAI reportedly added hundreds of millions of previously unindexed pages. For small business owners, this is a quiet but meaningful shift in how AI tools find and surface content — including, potentially, yours.
What happened
Cloudflare, which handles traffic and security for a large share of the internet's websites, ran a pilot with OpenAI that used network-level signals to help identify pages ChatGPT's crawler wasn't previously reaching. The result was a significant jump in the number of pages available to ChatGPT Search, according to Cloudflare's own account of the project.
This matters because AI crawlers and website owners have been in a standoff for the better part of two years. Many sites block AI bots outright through robots.txt files or bot-management settings, either to prevent content scraping, to preserve ad revenue, or simply out of caution. Cloudflare sits in the middle of that standoff — it's the infrastructure layer many sites use to manage who gets access.
The pilot appears to be less about new technology and more about untangling a permissions problem. Pages that were technically public but effectively invisible to OpenAI's crawler — because of blocking rules, server configurations, or crawl-budget limits — are now reportedly reachable. That's an indexing fix, not a new AI capability.
Why it matters
This fits a broader pattern of AI companies working to formalize their relationship with the sites they depend on for answers. Over the past year, Cloudflare has introduced tools letting site owners selectively allow or charge for AI crawler access, OpenAI has struck content deals with publishers, and lawsuits over unauthorized scraping remain unresolved in several cases. The friction hasn't gone away, but infrastructure providers are increasingly acting as brokers between AI companies and website owners rather than just gatekeepers.
It also reflects competitive pressure. ChatGPT Search is trying to be a credible alternative to Google, and a search tool is only as good as what it can see. Expanding the index is a direct way to close that gap.
What this means for small businesses
If your website runs through Cloudflare, this pilot may have changed whether ChatGPT can see your pages at all — without any action on your part. That's worth checking. Cloudflare's dashboard includes bot-management and AI-crawler settings where you can see what's currently allowed or blocked.
There's a genuine trade-off here, not a clear win. Being indexed means your business could show up in ChatGPT's answers when someone asks about your industry, products, or services — a new kind of visibility alongside traditional search rankings. But it also means ChatGPT can summarize your content directly in its answer, potentially reducing the number of people who click through to your actual site. This is the same "zero-click" tension that's been reshaping traditional SEO for years, now extending to AI search.
For businesses that rely on content marketing, blog traffic, or informational pages to drive leads, it's worth deciding deliberately whether you want AI crawlers indexing your site, rather than leaving it to default settings.
What to watch
Keep an eye on whether Cloudflare expands paid or permission-based crawling options — the company has already piloted "pay-per-crawl" arrangements — and whether OpenAI starts sharing referral or traffic data with site owners the way Google has historically done through Search Console. Also watch whether Google, Perplexity, or other AI search tools strike similar infrastructure deals, since that would suggest this becomes standard practice rather than a one-off.
The bottom line
If your site uses Cloudflare, log into your dashboard this week and check your AI bot and crawler permissions — you may now have more visibility in ChatGPT Search than you realized, for better or worse depending on your traffic model.