The AI Scraping Free-for-All Is Over: Welcome to the Era of Licensing (2026)

Introduction: The Zero-Click Problem

For years, search engines and AI displayed content directly on their platforms, satisfying user queries without ever sending traffic to the original site. This “Zero-Click” era left website owners asking:

If the platforms are getting the engagement, shouldn’t they pay for the content that fuels it?

AI companies treated the open web like free real estate. That era is over. We have entered the Era of Licensing, where data is treated less like “free information” and more like oil – a valuable resource that must be bought.

Yet while the “Big Gatekeepers” are cashing in with nine-figure deals, small websites remain largely uncompensated.

Who Is Actually Getting Paid? (The Gatekeepers)

Today’s largest AI companies – OpenAI, Google, Meta, and xAI – pay only a small set of data gatekeepers:

1. Social & Community Hubs

Platforms that own massive volumes of human conversation are among the biggest winners.

• Reddit and Stack Overflow dominate here
• Their discussions teach AI how humans actually talk, argue, joke, and explain things (and of course, sometimes the AI takes the advice of adding glue to your pizza sauce to keep the cheese from sliding off as fact, and not a joke!)
• Licensing deals with Google and OpenAI are estimated at $60M-$200M per year

These platforms don’t just host content – they own the data rights at scale.

2. Media Conglomerates

Instead of negotiating with individual journalists or publishers, AI companies license entire portfolios.

• News Corp (Wall Street Journal)
• Axel Springer (Business Insider, Politico)
• Condé Nast (Wired, Vogue)

Many of these agreements exceed $250M over five years, often following lawsuits that forced negotiations.

3. Data Brokers & Repositories

These are the wholesalers of AI training data.

• Appen
• Defined.ai
• Similar firms that hire humans to label data or acquire large, clean datasets

AI companies pay a premium for data that is already organized, labeled, and legally licensed.

Related: The $10 Billion “Echo”: Why Paying Experts to Train AI Might Be a Bridge to Nowhere

Why Individual Websites Aren’t Getting Paid (Yet)

If you run a blog, a niche site, or a small business website, chances are your content has already been scraped – without compensation.

Here’s why the system has been stacked against smaller players.

The Scaling Problem

It’s far easier to sign one deal with a company that owns 40 publications than to negotiate with 40,000 independent websites.

Lack of Leverage

A single site owner usually can’t:

• Detect advanced AI crawlers
• Enforce legal compliance
• Afford prolonged litigation

Large publishers can – and they have – which is why AI companies negotiate with them first.

The “Search Trap”

For over 20 years, websites wanted bots to crawl their content for search engine traffic.

AI companies exploited that same open-door policy:

• If Google could crawl it, so could an AI model
• Crawling permission quietly became training permission
• The RAG Shift (Retrieval-Augmented Generation): AI now scrapes your site live to answer questions instantly
• Real-Time Extraction: They use your current data to keep users off your site now

What helped SEO also enabled mass extraction.

The 2026 Shift: AI Tollbooths & Licensing

This year marks the first real turning point for independent websites.

Instead of begging to be paid later, site owners are beginning to lock the door first.

Cloudflare Pay-Per-Crawl

Cloudflare now allows site owners to:

• Automatically block known AI bots
• Require payment before access
• Enable micro-transactions per crawl or request
• Block known AI user agents (including the newer 2026 “stealth” bots)
• Use server-level rules (since ~40% of AI bots now ignore robots.txt)
• Add “No-AI” Meta Tags to your site headers for legal provenance

This alone gives small websites leverage they never had before.

Related: AI Gone Rogue Again? Perplexity Bots Bypass IP Blocks and Robots.txt

TollBit

TollBit acts as a clearinghouse between sites and AI companies.

• Websites register once
• AI companies pay a small “toll” when content is accessed
• No individual negotiations required

Really Simple Licensing (RSL)

An emerging standard (similar in spirit to RSS) that tells AI crawlers:

“You may read this content – but each query costs $0.01.”

Simple, machine-readable, and enforceable.

However, while RSL is the emerging industry standard for 2026, it is a ‘signaling’ tool. Think of it as the ‘No Trespassing’ sign on your lawn – it defines your rights legally, while tools like Cloudflare act as the physical fence that keeps people out.

How to Implement These AI Tollbooths

You don’t need to switch hosting to use these tools. Whether you use UltimateWB’s hosting plans or your own server, you can simply “layer” Cloudflare or TollBit on top of your site. Because UltimateWB gives you total control over your server rules and headers, you’re actually in a better position to implement these 2026 standards than users on more restrictive “closed” platforms.

Step-by-Step: Layering Cloudflare Security over UltimateWB

To “layer” Cloudflare over your UltimateWB hosting, you use what’s called a Full DNS Setup. This keeps your website files exactly where they are on the UltimateWB servers, but it routes your traffic through Cloudflare’s security “tollbooth” first.

1. Create a Free Cloudflare Account: Go to Cloudflare.com and add your domain name.
2. Scan DNS Records: Cloudflare will automatically find your current UltimateWB hosting records (A records and MX records).
   • Tip: Ensure the “Proxy Status” column has the Orange Cloud icon turned ON for your main domain and www records. This is what enables the AI blocking and Pay-Per-Crawl features.
3. Update Nameservers: Cloudflare will give you two new nameservers (e.g., dara.ns.cloudflare.com and olga.ns.cloudflare.com).
   • Log in to your Domain Registrar (where you bought your domain).
   • Replace your current nameservers with the ones Cloudflare provided.
4. Enable AI Protection: Once the DNS “propagates” (usually takes a few minutes to an hour), go to the Security > Bots tab in Cloudflare.
   • Toggle “AI Scrapers and Crawlers” to ON.
   • (Optional) Enable “Pay-Per-Crawl” to begin the process of monetizing any AI bots that you choose to let through.

How to Implement RSL: A 3-Step Guide for UltimateWB Users

Because you have full control over your files and headers with UltimateWB hosting plans, or on your own server, you can set up the 2026 “Paid Access” standard in under 5 minutes.

1. Create your license.xml file

This is the machine-readable contract. Create a file named license.xml and upload it to your root directory.

XML

<?xml version="1.0" encoding="UTF-8"?>
<rsl xmlns="https://rslstandard.org/rsl">
  <content url="/" server="https://api.rslcollective.org">
    <license>
      <permits type="usage">ai-train ai-input</permits>
      <payment type="inference">
        <amount currency="USD">0.01</amount>
        <standard>https://rslcollective.org/license</standard>
      </payment>
    </license>
  </content>
</rsl>

Note: Using the ai-input permit allows search engines to show your content but requires payment if it’s used to generate an AI answer.

2. Update your robots.txt

In 2026, robots.txt has been extended to include the License directive. Add this line to tell all bots where your terms are:

License: https://yourdomain.com/license.xml

*Replace yourdomain.com with your actual domain name.

3. Add the “Link” Tag to your Header

Using the UltimateWB built-in Ad(d)s app to add in the bottom of your <head> section – i .e. your Header & Meta Tags section, add this link. This ensures that even if a bot skips your robots.txt, it still “sees” your price tag as soon as it crawls your page.

<link rel="license" type="application/rsl+xml" href="https://yourdomain.com/license.xml">

RSL is still emerging, but it represents the direction the industry is moving.

The Reality Check in 2026

Player	Status	How They Get Paid
Big Platforms	✅ Winning	Direct multi-million-dollar licensing deals
Legacy Media	⚖️ Fighting	Lawsuits followed by licensing partnerships
Small Websites	🛠️ Transitioning	Blocking access until paid
Individual Creators	❌ Struggling	Platforms sell their data, not them

The imbalance still exists – but for the first time, small sites have real tools.

Of course, these tools that collect the payment from the AI are not free…

The Cost of the “Tollbooth” (2026 Pricing)

Service	Cost Category	Estimated Price (2026)
Cloudflare	Freemium	Free Plan: Includes basic Bot Fighting Mode. **Pro Plan ($20/mo):** Required for advanced WAF rules and full AI Crawl Control features.
TollBit	Revenue Share	Free to Join: TollBit typically takes a 20-30% cut of the “tolls” they collect from AI bots on your behalf. No upfront fee.
RSL Collective	Free / Membership	Free: To host your own license.xml. Membership: Small annual fee (~$50) if you want them to handle legal enforcement/billing.

The 2026 Adoption Report: Where These Tools Stand Today

Technology	Status (2026)	Adoption & Usage
Cloudflare Bot Management	✅ Mainstream Standard	Used by 30%-40% of the top 1 million websites. It is the gold standard for immediate, “hard” blocking of AI scrapers.
TollBit	⚖️ Scaling Rapidly	Now the primary clearinghouse for major media (Time, Vox, etc.). It’s the “Enterprise” choice for sites that want to be paid by OpenAI and Google directly.
Really Simple Licensing (RSL)	🚀 Official Industry Standard	Finalized as RSL 1.0 in late 2025. It is currently being adopted by ~1,500 major media organizations (including AP and Yahoo) and is the “proposed universal language” for the web.
Pay-Per-Crawl	🛠️ Emerging Feature	Currently in “Early Access” for most users. It is highly effective but requires a compatible CDN (like Cloudflare) to enforce the payment.

What You May Do Right Now: Lock the Door First

If you don’t like AI crawling your website for free…

Stop giving AI companies free access.

That means setting up a No AI Scraping protocol for your website.

At a minimum, this includes:

• Blocking known AI user agents
• Using server-level rules instead of relying solely on robots.txt
• Preparing your site to support paid access when licensing becomes standard

With platforms like UltimateWB, this is far easier to implement because you control:

• Your server rules
• Your headers (including No-AI meta tags)
• Your access logic

You don’t have to wait for a platform to decide what happens to your content.

1. The “No-AI” Meta Tag

Add this to the <head> section of your website. This is the legal signal that “well-behaved” AI crawlers (like those from Google or OpenAI) are required to respect.

<meta name="robots" content="noai, noimageai">

With the UltimateWB website builder, just use the built-in Ad(d)s app as mentioned above – go to List Ad(d)s and find the Ad(d) for the bottom of the head section, and paste this there, and click the Save button.

2. The Server-Level Block (.htaccess)

Since many 2026 bots ignore robots.txt, blocking them at the server level is much more effective. If you are on an Apache server (standard for most, like in the UltimateWB web hosting plans), you can add this to your .htaccess file:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (AI2Bot|GPTBot|Google-Extended|CCBot) [NC]
RewriteRule .* - [F,L]

Related: How to Block Unwanted Visitors? Use .htaccess for IP Blocking on Your Website

Webpage direct visits: bots, crawlers, or real visitors?

Performance, SEO, and the “Discovery” Trap

Many site owners hesitate to block bots because they fear two things: slowing down their site or becoming invisible. Here is the reality in 2026.

1. Will Cloudflare slow down my site?

Actually, the opposite is true. While Cloudflare adds a “filter” layer, it is famous for its CDN (Content Delivery Network), which stores copies of your site in thousands of locations globally.

• The “Clean Traffic” Bonus: By blocking AI bots, you stop them from eating up your server’s CPU and bandwidth. In 2026, AI bot traffic has quadrupled; blocking them may even result in a 20-40% speed increase for your actual human visitors because your server isn’t “distracted” by scrapers.
• The Result: Better Core Web Vitals, which Google uses as a major ranking factor.

Related: The Hidden SEO Power of Your Web Hosting Provider

2. If I block AI, will I lose my SEO rankings?

No. Cloudflare and the .htaccess rules we provided are “surgical.” They are designed to block Training Bots (like GPTBot) while allowing Search Bots (like Googlebot) to pass through freely.

• Search vs. Training: Googlebot still indexes your site for traditional search results even if you block Google-Extended (the AI training arm). Your rank on page 1 of Google stays safe.

3. The “Citation” Risk: Will AI still refer to me?

This is the hardest trade-off. In 2026, we see a “Crawl-to-Referral” imbalance:

• The Bad News: If you block all AI bots, ChatGPT or Perplexity might not be able to “read” your latest post to cite it in a conversation.
• The 2026 Strategy: This is why we recommend a Layered Approach.
  • Block Training Bots: Stop them from using your data to build their models for free.
  • Allow Retrieval Bots (Optional): Some owners choose to allow “Search-focused” AI bots (like Perplexity or Bing’s AI) because they are more likely to include a link back to your site.

Related: How to Check if Your Website is Showing Up in ChatGPT, Perplexity, Gemini, or Other AI Answers

How to Optimize Your Webpages So They Get Found by Search Engines and AI (LLMs)?

The Verdict: Visibility vs. Value

If you are a small business or niche blog, being cited by an AI can be good marketing. But if the AI is “answering” the question so well that the user never clicks your link (the Zero-Click problem), that citation is worthless.

By using the UltimateWB + Cloudflare setup, you get to choose: You can block the “greedy” bots that never send traffic and keep the door open for the “referral” bots that actually help your brand grow.

The 2026 “Safe List”: AI Bots That Actually Pay (In Traffic)

If you want to “Lock the Door” but keep a “Mail Slot” open for discovery, these are the bots you should allow-list. These companies have committed to a “Cite and Refer” model that actually sends users to your site.

The “Referral Friendly” List:

1. PerplexityBot (Perplexity AI): The leader in AI search; consistently provides high-visibility citations and source links. Consistently cited as the best “AI Citizen” for sending high-quality traffic.
2. Google-Search-Generative (Google Gemini): While controversial for “Zero-Clicks,” it is still the largest source of “AI-Referral” traffic in 2026. Essential for staying visible in Google’s AI Overviews.
3. Bingbot / MSB-AI (Microsoft Copilot): Respects licensing and provides clear links back to publishers.
4. Applebot-Extended (Apple Intelligence): Known for “on-device” citations that encourage users to visit the source.
5. ChatGPT-User and OAI-SearchBot: These are the “Search” bots. They only visit your site when a user specifically asks ChatGPT a question that requires a live web search (Retrieval-Augmented Generation). These bots are the ones that actually generate the blue links and citations inside the chat interface. If you block these, you become invisible in “ChatGPT Search.”

The “Block List” (Scrapers with No Traffic, Training Only):

• CCBot: The Public Resource (Common Crawl) – Often used for “Mass Training” where your data is swallowed, and you are never cited.
• Bytespider (TikTok/ByteDance): Aggressive scraping for internal models with almost zero outbound traffic to small sites.
• GPTBot: OpenAI’s training crawler. This bot is the “Vacuum.” It scrapes your site to train future versions of ChatGPT. It provides zero traffic and zero citations. Most publishers in 2026 (over 80% of major news sites) block this bot to protect their intellectual property.

The Bottom Line

The AI gold rush is over.

Data now has a price – and while the biggest players are cashing in first, the infrastructure for independent websites to get paid is finally emerging.

Until it fully arrives, you may decide the smartest strategy isn’t participation.

It’s protection.

Lock the door now on the training bots – and be ready when AI companies are forced to knock.

The next phase of the web won’t be defined by who can scrape the fastest –
but by who controls access.

Related: Is it a bad idea for SEO and search engine indexing and ranking to block bots and crawlers from accessing your website?

How to deal with bad bots and crawlers that waste your server resources and harm your website?

Ready to design & build your own website and charge AI training bots? Learn more about UltimateWB! We also offer web design packages if you would like your website designed and built for you.

Got a techy/website question? Whether it’s about UltimateWB or another website builder, web hosting, or other aspects of websites, just send in your question in the “Ask David!” form. We will email you when the answer is posted on the UltimateWB “Ask David!” section.