How to Whitelist Googlebot, Bingbot in CSF

One way to prevent server load spikes and maintain fast performance is to block problematic IP addresses and IP ranges. However, you have to make sure that legitimate crawlers, such as Googlebot and Bingbot, are not accidentally blocked from accessing your websites. Don’t crash your SEO! We covered this important step in, “How to Stop High Server Load from Distributed Botnets, Scrapers, and Database Queue Locks”. If you don’t know how to do this whitelisting, this tutorial is for you.

You might be thinking, Can I just use the terminal and the -i flag to ignore a specific IP address in the CSF, similar to how you use the -d flag to ban an IP? The answer is No – the -i flag in CSF is used to ignore a specific IP address (e.g., csf -i 1.2.3.4).

To add domain masks (like .googlebot.com), you have to manually add them to the configuration file.

How to Whitelist Via WHM

You can do this entirely through the WHM interface. It is often safer for beginners because it handles the file formatting and the firewall restart for you in a few clicks.

Here is the step-by-step for whitelisting Googlebot and Bingbot via WHM:

1. Access the CSF Plugin

1. Log in to WHM as root.
2. In the top-left search bar, type “firewall” or “csf”.
3. Click on ConfigServer Security & Firewall under the Plugins section.

2. Locate the “Ignore” Files

Scroll down the page until you see the section labeled lfd – Login Failure Daemon. This is the part of the firewall that tracks “bad behavior” and triggers blocks.

1. Look for the dropdown menu labeled Edit lfd ignore file.
2. Select csf.ignore from that list. (Note: Some experts use csf.rignore for reverse DNS, but csf.ignore works perfectly for domain masks in most cPanel setups).
3. Click the Edit button next to the dropdown.

3. Add the Domain Masks

The file will open in a text editor inside your browser.

1. Scroll to the very bottom of the text.
2. Paste the same lines you would use in the terminal: .googlebot.com .google.com .search.msn.com
3. Click the Change button at the bottom of the page.

4. Restart the Firewall

After you click “Change,” WHM will save the file, but the firewall is still running the old rules in its memory.

1. On the next screen, you will see a button labeled Restart csf+lfd.
2. Click it. The firewall will restart, and your new “Ignore” list will be active.

The Difference Between the “Ignore” files you can choose from?

Inside that WHM dropdown, you might see several files. Here is the difference:

• csf.ignore: Use this for IP addresses and Domain Masks (like .googlebot.com). It tells the server: “Never block this IP, even if it tries to log in 100 times.”
• csf.rignore: This is the “Reverse DNS Ignore” file. It is specifically built for patterns like Googlebot. If you put it here, the firewall does a deeper check to ensure the bot is actually Google and not a fake bot pretending to be Google.
• csf.pignore: Don’t use this for IPs. This is the “Process Ignore” file. You use this to stop the firewall from sending you emails like “High Resource Usage: /usr/bin/php”.

Recommendation: Adding them to csf.ignore is the most common and “bulletproof” way to ensure they aren’t blocked.

How to Whitelist via the Terminal

This method may be faster, but can be prone to more user error.

Here is the step-by-step terminal process:

1. Open the Ignore File

Open the csf.ignore file using the nano text editor:

nano /etc/csf/csf.ignore

2. Add the Domain Masks

Scroll to the bottom of the file and paste the following lines. These allow CSF to perform a reverse-DNS lookup to verify the IP actually belongs to Google or Bing before allowing it through:

.googlebot.com
.google.com
.search.msn.com

3. Save and Exit

• Press Ctrl + O then Enter to save.
• Press Ctrl + X to exit.

4. Restart the Firewall (CRITICAL)

The changes will not take effect until you restart CSF and LFD (Login Failure Daemon):

csf -r

Pro Tip: The “One-Liner” Shortcut

If you want to do this quickly without opening the editor, you can use the echo command to append the lines directly:

echo ".googlebot.com" >> /etc/csf/csf.ignore && echo ".google.com" >> /etc/csf/csf.ignore && echo ".search.msn.com" >> /etc/csf/csf.ignore && csf -r

Why do this instead of just whitelisting IPs?

Google and Bing use thousands of different IP addresses that change constantly. By adding the domain mask (starting with a dot), you are telling the server: “If an IP claims to be Google, check its DNS. If it ends in .googlebot.com, let it through regardless of how many connections it makes.” This prevents your firewall from accidentally blocking Google during a “crawl spike” that looks like an attack.

---

Related: How to Edit .htaccess on the Site Level or Add Rules Globally

---

Looking for a website builder that is flexible enough for a developer but easy enough for a beginner? Learn more about UltimateWB! We also offer web design packages if you would like your website designed and built for you.

Got a techy/website question? Whether it’s about UltimateWB or another website builder, web hosting, or other aspects of websites, just send in your question in the “Ask David!” form. We will email you when the answer is posted on the UltimateWB “Ask David!” section.