How to Stop High Server Load from Distributed Botnets, Scrapers, and Database Queue Locks

When a server load spikes to 30, 50, or 100+, most sysadmins panic and start haphazardly restarting services or blocking random IP addresses. However, modern attacks are rarely as simple as a single bad IP.

Today’s botnets use distributed Layer 7 (HTTP) floods, rotating through hundreds of unique IPs with only 1 connection each, exploiting web server application queues, fake WordPress scanners, background AJAX polling scripts, and WordPress admin-ajax.php endpoints that lock up MySQL.

This comprehensive guide covers how to diagnose, mitigate, and permanently protect your Linux/WHM web server during a high-load incident.

Phase 1: Diagnosing the Attack (Network & Process Layer)

1. Identify Top CPU-Hogging Processes

To see the exact processes, scripts, and system users consuming your CPU right now, run:

ps aux --sort=-%cpu | head -n 10

• What to look for: Look at the top process under COMMAND. If you see /usr/bin/php-cgi, /usr/sbin/mysqld, or admin-ajax.php taking 100%+ CPU, that specific user, script, or database engine is holding up your server load.

2. Check Connection Distribution

Run this pipeline to group active TCP connections by remote IP address:

netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n | tail -n 20

• Distributed Attack (Layer 7): Hundreds of lines showing 1 IP_ADDRESS. The attack is rotating through thousands of botnet IPs.
• Subnet Attack: Dozens of IPs coming from the exact same prefix (e.g., 47.79.13.x or 43.119.100.x).
• High 127.0.0.1 Count: Normal for reverse proxy setups (Nginx to Apache) or persistent backend database connections (~150 connections is typical). This is not an IP you want to block – it is your own server.

Phase 2: Locating & Reviewing Your Web Server Access Logs

Log file locations depend on your Linux distribution and control panel:

• Ubuntu / Debian (Apache): /var/log/apache2/access_log
• CentOS / AlmaLinux / RHEL (Apache): /var/log/httpd/access_log
• cPanel / WHM Domain Logs: /var/log/apache2/domlogs/yourdomain.com (Ubuntu) or /etc/httpd/domlogs/yourdomain.com (AlmaLinux)

How to Find Your Log Files Path Automatically:

find /var/log -type f -name "*access*" -mtime -1

How to Find Which Specific Domain is Being Targeted:

On multi-site servers (like WHM/cPanel), list domlog file sizes to instantly find the domain taking the heaviest hit:

ls -lhS /var/log/apache2/domlogs/ | head -n 10

(Look for log files that are unusually large, e.g., 100MB+).

it shows cumulative traffic since the last log rotation, which on most WHM/cPanel servers represents the current day, but can vary.

How WHM/cPanel Domlogs Work

1. Active Real-Time Files: Files directly inside /var/log/apache2/domlogs/ are the active log files Apache actively writes to.
2. Daily Rotation: By default, cPanel’s stats daemon (cpanellogd) processes and truncates/archives these logs during daily maintenance (usually overnight).
   • If rotation is working normally: The log file reflects traffic accumulated since the last rotation (typically today’s traffic).
   • If rotation is disabled or failed: The log file accumulates endlessly and contains days, weeks, or months of traffic data.

How to Verify the Exact Date Range

If you want to confirm whether a large log file is from an active attack right now or accumulated over time:

1. Check the First and Last Log Entries

Inspect the date timestamps at the start and end of the log file:

head -n 2 /var/log/apache2/domlogs/example.com
tail -n 2 /var/log/apache2/domlogs/example.com

2. Count Request Rate Right Now

To see if a domain is taking a heavy hit at this exact second, count live requests per second:

tail -f /var/log/apache2/domlogs/example.com | pv -l -r > /dev/null

Or run this command to watch incoming traffic in real time:

tail -f /var/log/apache2/domlogs/example.com

Press Ctrl + C on your keyboard to get out of the real time view and back to your shell prompt.

Note: If you are getting an error, pv: command not found, read the FAQ question below for the fix.

Phase 3: Firewall & Subnet Mitigation (CSF / WHM)

Manually blocking individual IPs during a distributed attack is ineffective. Instead, block entire abusive /24 subnets using ConfigServer Security & Firewall (CSF).

Blocking an Abusive Subnet in CSF:

csf -d 47.79.13.0/24 "Alibaba Cloud Botnet Range"
csf -d 43.119.100.0/24 "Tencent Cloud Botnet Range"
csf -r

The last line csf-r is necessary to restart the firewall service.

Phase 4: Application & Web Server Defense (.htaccess)

1. Stopping Fake WordPress Scans on Non-WordPress Sites (SAFE PERMANENT BLOCK)

If you run a custom CMS (like UltimateWB, or custom PHP) and are not using WordPress at all, automated bots will still flood your server searching for /wp-login.php, /wp-content/plugins/, or backdoor files like /file5.php.

Even if these return 404 errors, executing PHP to render a 404 page wastes CPU and database connections.

Add this to the VERY TOP (Line 1) of your root .htaccess file:

# --- SAFE PERMANENT BLOCK FOR NON-WORDPRESS SITES (LINE 1) ---
RedirectMatch 404 ^/wp-
RedirectMatch 404 ^/wordpress
RedirectMatch 404 ^/backup
RedirectMatch 404 ^/file[0-9]\.php

Result: Apache intercepts these requests in microseconds and returns a 280-byte response without launching PHP or MySQL.

2. Blocking WordPress xmlrpc.php Attacks (SAFE PERMANENT BLOCK)

Unless you use Jetpack or the mobile WordPress app, xmlrpc.php is an obsolete file targeted almost exclusively by automated botnets to execute brute-force amplification attacks.

Add this to /blog/.htaccess or your root .htaccess:

# --- SAFE PERMANENT BLOCK FOR ALL SITES ---
RedirectMatch 404 ^/xmlrpc\.php

3. Emergency Circuit Breaker for WordPress admin-ajax.php (TEMPORARY USE ONLY)

⚠️ CRITICAL WARNING FOR WORDPRESS SITES: Do NOT leave admin-ajax.php blocked permanently on a live WordPress site! admin-ajax.php is required for legitimate features like contact form submissions (Contact Form 7/WPForms), WooCommerce cart updates, live search, and post auto-saving.

Use this rule ONLY as an emergency 15-minute circuit breaker during an active server crash to drop CPU load while you block bad IP subnets in CSF. Remove it immediately after server load stabilizes!

# --- EMERGENCY TEMPORARY BLOCK ONLY (REMOVE AFTER LOAD DROPS) ---
RedirectMatch 403 ^/blog/wp-admin/admin-ajax\.php

4. Taming Live Chat / AJAX Polling Loops

Live support software (like Mibew Messenger) uses background AJAX polling (e.g., /mibew/operator/users/update) to check for new messages every 1–2 seconds.

• The Problem: Leaving an admin/operator panel open during an attack sends 1,800 database queries per hour per open tab. Under heavy server load, these queries lock up MySQL.
• The Solution: In your live chat admin settings, increase Operator Refresh Time from 2 seconds to 10 seconds. This reduces database polling load by 80%.

5. Managing Autodiscover & Email Client Overload

If your log analysis shows heavy traffic to /autodiscover/autodiscover.xml or /Microsoft-Server-ActiveSync, your server may be receiving automated mail-discovery probes, email-client requests, or excessive requests to these endpoints.

The Problem: Microsoft Outlook and other mail clients can make Autodiscover requests when configuring or troubleshooting an email account. At the same time, automated scanners and bots routinely probe /autodiscover/autodiscover.xml and other mail-related endpoints.

The Solution: If these requests are contributing significantly to server load, you can block the HTTP endpoints at the web-server level. This will not break existing email connections; it only means that new email setups must be configured manually.

Add this to your root .htaccess file:

# --- BLOCK AUTODISCOVER PROBES ---
RedirectMatch 404 ^/autodiscover/autodiscover\.xml$

# --- BLOCK ACTIVESYNC PROBES ---
RedirectMatch 404 ^/Microsoft-Server-ActiveSync(?:/.*)?$

Important: These rules only affect HTTP requests that reach the Apache virtual host where the .htaccess file is being applied. They do not disable cPanel’s Autodiscover functionality globally. cPanel can expose Autodiscover through a service subdomain such as autodiscover.example.com, so whether a particular request is handled by this .htaccess rule depends on which virtual host or service-subdomain path the request reaches.

Why do this?

• Potentially reduces request-processing overhead: Blocking unwanted HTTP requests early can reduce the amount of web-server or application processing associated with those requests. The actual benefit depends on how the server handles the endpoint and how much traffic it receives.
• Security by Obscurity: It hides your mail server configuration from automated scanners looking for vulnerabilities in your mail stack.
• Manual Setup: Users can still connect using their normal IMAP/SMTP settings if they configure their email accounts manually.

Can users just input mail settings manually?

Yes. Autodiscover is purely a convenience feature. If you block it:

1. Existing mail accounts already logged in will not stop working.
2. New users will simply get an “Auto-configuration failed” message.
3. They will then have to manually type in settings such as:
   • Incoming/Outgoing Server: mail.yourdomain.com
   • IMAP Port: 993 (SSL)
   • SMTP Port: 465 (SSL)

For most performance-focused sysadmins, the trade-off (slightly more manual setup vs. higher server stability) can be worth it, depending on how much traffic the endpoint is actually receiving and whether it is contributing meaningfully to load.

Note for WHM Users, if you want to disable this globally and permanently so it doesn’t even hit your logs: cPanel’s Autodiscover/AutoConfig support is controlled under WHM → Server Configuration → Tweak Settings → Domains → Thunderbird and Outlook autodiscover and autoconfig support (enables service subdomain and SRV record creation). Setting this option to Off prevents cPanel from creating the Autodiscover and AutoConfig service subdomains and their SRV records.

Do not disable Service subdomains just to disable Autodiscover. Service subdomains are used for other cPanel services as well.

cPanel notes that disabling Autodiscover/AutoConfig support can affect automatic configuration of email, calendars, or contacts. If you disable it, affected users may need to configure their accounts manually.

For a tutorial on how to add these rules to your site’s .htaccess files, or apply Apache rules globally, check out “How to Edit .htaccess on the Site Level or Add Rules Globally”.

Phase 5: Clearing MySQL Database Queue Locks

During a traffic flood, MySQL (mysqld) often gets stuck processing an enormous backlog of queued queries, consuming 100%+ CPU and gigabytes of RAM long after the network attack stops.

1. Check Active MySQL Queries:

mysqladmin proc status

• Healthy State: Threads: 2-5, queries finishing in Time: 0.
• Overloaded State: Dozens of threads stuck in Sending data, Updating, or Locked with Time > 30.

2. Flush the MySQL Memory Backlog:

If mysqld is taking 100%+ CPU in ps aux and has an uptime of several days, restart the database service to release trapped RAM (6GB+) and clear stuck thread locks:

systemctl restart mariadb || systemctl restart mysqld

Phase 6: Safe Automation & Googlebot Protection

Never risk blocking legitimate search engines (Googlebot, Bingbot).

1. Whitelist Googlebot, Bingbot in CSF

Add Google’s official domain masks to /etc/csf/csf.ignore:

.googlebot.com
.google.com
.search.msn.com

(CSF performs a reverse-DNS check and will NEVER block an IP originating from these hostnames).

For a tutorial on how to edit the csf.ignore file to whitelist these crawlers, read: “How to Whitelist Googlebot, Bingbot in CSF”

2. Verifying a Bot via Reverse DNS

To verify if an IP claiming to be Googlebot is real, run:

host 66.249.66.1
# Output MUST end in: .googlebot.com or .google.com

host crawl-66-249-66-1.googlebot.com
# Must match the original IP address

🛠️ Master Sysadmin Command Cheat Sheet

Table 1: Server Load & Network Traffic Diagnosis

Purpose	Exact Command	What to Look For
Check System Load	uptime	1-min, 5-min, 15-min load averages
Check Connections per IP	netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n | tail -n 20	IPs with high connection counts or subnets
Check Top CPU Processes	ps aux --sort=-%cpu | head -n 10	Find exact scripts/users consuming CPU
Check Real-Time CPU Idle	top -b -n 1 | grep "%Cpu"	Look at %id (Higher = More CPU free)
Find Missing Log Files	find /var/log -type f -name "*access*" -mtime -1	Active access log paths on server

Table 2: Web Server Log Analysis (Apache / cPanel)

Purpose	Exact Command	Description
Watch Traffic Live (Real-Time)	tail -f /var/log/apache2/access_log	Streams incoming web requests live as they hit the server
Find Target Domain	ls -lhS /var/log/apache2/domlogs/ | head -n 10	Identifies largest log file (domain being attacked)
Top Requested URLs (All Sites)	tail -q -n 10000 /var/log/apache2/domlogs/* 2>/dev/null | awk '{print $7}' | sort | uniq -c | sort -nr | head -n 15	Shows most attacked URLs across all domains
Top URLs (Specific Domain)	tail -n 10000 /var/log/apache2/domlogs/domain.com-ssl_log | awk '{print $7}' | sort | uniq -c | sort -nr | head -n 10	Shows top requested paths for one domain
Find IPs Hitting Specific URL	grep "/heavy-path" /var/log/apache2/domlogs/domain.com-ssl_log | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 10	Pinpoints exact IPs hitting a specific script
Extract User-Agent Strings	tail -n 10000 /var/log/apache2/access_log | awk -F'"' '{print $6}' | sort | uniq -c | sort -nr | head -n 10	Detects bad web crawler or scraper User-Agents

Table 3: Firewall & Subnet Blocking (CSF / IPTables / UFW)

Action	CSF Command	IPTables / UFW Alternative
Block Single IP	csf -d 1.2.3.4 "Attacker IP"	iptables -A INPUT -s 1.2.3.4 -j DROP
Block /24 Subnet	csf -d 47.79.13.0/24 "Botnet Subnet"	iptables -A INPUT -s 47.79.13.0/24 -j DROP
Search Block List	csf -g 66.249.	iptables -L -n | grep 66.249.
Remove Block	csf -dr 66.249.0.0/16	iptables -D INPUT -s 66.249.0.0/16 -j DROP
Restart Firewall	csf -r	systemctl restart iptables

Table 4: Database & Web Service Management

Action	Exact Command	Purpose
Check MySQL Status	mysqladmin proc status	Shows uptime, thread count, and slow queries
View Active SQL Queries	mysqladmin processlist	Shows running SQL queries and execution time
Restart Web Server	systemctl restart httpd	Clears stuck Apache/PHP worker queues
Restart Database	systemctl restart mariadb || systemctl restart mysqld	Releases trapped RAM (6GB+) & locked SQL threads

---

Frequently Asked Questions (FAQ)

Q: I am getting an error, pv: command not found. Is the command to count the request rate wrong: tail -f /var/log/apache2/domlogs/example.com | pv -l -r > /dev/null ?

The command itself is not wrong – in fact, it is a very clever way to monitor your real-time log velocity (which effectively tells you how many requests per second your Apache server is handling).

The error you are seeing simply means the Pipe Viewer (pv) utility is not installed on your server. It is not included by default on most Linux distributions.

To fix the error, you just need to install pv using your system’s package manager.

For Ubuntu / Debian:
sudo apt update
sudo apt install pv

For CentOS / RHEL / AlmaLinux / Rocky Linux: (Note: You may need to enable the EPEL repository first)
sudo dnf install epel-release
sudo dnf install pv

For macOS (if you are testing locally):
brew install pv

Once pv is installed, your command will work exactly as intended:

1. tail -f ... streams the live log file.
2. | pv -l -r takes that stream and measures the rate (-r) in lines (-l) per second, printing the speed to your terminal.
3. > /dev/null throws away the actual log text so your screen doesn’t get spammed with log entries, leaving only the pv speed meter visible.

Q: Is xmlrpc.php blocked by default in WordPress?

No, surprisingly it is NOT blocked by default. Fresh WordPress installations leave xmlrpc.php active out-of-the-box for backward compatibility with mobile apps and Jetpack.

Because it is open by default, hackers target every WordPress site’s xmlrpc.php file using a feature called system.multicall, which allows an attacker to guess 500 passwords in a single HTTP request. Adding RedirectMatch 404 ^/xmlrpc\.php to .htaccess is one of the most effective manual security hardening steps you can take.

Q: Can I permanently block admin-ajax.php on WordPress sites, or will it break features?

Do NOT permanently block admin-ajax.php on production WordPress sites.

admin-ajax.php is heavily used by legitimate plugins for contact form submissions (Contact Form 7/WPForms), WooCommerce cart updates, live search, and post auto-saving. Blocking it permanently will break those features for real visitors. Use RedirectMatch 403 ^/blog/wp-admin/admin-ajax\.php ONLY as a temporary emergency circuit breaker during an active flood attack.

Q: What if my server load keeps increasing and then decreasing (fluctuating)? Will it stabilize?

Yes, it will stabilize. Load fluctuations during recovery are completely normal and happen due to four reasons:

1. Rolling Averages (uptime): The 15-minute load average takes up to 15 minutes to decay and reflect recent fixes.
2. Background Cron Jobs: Automated system tasks (cPanel backups, log rotations, SSL checks, or antivirus scanners like Imunify360/Acronis) run on scheduled intervals and cause brief 10–30 second load bumps.
3. Cache Expiration Cycles: When cached pages expire, brief CPU spikes occur while fresh pages are generated.
4. AJAX Polling Waves: Open browser tabs (like admin dashboards) poll the server periodically.

How to verify you are safe: Run top -b -n 1 | grep "%Cpu". As long as your CPU idle rate (%id) stays high (80%+), these temporary load bumps are harmless and will smooth out.

Q: Why shouldn’t I use mod_evasive for Apache rate limiting?

mod_evasive is an outdated module from the early 2000s. Modern web applications load dozens of parallel assets and AJAX requests per page load. mod_evasive frequently mistakes legitimate human browsing or mobile CGNAT networks for DDoS attacks, returning 403 Forbidden errors to real users. Use Nginx rate-limiting, Cloudflare, or application-level caching instead.

Q: What is the ideal CT_LIMIT setting in CSF?

If using CSF Connection Tracking (CT_LIMIT), set it between 150 and 300, restrict it strictly to web ports (CT_PORTS = “80,443”), and use a temporary block time (CT_BLOCK_TIME = “300”). Setting it lower will block legitimate users on corporate networks or mobile carrier CGNATs.

Q: Does page caching break dynamic websites?

No. Modern caching (like LiteSpeed Cache, Nginx FastCGI, or Cloudflare) serves static HTML to unauthenticated guests and bots while automatically bypassing the cache for logged-in users, shopping carts, or active dynamic sessions.

Q: How do I know if an attack is at the network level or database level?

Run top -c. If httpd or php-cgi processes dominate CPU, the attack is overloading your web server. If mysqld dominates CPU (over 100%), stuck database queries are locking up memory. Restarting MySQL releases these locks immediately: systemctl restart mysqld

---

Case Study: Resolving a 300+ Server Load Emergency on a Multi-Site Production Server

Background

• Server Environment: WHM / cPanel on Linux (Apache + Oracle MySQL 8.0).
• Workload: Hosting multiple distinct web applications, including custom PHP CMS frameworks, active community forums, and WordPress blogs.
• Incident: Server load suddenly escalated from a healthy baseline to 300+, causing severe site unresponsiveness and high CPU utilization.

Incident Timeline & Root-Cause Breakdown

Phase 1: Initial Investigation (Distributed Botnet Detection)

Running netstat and ps aux revealed that the initial spike was not caused by a single rogue visitor, but rather a distributed Layer 7 (HTTP) attack across two distinct vectors:

1. Rotating Botnet Subnets: Hundreds of unique IPs opening 1 connection each, heavily concentrated in specific hosting subnets (43.119.100.0/24 and 47.79.13.0/24).
2. Fake Exploit Scanners: Automated scripts bombarding non-WordPress sites with requests for /wp-login.php, /wp-content/plugins/, and web shell backdoors like /file5.php. Even though these pages returned 404s, Apache was invoking PHP to process every single fake request.

Phase 2: Log Isolation & Targeted Firewall Rules

By sorting the domain access logs (domlogs) by file size using ls -lhS /var/log/apache2/domlogs/, the sysadmin identified the exact domains taking the brunt of the traffic.

• Action Taken:
  1. The abusive /24 hosting subnets were banned at the firewall level in ConfigServer Security & Firewall (csf -d 47.79.13.0/24).
  2. A lightweight .htaccess rule was placed at Line 1 of the primary site to intercept fake WordPress exploit scans instantly: ApacheRedirectMatch 404 ^/wp- RedirectMatch 404 ^/file[0-9]\.php

Phase 3: Uncovering Hidden Application-Level Bottlenecks

After blocking the initial botnet subnets, load fluctuated between 30 and 125. Further analysis using ps aux --sort=-%cpu and domlog inspect tools (tail -q -n 5000 /var/log/apache2/domlogs/*) uncovered three hidden application-level bottlenecks:

1. WordPress Comment Spam (wp-comments-post.php): Bots were submitting hundreds of automated comment spam payloads directly to a WordPress blog endpoint, forcing MySQL to execute heavy spam-table checks on every request.
2. WordPress AJAX Endpoint Floods (admin-ajax.php): Unauthenticated bot traffic was hammering admin-ajax.php, forcing the server to boot up full WordPress core in memory on every hit.
3. Database Queue Locks (mysqld): Oracle MySQL 8.0 (mysqld) was consuming 239% CPU and 6.8 GB of RAM, stuck processing an accumulated queue of database query locks generated during the initial traffic wave.

Phase 4: Resolution & Recovery

The administrator applied targeted .htaccess rules to block comment spam and AJAX floods at the web server edge:

# Block Comment Spam & WordPress AJAX Attacks
RedirectMatch 403 ^/blog/wp-comments-post\.php
RedirectMatch 403 ^/blog/wp-admin/admin-ajax\.php

Because Apache evaluates .htaccess in real-time, the moment these rules were saved to disk, Apache began dropping incoming comment spam and AJAX requests in 0.0001 seconds using 0% PHP/Database CPU.

Within minutes of cutting off the incoming request wave, MySQL completed its queued queries, released trapped thread locks, and the server load collapsed from 300+ down to a stable 5.0.

---

Key Takeaways & Lessons Learned

1. .htaccess Edge Blocking Saves PHP/MySQL: Returning a 404 or 403 at the Apache layer prevents PHP scripts from launching, keeping CPU usage near zero even under heavy request volume.
2. Identify Database Bottlenecks Early: When server load stays high despite network connections dropping, check ps aux for mysqld. Accumulated query queue locks will hold CPU load high until queries complete or the service is restarted.
3. Audit Background AJAX Polling: Live chat systems and operator dashboards perform AJAX polling every 1–2 seconds. Increasing operator console refresh intervals from 2 seconds to 10 seconds reduces background database load by 80%.
4. Target Subnets, Not Single IPs: In distributed attacks, individual IP blocking is ineffective. Identify common /24 CIDR blocks in netstat and drop the entire range via CSF or iptables.

---

Looking for a website builder that is flexible enough for a developer but easy enough for a beginner? Learn more about UltimateWB! We also offer web design packages if you would like your website designed and built for you.

Got a techy/website question? Whether it’s about UltimateWB or another website builder, web hosting, or other aspects of websites, just send in your question in the “Ask David!” form. We will email you when the answer is posted on the UltimateWB “Ask David!” section.