I found that my VPS is spending a lot of resources to serve the pages for the bots like Amazonbot, Bytespider, ClaudeBot, GPTBot, CCBot, SemrushBot, Baiduspider and YandexBot. Such bots are not important for me so I decided to block them by the user agent..

First of all I blocked them in my nginx configuration, so the webserver will respond them 444 error code (No response).

To do this you need to add regexp matching before your “location” definition in the “server” section of your nginx config, eg

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    ....
    # Ban bots
    if ($http_user_agent ~* (Amazonbot|Bytespider|ClaudeBot|GPTBot|CCBot|SemrushBot|Baiduspider|YandexBot)) {
        return 444;
    }

    # Locations
    location / {
        try_files $uri $uri/ /index.php$is_args$args;
    }
    ... 

Then test your config and restart Nginx

# nginx -t
# nginx -s reload

After this you will see in the access.log that the bots are blocked with the 444 status

116.179.32.71 - - [24/Mar/2026:12:58:12 +0200] "GET /?_r=577266341 HTTP/2.0" 444 0 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
47.128.127.106 - - [24/Mar/2026:12:58:12 +0200] "GET /?_r=3425415784 HTTP/2.0" 444 0 "-" "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)"
52.203.65.83 - - [24/Mar/2026:12:58:13 +0200] "GET /?_r=1208229002 HTTP/1.1" 444 0 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36"

It’s already enough, but I made a bit more – I installed fail2ban and configured it to drop all connections on firewall lelvel for one hour for all the IPs who get this 444 response. This will decrease a bit more the load for Nginx.

# apt install fail2ban

Create the filter and configure regexp

• /etc/fail2ban/filter.d/nginx-access-log.conf

[Definition]
failregex = ^<HOST> -.*"(?:GET|POST|HEAD) .* HTTP/.*" 444 0 .*
ignoreregex =

Test it with your access.log

# fail2ban-regex /var/www/logs/nginx-access.log /etc/fail2ban/filter.d/nginx-access-log.conf

Append the filter to the end of the jail.conf file

• /etc/fail2ban/jail.conf

[nginx-access-log]
enabled  = true
port     = http,https
filter   = nginx-access-log
logpath  = /var/www/logs/nginx-access.log
# The duration in seconds for the rate limit period (e.g., 300 = 5 min)
findtime = 300
# Max requests per IP before banning
maxretry = 3
# Ban time in seconds
bantime  = 3600

So if the bot is doing his attempts again and again despite getting 444 error, we just block it via firewall for about 1 hour.

Restart fail2ban

# systemctl restart fail2ban

Check fail2ban logs

# tail -n 10 /var/log/fail2ban.log 
2026-03-24 13:19:57,933 fail2ban.filter         [533222]: INFO    [nginx-access-log] Found 50.16.72.185 - 2026-03-24 13:19:57
2026-03-24 13:20:02,161 fail2ban.filter         [533222]: INFO    [nginx-access-log] Found 100.28.118.16 - 2026-03-24 13:20:01
2026-03-24 13:20:02,368 fail2ban.filter         [533222]: INFO    [nginx-access-log] Found 44.206.65.8 - 2026-03-24 13:20:02
2026-03-24 13:20:02,373 fail2ban.filter         [533222]: INFO    [nginx-access-log] Found 220.181.108.102 - 2026-03-24 13:20:02
2026-03-24 13:20:07,611 fail2ban.filter         [533222]: INFO    [nginx-access-log] Found 52.200.251.20 - 2026-03-24 13:20:07
2026-03-24 13:20:11,443 fail2ban.filter         [533222]: INFO    [nginx-access-log] Found 220.181.108.114 - 2026-03-24 13:20:11
2026-03-24 13:20:11,475 fail2ban.actions        [533222]: NOTICE  [nginx-access-log] Ban 220.181.108.114
2026-03-24 13:20:12,455 fail2ban.filter         [533222]: INFO    [nginx-access-log] Found 52.54.95.127 - 2026-03-24 13:20:12
2026-03-24 13:20:12,455 fail2ban.filter         [533222]: INFO    [nginx-access-log] Found 52.54.95.127 - 2026-03-24 13:20:12
2026-03-24 13:20:12,698 fail2ban.actions        [533222]: NOTICE  [nginx-access-log] Ban 52.54.95.127

Check banned IPs

# iptables -L -n
Chain f2b-nginx-access-log (1 references)
target     prot opt source               destination         
REJECT     all  --  35.169.102.85        0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  3.218.35.239         0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  54.156.124.2         0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  34.239.85.139        0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  52.3.104.214         0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  54.197.114.76        0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  98.83.10.183         0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  34.238.45.183        0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  54.225.98.148        0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  52.73.6.26           0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  54.221.203.24        0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  98.82.40.168         0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  44.218.170.184       0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  44.205.180.155       0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  54.89.90.224         0.0.0.0/0            reject-with icmp-port-unreachable