Some time ago, I wrote a post about blocking bots by their User Agent: Protect your website from the annoying bots. Today, I faced a new problem. Someone attacked one of my old services using thousands of random IP addresses. They make only 1-3 requests from a single IP and then change it. I decided to create a simple cookie trap. Here is how I did it..

Nginx configuration

Put this before regular “location /” and location for scripts handling

    # Protect /api from bots
    location ^~ /api {
        if ($cookie_ckt != "y") {
            access_log /var/www/site.com/logs/nginx-access-cookie-trap.log;
            add_header Set-Cookie "ckt=y; Path=/; Max-Age=3600";
            return 302 $scheme://$host$request_uri;
        }

        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_index index.php;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param QUERY_STRING    $query_string;
        fastcgi_pass unix:/run/php/php-fpm.sock;
    }

In this step, we check the cookie. If it doesn’t exist, we set it and redirect the visitor to the same page. This works fine for real visitors using browsers. However, most bots do not handle redirects, so they get stuck here.

But be careful, because Google and Bing bots (indexers) will be caught (filtered) as well. In my case, this is OK. However, in other cases, you need to exclude them using additional conditions. For example, you can create a good_bots map and use extra “if” statements.

I also save the logs to separate files. This way, I can use fail2ban to block these IPs if I find more than 3 redirects within one minute.

fail2ban configuration

Create fail2ban filter: /etc/fail2ban/filter.d/nginx-access-log-cookie-trap.conf

[Definition]
failregex = ^<HOST> - - .* "GET /api.*" 302
ignoreregex =

Append filter to the jail.conf: /etc/fail2ban/jail.conf

...
[nginx-access-log-cookie-trap]
enabled = true
port    = http,https
filter  = nginx-access-log-cookie-trap
logpath = /var/www/site.com/logs/nginx-access-cookie-trap.log
# If a bot triggers 3 redirects within 60 seconds without resolving the cookie, it will be blocked.
maxretry = 3
findtime = 60

Restart fail2ban

# systemctl restart fail2ban

Monitoring

You can check the logs in

• /var/www/site.com/logs/nginx-access-cookie-trap.log
• /var/log/fail2ban.log

Or monitor related chain via iptables

# iptables -L -n