Bot blocker not working for Apache/2.4.29 (Ubuntu) #139

contactr2m · 2019-12-19T21:17:16Z

Server version: Apache/2.4.29 (Ubuntu)
I tried following your given steps for Apache2.4 but when i try to test my site using curl, it seems bot blocker is not working.

Curl shows 301 may be because i have http redirect to https ?

curl -A "80legs" http://example.com
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://example.com/">here</a>.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at example.com Port 80</address>

But i i use curl -A "80legs" https://example.com
then entire page is loaded, instead of 403 ?

/etc/apache2# cat sites-enabled/000-default.conf

UseCanonicalName On

<VirtualHost *:80>
        ServerAdmin webmaster@localhost
        
        ServerName example.com
        ServerAlias www.example.com
        
        DocumentRoot /var/www/html

        <Directory /var/www/html/>
            Options FollowSymLinks
            AllowOverride All
            # Include /etc/apache2/custom.d/globalblacklist.conf
            Include custom.d/globalblacklist.conf
            Require all denied
        </Directory>

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
RewriteEngine on
RewriteCond %{SERVER_NAME} =example.com [OR]
RewriteCond %{SERVER_NAME} =www.example.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

sites-enabled/000-default-le-ssl.conf

<IfModule mod_ssl.c>
<VirtualHost *:443>
        ServerAdmin webmaster@localhost
        
        ServerName example.com
        ServerAlias www.example.com
        
        DocumentRoot /var/www/html

        <Directory /var/www/html/>
            Options FollowSymLinks
            AllowOverride All
            # Include /etc/apache2/custom.d/globalblacklist.conf
            Include custom.d/globalblacklist.conf
            Require all denied
        </Directory>

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
</VirtualHost>
</IfModule>

The text was updated successfully, but these errors were encountered:

ZerooCool · 2020-02-25T16:23:49Z

I have the same, with :
Require all denied

If i add
Include custom.d/globalblacklist.conf

At the begin or at the end end of
<Directory /var/www/server-ip>
Require all denied

Then, my 403 doesn't work and i can see index.html

My Vhost : https://wiki.visionduweb.fr/index.php?title=VirtualHosts_des_domaines_enregistr%C3%A9s#139.99.173.195_.C3.A9coute_du_port_HTTP_80

VirtualHosts des domaines enregistrés — Analyse Développement Hacking

cafn · 2020-05-03T19:47:04Z

No response. I too am having the same problems. I just installed, actually updated to the newest version as of this date, and I can not get the test responses to reply as they are supposed to. My responses:

curl -A "googlebot" https://mydomain.com
Should respond with 200 OK, but instead I get no response on http://, https:// loads the page.

curl -A "80legs" https://mydomain.com
curl -A "masscan" https://mydomain.com
Should respond with 403 Forbidden, but instead I get no response on http://, https:// loads the page.

curl -I https://mydomain.com -e http://100dollars-seo.com
curl -I https://mydomain.com -e http://zx6.ru
Should respond with 403 Forbidden, but I get "HTTP/2 200" back on both http:// and https:// . The connection is made.

Granted the URL is forwarded to https:// so I did not expect to get a reply back there, but I did expect to get the expected results when using https:// also. This did not happen.
I installed the previous version about 18 months ago and it worked fine when installed, but I had not checked it since. I just re-downloaded the new files as per the instructions and left the vhost include statement as it was. This does not appear to be working, for me and I would really like for it to work. I have read the instructions a couple of additional times and made sure all files were in place.

Anybody got any ideas?

ZerooCool · 2020-05-05T17:10:51Z

To my log, i have the same for AspiegelBot
Should respond with 403 Forbidden (Or 302 redirect?), but I get "200".

curl -A "80legs" https://www.visionduweb.fr
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="https://www.visionduweb.fr/403-forbidden.php">here</a>.</p>
</body></html>


curl -I https://www.visionduweb.fr -e http://100dollars-seo.com
HTTP/1.1 302 Found
Date: Tue, 05 May 2020 17:04:17 GMT
Server: Apache
Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
X-Frame-Options: SAMEORIGIN
Referrer-Policy: no-referrer-when-downgrade
Feature-Policy: geolocation none;midi none;notifications none;push none;sync-xhr self;microphone none;camera none;magnetometer none;gyroscope none;speaker self;vibrate none;fullscreen self;payment none;
Location: https://www.visionduweb.fr/403-forbidden.php
Cache-Control: max-age=604800
Expires: Tue, 12 May 2020 17:04:17 GMT
Content-Type: text/html; charset=iso-8859-1


curl -A "AspiegelBot" https://www.visionduweb.fr
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="https://www.visionduweb.fr/403-forbidden.php">here</a>.</p>
</body></html>


curl -A "AspiegelBot" -L https://www.visionduweb.fr
<!doctype html>
<html lang="fr">
<head>
<meta charset="utf-8"/>
</head>
<body>
<h1 style="text-align:center;">403 - Forbidden</h1>
<h2 style="text-align:center;">L'administrateur du serveur n'autorise pas la consultation de cette page.</h2>
<p style="text-align:center;"><img src="/images/structure/403-forbidden.jpg"/></p>
<p>Si vous pensez qu'il s'agit d'une erreur, merci de contacter l'administrateur en lui signifiant l'adresse URL de provenance.</p>
<p>Merci de transmettre des informations sur cette erreur, ainsi que la date et l'heure.</p>
<p>Le mail de l'administrateur : [email protected]</p>
<p><a href="https://www.visionduweb.fr">Retour sur la page d'accueil</a></p>
</body>
</html>

I have installed Bad Bot Blocker with this method : https://wiki.visionduweb.fr/index.php?title=Configurer_le_fichier_.htaccess#Bloquer_des_Bots_et_des_URL_ind.C3.A9sirables_avec_Bad_Bot_Blocker

My VirtualHost : https://wiki.visionduweb.fr/index.php?title=VirtualHosts_des_domaines_enregistr%C3%A9s

ctlui · 2020-05-15T17:32:11Z

I was getting same of you, so I try with v2.2 instructions and it's ok now.
About "AspiegelBot", he's not on the globalblacklist.conf as "zx6.ru"

ZerooCool · 2020-05-16T01:49:37Z

I have add AspiegelBot for my conf ;)

Where is the v2.2 instructions ? But, we have Apache 2.4.

For my last test, for AspiegelBot, i use the -L option for curl, then i can make the same with 80legs :
Strange, if i use curl -A .. -L .. the redirection work good.

The curl option -L is the good answer for you and me.

curl -A "80legs" -L https://www.visionduweb.fr
<!doctype html>
<html lang="fr">
<head>
<meta charset="utf-8"/>
</head>
<body>

<h1 style="text-align:center;">403 - Forbidden</h1>
<h2 style="text-align:center;">L'administrateur du serveur n'autorise pas la consultation de cette page.</h2>

<p style="text-align:center;"><img src="/images/structure/403-forbidden.jpg"/></p>

<p>Si vous pensez qu'il s'agit d'une erreur, merci de contacter l'administrateur en lui signifiant l'adresse URL de provenance.</p>
<p>Merci de transmettre des informations sur cette erreur, ainsi que la date et l'heure.</p>
<p>Le mail de l'administrateur : [email protected]</p>

<p><a href="https://www.visionduweb.fr">Retour sur la page d'accueil</a></p>
</body>
</html>

dchmelik · 2020-08-04T08:52:09Z

Not just an Ubuntu problem, but general Apache 2.4 problem; at least on other POSIX-based OSs (such as stable/up-to-date Slackware GNU/Linux 14.2 w/Apache 2.4.43the same problem: setup instructions' tests for bad bots result in Apache serving pages fine (with normal 200 code.)

mitchellkrogza · 2020-08-04T09:02:45Z

The blocker does work and is tested across 2.2 up to 2.4.23 - see the build log and tests for yourself
https://travis-ci.org/github/mitchellkrogza/apache-ultimate-bad-bot-blocker

If you mess up your Apache 2.4 permission structure in any way at all higher up the chain you break everything below it including the blocker.

Travis CI - Test and Deploy Your Code with Confidence
Travis CI enables your team to test and ship your apps with confidence. Easily sync your projects with Travis CI and you'll be testing your code in minutes.

dchmelik · 2020-08-07T09:08:19Z

The blocker does work and is tested across 2.2 up to 2.4.23 - see the build log and tests for yourself [...]

Fine, but I stated same problem except w/more server-focused OSes: any such setup/test logs?

If you mess up your Apache 2.4 permission structure [...]

Unclear to me what that means for Apache... can anyone suggest documentation or elaborate how to debug?

mitchellkrogza · 2020-08-07T10:10:58Z

Start off by comparing your apache2.conf with the version used in tests - https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/blob/master/.dev-tools/_test_results/_conf_files_2.4/apache2.conf

Specifically these blocks

https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/blob/master/.dev-tools/_test_results/_conf_files_2.4/apache2.conf#L159-L180

GitHub
mitchellkrogza/apache-ultimate-bad-bot-blocker
Apache Block Bad Bots, (Referer) Spam Referrer Blocker, Vulnerability Scanners, Malware, Adware, Ransomware, Malicious Sites, Wordpress Theme Detectors and Fail2Ban Jail for Repeat Offenders - mitc...

mitchellkrogza · 2020-08-07T13:05:24Z

Then also make sure your vhost config follows the same configuration for its main directory block
https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/blob/master/.dev-tools/_test_results/_conf_files_2.4/testsite.conf

GitHub
mitchellkrogza/apache-ultimate-bad-bot-blocker
Apache Block Bad Bots, (Referer) Spam Referrer Blocker, Vulnerability Scanners, Malware, Adware, Ransomware, Malicious Sites, Wordpress Theme Detectors and Fail2Ban Jail for Repeat Offenders - mitc...

skonesam · 2022-08-26T15:26:37Z

The links above don't work, but found the tests anyway. Only (significant?) change I see is RewriteEngine On is in the test but not the instructions. It doesn't seem to change the outcome.

amitash · 2023-02-06T15:25:46Z

I had the same issue but I was whitelisting my own IP, so everything was allowed. Testing from another machine worked well.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bot blocker not working for Apache/2.4.29 (Ubuntu) #139

Bot blocker not working for Apache/2.4.29 (Ubuntu) #139

contactr2m commented Dec 19, 2019

ZerooCool commented Feb 25, 2020 •

edited by unfurl-links bot

cafn commented May 3, 2020 •

edited

ZerooCool commented May 5, 2020 •

edited

ctlui commented May 15, 2020 •

edited

ZerooCool commented May 16, 2020 •

edited

dchmelik commented Aug 4, 2020

mitchellkrogza commented Aug 4, 2020 •

edited by unfurl-links bot

dchmelik commented Aug 7, 2020 •

edited

mitchellkrogza commented Aug 7, 2020 •

edited

mitchellkrogza commented Aug 7, 2020 •

edited by unfurl-links bot

skonesam commented Aug 26, 2022

amitash commented Feb 6, 2023

Bot blocker not working for Apache/2.4.29 (Ubuntu) #139

Bot blocker not working for Apache/2.4.29 (Ubuntu) #139

Comments

contactr2m commented Dec 19, 2019

ZerooCool commented Feb 25, 2020 • edited by unfurl-links bot

cafn commented May 3, 2020 • edited

ZerooCool commented May 5, 2020 • edited

ctlui commented May 15, 2020 • edited

ZerooCool commented May 16, 2020 • edited

dchmelik commented Aug 4, 2020

mitchellkrogza commented Aug 4, 2020 • edited by unfurl-links bot

dchmelik commented Aug 7, 2020 • edited

mitchellkrogza commented Aug 7, 2020 • edited

mitchellkrogza commented Aug 7, 2020 • edited by unfurl-links bot

skonesam commented Aug 26, 2022

amitash commented Feb 6, 2023

ZerooCool commented Feb 25, 2020 •

edited by unfurl-links bot

cafn commented May 3, 2020 •

edited

ZerooCool commented May 5, 2020 •

edited

ctlui commented May 15, 2020 •

edited

ZerooCool commented May 16, 2020 •

edited

mitchellkrogza commented Aug 4, 2020 •

edited by unfurl-links bot

dchmelik commented Aug 7, 2020 •

edited

mitchellkrogza commented Aug 7, 2020 •

edited

mitchellkrogza commented Aug 7, 2020 •

edited by unfurl-links bot