How bad the Bot hits are on the Forum

[Alex]

A real time clip of the nginx access logs for requests specifically for "/forum"

A sampling of count hits specifically for "/forum" over a time interval returned 1,327,557

A query of the proxy_access_ssl_log for requests coming from bots returned insane numbers:

1574687 "Mozilla/5.0
18035 "Googlebot-Image/1.0"
12896 "Mediapartners-Google"
8918 "Sogou
7567 "meta-externalagent/1.1
2038 "Mozilla/4.0 1468 "ChatGPT/1.2025.294
1242 "python-requests/2.32.3"
422 "Uptime-Kuma/2.0.0-beta.4"
374 "AliyunSecBot/Aliyun

That's not actual site visitors... Per Google Search Console, the entire website is lucky if it even accrues 1,000 web search clicks per day.
Overall AARoads traffic has been abysmal for most of this year and is only 30% of what it was during Summer 2024. More often these days I find myself saying "why bother?"

Having spent another 4 hours trying out various code with nginx directives, Fail2Ban, ModSecurity, etc. with no results, I am now leveraging Cloudflare proxying to deal with Bot-fest. Unlike my last attempt at using Cloudflare, this time I addressed the infinite loop problem that occurred before. The settings in place now are aggressive, but hopefully will provide a noticeable drop in the bogus traffic and site downtime.

[kphoger]

We really do appreciate your tireless attention to fixing things when they go wrong.  The information superhighway sure does have its Breezewoods, doesn't it?

[Max Rockatansky]

I guess that explains why I'm suddenly being occasionally verified as "not a robot" today.

[LilianaUwU]

I guess that explains why I'm suddenly being occasionally verified as "not a robot" today.

Little did they know I'm secretly a robot girl...

[Plutonic Panda]

Alex, I can tell you I really enjoy this forum and would be pretty bummed if it went away. That said I'm sure it is a lot of work keeping it going and I sure appreciate it!

[hotdogPi]

Blocking Mozilla 5.0 will cut down 98% of the bots (and no legitimate users because the current version is in the triple digits).

[Rothman]

Thanks, Alex.

[thspfc]

Overall AARoads traffic has been abysmal for most of this year and is only 30% of what it was during Summer 2024. More often these days I find myself saying "why bother?"

I have noticed the decline in activity as well, but would never have guessed 70%.

My activity here has dropped from almost 1300 posts in 2021, to 900 in 2022, 550 in 2023, and 250 last year. I think there just aren't as many interesting threads as there used to be.

[kphoger]

I guess that explains why I'm suddenly being occasionally verified as "not a robot" today.

It's a minor thing that bothers me when a website tells me it needs to confirm I'm a human, but it's a minor thing that pleases me when it does confirm it without my having to click every tile with a bus in it.

[1995hoo]

If I were in a location closer to you, Alex, I would buy you a beer or dinner. Like others, I greatly appreciate your efforts.

[gonealookin]

Pausing that access log stream, I guess we know now where Claude is getting its information about roads.

And checking a few of the IP addresses, there seem to be an awful lot of hits from China (no surprise) and Brazil (maybe more surprising) compared to the number of forum members from those places.

[kphoger]

If I were in a location closer to you, Alex, I would buy you a beer or dinner. Like others, I greatly appreciate your efforts.

I'm pretty sure one of us could, if we wanted, order a dinner for delivery to his house.  Beer, it probably depends on state law.  Of course, half the reason of buying someone dinner or a beer is to personally share it with him...

[kphoger]

And checking a few of the IP addresses, there seem to be an awful lot of hits from China (no surprise) and Brazil (maybe more surprising) compared to the number of forum members from those places.

Approximately 30% of all cyber-attacks originating in Latin America come from Brazil.

[CovalenceSTU]

Blocking Mozilla 5.0 will cut down 98% of the bots (and no legitimate users because the current version is in the triple digits).

User agent strings (as shown in the access logs) have a lengthy, messy history causing almost every browser and bot to first identify itself as "Mozilla/5.0" before anything else. (For example, the user agent for Chrome on Windows is "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36".)

Seeing "Mozilla/4.0" make the list is a surprise though, as the last legitimate browser (afaik) to use it was IE8 in 2009.

[GaryV]

I am so grateful that this forum was able to confirm I am human, without any intervention on my part. Sometimes I've begun to wonder ...

[wanderer2575]

I am so grateful that this forum was able to confirm I am human, without any intervention on my part. Sometimes I've begun to wonder ...

Me too.  Sometimes it's the nicest thing anyone says to me all day.

[Max Rockatansky]

I am so grateful that this forum was able to confirm I am human, without any intervention on my part. Sometimes I've begun to wonder ...

Me too.  Sometimes it's the nicest thing anyone says to me all day.

Sure would have resolved much of the plot of Alien: Earth early on.

[kphoger]

I blame all our site's technical problems on these three forum members:

tadams — joined in OCT 2012 — 42 days 10 hours total time online — 0 posts — last online today

EricLewisAHTD — joined in JUN 2015 — 84 days 1 hour total time online — 0 posts — last online today

DarkLynx108 — joined in JUL 2015 — 27 days 21 hours total time online — 0 posts — last online today

[seicer]

Site metrics will only get worse as search engines scrape data for their AI-powered results. My sites have been slowly declining in metrics for about a year now. Coupled with people getting information on social media (less on Facebook and more on short-form reel-focused platforms), forums and group-like sites are increasingly seen as irrelevant. So many of the Facebook groups I belong to on niche topics are all but dead, and even the ones with impressive user counts have very poor engagement. Part of that is Facebook (as an example) prioritizing traffic to AI-slop groups.

I really miss the earlier iterations of the internet.

[hbelkins]

I don't post to the forum from anything other than a computer.

However, I do get email notifications whenever someone quotes one of my posts, and I will occasionally click the links in the Gmail app on my phone to read what the response was.

I noticed several Cloudflare verification popups yesterday and wondered what was up, as I'd never seen those before.

[Molandfreak]

I blame all our site's technical problems on these three forum members:

tadams — joined in OCT 2012 — 42 days 10 hours total time online — 0 posts — last online today

EricLewisAHTD — joined in JUN 2015 — 84 days 1 hour total time online — 0 posts — last online today

DarkLynx108 — joined in JUL 2015 — 27 days 21 hours total time online — 0 posts — last online today

@Stone027 Total time spent Online: 112 days, 5 hours and 31 minutes, 0 posts.

[ZLoth]

For whatever it's worth, this is what I'm seeing from my IP address for my reverse proxy for the calendar month of October, 2025.




Keep in mind the following...

• This is from my reverse proxy service to access services hosted on my home server.
• I'm using a fully qualified domain name. However, the root level domain is redirecting to Amazon.
• The DNS records are hosted at Cloudflare, and I have it set so that http requests are redirected to their https versions, and I'm using Let's Encrypt certificates.
• With an exception of a subdomain that I'm using here for image hosting, I'm using GeoIP to block visitors who are not from the United States

Yet, I'm getting visitors from around the world.

[Scott5114]

Would it be possible to tweak the Cloudflare settings to increase the time between re-verification? I tend to view a few threads, go do something else in another tab for a little while, and come back, but the verification triggers every time I do that. Maybe set it to something around two hours?

[rschen7754]

Would it be possible to tweak the Cloudflare settings to increase the time between re-verification? I tend to view a few threads, go do something else in another tab for a little while, and come back, but the verification triggers every time I do that. Maybe set it to something around two hours?

I would suggest even longer on the wiki - I keep multiple AARW pinned tabs open on multiple machines. I haven't yet had a scenario where the re-verification happens as I try to save an edit, but am concerned what might happen when that occurs.

[vdeane]

Would it be possible to tweak the Cloudflare settings to increase the time between re-verification? I tend to view a few threads, go do something else in another tab for a little while, and come back, but the verification triggers every time I do that. Maybe set it to something around two hours?

Even without going to do something else, I had it re-verify no less than three times in my evening forum check today.  Usually I find out I need to do so when something breaks or when I try to queue multiple tabs and have them go "just a moment" all at once.