It's to a point where every night the forum is essentially DDOSed. What should be done to counter this?
Quote from: LilianaUwU on May 05, 2025, 10:52:22 AMIt's to a point where every night the forum is essentially DDOSed. What should be done to counter this?
I've noticed that too. I keep getting "this site is not secure" errors pretty much every night and every morning.
I haven't received an error accessing this site in a few weeks. Interesting that I haven't been affected like some others have.
Quote from: SectorZ on May 05, 2025, 01:04:11 PMI haven't received an error accessing this site in a few weeks. Interesting that I haven't been affected like some others have.
Happened to me briefly early this morning.
Oddly, I got an error on my iPad this morning, but it worked fine on my PC a few minutes later.
I was thinking the same thing last night when I got home and my wife had twelve people over.
Quote from: Max Rockatansky on May 05, 2025, 03:51:19 PMI was thinking the same thing last night when I got home and my wife had twelve people over.
You're making me remember an old joke:
A wealthy but very rednecky man marries a hot young wife. The first day after the marriage, he says, "OK, look, I want to make sure you understand the rules right off. Every day I get home from work at 5:00 PM. I want the TV to be tuned to ESPN, I want my cold beer waiting, and I want to sit like a lump until you have dinner ready at 6:00."
His wife looks at him and says, "That's fine. And I want you to understand my day. I will have sex in this house every afternoon at 4:00 whether you're here or not."
I think the "guests" are bots scraping the site for stuff to train "AI" models on. Apparently those just straight-up ignore robots.txt on the server, and don't try to avoid overloading servers. I know the forum I help run and SABRE have also been having some issues with the same things lately. I think on my end, I just banned a bunch of IPs until the number of "guests" got to a reasonable level while I think SABRE's implemented more robust anti-bot measures.
I'm getting a completely different error. When I try to access the forum directly, I get redirected to a Plesk site, via https://forum.aaroads.com/login_up.php?success_redirect_url=%2Findex.php When I go to the aaroads.com main page, its link to the forum takes me here. What gives?
I don't know if the Plesk site is a new and improved version of the forum, a bogus phishing site, or malware. I'm not trying it out until I'm sure it's safe.
All I get is a blank screen when I try to click on a link, and sometimes the site even looks funny, as in the normal design isn't there, but instead, it looks more like a poorly-formatted Word document.
Just got an "offline" message about ten minutes ago -- said the site was down.
The code for the forum is depreciating and the downtime on the server and AARoads website is a result of GET requests executing too slow. A snip of the error log just now shows continuous problems:
[06-May-2025 07:31:17] WARNING: [pool aaroads.com] child 1966, script '/var/www/vhosts/aaroads.com/httpdocs/forum/index.php' (request: "GET /forum/index.php?PHPSESSID=6049a912ca767dfb97152ad3c7f8f774&topic=11718.msg2653033") executing too slow (1.066481 sec), logging
[06-May-2025 07:31:17] WARNING: [pool aaroads.com] child 1929, script '/var/www/vhosts/aaroads.com/httpdocs/forum/index.php' (request: "GET /forum/index.php?PHPSESSID=df4020088fabc59cea256ba595c36904&topic=99.msg2903310") executing too slow (1.050513 sec), logging
[06-May-2025 07:31:17] WARNING: [pool aaroads.com] child 1967, script '/var/www/vhosts/aaroads.com/httpdocs/forum/index.php' (request: "GET /forum/index.php?PHPSESSID=9243f639c20e1223d42ac858e8095c99&action=printpage&topic=7920.0") executing too slow (1.307300 sec), logging
[06-May-2025 07:31:18] WARNING: [pool aaroads.com] child 1876, script '/var/www/vhosts/aaroads.com/httpdocs/forum/index.php' (request: "GET /forum/index.php?PHPSESSID=ad8238489b8b8ffcbed8d0400e616855;topic=32158.msg2770981") executing too slow (1.330181 sec), logging
I enabled a slow log showing which PHP transactions are problematic and 95% of them are from the forum index.php script. Here's a snip from that:
[06-May-2025 07:37:57] [pool aaroads.com] pid 2834997
script_filename = /var/www/vhosts/aaroads.com/httpdocs/forum/index.php
[0x000075f42ca13c30] shell_exec() /var/www/vhosts/aaroads.com/httpdocs/forum/Sources/Subs.php:5270
[0x000075f42ca13b50] host_from_ip() /var/www/vhosts/aaroads.com/httpdocs/forum/Sources/Security.php:191
[0x000075f42ca13940] is_not_banned() /var/www/vhosts/aaroads.com/httpdocs/forum/index.php:114
[0x000075f42ca13120] smf_main() /var/www/vhosts/aaroads.com/httpdocs/forum/index.php:90
[06-May-2025 07:37:59] [pool aaroads.com] pid 2835999
script_filename = /var/www/vhosts/aaroads.com/httpdocs/forum/index.php
[0x000075f42ca13c30] shell_exec() /var/www/vhosts/aaroads.com/httpdocs/forum/Sources/Subs.php:5270
[0x000075f42ca13b50] host_from_ip() /var/www/vhosts/aaroads.com/httpdocs/forum/Sources/Security.php:191
[0x000075f42ca13940] is_not_banned() /var/www/vhosts/aaroads.com/httpdocs/forum/index.php:114
[0x000075f42ca13120] smf_main() /var/www/vhosts/aaroads.com/httpdocs/forum/index.php:90
[06-May-2025 07:38:00] [pool aaroads.com] pid 2834998
script_filename = /var/www/vhosts/aaroads.com/httpdocs/forum/index.php
[0x000075f42ca13c30] shell_exec() /var/www/vhosts/aaroads.com/httpdocs/forum/Sources/Subs.php:5270
[0x000075f42ca13b50] host_from_ip() /var/www/vhosts/aaroads.com/httpdocs/forum/Sources/Security.php:191
[0x000075f42ca13940] is_not_banned() /var/www/vhosts/aaroads.com/httpdocs/forum/index.php:114
[0x000075f42ca13120] smf_main() /var/www/vhosts/aaroads.com/httpdocs/forum/index.php:90
I have tried implementing the recommended level of caching, forcing PHP to downgrade to 7.4 (which is what the latest SMF software is compatible with), working with AI to try to address broken code, etc. By no means is this my area of expertise, and the level of frustration and time invested into this is wearing on me.
The Plesk issues are partly because I have not finalized settings, such as 404. I'm nearly complete with my object-oriented overhaul of the AARoads back end, which has been the priority of my focus for the last 4 months. I want to get through this, but the forum downtime on the server has been problematic with both script debugging and data entry. Also as a result, I have the server rebooting daily at 2 AM now, as when you get the "this site is offline" message, that seems to be the one solution that works.
Today, the record for most people online was DOUBLED from 48k to 95k. The forum is very slow, and it threw me a database error once.
We're just too friendly and welcoming.
BOTs are flooding the forum with GET requests. 32 in the last 21 seconds. I have consulted AI to try to find a solution, but found nothing. This is also affecting the rest of the server, and I may resort to taking the Forum offline again unless someone can provide me with guidance on how to address this.
Quote from: Rothman on May 08, 2025, 04:31:53 PMWe're just too friendly and welcoming.
I think from previous conversation that is actually part of the problem. Non-registered guests can view much of the forum. Problem is that it is causing chaos because of all the bots.
Might be high time to make the forum viewable to only registered users. I haven't seen anyone propose a viable solution and this bot stuff keeps happening.
Quote from: Max Rockatansky on May 08, 2025, 04:43:18 PMQuote from: Rothman on May 08, 2025, 04:31:53 PMWe're just too friendly and welcoming.
I think from previous conversation that is actually part of the problem. Non-registered guests can view much of the forum. Problem is that it is causing chaos because of all the bots.
Might be high time to make the forum viewable to only registered users. I haven't seen anyone propose a viable solution and this bot stuff keeps happening.
That also makes it very difficult to join this forum.
Quote from: formulanone on May 08, 2025, 04:49:00 PMQuote from: Max Rockatansky on May 08, 2025, 04:43:18 PMQuote from: Rothman on May 08, 2025, 04:31:53 PMWe're just too friendly and welcoming.
I think from previous conversation that is actually part of the problem. Non-registered guests can view much of the forum. Problem is that it is causing chaos because of all the bots.
Might be high time to make the forum viewable to only registered users. I haven't seen anyone propose a viable solution and this bot stuff keeps happening.
That also makes it very difficult to join this forum.
Time and other social media platforms seem to be chipping away on the demand for forums such as this anyways. If I had to choose between forum death by bots or increasing entropy the latter is far more appealing.
Quote from: Max Rockatansky on May 08, 2025, 04:43:18 PMMight be high time to make the forum viewable to only registered users. I haven't seen anyone propose a viable solution and this bot stuff keeps happening.
I've BEEN suggesting exactly that.
Quote from: Max Rockatansky on May 08, 2025, 04:43:18 PMQuote from: Rothman on May 08, 2025, 04:31:53 PMWe're just too friendly and welcoming.
I think from previous conversation that is actually part of the problem. Non-registered guests can view much of the forum. Problem is that it is causing chaos because of all the bots.
Might be high time to make the forum viewable to only registered users. I haven't seen anyone propose a viable solution and this bot stuff keeps happening.
I tried changing the settings to make the forum viewable to only registered users. Didn't stop the GET requests at all.
Quote from: Alex on May 08, 2025, 05:20:57 PMQuote from: Max Rockatansky on May 08, 2025, 04:43:18 PMQuote from: Rothman on May 08, 2025, 04:31:53 PMWe're just too friendly and welcoming.
I think from previous conversation that is actually part of the problem. Non-registered guests can view much of the forum. Problem is that it is causing chaos because of all the bots.
Might be high time to make the forum viewable to only registered users. I haven't seen anyone propose a viable solution and this bot stuff keeps happening.
I tried changing the settings to make the forum viewable to only registered users. Didn't stop the GET requests at all.
Ah, that is an element I wasn't aware of.
I think I am about done hosting/running the Forum...
(https://www.aaroads.com/forum_images/bots.jpg) (https://www.aaroads.com/forum_images/bots.jpg)
I'm looking for similar ideas to fight this on a much smaller scale for the TM forum and the TM site in general. I added some more robots.txt files but the problematic bots are unlikely to honor them. I'm thinking about whether to put on a site-wide apache username/password that I'd have to share somehow with legitimate users without having it get brought into the automated requests by the bots.
IIRC, there are services out there that should help hinder/limit the number of requests from bots/scrapers?
Quote from: freebrickproductions on May 08, 2025, 05:43:17 PMIIRC, there are services out there that should help hinder/limit the number of requests from bots/scrapers?
I implemented code AI gave me with an .httaccess file earlier today. Didn't reduce the requests.
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (bot|crawler|spider|scraper) [NC]
RewriteRule ^ - [F]
Quote from: freebrickproductions on May 08, 2025, 05:43:17 PMIIRC, there are services out there that should help hinder/limit the number of requests from bots/scrapers?
I know Cloudflare has a DDOS protection thing that's basically a captcha.
I saw a mention of feeding the bots nonsense instead of the legitimate pages: https://marcusb.org/hacks/quixotic.html
Also something about responding to them with HTTP code 429 instead of real pages.
Quote from: LilianaUwU on May 08, 2025, 06:04:05 PMQuote from: freebrickproductions on May 08, 2025, 05:43:17 PMIIRC, there are services out there that should help hinder/limit the number of requests from bots/scrapers?
I know Cloudflare has a DDOS protection thing that's basically a captcha.
Cloudflare caused the redirect loop. It might still be worth attempting, however, so long as all existing redirects are disabled for the attempt. From what I read, it seemed like the issue was because the existing redirects would trigger the Cloudflare redirect, which would trigger the existing redirect, etc.
I don't know if you have access to the network stack, but the only way I solved this on my own sites was IP-blocking a lot of networks. For several days I blocked the whole of 47.*.*.* until the site calmed down, and then cut the netblock down to the remnant segments that kept popping up. It's not enough to block them at the web server - you have to block them at the firewall or IP filter level.
The problem with feeding them nonsense is they will simply download the nonsense. Half of these are from Chinese startups anyway. They don't know if they're getting Shakespeare or sh... well, you know.
Quote from: Alex on May 08, 2025, 04:36:16 PMI have consulted AI to try to find a solution, but found nothing.
The conspiratorially-minded would be quick to point out that the bots are there in the first place because they're feeding the forum's contents to the AI, so of course the AI wouldn't tell you how to stop it...
Quote from: ClassicHasClass on May 08, 2025, 09:17:44 PMThe problem with feeding them nonsense is they will simply download the nonsense. Half of these are from Chinese startups anyway. They don't know if they're getting Shakespeare or sh... well, you know.
The goal of feeding them nonsense would be that it would make the "throw a million bots at the problem" approach ultimately give them a garbage result, so they are motivated to slow down to something more like the pace of a search engine crawler. But that is a long-term approach that doesn't solve the problem of traffic coming in so fast and furious that it causes the server to keel over.
I don't know the details of your hosting arrangements, but usually 'get scraping' requires some kind of rule that limits the number of requests per minute or per hour by IP or IP block.
These are usually implemented in the load balancer for your web services provider, or a policy enforcement appliance your host provider may use.
If you are being hosted in a basic services site, where it is up to the web admin to sort it out at the host which is the target of the requests, you could use a reverse proxy.
You can install nginx or apache, set up a reverse proxy on your web server, and setup rules.
It will add some overhead, but cant be as bad as dealing with the scraping issue.
Just some thoughts.
Quote from: ClassicHasClass on May 08, 2025, 09:17:44 PMThe problem with feeding them nonsense is they will simply download the nonsense. Half of these are from Chinese startups anyway. They don't know if they're getting Shakespeare or sh... well, you know.
Wouldn't it still at least keep the forum database from getting hammered since they wouldn't get the real content? And quite frankly, if they're essentially DDOSing the sites they crawl, they deserve to have their AI ruined by garbage data.
I was under the impression that the bots aren't actually causing any issues. We've had 90000 guests at times in the last few days, but the issues with the forum randomly going down and search not working unless you do a hyper-specific advanced search are caused by a PHP version mismatch, not the bots.
For example, I'm seeing 82,187 guests now and no slowdowns.
Quote from: vdeane on May 09, 2025, 12:43:17 PMQuote from: ClassicHasClass on May 08, 2025, 09:17:44 PMThe problem with feeding them nonsense is they will simply download the nonsense. Half of these are from Chinese startups anyway. They don't know if they're getting Shakespeare or sh... well, you know.
Wouldn't it still at least keep the forum database from getting hammered since they wouldn't get the real content? And quite frankly, if they're essentially DDOSing the sites they crawl, they deserve to have their AI ruined by garbage data.
It would presumably keep mysqld from getting hammered, but httpd would still have to serve the requests, and it would still use a portion of the server's bandwidth.
Quote from: hotdogPi on May 09, 2025, 12:49:03 PMI was under the impression that the bots aren't actually causing any issues. We've had 90000 guests at times in the last few days, but the issues with the forum randomly going down and search not working unless you do a hyper-specific advanced search are caused by a PHP version mismatch, not the bots.
For example, I'm seeing 82,187 guests now and no slowdowns.
Alex can correct me if I'm wrong, but I believe the forum is randomly going down because it's running out of memory due to all of the requests.
Quote from: Scott5114 on May 09, 2025, 09:59:46 PMQuote from: vdeane on May 09, 2025, 12:43:17 PMQuote from: ClassicHasClass on May 08, 2025, 09:17:44 PMThe problem with feeding them nonsense is they will simply download the nonsense. Half of these are from Chinese startups anyway. They don't know if they're getting Shakespeare or sh... well, you know.
Wouldn't it still at least keep the forum database from getting hammered since they wouldn't get the real content? And quite frankly, if they're essentially DDOSing the sites they crawl, they deserve to have their AI ruined by garbage data.
It would presumably keep mysqld from getting hammered, but httpd would still have to serve the requests, and it would still use a portion of the server's bandwidth.
Quote from: hotdogPi on May 09, 2025, 12:49:03 PMI was under the impression that the bots aren't actually causing any issues. We've had 90000 guests at times in the last few days, but the issues with the forum randomly going down and search not working unless you do a hyper-specific advanced search are caused by a PHP version mismatch, not the bots.
For example, I'm seeing 82,187 guests now and no slowdowns.
Alex can correct me if I'm wrong, but I believe the forum is randomly going down because it's running out of memory due to all of the requests.
That is exactly what the problem is, the RAM is being tapped out. You can see where the server restarts in the memory usage graph.
(https://www.aaroads.com/forum_images/ram-usage.jpg)
The rest of the site certainly is not drawing this kind of traffic, especially considering that there have been no content updates for months and that Wordpress is increasingly making the browsing experience a test of patience.
Quote from: ClassicHasClass on May 08, 2025, 09:17:44 PMI don't know if you have access to the network stack, but the only way I solved this on my own sites was IP-blocking a lot of networks. For several days I blocked the whole of 47.*.*.* until the site calmed down, and then cut the netblock down to the remnant segments that kept popping up. It's not enough to block them at the web server - you have to block them at the firewall or IP filter level.
The problem with feeding them nonsense is they will simply download the nonsense. Half of these are from Chinese startups anyway. They don't know if they're getting Shakespeare or sh... well, you know.
What I have done on the AARoads Wiki side is feed suspicious IPs into https://ipinfo.io/. If it comes up "hosting" or "cloud", I block the entire range on the firewall. It usually takes care of the problem, at least for a few days.
New problem: Users aren't able to log in. I got a report from another user here on another forum that he wasn't able to login. When I attempted to do so on another computer, an error "Sorry Guest, you are banned from using this forum!" pops up instead of the login screen. I realize this may be part of the DDoS mitigation measures, but maybe they are TOO good.
Quote from: NJRoadfan on May 11, 2025, 10:10:33 PMNew problem: Users aren't able to log in. I got a report from another user here on another forum that he wasn't able to login. When I attempted to do so on another computer, an error "Sorry Guest, you are banned from using this forum!" pops up instead of the login screen. I realize this may be part of the DDoS mitigation measures, but maybe they are TOO good.
Maybe they were banned...
Sounds like their IP address shifted to a range that had formerly been used by spammers or a banned user.
Without knowing the IP address (or even the username) of the person affected, though, I wouldn't know what to unban.
It may be a general issue. I clicked "Log In" from my phone, and could not even attempt to log in before seeing the banned message.
This is because one of the htaccess rules I placed yesterday to mitigate the 100,000 plus BOT GET requests included the login page
RewriteCond %{QUERY_STRING} (action=reminder|action=login|action=register|action=printpage|action=profile|action=search) [NC]
I'll remove action=login from this and see if the flood gates open again.
And to show that the bot GET requests are not stopping, this error again appears from SMF:
UPDATE smf_log_activity
SET
hits = 1,
most_on = 88168
WHERE date = '2025-05-12'
which is because that column of the SQL database table is set to smallINT(5), which even with UNSIGNED as an attribute is capped at 65535. The fact that the number needs to exceed 88,168 is indicative of how much bad attention this form is garnering.
To that end I have tried a number of efforts to block these requests and otherwise mitigate them. Went to SMF forums and tried out .htaccess code from developers there. Manually edited some of the SMF scripts directly to remove guest email access options, such as restricting the RemindMe() function to only members. Activated the firewall through Plesk, set up Monitoring via Grafana, increased Fail2Ban rules for blocking IP addresses, etc.
I confirmed that the htaccess restrictions I enabled yesterday were working, as the number of slow.log entries from the forum decreased dramatically. Nonetheless, the bots are still somewhat hammering the Forum, as the memory usage is still somewhat high.
(https://www.aaroads.com/forum_images/ram2.jpg)
Keep in mind that I am not a software developer, and a lot of what I am trying I am either learning on the fly or relying upon AI and SMF forum posts for advice.
Unfortunately if this keeps up, I will be forced to take the Forum down as the increasing memory usage of the GET requests results in slow loading scripts, time-outs, gateway errors, and "this site is offline" messages. Been trying to finish this 4-month long reprogramming of the back-end of AARoads, and lately the Forum issues have consistently taking me away from completing the debugging and data entry for that.
Can't complain. It's been a good run.
We have a solution in place on the wiki that we're testing out. (Go check out https://wiki.aaroads.com/wiki/Chickasaw_Turnpike or something to see it in action—you'll have to look quick though because every time I've seen it it just flashes on the screen for a second or so.) If that quells the bot flood over there, then we can probably use the same thing on the forum. Last I heard the CPU on the wiki server was holding steady at 3% or so, so that's a good sign.
I hope this solution works, but if it doesn't, would simply not allowing any new registrant work?
Quote from: SEWIGuy on May 12, 2025, 09:29:31 AMI hope this solution works, but if it doesn't, would simply not allowing any new registrant work?
The problem isn't registrants, it's that AI companies are having 101,235 computers (actual number, not an exaggeration) try to load the site at the same time. The server, understandably, cannot cope with that. Traditionally, you could put together a file called
robots.txt that would essentially serve as a gentleman's agreement for how bots should interact with the site to prevent this from happening, but the AI companies just blatantly disrespect the rules outlined in that. I guess the $$$$ they're theoretically making (not actually making, mind you) is more important to them than whether the website they're ingesting is able to stay online while they're doing it.
This is not just a problem for our site, either—it's happening Internet-wide.
Quote from: Scott5114 on May 11, 2025, 10:28:25 PMSounds like their IP address shifted to a range that had formerly been used by spammers or a banned user.
Without knowing the IP address (or even the username) of the person affected, though, I wouldn't know what to unban.
It was me. Haven't been banned yet. :bigass:
There was an article in the Register recently about the IETF coming up with a different take on robots.txt to address AI crawlers, but that just seems like rearranging the deck chairs while the ship gets bombarded. I guess we'll find out in August. https://www.theregister.com/2025/04/09/ietf_ai_preferences_working_group/
I hate that this is happening, especially the problem from yesterday falsely informing us that we were banned from the forum (myself included). That being said, I hope that the admins find a solution that will keep the site running smoothly, because I'd hate to lose the biggest part of my roadgeeking life forever.
It's fucked that there is no definitive solution short of burning down the data centers that host the bots.
(Somehow this originally posted in the Buc-ee's thread below even though I had clicked on this one?)
Makes me wonder what I'd save from the forum. Truth be told, I'm not so sure if anything.
Quote from: Rothman on May 12, 2025, 10:07:51 PMMakes me wonder what I'd save from the forum. Truth be told, I'm not so sure if anything.
I'd say Alanland, but we did that already (https://quindaropedia.fandom.com/wiki/Quindaropedia_Wiki).
Quote from: Rothman on May 12, 2025, 10:07:51 PMMakes me wonder what I'd save from the forum. Truth be told, I'm not so sure if anything.
Anything dealing with that one night at band camp.
I'd imagine there's a lot of info on the forum that hasn't (yet) been ported over to the wiki or other websites yet. Plus, I generally find it easier to search on this forum than within groups on Facebook.
I seem to recall claims that all the bots are from Hong Kong. Is there a way to block all of Hong Kong from making GET requests to the forum?
Are we back?
Seems that the 404 errors Alex was talking about off forum got resolved.
Good to have the forum back, hope the bots are being kept at bay successfully.
Quote from: Henry on May 12, 2025, 03:45:24 PMfalsely informing us that we were banned from the forum
When I'm using an incognito window and get that message before even getting to the login page, I know it's not to be taken literally.
Quote from: Jim on May 16, 2025, 12:26:36 PMGood to have the forum back, hope the bots are being kept at bay successfully.
I'm still seeing functionality issues, but I don't know if they're part of the problem or part of the fix. Quoting multiple posts at once like this, for example, took some creativity on my part, presumably because of the multiple requests it required.
I am occasionally encountering a 502 error, but it comes back to normal on the next try.
Quote from: Big John on May 16, 2025, 02:41:40 PMI am occasionally encountering a 502 error, but it comes back to normal on the next try.
I'm getting that error if I click a link in the "Recent Posts" list, but then if I go into the appropriate subforum and open the thread everything displays correctly.
Edited to add: I also just got that error when I clicked the "New" button next to a thread.
Quote from: Big John on May 16, 2025, 02:41:40 PMI am occasionally encountering a 502 error, but it comes back to normal on the next try.
I think it's still getting banged on. :(