How bad the Bot hits are on the Forum

[Chris]

Getting back to topic, I have seen noticeable traffic drops at many other forums I have frequented over the years. I was reviewing data for two, and this problem isn't unique for AARoads.

Many European country-specific road forums became inactive over the past 10 years, due to the shift of internet usage to mobile devices and social media apps. This predated AI.

Another factor is the significant increase in web hosting cost, especially with the AI scraping. Website owners pay far higher cost while getting increasingly little traffic. It's killing the internet as we knew it.

Out of all the forums I used to frequent, AAroads is one of the few that's still pretty active. The Highways & Autobahns forum on Skyscrapercity is also pretty active, but it tends to be more focused on Europe. Many regional SSC forums have seen much reduced activity in recent years, some are almost dead.

[bandit957]

This is why antitrust enforcement needs to be stepped up.

[Max Rockatansky]

This is why antitrust enforcement needs to be stepped up.

How does that relate to bots spiking the forum?

[LilianaUwU]

Blaming Alex for that is a leap too far.

I didn't take it as such, but can understand the frustration and disdain for the company involved.

And it wasn't targeted at you either. Sorry if anyone took it as such. But yeah, it blows that the best company for the job has a slew of controversies.

[Scott5114]

I suppose an option we could keep in our back pocket if Cloudflare doesn't work out long term is Anubis on both sides instead.

[Jim]

Is the cloudflare check taking more time for others than it does for me?  It just occasionally redirects to a "checking if you are human" page and before I can even read the whole page it's usually already sending me right back to the forum.  Nearly non-existent disruption on my end.

[Scott5114]

Is the cloudflare check taking more time for others than it does for me?  It just occasionally redirects to a "checking if you are human" page and before I can even read the whole page it's usually already sending me right back to the forum.  Nearly non-existent disruption on my end.

I have to manually click a checkbox every time. My forum usage pattern is to read a few threads, go do something else for 30 min-1 hour, come back, read a few threads, etc. Pretty much every time I come back I have to click the box again.

I'm not particularly fussed by it, because I'd rather be clicking boxes every hour than not have a forum or wiki, but I can see how it would drive people mad.

[vdeane]

Is the cloudflare check taking more time for others than it does for me?  It just occasionally redirects to a "checking if you are human" page and before I can even read the whole page it's usually already sending me right back to the forum.  Nearly non-existent disruption on my end.

If I'm signing in on my lunch break, it's a lot like that, but if I'm coming in from home, it's spinning for long enough to read the whole page, and then I have to click the box.  I guess my home computer is locked down enough that it looks kinda like a bot.  Or maybe they just don't like Linux.

So yeah... definitely better than having no forum, but still a minor annoyance.

At least this post confirmed that replies can be posted through a verification check.

[Thing 342]

You can set it to where Scott can approve new members

The last time I was in charge of anything having to do with that, I somehow managed to ban the entire East Coast.

Banning the IP range 0.*.*.* through 63.*.*.* is never the answer.

On the brighter side, the forum at least gets IPv6 support now.

[1995hoo]

Is the cloudflare check taking more time for others than it does for me?  It just occasionally redirects to a "checking if you are human" page and before I can even read the whole page it's usually already sending me right back to the forum.  Nearly non-existent disruption on my end.

I had to check a box once yesterday, but otherwise it just reloads, usually fairly quickly. I'd far rather deal with this process than with clicking all the boxes that contain crosswalks or the like.

[PColumbus73]

Would the Wikipedia model of asking for donations / crowdfunding be applicable?

[kphoger]

But yeah, it blows that the best company for the job has a slew of controversies.

Most every company these days ends up in some sort of controversy, it seems.  I stopped paying attention long ago.  If a company provides a service that I find valuable, then good for them—even if I don't personally agree with the stands they take on this issue or that, and even if they donate money to causes I don't support.  If I pull up to a farmer selling strawberries from the back of his pickup at a highway junction, I don't make him fill out a questionnaire about his political, social, religious, and financial associations before buying the strawberries from him;  same with buying a box of screws from a hardware store, a shirt from a department store, or a meal at a restaurant—or, in this case, a cybersecurity product from a CDN company.

I had to check a box once yesterday, but otherwise it just reloads, usually fairly quickly. I'd far rather deal with this process than with clicking all the boxes that contain crosswalks or the like.

Same here.  I rarely have to actually click the checkbox.  Most times, it just lets me in after a few seconds.

[Jim]

I got my first checkbox to prove I'm human tonight.  Even so, it couldn't have delayed viewing the forum page I was loading by more than 2 or 3 seconds. 

[Plutonic Panda]

Yeah it doesn't seem like something I can't get used to especially if that means keeping the form up. I'll gladly check the box every hour or so.

[Alex]

For as much of a pain or annoyance (count me in on that) that the "verifying you are human" confirmation that Cloudflare implements, I am very happy to report that the number of hits on the server for '/forum' dropped from 1,327,557 last week to 16,930 just now. Additionally the bot entries for Mozilla/5 dropped from 1,574,687 last week to 46,304 just now.

[wriddle082]

For as much of a pain or annoyance (count me in on that) that the "verifying you are human" confirmation that Cloudflare implements, I am very happy to report that the number of hits on the server for '/forum' dropped from 1,327,557 last week to 16,930 just now. Additionally the bot entries for Mozilla/5 dropped from 1,574,687 last week to 46,304 just now.

I don't think it's a pain at all.  If it stops or drastically reduces the bot hits, I'll gladly put up with it.

Thank you, Alex, for all that you are doing to keep the forum up and stable!

[hbelkins]

I've been getting a number of 502 Bad Gateway errors lately.

[kphoger]

For as much of a pain or annoyance (count me in on that) that the "verifying you are human" confirmation that Cloudflare implements, I am very happy to report that the number of hits on the server for '/forum' dropped from 1,327,557 last week to 16,930 just now. Additionally the bot entries for Mozilla/5 dropped from 1,574,687 last week to 46,304 just now.

That's a huge difference!

I've been getting a number of 502 Bad Gateway errors lately.

I've been getting a never-ending 'Loading' banner occasionally.  It appears to happen when I go to preview my post but the site has decided in the meantime that I might not be human.  Getting unstuck just requires a refresh on the browser tab.

I had a similar problem a day or two ago when I clicked on the "likes" icon, but the site had decided in the meantime that I might not be human.  Instead of popping up a 'Loading' banner, it instead told me that I was banned from posting in that particular thread.  Because it was an old locked thread that I was viewing, it didn't immediately occur to me that that's what was going on.

Weird little quirks, but no big deal.

[hotdogPi]

I've never gotten an outright refusal in any form, but I do have to click the checkbox whenever it pops up (about every 10 minutes).

I've also gotten "banned", and the ban was for the entire forum, not just the one thread.

[kphoger]

I also occasionally get a never-ending "Fetching preview", because apparently the site decided in the meantime that I might not be human.

Again, I just do a Ctrl+C of whatever I had typed up, then refresh my browser.

[Roadgeekteen]

This forum randomly refused to load once, but most of the time it works. I do most of the time have to click the button.

[ZLoth]

FYI.... https://www.cloudflarestatus.com/incidents/8gmgl950y3h7

[kphoger]

Let me just say that it feels really nice to know that a team of paid professionals was working on this fix, rather than Alex beating his head against the desk for free.

[vdeane]

This morning is a good example of why half the internet being reliant on one company is a bad thing.  "Let's do a quick forum skim... oh deah, Cloudflare is down.  Maybe read some fanfic... oh, AO3 also uses Cloudflare.  But the story I'm reading is also on FFN... which also uses Cloudflare".

[PColumbus73]

This morning is a good example of why half the internet being reliant on one company is a bad thing.  "Let's do a quick forum skim... oh deah, Cloudflare is down.  Maybe read some fanfic... oh, AO3 also uses Cloudflare.  But the story I'm reading is also on FFN... which also uses Cloudflare".

One day Cloudflare and CloudStrike will join forces and send humanity back to the 1980s.