Server Downtime - May 13th-16th, 2025

[vdeane]

Country flags are gone too, though I think a lot of us stopped using them by this point.  Also, changing your profile details apparently nukes your avatar, with no way to get it back unless you have external hosting.

EDIT: Managed to find it in the gallery listing.  Also, it's hilarious that there's still a spot for Skype usernames with Skype no longer being a thing.

[cahwyguy]

Just a note to say THANK YOU to Alex for all his hard work on the forum. The Pacific Southwest forum is one I depend upon as a primary source of information for my pages, as well as a primary place to promote when I do California Highway Headline posts, posts of updates to my pages, and posts about the California Highways podcast. It has always served as a relatively high quality source of information.

I know the incredible amount of work that goes into maintain all of this, and I just want to let @Alex know that I appreciate it.

The one suggestion I might make is to create a read-only, non-forum page off AARoads (e.g. .../forumstatus.html) that just provides a status update on the forum for downtimes like this. The AARoads FB page, which would be a logical place for that information, isn't a good source.

Thank you again.

[Molandfreak]

I miss the like button 

Like

See, this is why it's handy. If someone posts something thoughtful or interesting that I have no further comment on, I like using it since we can no longer just post an emoticon in response to someone's post.

[Alex]

Just a note to say THANK YOU to Alex for all his hard work on the forum. The Pacific Southwest forum is one I depend upon as a primary source of information for my pages, as well as a primary place to promote when I do California Highway Headline posts, posts of updates to my pages, and posts about the California Highways podcast. It has always served as a relatively high quality source of information.

I know the incredible amount of work that goes into maintain all of this, and I just want to let @Alex know that I appreciate it.

The one suggestion I might make is to create a read-only, non-forum page off AARoads (e.g. .../forumstatus.html) that just provides a status update on the forum for downtimes like this. The AARoads FB page, which would be a logical place for that information, isn't a good source.

Thank you again.

Much appreciated Dan. I like your idea and will add that as an addition to my nearly complete site reprogramming.

I miss the like button 

Like

See, this is why it's handy. If someone posts something thoughtful or interesting that I have no further comment on, I like using it since we can no longer just post an emoticon in response to someone's post.

I don't disagree with you on this. It is that during the course of my debugging and code adjustments, the Likes script was part of a recursion error that generated incessantly:

Code Select Expand

2022 | 1747489305 |      6594 | D
▒g  | ?topic=18020.1750             | 2: Unknown: Cannot call session save handler in a recursive manner | 9bf262bec0956a41b08875ca9536d734 | general    | Unknown |    0 | [{"function":"smf_error_handler"}] |
|     2021 | 1747488610 |         0 | (M▒▒   | ?topic=10906.msg261197        | 2: Unknown: Cannot call session save handler in a recursive manner | 959f589879e9afad53248ff3c71675f9 | general    | Unknown |    0 | [{"function":"smf_error_handler"}] |
|     2020 | 1747487382 |         0 | 4▒▒▒    | ?topic=22572.msg2317834       | 2: Unknown: Cannot call session save handler in a recursive manner | f3e7f1bfb28018d2e3335073d4650b24 | general    | Unknown |    0 | [{"function":"smf_error_handler"}] |
|     2019 | 1747487209 |         0 | ▒˾▒    | ?topic=12138.0                | 2: Unknown: Cannot call session save handler in a recursive manner | 9a69812234fb96c3683991378bf0ce6d | general    | Unknown |    0 | [{"function":"smf_error_handler"}] |
|     2018 | 1747487208 |         0 | ▒˾▒    | ?topic=12138.0                | 2: Unknown: Cannot call session save handler in a recursive manner | 7c13142ee7c9d82c8af59de1aef4a8c5 | general    | Unknown |    0 | [{"function":"smf_error_handler"}] |
|     2017 | 1747487208 |         0 | ▒˾▒    | ?topic=12138.0                | 2: Unknown: Cannot call session save handler in a recursive manner | a26d2231f1f3f6808fb37bd747c8c851 | general    | Unknown |    0 | [{"function":"smf_error_handler"}] |
|     2016 | 1747487198 |         0 | 4▒▒▒    | ?topic=1224.msg2979417        | 2: Unknown: Cannot call session save handler in a recursive manner | b85130c98773c858c7f1a81cd84a7fce | general    | Unknown |    0 | [{"function":"smf_error_handler"}] |
|     2015 | 1747487102 |         0 | B▒D  | ?topic=11590.0;prev_next=prev | 2: Unknown: Cannot call session save handler in a recursive manner | e1bc74829e96601cb5c14bc013d45ed7 | general    | Unknown |    0 | [{"function":"smf_error_handler"}] |
|     2014 | 1747486515 |         0 | 4▒▒▒    | ?topic=22799.msg2541607       | 2: Unknown: Cannot call session save handler in a recursive manner | 793979d9cc5cb5d17f792a2caa87b584 | general    | Unknown |    0 | [{"function":"smf_error_handler"}] |
|     2013 | 1747486371 |         0 | 3▒▒   | ?topic=1492.msg39701          | 2: Unknown: Cannot call session save handler in a recursive manner | fa88003ecde62d7c793a8481b5f76c26 | general    | Unknown |    0 | [{"function":"smf_error_handler"}] |

Error Log (09:55:02 and later)
Stack traces show two errors related to checkSession() in Security.php, called from Likes.php:147:

Code Select Expand

[0x00007d2bd8013340] fatal_lang_error() /var/www/vhosts/aaroads.com/httpdocs/forum/Sources/Errors.php:237
[0x00007d2bd80131c0] checkSession() /var/www/vhosts/aaroads.com/httpdocs/forum/Sources/Security.php:710
[0x00007d2bd8013130] call() /var/www/vhosts/aaroads.com/httpdocs/forum/Sources/Likes.php:147

The stack trace points to Likes.php:147, where checkSession() is called during a "Like" action. SMF's checkSession() function (in Security.php) ensures a valid session before processing user actions. If an AJAX request (e.g., liking a post) triggers a session save while another session operation is active, PHP throws the recursion error.
This can happen if:
Multiple AJAX requests fire simultaneously (e.g., a user rapidly clicks "Like").
SMF's session handler is misconfigured or interfered with by another process.

Immediate Fix:
Modify Likes.php to avoid recursive session calls:

I attempted some code modifcation to address this, but the problem continued. I eventually decided to just deactivate Likes so I could move forward with getting the Forum to a stable state.

Session Recursion Error Resolution
Deactivating "Likes" (via Admin > Posts and Topics > Likes) was a smart move. The stack traces in the slow log (e.g., Likes.php:147 triggering checkSession()) confirm the error was tied to that feature. Since no new errors appear in the logs after 09:47:46, the issue is likely resolved.

I believe James was who activated the Likes aspect of SMF, so maybe he can restore it when he has some time.

[Alex]

The one suggestion I might make is to create a read-only, non-forum page off AARoads (e.g. .../forumstatus.html) that just provides a status update on the forum for downtimes like this. The AARoads FB page, which would be a logical place for that information, isn't a good source.

I have a script ready, and it will debut once I finish the AARoads backend reprogramming. It will query the SMF database and output users online and the most recent five posts in addition to a status message regarding if any SMF errors are currently logged. I'll adjust the text above it when forum downtime will occur of if I have to take it offline to address an ongoing problem with bots and the like.

I set up a cron job so that this will not be an actual live look at the forum, but instead be a snapshot generated every 5 minutes. This way the script is not actively querying the database anytime someone is accessing the forum status webpage. Programmingwise, this is encapsulated as a method to make it more efficient as well.

[MaxConcrete]

Thanks for the efforts to get the site back up.

Now I need to dedicate effort to restoring one of my web hosts.

Early this morning I received a message from the host of my DallasFreeways.com domain that "User dallasfr has used up 460% of their bandwidth and 88.4% of their allocated disk space" and "User dallasfr has been suspended for bandwidth overusage". This situation has never occurred in the 10 years I've used this host (webhostingpad). The long term trend for bandwidth on my sites is flat or down, so a bandwidth surge is unexpected.

So all content hosted on that site is unavailable. My host is totally suspended - I cannot even log in with FTP.

I use a directory on DallasFreeways.com for images which I posted on AA Roads. It appears that the bot is following links in posts on AA Roads and also instigating a DDoS attack on those links.

I have a few items on AA roads which are linked to files hosted on Azure. On May 5 and 6 file requests were 5 to 10 times normal rates. There are also spikes about 5x normal on May 12 and May 14. I can't say for a fact that these spikes were caused by the AA Roads DDoS bot, since I have only a few files linked on AA Roads and the overwhelming majority of requests to Azure are direct from dfwfreeways.com and houstonfreeways.com.

I've been in the process of migrating to Azure blob hosting, so I think this will force me to complete that process.

UPDATE: my host says all the requests are coming from a single ip, 166.194.200.53, which is an ATT/Cingular ip in Allen, Texas. Is that consistent with the attack source on AARoads, or is my situation separate and unrelated?

Hello,

I have reviewed your request and found that your account has consumed approximately 450 GB of bandwidth so far. Upon analyzing the account statistics, I discovered that Apache was responsible for the bandwidth consumption.

I further examined the domain access logs and identified that the IP caused the issue.
166.194.200.53

Are you aware of this activity?

Please let us know if there's anything else we can assist you with.

[wxfree]

I was unable to get the forum on my mobile browser for days.  On the computer, it came up most of the time.  It would be available on the computer and not available on mobile at the same time.  Even recently, when the number of guests has been reasonable and the site is working, I couldn't get it on mobile.  It was redirecting from aaroads.com/forum to forum.aaroads.com.  The problem was on my side.  I cleared the browser cache (I'm old enough to remember when that sometimes fixed things, although I haven't done it in a long time) and it works now.  Also, the redirect is gone.

[vdeane]

Interesting that the like button and the 502 bad gateway errors seem to have been related, as the absence of both makes the lack of "new" indicators on posts a less pressing issue.  Without the random chance of starting at the top of a page due to an error or scrolling up to find and look at post likes, I can just start browsing from the new post without worrying about needing to keep track of where it is.

[hotdogPi]

I was unable to get the forum on my mobile browser for days.  On the computer, it came up most of the time.  It would be available on the computer and not available on mobile at the same time.  Even recently, when the number of guests has been reasonable and the site is working, I couldn't get it on mobile.  It was redirecting from aaroads.com/forum to forum.aaroads.com.  The problem was on my side.  I cleared the browser cache (I'm old enough to remember when that sometimes fixed things, although I haven't done it in a long time) and it works now.  Also, the redirect is gone.

What trick did you use to get onto the forum, since it was down serverside?

[wxfree]

I was unable to get the forum on my mobile browser for days.  On the computer, it came up most of the time.  It would be available on the computer and not available on mobile at the same time.  Even recently, when the number of guests has been reasonable and the site is working, I couldn't get it on mobile.  It was redirecting from aaroads.com/forum to forum.aaroads.com.  The problem was on my side.  I cleared the browser cache (I'm old enough to remember when that sometimes fixed things, although I haven't done it in a long time) and it works now.  Also, the redirect is gone.

What trick did you use to get onto the forum, since it was down serverside?

I think what happened was that the forum.aaroads.com redirect got cached and the mobile browser kept going to it.  It seems the redirect wasn't active very long, so it lead to a dead end.  Once I cleared the cache today, it started working normally.  My computer browser never got that redirect and it either loaded the forum or didn't, based on the server conditions.

[kphoger]

Alex:  Do you ever feel like you're playing Whack-A-Mole with a Band-Aid?

I miss the like button

Yes, I had noticed that you were 'liking' approx. 95% of all forum posts...  [/sarc]

If someone posts something thoughtful or interesting that I have no further comment on, I like using it ...

Oh, see, now, that makes total sense.  Instead of either posting or just leaving the author wondering if anyone cared what he wrote, you could do something in between.  I see the light now.

[Molandfreak]

I guess spreading a bit of positivity wasn't quite as well-received as I thought it would be.

[Scott5114]

I was one who wasn't exactly enthusiastic about the idea of the like button, until it was implemented. Indeed, it served a useful purpose as a form of light engagement that doesn't take space up on the page but still indicated appreciation for the post. It was always nice getting a bunch of likes whenever I made a stupid joke.

I never really knew what to think when I purpletexted at people and got likes on that, though.

[GaryV]

what to think when I purpletexted at people and got likes on that

People liked how you were maintaining control of the forum.

[JayhawkCO]

I was one who wasn't exactly enthusiastic about the idea of the like button, until it was implemented. Indeed, it served a useful purpose as a form of light engagement that doesn't take space up on the page but still indicated appreciation for the post. It was always nice getting a bunch of likes whenever I made a stupid joke.

I never really knew what to think when I purpletexted at people and got likes on that, though.

People wish they had the power to shut others up.

[rickmastfan67]

Working fine for me so far; no "NEW" icons is tolerable.

This has been fixed.

[Scott5114]

what to think when I purpletexted at people and got likes on that

People liked how you were maintaining control of the forum.

That was the conclusion I eventually came to, but it was a little amusing comparing it to a gardener pruning a shrub or a policeman writing a traffic ticket and getting a round of applause from passersby for doing so.

[LilianaUwU]

a policeman writing a traffic ticket and getting a round of applause from passersby for doing so.

Sometimes people are known in the neighborhood for going way too fast.

[hotdogPi]

what to think when I purpletexted at people and got likes on that

People liked how you were maintaining control of the forum.

That was the conclusion I eventually came to, but it was a little amusing comparing it to a gardener pruning a shrub or a policeman writing a traffic ticket and getting a round of applause from passersby for doing so.

I saw it more like the Supreme Court making a ruling I agreed with.

[Scott5114]

what to think when I purpletexted at people and got likes on that

People liked how you were maintaining control of the forum.

That was the conclusion I eventually came to, but it was a little amusing comparing it to a gardener pruning a shrub or a policeman writing a traffic ticket and getting a round of applause from passersby for doing so.

I saw it more like the Supreme Court making a ruling I agreed with.

I'm not sure whether I should be offended by being compared to the Supreme Court or not.

[ClassicHasClass]

what to think when I purpletexted at people and got likes on that

People liked how you were maintaining control of the forum.

That was the conclusion I eventually came to, but it was a little amusing comparing it to a gardener pruning a shrub or a policeman writing a traffic ticket and getting a round of applause from passersby for doing so.

The CA 125 NB to I-8 WB ramp in San Diego has a notorious ramp signal during the AM commute and traffic would back up onto the 125 regularly. Most of us sat in line but would watch people zoom by on the shoulder and zip in that way. This ramp is a parclo, so there's no way to see what's at the other end.

One morning someone pulled the usual shoulder crap but as I got around the turn, I saw him pulled up about 50 feet shy of a CHiP, who was sitting in the shoulder as well. The cop was already out of his cruiser and the driver had no way out (and we certainly weren't going to let him back in), and the cop just beckoned him forward as if to say, "C'mon, you got caught, let's get this over with."

I'd have applauded him. I think pretty much all of us sitting there would have.

[webny99]

what to think when I purpletexted at people and got likes on that

People liked how you were maintaining control of the forum.

That was the conclusion I eventually came to, but it was a little amusing comparing it to a gardener pruning a shrub or a policeman writing a traffic ticket and getting a round of applause from passersby for doing so.

Aside from somehow making purple text posts ineligible for likes, getting likes on them is much better than not getting likes.

[kphoger]

I guess spreading a bit of positivity wasn't quite as well-received as I thought it would be.

Oh, I appreciated the positivity.  But, at first, considering I only checked to see who had liked my own posts, it did seem like you had a bit of a crush on me...

I was one who wasn't exactly enthusiastic about the idea of the like button, until it was implemented.

Same.  I thought it was a silly idea at first.  And even afterward, I'd mostly ignore the likes.  But, if a post of mine got six likes or whatever, then I did find it interesting to see who those six people were.

I never really knew what to think when I purpletexted at people and got likes on that, though.

it was a little amusing comparing it to a gardener pruning a shrub or a policeman writing a traffic ticket and getting a round of applause from passersby for doing so.

Just as others have said, sometimes we'd all like to applaud the cop who pulled over the dickhead driver.

Or how about this.  You're not paid to do anything on this forum.  You're a volunteer, basically.  If someone volunteers their time to go and prune shrubs at a battered women's shelter or whatever every so often, then chances are the people involved at the shelter would show some appreciation for that.

Aside from somehow making purple text posts ineligible for likes, getting likes on them is much better than not getting likes.

Imagine if we had down-votes, and people could down-vote purple text.

[GaryV]

whether I should be offended by being compared to the Supreme Court or not

Better than the other 2 branches.

[ZLoth]

Companies downloading the entire forum to feed it into their AI. Except instead of going through the pages one by one they have 136,000 computers connect all at once to download everything.

136,000 computers connecting at once? Doubtful. But, 136,000 virtual instances connecting at once to download everything in a server farm? That's more realistic.

I dunno, that's what the forum software said before it went down.

If they're all connecting from different IP addresses and displaying different user agents, then it really doesn't make all that much difference on our end whether they're running on different hardware or not. (And with IT stuff, I find it usually makes more sense to speak imprecisely when that conveys the gist of what's going when you have no idea to what degree the person you're talking to cares about the technical detail.)

Like I said, I really doubt there are that many computers connecting. When you consider virtualization, and the fact that you can have multiple virtual instances from a single server requesting information, the impact is still the same... the forum server is flatlined.