Can someone explain to me why someone wearing their pants low bothers you or is "unfortunate"?
Because your butt being visible for people to see is generally considered inappropriate. Showing your underwear is just one step below that. And, even if you can't actually see the butt/underwear, the sagging pants makes it obvious that you could if only the shirt were shorter.
Actually, I've been thinking about this, and I believe there's more at play.
Clothing norms are obviously culturally dependent, which is to say they vary across time and geography. There was once an anthropologist working among a tribe in Papua New Guinea (or somewhere in that general area) whose only typical clothing item was a cord around the waist for hanging tools on. Aside from that cord, they went around otherwise nude. One day, the anthropologist asked one of the men if anyone ever went without the cord around their waist. With shocked disbelief on the man's face, he answered,
And be naked!?? We can chuckle at that, but on some level we all understand that dress codes vary from culture to culture. For this reason, then, it's unavoidably a bit foolish to try and conform such norms to the rigors of logic and reason: at some point, we have to just admit that that's the way things are in our culture.
But clothing norms are also situationally dependent—even within the same culture. We can think nothing of seeing a woman in a bikini at the beach or the pool, yet we'd be startled at seeing her dressed like that at the office copy machine or taking communion at mass. I wouldn't let my sons wear gym shorts and an old T-shirt to a wedding, even though I have no problem sending them out to play in those clothes. Go to a hot springs pool, and you can take all your clothes off without shocking anyone; try heading downstairs to the hotel breakfast with no clothes on the next morning, however, and be prepared for them to kick you out in short order.
With that in mind, it is my expectation that people in normal everyday life will keep their pants on. If my pants were to fall down to my knees as I was heading into my boss's office at work, then that would be embarrassing for both of us. If you come over to my house for dinner, then I expect you to keep your pants from falling off. Knowing what brand and style of underwear you have isn't a big deal in the locker room, but it's not something people should find out while waiting for their order at Burger King.
And, when someone lets their pants sag to the point that it's obviously noticeable, then it basically looks like they're halfway to ending up undressed—and that they don't really care—and, moreover, that they actually
want that. It looks like, any second now, they might end up "letting it all hang out", and that they're fine with that, to the point that they didn't even bother to put on a belt.