I've always thought of it like this: imagine that the signal were mounted vertically on a vertical pole. Now bend the pole 90°. (Obviously, this only works for signals on the "normal" side of the road.)
That makes sense to me too. As long as you mean 90 degrees towards the road. For either driving side, mounted
from the driving side, this would always result in the red being closest to the center of the road (on right with LHT, on left with RHT).