Just from my personal observations here in Louisiana.....
On the state & US highways, it seems that the signs reflect distance between a specific intersection that may, or may not, be the town's center.
I live between Shreveport and Monroe. The distance signs on I-20 seem to reflect the distance to the first exits for Shreveport and Monroe. (exit 19A in Shreveport and exit 116B in Monroe) These exits also happen to be very close to their downtown centers, and both happen as soon as you cross a river. I haven't noticed what exit seems to be referenced coming from the other way into each city, but I will try next time I'm out that way.