It's been a while since I woke up a sleeping topic...
I looked at Google Maps to see the area, and I figured out that the TCH-104/Trunk 4 interchange (well, let's say "intersection" for now) northeast of Port Hawkesbury was promised to a much bigger project, that could bypass Pt. Hawkesbury and Pt. Hastings as well. Let's imagine that, from west to east.
First, connect back Trunk 4 and the actual TCH-104 east, following the remnants of the ancient road, and start a brand new TCH-104 east, just west of the actual one. Then let it stretch southeast, crossing Tower Rd., passing under the power lines, and crossing Hillside Dr., before turning east, passing south of O'Brien Rd, and crossing Marine Dr. (NS-344), the railway and Trunk 4 (actual TCH-104), before crossing Barrys River/Aulds Cove on a brand new bridge, northwest of the causeway. The new TCH-104 would cross Ceilidh Trail (Trunk 19) just south of the southern end of Fox Rd., would stretch northeast, then curve east just south of the NS Power site, cross TCH-105 and pass under the power lines, before bypassing Hector Lake by the north, and curve southeast, following the power lines, crossing Kings Rd. and Crandall Rd. on its way to connect to actual TCH-104 at the Trunk 4 interchange. Finally, add some interchanges along the way where needed, like at Sunrise Trail (Trunk 4), Ceilidh Trail (Trunk 19), and TCH-105.
The new bridge, crossing Barrys River/Aulds Cove, would be one lane per direction, and be twinned when needed.
That kind of project would be expensive, for sure, but will separate transit and local traffics, and, foremost, definitely put an end to the swing bridge wait.