I've driven through Toronto on the 401 without stopping (though I came fairly close to) on a Monday around noon.
Of course, customs may count as a stop too...
Customs very much counts as a stop.
It's like those on these forums who lament the lack of a full freeway for Hwy 137... it's an uninterrupted flow facility and has no capacity issues in and of itself. The toll bridge and customs would gum up a conventional highway just as much as it would a freeway.
There's no capacity benefit to upgrading it. And in terms of safety, yes, removing the five entrances/sideroads would reduce the chances of a conflict from those points. But suddenly you'd have freeway with significant queuing... that's just begging for rear end collisions, and at higher speeds. Such a terrible plan.