VA 132 is still there. It was, for some reason, only 162 and 163 that were dropped. (31 was as well, but it was concurrent with 5.)
My only logic for this is that the city wanted to take over maintenance of 2nd Street and Parkway Drive, which were both annexed into the city at some point in the 1960s or early 70s (maps before 1960 show just a bit of Parkway Drive enters city limits but is entirely within it today, and only a tiny bit of 2nd was in the city instead of the majority of it), but wanted to keep 132 and the piece of Capitol Landing Road between 60 and 143 in the primary system for whatever reason. Parkway Drive is a bit redundant as a primary route with VA 132Y also connecting the Colonial Parkway to the primary system in Williamsburg, and VA 132 also runs next to Colonial Williamsburg. I don’t know why the city didn’t want the rest of 162 to be primary. Maybe they also just wanted to take over maintenance on Richmond Road.
As to why they kept 5 and not 31, I guess that’s just because single digit numbers have an enhanced sense of importance in some people’s eyes. (Poor VA 4 aside.) 31’s original utility north of Williamsburg was to connect to a community that was absorbed by Camp Peary over the years.
The bigger mystery to me is why only half of the Monticello Avenue extension became primary.