Highway Data Discussion (CHM/TravelMapping)

[Jim]

We had a report in the old CHM forum about a Truck NY 32 in Glens Falls, and the discussion followed that truck routes are not official from by NYSDOT.  (http://clinched.s2.bizhat.com/viewtopic.php?t=2248&mforum=clinched) We have a handful of them in our DB but there are surely more.  Thoughts on whether signed truck routes should be included where we know they exist, or leave them all out?

[mapcat]

I've never understood why we include truck routes.  It's been suggested that there's something remarkable about Truck US-19 in Pittsburgh, but that alone doesn't convince me that the other, unremarkable truck routes are necessary.  In PA especially, aren't they posted only in places where the current bridges are structurally deficient?  Once those get upgraded, the truck routes will disappear, just like the one in Alliance, Ohio, did after an underpass was modified to allow higher-clearance vehicles.

[oscar]

Regarding New Mexico, it looks like the following documents have been updated since I last used similar documents in updating NM's national system routes and drafting NM's state highways:

http://dot.state.nm.us/content/dam/nmdot/Data_Management/NM_AADT_Listing.pdf
http://dot.state.nm.us/content/dam/nmdot/Data_Management/US_AADT_Listing.pdf
http://dot.state.nm.us/content/dam/nmdot/Data_Management/Interstate_AADT_Listing.pdf

Do we think these are a definitive source about what routes exist and where?

Just a casual glance, for routes on which I've previously weighed in.

If the lists are authoritative and complete (I have not worked out whether there are routes we know to be in the state system but not in the reports):

-- the on-again off-again NM 497 in Deming looks to be back off again, so we can leave that alone

-- NM 332, mentioned in the NM 11 route file but has no route file of its own, is not a state route

-- NM 300, which we offed after I reported it as unsigned in 2013 (except as a bike route with a different number), is listed in the report, with AADTs reported for years both before and after my report. 

[rickmastfan67]

It's been suggested that there's something remarkable about Truck US-19 in Pittsburgh, but that alone doesn't convince me that the other, unremarkable truck routes are necessary.

Well, it's the only known officially approved Truck US Highway by the AASHTO since it's in the logs.

[Rothman]

We had a report in the old CHM forum about a Truck NY 32 in Glens Falls, and the discussion followed that truck routes are not official from by NYSDOT.  (http://clinched.s2.bizhat.com/viewtopic.php?t=2248&mforum=clinched) We have a handful of them in our DB but there are surely more.  Thoughts on whether signed truck routes should be included where we know they exist, or leave them all out?

Makes me wonder about Truck NY 5 as well in Schenectady.

[Mapmikey]

It's been suggested that there's something remarkable about Truck US-19 in Pittsburgh, but that alone doesn't convince me that the other, unremarkable truck routes are necessary.

Well, it's the only known officially approved Truck US Highway by the AASHTO since it's in the logs.

US 9 Truck in Delaware was also approved by AASHTO in 1983...

Mike

[Jim]

We had a report in the old CHM forum about a Truck NY 32 in Glens Falls, and the discussion followed that truck routes are not official from by NYSDOT.  (http://clinched.s2.bizhat.com/viewtopic.php?t=2248&mforum=clinched) We have a handful of them in our DB but there are surely more.  Thoughts on whether signed truck routes should be included where we know they exist, or leave them all out?

Makes me wonder about Truck NY 5 as well in Schenectady.

We have that one, and 6 others.  We know of at least NY 32 Truck in Glens Falls, but surely there are others we have not discovered.

Code Select Expand

usany;NY;NY5;Trk;Sch;Schenectady;ny.ny005trksch;
usany;NY;NY9P;Trk;Sar;Saratoga Springs;ny.ny009ptrksar;
usany;NY;NY14;Trk;Gen;Geneva;ny.ny014trkgen;
usany;NY;NY19;Trk;Bro;Brockport;ny.ny019trkbro;
usany;NY;NY29;Trk;Sar;Saratoga Springs;ny.ny029trksar;
usany;NY;NY30;Trk;Sch;Schoharie;ny.ny030trksch;
usany;NY;NY298;Trk;Syr;Syracuse;ny.ny298trksyr;


Some add little or no "clinchable" mileage, as they run along other signed routes for much of their length.

I am not proposing any changes to what we do with US highway truck rounds approved and well signed and shown in the documents we use for references.  I am thinking about what we should do with the NY truck routes.

[vdeane]

After discovering Truck 32 I went to Mike Fay in Highway Data Services and asked him if he had a list of them; that's when I found out they weren't considered official by Main Office.  I think the municipalities come up with them.  We don't do pavement sufficiency on them (or traffic counts if they're function class 8, 9, or 19), so I think we can consider them as not being state highways.

Truck 5 and Truck 14 were also my discoveries.

[Rothman]

After discovering Truck 32 I went to Mike Fay in Highway Data Services and asked him if he had a list of them; that's when I found out they weren't considered official by Main Office.  I think the municipalities come up with them.  We don't do pavement sufficiency on them (or traffic counts if they're function class 8, 9, or 19), so I think we can consider them as not being state highways.

Truck 5 and Truck 14 were also my discoveries.

So, Mike Fay doesn't consider them official and yet we spend the money to sign them...kind of an interesting endeavor to pursue when they aren't official.

[SD Mapman]

Regarding New Mexico, it looks like the following documents have been updated since I last used similar documents in updating NM's national system routes and drafting NM's state highways:

http://dot.state.nm.us/content/dam/nmdot/Data_Management/NM_AADT_Listing.pdf
http://dot.state.nm.us/content/dam/nmdot/Data_Management/US_AADT_Listing.pdf
http://dot.state.nm.us/content/dam/nmdot/Data_Management/Interstate_AADT_Listing.pdf

Do we think these are a definitive source about what routes exist and where?

Just a casual glance, for routes on which I've previously weighed in.

If the lists are authoritative and complete (I have not worked out whether there are routes we know to be in the state system but not in the reports):

-- the on-again off-again NM 497 in Deming looks to be back off again, so we can leave that alone

-- NM 332, mentioned in the NM 11 route file but has no route file of its own, is not a state route

-- NM 300, which we offed after I reported it as unsigned in 2013 (except as a bike route with a different number), is listed in the report, with AADTs reported for years both before and after my report.

To add on to that, the in-city portion of NM 72 that I reported last year isn't in the logs. That never got added in, though.
It's probably still signed, because New Mexico.

[oscar]

After discovering Truck 32 I went to Mike Fay in Highway Data Services and asked him if he had a list of them; that's when I found out they weren't considered official by Main Office.  I think the municipalities come up with them.  We don't do pavement sufficiency on them (or traffic counts if they're function class 8, 9, or 19), so I think we can consider them as not being state highways.

I think the "not considered official by Main Office" is the clincher. No pavement sufficiency checks or traffic counts might not be enough by themselves, since we're including some locally-maintained but state-designated roads (Vermont especially, but also other states) in state route sets.

[Jim]

I'm declaring the current datacheck page and the mechanism to submit false positives to be good enough to have everyone start going through their regions and making reports.  I hope this is as simple as looking through the errors for active routes and matching them up to FPs already reported in CHM to start.  I wish this could have been automated, but I just didn't have access to the FP list data in a reasonably convenient format to be able to do so. 

I expect very few new datacheck errors that need actual fixes in the data for active routes.  Let's focus on the active routes to start, then the in-development systems close to activation.

Like the updates.csv file, it will likely cause a lot of edit conflicts if we all edit the file directly and make pull requests.  So let's do like that one, and submit your csv entries for the datacheckfps.csv file to me to add to the file.  Posting here, posting in a GitHub issue in the HighwayData repository, or sending to me by email are all good ways to get them to me.  Later, once the initial rush is done, I'll sort the file alphabetically so we'd each be making changes in different sections.  Git would have no problem with merges in that case.

[yakra]

Datacheck doesn't seem to be picking up all of the label references own route error type.

The following OK examples, where the entire label matches the route name, were flagged:
ok.ok077dced;OK77D;;;LABEL_SELFREF;
ok.ok077smar;OK77S;;;LABEL_SELFREF;
ok.ok077sove;OK77S;;;LABEL_SELFREF;

These false positives from the old page at CHM were not:
NB NB772 NB772_E label references own route
NB NB772 NB772_W label references own route
NB NB845 NB845_E label references own route
NB NB845 NB845_W label references own route
ND US52 US52/281Byp label references own route

[rickmastfan67]

We had a report in the old CHM forum about a Truck NY 32 in Glens Falls, and the discussion followed that truck routes are not official from by NYSDOT.  (http://clinched.s2.bizhat.com/viewtopic.php?t=2248&mforum=clinched) We have a handful of them in our DB but there are surely more.  Thoughts on whether signed truck routes should be included where we know they exist, or leave them all out?

Makes me wonder about Truck NY 5 as well in Schenectady.

We have that one, and 6 others.  We know of at least NY 32 Truck in Glens Falls, but surely there are others we have not discovered.

Code Select Expand

usany;NY;NY5;Trk;Sch;Schenectady;ny.ny005trksch;
usany;NY;NY9P;Trk;Sar;Saratoga Springs;ny.ny009ptrksar;
usany;NY;NY14;Trk;Gen;Geneva;ny.ny014trkgen;
usany;NY;NY19;Trk;Bro;Brockport;ny.ny019trkbro;
usany;NY;NY29;Trk;Sar;Saratoga Springs;ny.ny029trksar;
usany;NY;NY30;Trk;Sch;Schoharie;ny.ny030trksch;
usany;NY;NY298;Trk;Syr;Syracuse;ny.ny298trksyr;


Some add little or no "clinchable" mileage, as they run along other signed routes for much of their length.

I am not proposing any changes to what we do with US highway truck rounds approved and well signed and shown in the documents we use for references.  I am thinking about what we should do with the NY truck routes.

Don't forget about US-20 Truck in Silver Creek, Jim.
http://cmap.m-plex.com/hb/hwymap.php?r=ny.us020trksil&sys=usausb&rg=all

[rickmastfan67]

Jim, here's another problem with the datacheck page.

I honestly think it needs to be fixed before a full scale FP cleanup is done.
https://github.com/TravelMapping/DataProcessing/issues/2

[english si]

I assumed it was the logical duplicate on loop routes, though you have a point.

The most obvious alteration to the page - even before sorting by region - is links on the root field (like the updates page), so people can click and easily see!

[Jim]

Let's wait before putting much effort into FP reporting.  I'm looking at the CHM FP list (over 4300 of them) and I think I can programmatically get the majority of them flagged in TravelMapping.

[Jim]

First batch of grabbing CHM datacheck FPs and converting them is complete.  There are now about 1800 more visible distance FPs in our database than there were before.  Not everything matched up perfectly since our data has diverged some, but it's a big chunk of work no one has to do to look at those 1800.  I should be able to do the same for a couple other error types, notably the sharp angle errors.  I also will at some point add a list of items in the FP csv list that didn't match any actual error, which might also give some indication of more errors that can be flagged as FPs without much effort (route name changes, label changes or small repositionings).  That will help as much down the road as now.  So again, hold off on hunting down FPs until I do as much as I can automating the process.  I doubt I'll have any time to work on it tomorrow but maybe Sunday.

[Jim]

A couple quick improvements to the datacheck page: each datacheck page entry now has an HB link, and the three (one with all datacheck errors not flagged as FPs in active systems, one with all datacheck errors not flagged as FPs in in-devel systems, and one with datacheck errors that were successfully marked as FPs) and are color-coded: pink for active unflagged, blue for in-devel unflagged, grey for flagged.  But continue to hold off dealing with errors until I can import more from CHM automatically.

[Jim]

Sharp angle datacheck FP entries were almost trivial to import from CHM, so they're in.  Another 450+ moved from "to do" to "done"!

3167 of the 4471 currently detected errors are now flagged.  That's still leaves a lot (1304, in fact) but we're making quick progress here.

[yakra]

I like the color coding. It makes it easy to hold PageUp/PageDown until I'm looking at the table I want.

A thought I had last night: have links at the top of the page to datacheck.php#InDevel and datacheck.php#FP

[Jim]

Next time I can put a little time in, I'm planning to look for those situations where we have a datacheck error that's not quite the same as the FP entry that was converted from CHM.  So if a sharp angle or visible distance error is a match except for the angle or distance, I'll report those specially somehow, so they can be given a quick look and hopefully fixed.

Don't worry about duplicates yet - if you sent in new FP reports and there are also now CHM-imported reports.  I got rid of all exact duplicates but I'm sure there are a bunch that will be singled out by the check described above.

[Jim]

I knocked off another 300 or so datacheck false positives by taking the list of those imported from CHM which did not match any TravelMapping error exactly, but did match in all except the distance or angle field.  If they were off by less than a half mile or a couple degrees, they were fixed to match the current distance or angle.  I feel like this is pretty safe.  Most of what remain of visible distance and sharp angle errors in active systems are in Europe.  I suspect this is because of Si's recent revamp of so many things over there, meaning my import of CHM FPs didn't help much.

As part of this process, the site update program now produces a new log file which contains all FP reports in the TravelMapping datacheckfps.csv file that didn't match any actual error:

http://www.teresco.org/~terescoj/travelmapping/logs/unmatchedfps.log

Hopefully these will be helpful in identifying more FPs, but if they're really not relevant, their entries should be removed from datacheckfps.csv.  Low priority work, for sure.

I think this is all I can do automatically with visible distance and sharp angle errors.  Please feel free to go back to reporting FPs of those kinds of errors.  There's still some thing I can and plan to do with other error types to mark more FPs automatically from CHM's list, so no need to address those just yet.

[Jim]

Duplicate coordinates datacheck errors now include point labels, and I updated the existing FPs for this in our csv file.  I hope to be able to use this new feature to be able to knock off another set of FPs from CHM's list.

[Jim]

All "Exit 0" datacheck errors and most "duplicate coordinate" errors have now been marked as FPs thanks to conversion from CHM's list.

All highway data maintainers, please address all remaining duplicate coordinate, duplicate label, sharp angle, and visible distance datacheck errors by reporting as FPs or fixing the highway data.  Si, I am sure that most if not all of the visible distance errors in Russia, Kazakhstan, Tajikistan, and some of the other countries currently generating most of the remaining errors in active systems are false positives.  Rather than copying and pasting line by line from the tables on the web, I can get you a list of those errors in csv format for you to check out and then paste into the FP list if that's easier.  I do think they all need at least a quick look, though, as a quick spot check quickly showed that tjk.e006, for example, looks to have a point out of order generating a few legitimate errors.

Please continue to ignore all datacheck errors of types other than those listed above until I have had a chance to refine my error flagging for those, and then to convert FP reports from the old CHM list where possible.