Highway Data Discussion (CHM/TravelMapping)

[english si]

I'm starting to look at Si's new in-dev systems (so far, just to fix my draft list file entries), and already have one comment on CO 2 and CO 22.

Already dealt with three comments on usaco. The old forum, private messages, here - wherever.

A thread here is probably best.

[Jim]

But first, a meta-question:  CHM put discussions of in-dev systems in a collaborators-only area of its forum. Anyone know why it was done that way? We could set up new topics in that area of the CHM forum. But I don't see why we should do that, which could make it harder to get useful input from users who weren't on the CHM collaborator team. OTOH, if we're about to set up our own forum separate from the aaroads forum, we could wait and set up the new threads there instead.

When we get a project-specific forum up and running, I'd like to see it all or almost all public.  We can always use private messages or other mechanisms if something needs to be discussed that for some reason should not be seen by the end users.  The only place I can really think that was a slight positive in the old CHM forum was keeping the point request thread private so we could avoid a flood of point requests from end users for every little turnaround.  I suppose it wouldn't hurt to have a private subforum there but I'd prefer it to be used rarely.

[oscar]

We can always use private messages or other mechanisms if something needs to be discussed that for some reason should not be seen by the end users.  The only place I can really think that was a slight positive in the old CHM forum was keeping the point request thread private so we could avoid a flood of point requests from end users for every little turnaround.  I suppose it wouldn't hurt to have a private subforum there but I'd prefer it to be used rarely.

I'd distinguish between "point requests" and "point suggestions". "Point requests" were basically a "collaborator perk", rejected only if unreasonable, but only collaborators could make them. But CHM also stiff-armed "point suggestions" made in the public part of the forum. I would be more open to such suggestions, if they are something likely to be helpful to other users rather than just a personal favor to the requester, though it would be entirely up to the relevant team member whether to adopt a suggestion.

[Jim]

Now that we have several people well underway using GitHub to submit highway data updates, I'd like to bring up a few things.

1) It would be good if someone can take charge of a sources spreadsheet/CSV.  I think we want to start with CHM's list and make sure we keep it up.  I think it should just be another .csv in the HighwayData repository like the updates one.

2) The fork/pull model of data updates has worked far better than I hoped from my perspective as the primary person managing those changes.  I'm hoping that those contributing highway data updates are finding the process at least tolerable if not always intuitive.  This has to be worlds easier than what Tim had to do with everyone sending zip files.  That said, it would be nice if some of the typos and other problems could be discovered before I pull them in and try running an update.  If you have a computer capable of running Python 3.4 (this is you, most likely) and can get copies of our DataProcessing, HighwayData, and UserData repositories, it shouldn't be too bad.  Basically, you would want to point a few paths in the read_data.py program to the right places to find all the data and then run it.  If you get errors or warnings, you can fix them and try again.  There's also nothing stopping anyone from setting up their own copy of the DB and a web server with a copy of our things in the Web repository and seeing your results as well.  But that's not likely of interest to as many people.

3) We can probably move the project to a more permanent web home soon.  Assuming we want to use the code I've been working on and using, we'd need a place that has ssh shell access, a modern apache web server with PHP, and a recent mysql database.  If I have a chance, I might try setting up a copy of this in my hostmonster account to see if it will work.  I know we have a domain name, but I don't think there's anything else yet.  I'm perfectly willing to host the new domain on my FreeBSD server, but that leaves me and that server and a pretty small pipe to the Internet all as single points of potential failure.

4) I mentioned datacheck errors recently.  I need to track down a problem that's keeping them from loading into the DB now, but once that's done, one of my priorities related to the project (second to processing updates every day or two) is to come up with the mechanism to mark false positives so we can aim for a clean slate for all active systems, and know what needs to be fixed in the in-development systems.  One of the big things here is to make sure things already marked in CHM that haven't changed in our data can remain FPs.  Once a mechanism is in place, I think it will make sense for someone to be the primary person in charge of maintaining the list of FPs and watching the error list for things that need attention.  My semester starts very soon so it will be hard to dedicate much time to it for a while now.

[oscar]

1) It would be good if someone can take charge of a sources spreadsheet/CSV.  I think we want to start with CHM's list and make sure we keep it up.  I think it should just be another .csv in the HighwayData repository like the updates one.

I should be able to take care of this, especially since I'm almost done updating my jurisdictions. But I first have to take care of my non-roads websites, which are on a webspace my ISP is shutting down in a few weeks, so I need to find a new home for them (perhaps on a webspace I control and isn't affected by the shutdown). 

2) The fork/pull model of data updates has worked far better than I hoped from my perspective as the primary person managing those changes.  I'm hoping that those contributing highway data updates are finding the process at least tolerable if not always intuitive.  This has to be worlds easier than what Tim had to do with everyone sending zip files.  That said, it would be nice if some of the typos and other problems could be discovered before I pull them in and try running an update.  If you have a computer capable of running Python 3.4 (this is you, most likely) and can get copies of our DataProcessing, HighwayData, and UserData repositories, it shouldn't be too bad.  Basically, you would want to point a few paths in the read_data.py program to the right places to find all the data and then run it.  If you get errors or warnings, you can fix them and try again.  There's also nothing stopping anyone from setting up their own copy of the DB and a web server with a copy of our things in the Web repository and seeing your results as well.  But that's not likely of interest to as many people.

That's probably beyond both my technical abilities and my hardware (all my computers are Windows boxes, one of them an ancient one still on Windows Vista). I'll just have to be more careful in assembling my pull requests, and try to keep them bite-sized to minimize the risks of errors in my submissions.

[Duke87]

Things like the Lincoln Highway have also been suggested (system name of something like "United States Select Scenic and Historic Highways").

The challenge here is "which alignment of it?". At various points in its history any "historic route" will have been realigned, so there is no one single alignment to follow.

Then you have the many cases where modernization of a highway has resulted in segments of what was the old route left as odd and perhaps discontinuous jogs - see for example all the little bits of "Old (Boston) Post Road", "Old Kings Highway", etc. in Connecticut. To what degree, if any, is it appropriate to mark the historic route as following the old alignment(s) that were actually signed as it back in the day versus the modern alignment which was never signed as it but is now the de facto through road along the corridor?

Consider as well that in some cases what was the old route is not driveable. The original routing of US 1 (and also the Boston Post Road) through Stamford CT is still more or less clinchable on foot, but it involves walking through a park and a mall as well as using a bridge which is closed to vehicular traffic. Would we route "CT BosPostRd" along that, or along the newer road which was created in the 1970s?

[sipes23]

Consider as well that in some cases what was the old route is not driveable. The original routing of US 1 (and also the Boston Post Road) through Stamford CT is still more or less clinchable on foot, but it involves walking through a park and a mall as well as using a bridge which is closed to vehicular traffic. Would we route "CT BosPostRd" along that, or along the newer road which was created in the 1970s?

My feeling is this: if it is a historic route, follow the historic route. Of course, this leaves the problem that you raise: at which time period? I know US 66 has a few alignments through Illinois that depend on just when you're talking about. But for something like the Boston Post Road, I'd be prone to "the most historic route" of the historic route. If it's only clinchable by foot, well, that's the alignment. The newer 1970's road is not a "historic route". Yet.

This exact situation of undriveable alignments (potentially with no modern alignment available to boot) also applies to things like the Old Spanish Trail too. So it's hardly a unique situation.

But I suspect this is, as of right now, small potatoes.

[Jim]

If we do historic routes, I think those that are signed somehow make the most sense.  Even if there are multiple signed alternatives, no harm in including them all.  For example, I recall there being multiple signed Historic US 66 routes through New Mexico, one of which went up through Santa Fe, another more directly to Albuquerque.  Nothing would stop us from including all such routings, using the same kinds of mechanisms we use for multiple segment routes in current systems.

I don't think it's worth trying to get old routings that have no official designation or signage.  There's going to be too much guesswork, both in trying to map the routes for the project, and for travelers trying to clinch them.

[english si]

If we do historic routes, I think those that are signed somehow make the most sense.  Even if there are multiple signed alternatives, no harm in including them all.  For example, I recall there being multiple signed Historic US 66 routes through New Mexico, one of which went up through Santa Fe, another more directly to Albuquerque.  Nothing would stop us from including all such routings, using the same kinds of mechanisms we use for multiple segment routes in current systems.

I don't think it's worth trying to get old routings that have no official designation or signage.  There's going to be too much guesswork, both in trying to map the routes for the project, and for travelers trying to clinch them.

Indeed. I gather parts of the Lincoln Highway are signed (at least patchily).

[mapcat]

Indeed. I gather parts of the Lincoln Highway are signed (at least patchily).

Yes, it is in many places.  Same with parts of the National Road, plus the multiple routings of US66, US99, etc.

Then you have the Oregon Trail, Santa Fe Trail, Trail of Tears, and so forth, that mostly follow other numbered highways.  Would those be part of this system, another, or ignored?

[Duke87]

Indeed. I gather parts of the Lincoln Highway are signed (at least patchily).

Yes, it is in many places.  Same with parts of the National Road, plus the multiple routings of US66, US99, etc.

Then you have the Oregon Trail, Santa Fe Trail, Trail of Tears, and so forth, that mostly follow other numbered highways.  Would those be part of this system, another, or ignored?

Guess it depends on how you define "signed". Boston Post Road has no shields, but many segments of it retain the name "Boston Post Road" on street signs. And in several places there are original 18th century stone mile posts still there by the side of the road, although they're too small and low to the ground to easily spot from a car moving at modern speeds, and at least one has been moved from its original location.

But I suppose unlike things like the Lincoln Highway it is not and never was signed with the intent that people on road trips would be able to follow it by following the signs. Being able to properly follow the original route requires doing some research before setting out to do so.

[Jim]

As mentioned in other places here, it's time to start addressing datacheck errors.  This post describes what I've done tonight toward this goal, and asks for one or two volunteers from among our highway data managers to try it out on a small scale before we expand to all data.

I've introduced a new file to the HighwayData repository, datacheckfps.csv, which should contain an entry for each datacheck error that has been verified to be a false positive (FP).  At least for now, the definitions of a datacheck error, and what qualifies as an FP, remain the same as CHM.  This file is consulted during a site update to flag errors as they are detected.  The datacheck page in its current, very basic format (http://www.teresco.org/~terescoj/travelmapping/devel/datacheck.php), shows all datacheck errors, with a column indicating FPs.  I've added a new column that prints, for each error, the .csv line that would be added to the datacheckfps.csv file to mark it as an FP (for easy copy and paste).  My next project goal, other than continuing to process data and list updates, is to enhance the datacheck page to have 3 tables instead of 1.  1) Errors in active systems that are not flagged as FPs, which should be addressed with high priority, 2) Errors in in-development systems that would need to be addressed or flagged as FPs before activation, and 3) Errors that have been flagged as FPs in any systems.  Later, I think it will be useful to include a list of entries in datacheckfps.csv that didn't match any detected error, as these might be indications of new errors introduced.

Anyway, for now, I'd like a few of our highway data managers to take one of your smaller systems and send me, by email, the csv entries for false positives in that system, so I can further test my code that deals with this stuff.  Once I'm satisfied that things are working, and maybe with some enhancements to the system to make this process easier, we'll try to get all active systems to show up as clean.  I found it useful to compare with entries on the old CHM datacheck page when I processed Massachusetts.

[rickmastfan67]

Anyway, for now, I'd like a few of our highway data managers to take one of your smaller systems and send me, by email, the csv entries for false positives in that system, so I can further test my code that deals with this stuff.  Once I'm satisfied that things are working, and maybe with some enhancements to the system to make this process easier, we'll try to get all active systems to show up as clean.  I found it useful to compare with entries on the old CHM datacheck page when I processed Massachusetts.

Here's one very quick FP from a state I'm in control of currently.  I'll check more later and e-mail those instead.

Code Select Expand

wv.i070;0A;;;EXIT0;

Also Jim, is there anyway to have them sorted by 'region' first, and then system, and then route number?  Very hard to find routes for one state when they are all over the place.

[Jim]

Also Jim, is there anyway to have them sorted by 'region' first, and then system, and then route number?  Very hard to find routes for one state when they are all over the place.

Thanks - right now they're just listed by however the DB gives them to me, but I plan to make it easier to navigate, filter, order, later on.  Ideally, the list will be nearly empty for active routes after the initial pass for each person's routes.  I'm thinking "filter by region" is going to be the easiest and most useful improvement.

I also wonder about how useful the "visible distance too long" errors really are.

[oscar]

I also wonder about how useful the "visible distance too long" errors really are.

Are you checking for "long segment" errors (>20 mi. between waypoints, whether visible or not)? That was to head off technical problems with mapping long interrupted segments, so Tim made us insert shaping points even if not otherwise needed, just to break up the segment. An LS check could be a good fallback for a VD check, and has flagged for me once or twice when my draft route file mistakenly left a long stretch of highway completely unmapped.

I think a VD check is somewhat useful to encourage waypoints every 10 mi. or so (we could tweak the threshold upward) for travelers who turn off or around at places we didn't expect when we drafted the route files. But it can be taken too far, as Tim chided me for taking GPS reads for, and including in my draft Arctic route sets, unnamed and unmapped minor roads such as river accesses just to meet VD requirements. I would favor keeping a VD check, with the understandings that (a) most of them will be crossed off just once as false positives, and (b) we tell new team members not to get desperate about placing waypoints every 10 miles.

[yakra]

Tim chided me for taking GPS reads for, and including in my draft Arctic route sets, unnamed and unmapped minor roads such as river accesses just to meet VD requirements.

i guess I missed that discussion. I added minor ranch access roads in Texas for the same reason.

[oscar]

Tim chided me for taking GPS reads for, and including in my draft Arctic route sets, unnamed and unmapped minor roads such as river accesses just to meet VD requirements.

i guess I missed that discussion. I added minor ranch access roads in Texas for the same reason.

I think that discussion was entirely by e-mail.

Tim's issue was with my inventing names for the access roads I wanted to use, beyond what our waypoint labeling rules allow. Look at the Datacheck entries for Canada's Arctic territories, you'll see a few entries for distances of over 60 miles between visible waypoints, which bothered Tim less than the invented-names points I proposed to reduce those errors.

[Jim]

Yes, same rules apply at this point as CHM.  Any visible distance greater than 10 and long segment greater than 20 is flagged.

[rickmastfan67]

Hey Jim, I just noticed that there isn't a mention for a version of the 'Exit 0' mention.

WV US-250 has a 'Exit #0' mentioned ( I-70(0A) ), but it isn't flagged like the I-70 tag was.  Maybe there should be something in your code that could flag any '(0)' or '(0x)' tag as well so it can be double checked too.

[oscar]

Hey Jim, I just noticed that there isn't a mention for a version of the 'Exit 0' mention.

WV US-250 has a 'Exit #0' mentioned ( I-70(0A) ), but it isn't flagged like the I-70 tag was.  Maybe there should be something in your code that could flag any '(0)' or '(0x)' tag as well so it can be double checked too.

Aren't 0x tags less suspicious than a 0 tag? The latter could be a residue from the days when we routinely used 0 and 999 as artificial endpoints in route files (at least for Interstate segments). A 0x tag seems more likely to reflect signage in the field.

[Jim]

Hey Jim, I just noticed that there isn't a mention for a version of the 'Exit 0' mention.

WV US-250 has a 'Exit #0' mentioned ( I-70(0A) ), but it isn't flagged like the I-70 tag was.  Maybe there should be something in your code that could flag any '(0)' or '(0x)' tag as well so it can be double checked too.

Thanks - I opened an issue into Github so I'll remember to address this.

[rickmastfan67]

Hey Jim, I just noticed that there isn't a mention for a version of the 'Exit 0' mention.

WV US-250 has a 'Exit #0' mentioned ( I-70(0A) ), but it isn't flagged like the I-70 tag was.  Maybe there should be something in your code that could flag any '(0)' or '(0x)' tag as well so it can be double checked too.

Aren't 0x tags less suspicious than a 0 tag? The latter could be a residue from the days when we routinely used 0 and 999 as artificial endpoints in route files (at least for Interstate segments). A 0x tag seems more likely to reflect signage in the field.

True.  Still, if the '0' is getting flagged, so should the '(0)' just in case.  I do know that the I-70 example is really posted as '0' in the field, but because of the old '0' & '999' being used for the end points at state lines, we were forced to make it '0A'.

[yakra]

Tim's issue was with my inventing names for the access roads I wanted to use, beyond what our waypoint labeling rules allow. Look at the Datacheck entries for Canada's Arctic territories, you'll see a few entries for distances of over 60 miles between visible waypoints, which bothered Tim less than the invented-names points I proposed to reduce those errors.

In my case, I kinda wibbly-wobblied "Truncated, nearby/distant town name" into "Truncated, nearby/distant placename". I used placenames included on topo maps. LomaVisRan, anyone?

[Jim]

The datacheck page now has three tables:

- errors in active systems not flagged as FPs (i.e., the things we need to mark as FPs or fix -- this table should be empty if things are in a good state).

- errors in in-development systems not flagged as FPs (i.e., the things that will need to be marked as FPs or fixed before a system goes active).

- errors in all systems that have been flagged as FPs.

[Jim]

Regarding New Mexico, it looks like the following documents have been updated since I last used similar documents in updating NM's national system routes and drafting NM's state highways:

http://dot.state.nm.us/content/dam/nmdot/Data_Management/NM_AADT_Listing.pdf
http://dot.state.nm.us/content/dam/nmdot/Data_Management/US_AADT_Listing.pdf
http://dot.state.nm.us/content/dam/nmdot/Data_Management/Interstate_AADT_Listing.pdf

Do we think these are a definitive source about what routes exist and where?

I won't be able to dig through these looking for needed changes in the near term, so it would be great if anyone could at least verify that the highways I have plotted in usanm match those in the first document above, and that termini that are in any way questionable are correct in all routes in the NM region.

I would like this process to be complete before we activate usanm.