Status of Travel Mapping

[Jim]

First per-user stats are now being computed and placed in the same log files as .list errors.  I still need to add per-route stats and store things in the DB for display on the web, but it's another nice thing to be able to check out and a big step forward in overall functionality.

All .list files received as of 11:30 PM EDT are also in the DB now, so a few new people can see maps and even a start on stats.

Enjoy!

[Jim]

On behalf of those of us who aren't actively involved with this project, I'd like to throw out a few questions to every working on the new travel mapping site.

1. Is the intent to keep the current method of uploading .list files? Have other methods for updating been considered or is that being delayed pending full functionality?
2. When is it appropriate to begin adding routes in states/regions under development? (Based on recent posts, it appears this will be phased in over time).
3. What is the likelihood of current waypoints changing thus requiring revisions to existing .list files?
4. How will we be able to find the correct waypoints for new systems or if existing ones change? (currently, I'm still going to CHM to look at the highway browser).
5. What other pieces of the old CHM site will be incorporated? (I love Jim's new maps). I imagine the highway browser and traveler stats are high on the list of priorities. (Here is one part of CHM that I look at often and would like to see http://cmap.m-plex.com/stat/region.php?u=bejacob&c=usa&rg=oh&du=mi&sort=ra
6. Will concurrencies be automatically updated as they are on CHM now? (I think this is already happening on the maps, but I'm curious how it will show up on the stats--see link in the previous question).
7. Are there things that those of who are not programmers can do to help (other than send Jim .list files and cheer everyone on  )?

I'll take my stab at these and I'm sure others will as well.

1) I expect we'll have multiple convenient options to update .list files.  At this point, I'm willing to accept anyone's .list files by email (PM for address) as I am interested in testing at scale now.

2) I think this was addressed in a few places, but I believe the plan is to make many needed fixes to existing systems, then add in the nearly-complete systems, then look to expand.  Maintaining and improving quality should take precedence over expansion.

3-4) I expect we'll adopt the CHM model: try not to break .list files but when it's necessary, there will be a change log.

5) I expect we'll have all CHM functionality and more.  HB exists, just needs to look pretty.  Stats are almost in place, need to make them show up on end-user pages.  My overlay maps are good to go (need to add stats) and others are working on the static maps like CHM had.  I'm hoping to have a lot more (and more configurable) stats pages.

6) Concurrencies should all be accounted for correctly in maps and stats now.

7) Look for errors in the data, maps, stats, being produced so far.

[yakra]

1. Is the intent to keep the current method of uploading .list files?

Yes. For now, at the very least...

Have other methods for updating been considered or is that being delayed pending full functionality?

Considered, yes. Though I don't think very seriously or very in-depth yet. Most of that discussion has been here on AARoads, from what I remember. In any case, that would be something to think about after we have a fully functional site down.

2. When is it appropriate to begin adding routes in states/regions under development? (Based on recent posts, it appears this will be phased in over time).

Edit: I initially read this as "When is it appropriate to begin adding routes ... to my .list file?", which may or may not have actually been your intent. I wrote my reply accordingly. So, uhh... yeah. Nota Bene.
Whenever you want, really. If the system's not active yet, you'll just get a line in your log file such as "Ignoring line matching highway in inactive system: MT MT2 I-15/90 EliWarAve". It's just that you run the risk of, say, having to re-enter lines if waypoint labels change while the system is still in development. No big woof though.

3. What is the likelihood of current waypoints changing thus requiring revisions to existing .list files?

It does happen, from time to time. When labels change, we try to keep the old label, if it's in use, in the file as a deprecated label.
For example, ME ME123 ME24 needs to be relabeled as ME24Bus. When I make that change, ME24 will stay in the file as a deprecated label, so existing lists using that point will work and changes will not be necessary.
Other times, a waypoint label will move a bit. TX US80 TX34 became TX34Bus. TX34 couldn't stay in place as a deprecated label, as it was needed for the new TX34 two blocks away. So, anyone using the old TX34 point will still have that line of their list file processed OK, but it will point to the wrong point, the new TX34, and their stats/maps will be off by a tiny bit. A .list revision will be needed.
Sometimes, changes are needed that will break a .list file line, usually when a highway is relocated. For example, CT US7 CT25 is no longer a valid waypoint; anyone still using it will have an error in his .log file.
In these last 2 scenarios, these changes have traditionally bee listed on the Highway Data Updates page. The new project will need some kind of equivalent to this, but I haven't seen any discussion about it yet. I guess it's another fairly simple thing to get going after we have an operational site.

4. How will we be able to find the correct waypoints for new systems or if existing ones change? (currently, I'm still going to CHM to look at the highway browser).

The current thinking is to have the new site start out with the same data as on the existing site, so all waypoint labels should start out the same. We'll keep on with efforts to keep the breaking of .list files to a minimum, so waypoints shouldn't change unless necessary. For new systems, and changes to existing ones, I'd say, the new HB, once it's operational.

5. What other pieces of the old CHM site will be incorporated? (I love Jim's new maps). I imagine the highway browser and traveler stats are high on the list of priorities. (Here is one part of CHM that I look at often and would like to see http://cmap.m-plex.com/stat/region.php?u=bejacob&c=usa&rg=oh&du=mi&sort=ra

I can't imagine that not being high on the list.

6. Will concurrencies be automatically updated as they are on CHM now? (I think this is already happening on the maps, but I'm curious how it will show up on the stats--see link in the previous question).

I believe the intent is to have concurrencies automatically updated.

7. Are there things that those of who are not programmers can do to help (other than send Jim .list files and cheer everyone on  )?

Cheering us on is great! I would say, look for & report errors in the highway data, but I'd be getting ahead of myself; that's best saved for once the dust settles down and everything is online.

[bejacob]

Edit: I initially read this as "When is it appropriate to begin adding routes ... to my .list file?", which may or may not have actually been your intent. I wrote my reply accordingly. So, uhh... yeah. Nota Bene.
Whenever you want, really. If the system's not active yet, you'll just get a line in your log file such as "Ignoring line matching highway in inactive system: MT MT2 I-15/90 EliWarAve". It's just that you run the risk of, say, having to re-enter lines if waypoint labels change while the system is still in development. No big woof though.

That's exactly what I meant. Sorry about not being clear. I've got a separate working file with routes under development. I'll plan on adding them to my .list at some point, though I may wait until I'm sure I've corrected all the missing/incorrect routes in my current file.

Thank you for the response (and to all who replied).

[Jim]

More stats!  Per-route clinching stats are now in log files.  Please let me know if you notice problems.  My spot checks matched CHM for my routes except in cases where my .list has been updated or the data changed (like the NM 599 example mentioned upthread).  I'm unlikely to get the stats into the DB tonight but that's my next project, so we'll be able to see things on web pages.  Please remember that the stats in the logs aren't pretty nor are they intended to be.  It's just a convenient way to see them, and I'm tempted to leave them in there even when they're available through other means.  Or maybe they could be in a separate log file or something like that.

[sipes23]

More stats!  Per-route clinching stats are now in log files.  Please let me know if you notice problems.  My spot checks matched CHM for my routes except in cases where my .list has been updated or the data changed (like the NM 599 example mentioned upthread).

My log looked accurate based on spot checks. The mismatches I found were known mismatches due to updates and further travel.

[froggie]

MS: not officially in development. No significant obstacles to implementation.

Like Arkansas, there are several discontinuous segments.  And we'd have to figure out how to handle the MS 7xx/8xx/9xx routes, most of which are unsigned.

[bejacob]

More stats!  Per-route clinching stats are now in log files.

I like this addition. It's been a huge help discovering concurrencies I overlooked. I'm using these stats to add routes to my .list (for completeness sake). It's also interesting to see which routes are almost to 100% (i.e. ones I might want to clinch next time I'm in the area  ).

Since my CHM stats are way off, I haven't bothered comparing, though did a few spot check on routes that I know are correct in CHM. Everything looks good.

[oscar]

I like this addition. It's been a huge help discovering concurrencies I overlooked. I'm using these stats to add routes to my .list (for completeness sake).

If Travel Mapping is picking up, and automatically crediting you for, concurrent mileage, you don't need to add lines to your .list file just for the concurrences. Only if you also racked up non-concurrent mileage on a route is a new line necessary.

My own .list file is bloated out by the concurrence-only lines I added long ago when CHM did not at first automatically credit concurrences. I'm not adding any new ones, and if I'm cleaning up old entries for a state I'll try to remove the old concurrence-only lines.

While I'm on the road I've taken a brief glance at my log file to look for under- or over-mileage in the user stats, starting with the systems for which I'm expecting either 100% or 0% mileage in a (sub)region. So far, the only errors I've picked up are from errors in my own .list file, which I'll fix in a few weeks, and a coordinates mismatch in Quebec at the A-20/A-85 junction (which is also in CHM), which I'll fix when we start updating route files (A-85 needs a lot of other work too).

The user stats as they now stand give me one item of information I was particularly looking for, on my overall coverage of U.S. routes.

[Jim]

I believe CHM has a per-user concurrency detection report.  I never found it especially helpful so I went with a general concurrency detection shown in the log file.  If people think it would be useful, I should be able to transform that report from every concurrent segment (that is, every connection between two waypoints that's shared by at least 2 highways) to reporting entire subsets of the highways that are concurrent.  I don't think it's that hard, just hasn't been a priority yet.

[SSOWorld]

I believe CHM has a per-user concurrency detection report.  I never found it especially helpful so I went with a general concurrency detection shown in the log file.  If people think it would be useful, I should be able to transform that report from every concurrent segment (that is, every connection between two waypoints that's shared by at least 2 highways) to reporting entire subsets of the highways that are concurrent.  I don't think it's that hard, just hasn't been a priority yet.

I suggest waiting with that until later.  We have a decent algorithm in place now, but it might benefit later (how is the question)

[bejacob]

Per-route clinching stats are now in log files.  Please let me know if you notice problems. 

One route that popped up on my log is SC31 near Myrtle Beach. I can't figure out how. I drove US501 from into town from Conway and picked up US17 heading south. Perhaps there is a concurrency I'm missing  . Here's my SC map http://www.teresco.org/~terescoj/travelmapping/hbtest/mapview.php?rg=SC&u=bejacob in case someone wants to take a look and can explain how that extra teal colored line showed up near Myrtle Beach.

The rest of the state looks fine except for the gaps left where I drove on SC state highways not yet part of the database (which I'm perfectly fine with since I know the reason). I didn't notice this route (SC31) when I first glanced at my map, but found it while reviewing the per-route stats. (So Jim, they are being put to good use.  ).

[rickmastfan67]

Per-route clinching stats are now in log files.  Please let me know if you notice problems. 

One route that popped up on my log is SC31 near Myrtle Beach. I can't figure out how. I drove US501 from into town from Conway and picked up US17 heading south. Perhaps there is a concurrency I'm missing  . Here's my SC map http://www.teresco.org/~terescoj/travelmapping/hbtest/mapview.php?rg=SC&u=bejacob in case someone wants to take a look and can explain how that extra teal colored line showed up near Myrtle Beach.

The rest of the state looks fine except for the gaps left where I drove on SC state highways not yet part of the database (which I'm perfectly fine with since I know the reason). I didn't notice this route (SC31) when I first glanced at my map, but found it while reviewing the per-route stats. (So Jim, they are being put to good use.  ).

Did you add 'Future I-74' there to your list file?

[bejacob]

Did you add 'Future I-74' there to your list file?

It appears that I did. I knew it had to be something wrong in my list, but I couldn't track down which route was causing my problem. I'll get the offending Future I-74 removed. 

Thanks for the help.

[bejacob]

More stats!  Per-route clinching stats are now in log files.  Please let me know if you notice problems.  My spot checks matched CHM for my routes except in cases where my .list has been updated or the data changed (like the NM 599 example mentioned upthread).  I'm unlikely to get the stats into the DB tonight but that's my next project, so we'll be able to see things on web pages.  Please remember that the stats in the logs aren't pretty nor are they intended to be.  It's just a convenient way to see them, and I'm tempted to leave them in there even when they're available through other means.  Or maybe they could be in a separate log file or something like that.

The map overlays look great and are particularly helping in spotting errors (as I have found numerous times). The stats appear to be accurate. It looks as though the engine behind the new site is functioning properly.

So what comes next?

Adding new systems? (I saw several posts discussing which systems were nearest completion).
Getting stats into a web based report that users can easily read and understand?
Some other functionality?

Enquiring minds want to know.

[Jim]

For my part, I plan to get the computed stats into the DB, so they're available for viewing on web pages.  I was hoping this would happen tonight, but the evening is slipping away on me quickly...  I also hope I can get the site update program reading from the new directory organization, which will make it possible to archive the "chm_final" data and start making fixes and improvements.  As for new systems, I think it's best to get the current active systems up to date first before anything new is activated. 

Then I'll be traveling for a while and won't contribute much (but do hope to check in on the discussion when possible).

[Jim]

For those wondering, there's been some important behind-the-scenes progress made this week.  I think all of the stats I compute (including everything needed to support all of the stats and rankings CHM provided, unless I'm missing something) are now in the DB.  I haven't updated any web-facing code to use them, but I might tinker with that tonight or tomorrow night.  Or I might update the script to use the new directory organization first.

Some stats about the system so far:

• The Python program that analyzes the input files, generates logs and SQL takes about 3 1/2 minutes to run on my MacBook Pro.
• The SQL file contains 1,681,834 lines.  Moving to the server takes about a minute to compress (bzip2 -9), transmit, and uncompress.
• It takes mysql less than a minute and half to ingest that SQL file.  This is the only time when the site is not functional.

[Jim]

Well, before I do anything else, I need to fix up a problem I introduced that seems to show up as completely incorrect mapping results for state highway systems.  I'll post again if and when I can get it fixed.  In the meantime, ignore your maps...

[sipes23]

Well, before I do anything else, I need to fix up a problem I introduced that seems to show up as completely incorrect mapping results for state highway systems.  I'll post again if and when I can get it fixed.  In the meantime, ignore your maps...

And every system that isn't an interstate, so far as I can see. Interstate Business Loops are extremely odd. Like it's showing me that I've done an isolated chunk of I-25BLDou, which isn't in my .list at all yet. I'd have never caught it had I not been just *now* updating my .list file. After all, gotta log in the day's travel before I go and forget.

Edit to add: SSOWorld clears up the problem. (And I really appreciate all of the work that's gone in to an internet function I love.)

[SSOWorld]

That be the database crunching numbers

[Jim]

Should be good now.  I was relying on two DB queries to return things in the same order, and one stopped doing so with a recent change (dealing with connected routes).  Once I tracked it down, it was an easy fix to the DB query to force both to have the same ordering.  After tracking it down, I remembered worrying that I was relying on something that was probably not guaranteed, and sure enough, I was.

Let me know if you still see any errors.

[theFXexpert]

Should be good now.  I was relying on two DB queries to return things in the same order, and one stopped doing so with a recent change (dealing with connected routes).  Once I tracked it down, it was an easy fix to the DB query to force both to have the same ordering.  After tracking it down, I remembered worrying that I was relying on something that was probably not guaranteed, and sure enough, I was.

Let me know if you still see any errors.

In my Florida region map, there is a gap in I-75 between waypoint 240 and what I guess is supposed to be a hidden waypoint. A straight line also displays between its two endpoints. My stats appear to be inflated due to the error as well. (It should be ~49%)

[mapcat]

In my Florida region map, there is a gap in I-75 between waypoint 240 and what I guess is supposed to be a hidden waypoint. A straight line also displays between its two endpoints. My stats appear to be inflated due to the error as well. (It should be ~49%)

Looks like this is happening on all users' Florida maps.

[Jim]

In my Florida region map, there is a gap in I-75 between waypoint 240 and what I guess is supposed to be a hidden waypoint. A straight line also displays between its two endpoints. My stats appear to be inflated due to the error as well. (It should be ~49%)

Looks like this is happening on all users' Florida maps.

Bug squashed, thanks.  It's another place that I was relying on a non-guaranteed ordering of values coming out of the DB.  For some reason, it decided to provide I-75 FL starting with 240, down to 1, then back up to FL/GA then down the northern half.  So I forced it to sort within a route by the waypoint ids created when the DB is populated.  Something I should have done when I first implemented.

Keep any reports of odd behavior coming in.

[Purgatory On Wheels]

Is there a way to display a specific road in the maps?  Say I wanted to just look at what I'd driven on I-5.  Could I add something similar to CHM's "r=ca.i005"?