Technical/Design/Implementation Discussions (CHM/Travel Mapping)

[Bickendan]

Here's mine: https://github.com/Bickendan/CHM

[froggie]

Are you looking for smaller lists for this test, or a larger one?  I ask since both Oscar and I have two of the largest .list files in the project.

[oscar]

I'd like to get a handful of .list files I can use to experiment with populating my first attempt at a database of the travel mapping project data.  I've already picked a fairly small subset of the highway data, and I'll put in my own .list.  Rather than selecting some random ones, I figured I'd see who might like theirs to be part of initial development and testing.  Send CHM username if your .list is up-to-date on the old site, or email your .list directly to me (send private message on this board if you don't have my email).

You can use my old list file, on the CHM server. I'll also send an updated file by email, once my current round of travels are done next weekend, though it will be even more of a "stress test" than the old version (including some bad lines from travels on new roads not in CHM route sets).

[Jim]

Are you looking for smaller lists for this test, or a larger one?  I ask since both Oscar and I have two of the largest .list files in the project.

I don't think it should matter much.  Mine's also pretty large.  Part of this is to get a feel for whether the approach I'm trying has any chance to scale, so having some of the bigger files in there would be good.  Mainly, I just want to pick a few people's .list files who might have interest in seeing their data show up in early tests.

I'm hoping in the next few days to have a very basic highway browser up that can read data from the database.  Daytime work has to be real work, though, so I make no promises on timing.

[sipes23]

My .list isn't super current (and has a few bad lines which I'm leaving in for my purposes), but it is here:

http://cmap.m-plex.com/list/sipes23.list

If you need something more current, I can get it to you.

[vdeane]

Here's my current file: http://nysroads.com/files/aaroads/vdeane.list

[rickmastfan67]

My list is pretty much up-to-date on the CHM site if you want to use it for testing Jim.
http://cmap.m-plex.com/list/rickmastfan67.list

[Jim]

Si, or anyone else who's willing to fix, I noticed a bunch of the .wpt files (113 by my count) in the chm_final/usausb directory on Github contain an error message instead of the waypoint data.  Could you check it out?  A spot check shows the files do exist in the CHM data folder, and I can see them in a browser, but when I tried to grab them via wget with a script, I get the same error message.  I don't see any problems in any other directory.

Thanks.

[english si]

I see what's happened. The sheer size of the folder meant that my download-all program couldn't quite get there and started saving the files with error text in instead. Re-downloading the just over 100 files manually.

Given they are all at the end of the folder alphabetically, and usausb is the biggest folder, we should be alright. Other big folders, having checked, are fine.

PS: list file one and list file two (that simply shows green sign colour in the UK)

[english si]

OK, Github's GUI on my PC won't let me do anything (commit, make a new branch, etc, etc). It's the VA, VT, WA, WI, WV and WY files in usausb that have issues.

I've tried using the shell, but it's gobbledy-gook.

[Dr Frankenstein]

If you still need list files... mine is up to date (well, up to when updates stopped).

http://cmap.m-plex.com/list/drfrankenstein.list

[Jim]

Anyone aware of a .csv or other data file we can get our hands on from the old CHM project that has a list of active and in-development systems with their human readable name equivalents?  If not, anyone willing to make one?

[english si]

Something like this I just knocked up? Obviously the order, colors and fields can be changed to suit our needs and desires.

Code Select Expand

System;CountryCode;Name;Color;InDev
alba;ALB;Albania Motorways;blue;
auta;AUT;Austria Motorways;blue;
auts;AUT;Austria Expressways;teal;
bela;BEL;Belgium Motorways;blue;
biha;BIH;Bosnia and Herzegovina Motorways;blue;
bgra;BGR;Bulgaria Motorways;blue;
cannsf;CAN;Nova Scotia Provincial Freeways;blue;
canonf;CAN;Ontario Provincial Freeways;blue;
canqca;CAN;Quebec Freeways;blue;
cantch;CAN;Trans-Canada Highway;red;
cannb;CAN;New Brunswick Provincial Highways;brown;
cannst;CAN;Nova Scotia Trunk Routes;brown;
cannsc;CAN;Nova Scotia Collector Routes;yellow;
cannt;CAN;Northwest Territorial Highways;brown;
canon;CAN;Ontario King's Highways;brown;
canpe;CAN;Prince Edward Island Provincial Highways;brown;
canyt;CAN;Yukon Territorial Highways;brown;
chea;CHE;Switzerland Motorways;blue;
cypa;CYP;Cyprus Motorways;blue;
czed;CZE;Czech Republic Motorways;blue;
czer;CZE;Czech Republic Expressways;teal;
deua;DEU;Germany Motorways;blue;
eure;EUR;International E-roads;red;
espa;ESP;Spain National Motorways;blue;
espaca;ESP;Spain Select Autonomous Community Motorways;teal;
fraa;FRA;France Motorways;blue;
mtqa;FRA;Martinique Motorways;blue;
gbnm;GBR;Great Britain Motorways;blue;
gbnam;GBR;Great Britain A Motorways;blue;
gbna;GBR;Great Britain A Roads (Zones 2-5,7);brown;
nirm;GBR;Northern Ireland Motorways;blue;
niram;GBR;Northern Ireland A Motorways;blue;
nira;GBR;Northern Ireland A Roads;brown;
grca;GRC;Greece Motorways;blue;
hrva;HRV;Croatia Motorways;blue;
hunm;HUN;Hungary Motorways;blue;
imna;IMN;Isle of Man A Roads;brown;
irlm;IRL;Ireland Motorways;blue;
irln;IRL;Ireland N Roads;brown;
itaa;ITA;Italy Motorways;blue;
itasm;ITA;Italy Select Named Motorways;teal;
jamt;JAM;Jamaica Motorways;blue;
jeya;JEY;Jersey A Roads;brown;
luxa;LUX;Luxembourg Motorways;blue;
luxb;LUX;Luxembourg Expressways;teal;
mexd;MEX;Mexico Federal Expressways;blue;
mexed;MEX;Mexico State Expressways;teal;
nlda;NLD;Netherlands Motorways;blue;
pola;POL;Poland Motorways;blue;
pols;POL;Poland Expressways;teal;
prta;PRT;Portugal Motorways;blue;
rksr;RKS;Kosovo Motorways;blue;
roua;ROU;Romania Motorways;blue;
svkd;SVK;Slovakia Motorways;blue;
svkr;SVK;Slovakia Expressways;teal;
svna;SVN;Slovenia Motorways;blue;
svnh;SVN;Slovenia Expressways;teal;
turo;TUR;Turkey Motorways;blue;
usai;USA;United States Interstate Highways;blue;
usaib;USA;United States Business Interstate Highways;green;
usaif;USA;United States Future Interstate Highways;teal;
usansf;USA;United States Select Numbered State Freeways;teal;
usasf;USA;United States Select Named Freeways;teal;
usaus;USA;United States Numbered Highways;red;
usausb;USA;United States Auxillary Numbered Highways;magenta;
usaaz;USA;Arizona State Highways;brown;
usact;USA;Connecticut State Highways;brown;
usadc;USA;District of Columbia District Highways;brown;
usade;USA;Delaware State Highways;brown;
usahi;USA;Hawaii State Highways;brown;
usaia;USA;Iowa State Highways;brown;
usaid;USA;Idaho State Highways;brown;
usail;USA;Illinois State Highways;brown;
usaks;USA;Kansas State Highways;brown;
usaky;USA;Kentucky State Highways 1-999;brown;
usame;USA;Maine State Highways;brown;
usamd;USA;Maryland State Highways;brown;
usama;USA;Massachusetts State Highways;brown;
usami;USA;Michigan State Highways;brown;
usamn;USA;Minnesota State Highways;brown;
usamo;USA;Missouri State Highways;brown;
usane;USA;Nebraska State Highways;brown;
usanv;USA;Nevada State Highways;brown;
usanh;USA;New Hampshire State Highways;brown;
usanj;USA;New Jersey State Highways;brown;
usany;USA;New York State Highways;brown;
usanc;USA;North Carolina State Highways;brown;
usand;USA;North Dakota State Highways;brown;
usaoh;USA;Ohio State Highways;brown;
usaok;USA;Oklahoma State Highways;brown;
usaor;USA;Oregon State Highways;brown;
usapa;USA;Pennsylvania State Highways;brown;
usari;USA;Rhode Island State Highways;brown;
usawa;USA;Washington State Highways;brown;
usawv;USA;West Virginia State Highways;brown;
usawi;USA;Wisconsin State Highways;brown;
cannf;CAN;Canada Select Named Freeways;teal;yes
cansph;CAN;Canada Select Provincial Highways;brown;yes
gbna1;GBR;Great Britain A Roads (Zone 1);brown;yes
gbna6;GBR;Great Britain A Roads (Zone 6);brown;yes
gbna8;GBR;Great Britain A Roads (Zone 8);brown;yes
gbna9;GBR;Great Britain A Roads (Zone 9);brown;yes
usaak;USA;Alaska State Highways;brown;yes
usaca;USA;California State Highways;brown;yes
usafl;USA;Florida State Highways;brown;yes
usaky3;USA;Kentucky State Highways 1000-1499;brown;yes
usaky4;USA;Kentucky State Highways 1500-1999;brown;yes
usala1;USA;Louisiana State Highways 1-499;brown;yes
usamt;USA;Montana Primary State Highways;brown;yes
usanm;USA;New Mexico State Highways;brown;yes
usasc;USA;South Carolina State Highways;brown;yes
usaut;USA;Utah State Highways;brown;yes
usavt;USA;Vermont State Highways;brown;yes

[Jim]

Something like this I just knocked up?

Looks like it has the essentials to get us started, thanks.

[SSOWorld]

http://cmap.m-plex.com/list/master_son.list is my latest published list file.  Note that it has highways systems under development (such as CA and UT) and would produce errors if points are not matched.

[Jim]

I've grabbed a bunch of the .list files from some of those who contacted me to be a part of testing.  My program generates log files along the lines of those on the old CHM site, but with what I hope is some additional useful information.  For example, I report a note when an old route name is in the .list and I'm instead using the new canonical name.

My spot check of the logs have led to several improvements (better handling of empty lines, lines with extraneous whitespace, etc).  I am posting all of the logs from a recent run at

http://www.teresco.org/~terescoj/travelmapping/logs/

If you see anything unexpected there or have any other questions or suggestions, let me know.

One thing I noticed is that some .list files have the "*" entries for former waypoints included in waypoint names.  For both those and hidden points starting with "+", I was just matching .list entry waypoints with actual waypoint names with any leading "+" or "*" stripped off.  I think it has to be that way for "+" names, since we add a "+" to alternate old names in many cases (see all the +0 and +999 entries in the interstates).

I'm sure there's more to fix, but at least .list files are being parsed and lines are being matched with actual waypoints from actual routes.  Progress!

[rickmastfan67]

I saw "Note: replacing deprecated route name" in my log file.  So, I'll update my list file on my HD so that doesn't show up again in the future.

And Jim, might want to add a time code as to when the log file was created like what happens on the CHM site.  Lets people know when it was last checked.

[Duke87]

I like how "Ignoring line matching highway in inactive system" is now a separate error from "Unknown region/highway combo".

I'm actually in the gradual process of mapping my travels over the last two weeks using your highway browser. I'm liking the simplicity of the URL format since it makes it quicker to go from one highway to another simply by editing the URL rather than hitting back and then having to scroll and click through to other things.

[Dr Frankenstein]

I like how "Ignoring line matching highway in inactive system" is now a separate error from "Unknown region/highway combo".

I really like that too.

By the way, I think this has been requested in the original app before, but supporting comments in the file would be a great addition.

[Jim]

By the way, I think this has been requested in the original app before, but supporting comments in the file would be a great addition.

That's simple enough, and I'll add it.  I don't see why we can't support comments in .csv and .wpt files too.

Any objection to '#' starting single-line comments in any or all of the project's files?  I don't see need for direct multi-line comment support.

[Dr Frankenstein]

That's exactly what I had in mind, provided no routes or waypoints use the '#' character.

CSV is standardized with no support for comments, but some parsers stray from the RFC in that regard.

[SSOWorld]

As long as the .list file could support comments

[Jim]

I'm happy to report a lot more data processing progress, though none that produces any new visual results.  All colocated waypoints are now detected, which will allow me to detect concurrencies efficiently, which is, I think, the next thing on the list.  It's also what's needed to support the links to intersecting and continuing routes that are not in the draft highway browser yet.  I believe Tim used to have a tolerance to consider very nearby points colocated, but I am requiring exact coordinate matches.  If this requirement remains, it means some work to fix up some of the data, but much of that can be done automatically.  We'd just need a manual check to make sure no points are merged which were kept separate intentionally.

Tonight's work was also a big win for efficient data structures.  The brute force check to find waypoints with matching coordinates was on pace to take a few hours to complete.  By coding up and using a quadtree structure to hold the waypoints, the whole program is back down to about a minute of execution time.  FYI, the wpt files currently being loaded in contain 365,937 waypoints.  So pay attention in your data structures class.

[Dr Frankenstein]

So pay attention in your data structures class.

Indeed.

(As an aside, I'm tired of fixing my colleagues' O(n) and O(n²) algorithms at work. All of them have university degrees, including one with an engineer degree. I'm the only one with a community college degree. Pay attention in class, folks!</rant>)

[sammi]

So pay attention in your data structures class.

We never learned about quadtrees in my data structures class. Maybe next year I might.