I've been thinking about how to get started on something useful for the new travel mapping project, and it keeps coming back to having a database design. I'm not sure if it makes more sense to have the raw .csv, .wpt, and .list files built into a database directly and everything else works off the "raw" database, or if there should be some program that does some work on the data before it goes into the database. I'm not sure how the old CHM system handles all of this, but I'm confident that it at least operates everything from the database after an update runs to bring in new highway and user data. But I really don't know what that database looks like. For example, is concurrency detection done as the files are loaded into the database, or when that data is used to generate some stats or a map?
SQL Queries are what I think it worked off of (but having no knowledge of what it did, I am making a guess.)
SQL (likely MySQL in this case) is a language that is used to build, search, populate, whatever you do to... database data tables in said free DBMS or similar (SQLite, MSSQL Server, etc). PHP or any other language can be used to write SQL queries on the fly and access the data for read/write purposes. (lot of reasons for PHP being vulnerable to security risks is its ease of use for this.)
The database appears to be a cache of the raw files allowing everything to be displayed online without processing everything whenever someone views a page. We'll probably want to make tables that can hold what we're using to get the JavaScript canvas maps to work as well as the highway browser, stats, and narrowing down the map view.
That makes sense. So the database would have everything in all forms needed for the various pages (maps, stats, highway browser). Meaning the "site update" programs do more processing to take all of the .csv. .wpt, and .list entries and store them into the database in whatever form(s) is/are convenient for the various web-facing components.
I'm thinking about taking a stab at writing some code, either in Java or Python, which would read in all of the data files and get at least a subset of it into a database ready for someone's web-facing code to read. Anyone want to request a table and fields needed for one of those?
Do I remember someone mentioning that they'd grabbed at least a subset of current .list files? I suppose it's not hard to write something to grab them but if it's done already no sense redoing. I already have scripts that pull down the current data folder contents, which unfortunately are probably unchanged since the last time I pulled it all down last summer.
I've just created a Github organization called TravelMapping that I propose we use to manage the project. Github's organization structure seems to give us the level of control we'd need in terms of allowing different groups of users different level of access to the repositories. See https://github.com/blog/674-introducing-organizations for some information about it. It's free as long as we keep the project open source, which is the intent anyway from what I gather from our conversations to date.
I think we'll want just a few repositories within the organization. It seems to make sense to me to have separate repositories for the "site update" code, the web site, the highway data, and user .list files. I haven't created any of these yet. I think we'd have different Github teams within the organization who have different levels of access to all of this. No matter what, anyone would be able to fork any part of the project, but would not be able to commit/push (I really need to get up to speed on the correct git/Github terms) unless part of the team with write access to a particular repository.
For now, I'd like to invite those who would like to be involved in the initial implementation efforts (site update data processing, web site development) to join the organization. You'll need a Github account if you don't already have one (also free). Send me your Github username and I can add you to our organization.
Once we get good at it, we can invite those whose main contributions will be through highway data updates, and finally, should we decide to use Github as a mechanism for .list updates, end user accounts could be added to the organization.
Quote from: Jim on April 05, 2015, 01:28:44 PM
Do I remember someone mentioning that they'd grabbed at least a subset of current .list files? I suppose it's not hard to write something to grab them but if it's done already no sense redoing.
You could also ask people to send in the list files they've been updating over the last few months. My large file has grown larger in that time, and so could be an even better "stress test" of new code.
Quote from: Jim on April 05, 2015, 02:25:45 PM
No matter what, anyone would be able to fork any part of the project, but would not be able to commit/push (I really need to get up to speed on the correct git/Github terms) unless part of the team with write access to a particular repository.
"Push" is the correct git term. It matches "pull", which takes the current remote repository and merges its changes into your working branch.
Quote from: Jim on April 05, 2015, 02:25:45 PM
For now, I'd like to invite those who would like to be involved in the initial implementation efforts (site update data processing, web site development) to join the organization. You'll need a Github account if you don't already have one (also free). Send me your Github username and I can add you to our organization.
My github username is the same as the one here.
Nick
I just created a github account with the same username as here. I'm willing to offer my .list file for testing, though there is the question of whether or not I should add the Vermont state highways to my main file (they're currently in a separate file); the current prototype for the maps has them in.
I have a GitHub account with the same username.
go ahead and ad me to the organization, same username as here but all lower case.
Quote from: vdeane on April 05, 2015, 09:36:25 PM
I'm willing to offer my .list file for testing, though there is the question of whether or not I should add the Vermont state highways to my main file (they're currently in a separate file); the current prototype for the maps has them in.
New Mexico, Utah, and Montana state routes, all of which have gone through peer review, seem closer to ready to add to the list of mappable route sets. I have draft list file entries for all three as well as Vermont.
But I suggest that for system testing purposes, we have plenty of existing state route sets to work with, including for starters Jim's large and complex New York route set. So we can wait to add others to the mappable list, the prototype system, and to user list files until we have the basic system up and running.
I was not planning on getting a github account, at least right away, given my total ignorance of github or other aspects of programming. There are things I could be doing while the system is set up, but not sure any of it would require github access.
I've sent Github invitations to join the organization to the three who posted here so far.
In the old CHM system, there was no harm (other than a whole bunch of warnings in the log as in http://cmap.m-plex.com/trav/terescoj.log.txt) to having in-development or even nonexistent routes in your .list, and I hope that will continue to be the case. In addition to keeping my own travels tracked for in-development systems, I also added what I expected entries would be called for traveled segments in regions where no work had yet started so I would remember where I had been if and when the project expanded to those regions.
Quote from: Jim on April 05, 2015, 02:25:45 PM
For now, I'd like to invite those who would like to be involved in the initial implementation efforts (site update data processing, web site development) to join the organization. You'll need a Github account if you don't already have one (also free). Send me your Github username and I can add you to our organization.
Same exact user name as here Jim for Github.
Set up, same user name.
I've just registered with github as si404.
For a stress test for .wpt file size, the CA1 file is the biggest (49kB), beating things like Norway's E6 and Russia's E105 (my redone file that has no subregions is 'only' 45kB).
Quote from: Jim on April 05, 2015, 01:28:44 PM
I'm thinking about taking a stab at writing some code, either in Java or Python, which would read in all of the data files and get at least a subset of it into a database ready for someone's web-facing code to read. Anyone want to request a table and fields needed for one of those?
I suggest Python. Java often tries to use end-user computer clients to run and hogs resources.
Quote from: SSOWorld on April 06, 2015, 06:29:53 AM
Quote from: Jim on April 05, 2015, 01:28:44 PM
I'm thinking about taking a stab at writing some code, either in Java or Python, which would read in all of the data files and get at least a subset of it into a database ready for someone's web-facing code to read. Anyone want to request a table and fields needed for one of those?
I suggest Python. Java often tries to use end-user computer clients to run and hogs resources.
I have a first pass at a Python program that can read in .csv and .wpt data now in Github. I'm a much more experienced Java programmer than Python, but I think it makes a lot of sense to use Python here and it's a good excuse for me to get better at it.
Quote from: english si on April 06, 2015, 06:22:18 AM
I've just registered with github as si404.
For a stress test for .wpt file size, the CA1 file is the biggest (49kB), beating things like Norway's E6 and Russia's E105 (my redone file that has no subregions is 'only' 45kB).
The current version of CA 1 is a good stress test. But I've already started on shrinking that route file, which should make it less "stress"-ful even if we don't end up chopping it up due to relinquishments in Santa Monica and a few other coastal cities.
Quote from: Jim on April 06, 2015, 09:23:36 AM
Quote from: SSOWorld on April 06, 2015, 06:29:53 AM
Quote from: Jim on April 05, 2015, 01:28:44 PM
I'm thinking about taking a stab at writing some code, either in Java or Python, which would read in all of the data files and get at least a subset of it into a database ready for someone's web-facing code to read. Anyone want to request a table and fields needed for one of those?
I suggest Python. Java often tries to use end-user computer clients to run and hogs resources.
I have a first pass at a Python program that can read in .csv and .wpt data now in Github. I'm a much more experienced Java programmer than Python, but I think it makes a lot of sense to use Python here and it's a good excuse for me to get better at it.
:bigass:
I have this philosophy about being a software engineer/computer scientist
"As you gain extensive experience in one language, the others will catch up to you"
I work mostly in and have extensive experience in C, C++ and C#, but I have (with some help from the manuals) been able to work other languages regularly. My biggest challenges over the last year was picking up a language that's a hodgepodge of BASIC and C that's proprietary to a robot and being able to work with it and also work with a Flow chart driven program that had C-like back end language in it (also proprietary to a controller)
but that's enough off-topic matter.
Regardless, I can be of good help to you - however as mentioned before - I will have to work around my work schedule (and for the next two months - house buying related stuff).
I left my GitHub username above (it's not the same username as for CHM, but I figure it should work)
I've been thinking about an organizational hierarchy for highways, and I have a proposal for how to do it:
At the top you have "Countries", which represent countries and international route systems (eg E-Routes).
Within Countries, there are "Networks", which represent either regional systems (such as state highway networks) or national networks (such as Motorways, Interstates, or the TCH)
Within Networks, there are "Systems", which represent individual systems within a network (such as state routes and parkways within a state) or different classifications of routes within a national network (such as mainline versus bannered US Highways)
Within Systems, we have the individual highways, which are composed of wpt segments.
Each folder would contain a metadata.json file which would contain some information about user-facing text, as well as potential support for translations. An example for a country would be:
{
"abbr":"US"
"name:"United States"
"name_en":"United States"
"name_es":"Estados Unidos"
}
Each route would contain its wpt segments and a shield.png file which would be the shield used for the route, as well as a modified metadata.json which would contain information about individual segments of the route. A route metadata.json would look somewhat like this:
{
"name": "Interstate 74",
"segments": [
{
"region": "IA",
"num":0,
"mileage": 15,
"waypoints": "ia.wpt"
},
{
"region": "IL",
"num":1,
"mileage": 220,
"waypoints":"il.wpt"
},
{
"region": "IN",
"num":2,
"mileage": 180,
"waypoints":"in.wpt"
},
{
"region": "OH",
"num":3,
"mileage": 35,
"waypoints": "oh.wpt"
},
{
"region": "NC",
"num": 4,
"locale": "Mt. Airy, NC",
"mileage": 15,
"waypoints": "nc_mtairy.wpt"
},
{
"region": "NC",
"num": 5,
"locale": "Ellerbe, NC",
"mileage": 70,
"waypoints": "nc_ellerbe.wpt"
},
{
"region": "NC",
"num": 6,
"locale": "Lumberton, NC",
"mileage": 15,
"waypoints": "nc_lumberton.wpt"
},
]
}
The 'locale' field would be used in the case of routes with multiple segments within a single region.
Here's an example tree for this proposed hierarchy (obviously not complete):
highway
├── CN
├── EU
├── UK
└── US
├── AK
├── AL
├── I
│ ├── bus
│ │ ├── 95
│ │ │ ├── ga_savannah.wpt
│ │ │ ├── metadata.json
│ │ │ ├── nc_fayetteville.json
│ │ │ └── shield.png
│ │ └── metadata.json
│ ├── future
│ │ └── 74
│ │ ├── metadata.json
│ │ ├── nc_laurinburg.wpt
│ │ ├── nc_pilotmtn.wpt
│ │ └── shield.png
│ ├── main
│ │ ├── 2
│ │ │ ├── metadata.json
│ │ │ ├── shield.png
│ │ │ └── tx.wpt
│ │ ├── 4
│ │ │ ├── fl.wpt
│ │ │ ├── metadata.json
│ │ │ └── shield.png
│ │ ├── 5
│ │ │ ├── ca.wpt
│ │ │ ├── metadata.json
│ │ │ ├── or.wpt
│ │ │ ├── shield.png
│ │ │ └── wa.wpt
│ │ └── metadata.json
│ └── metadata.json
├── KY
│ ├── banner
│ ├── main
│ ├── metadata.json
│ └── parkway
├── metadata.json
├── named_freeways
└── US
├── banner
├── main
│ ├── 1
│ │ ├── ct.wpt
│ │ ├── dc.wpt
│ │ ├── fl.wpt
│ │ ├── ga.wpt
│ │ ├── ma.wpt
│ │ ├── md.wpt
│ │ ├── metadata.json
│ │ ├── me.wpt
│ │ ├── nc.wpt
│ │ ├── nh.wpt
│ │ ├── nj.wpt
│ │ ├── ny.wpt
│ │ ├── pa.wpt
│ │ ├── ri.wpt
│ │ ├── sc.wpt
│ │ ├── shield.png
│ │ └── va.wpt
│ └── metadata.json
└── metadata.json
Thoughts?
Quote from: oscar on April 06, 2015, 09:51:26 AM
Quote from: english si on April 06, 2015, 06:22:18 AM
I've just registered with github as si404.
For a stress test for .wpt file size, the CA1 file is the biggest (49kB), beating things like Norway's E6 and Russia's E105 (my redone file that has no subregions is 'only' 45kB).
The current version of CA 1 is a good stress test. But I've already started on shrinking that route file, which should make it less "stress"-ful even if we don't end up chopping it up due to relinquishments in Santa Monica and a few other coastal cities.
About those relinquishments... since those cities are supposed to maintain guide banners directing motorists to the next segment (even if they don't, which is usually the case and against the route definitions), I'd argue that we shouldn't break up CA highways that have turnbacks in the middle of the route. Otherwise we'd have to also cut the routes down wherever they overlap with another for consistency (eg, CA 99 along US 50 and I-5 in Sacramento).
---
From a mapping standpoint, would instead of using two colors for traveled and untravelled, we use a single color and change the line thickness work?
Quote from: Thing 342 on April 06, 2015, 10:59:02 AM
I've been thinking about an organizational hierarchy for highways, and I have a proposal for how to do it:
At the top you have "Countries", which represent countries and international route systems (eg E-Routes).
<snip>
Thoughts?
We currently have the following hierarchy for systems with (with the exception of E roads) the map showing colours higher up the hierarchy:
phase 1 (blue): Interstates, QC Autoroutes, NS freeways, ON freeways, MEX federal expressways, European 'Motorways' systems
phase 1a (teal): European 'Expressways' systems, Ax(M), Select Named Motorways, Select Numbered State/Provincial/Autonomous Community Freeways
phase 1b (yellow): Future Interstates
phase 1c (green): Business Interstates
phase 2 (red): US routes, Trans-Canada Highway, E roads (mapped below phase 3)
phase 2a (orange): Bannered US routes
phase 3 (brown): British Isles A roads/N Roads, state/provincial/territorial/district highways
I should point out that that phase 1 and 2 (even E roads) are shown at higher zoom outs and have thicker lines.
I'd make a continental split, with possibly a split of North America into two (Canada-USA and Caribbean-Central America) and move future interstates, giving (with the current in-browser systems):
Europe
phase 1 - blue (motorway systems)
Albania Motorways: alba
Austria Motorways: auta
Belgium Motorways: bela
Bosnia and Herzegovina Motorways: biha
Bulgaria Motorways: bgra
Switzerland Motorways: chea
Cyprus Motorways: cypa
Czech Republic Motorways: czed
Germany Motorways: deua
Spain National Motorways: espa
France Motorways: fraa
Great Britain Motorways: gbnm
Northern Ireland Motorways: nirm
Greece Motorways: grca
Croatia Motorways: hrva
Hungary Motorways: hunm
Ireland Motorways: irlm
Italy Motorways: itaa
Luxembourg Motorways: luxa
Netherlands Motorways: nlda
Poland Motorways: pola
Portugal Motorways: prta
Kosovo Motorways: rksr
Romania Motorways: roua
Slovakia Motorways: svkd
Slovenia Motorways: svna
Turkey Motorways: turo
phase 1a - teal (expressway systems, select expressways)
Austria Expressways: auts
Czech Republic Expressways: czer
Spain Select Autonomous Community Motorways: espaca
Great Britain A Motorways: gbnam
Northern Ireland A Motorways: niram
Italy Select Named Motorways: itasm
Luxembourg Expressways: luxb
Poland Expressways: pols
Slovakia Expressways: svkr
Slovenia Expressways: svnh
phase 2 - red (continental systems)
International E-roads: eure
phase 3 - brown (national systems)
Great Britain A Roads (Zones 2-5, 7)
(Great Britain A Roads (Zone 1): gbna1)
(Great Britain A Roads (Zone 6): gbna6)
(Great Britain A Roads (Zone 8): gbna8)
(Great Britain A Roads (Zone 9): gbna9)
Northern Ireland A Roads: nira
Ireland N Roads: irln
Isle of Man A Roads: imna
Jersey A Roads: jeya
North America
phase 1 - blue (motorway systems)
Nova Scotia Provincial Freeways: cannsf
Ontario Provincial Freeways: canonf
Quebec Freeways: canqca
Martinique Motorways: mtqa
Jamaica Motorways: jamt
Mexico Federal Expressways: mexd
United States Interstate Highways: usai
phase 1a - teal (expressway systems, select expressways)
(Canada Select Named Freeways: cannf)
Mexico State Expressways: mexed
United States Future Interstate Highways: usaif
United States Select Named Freeways: usasf
United States Select Numbered State Freeways: usansf
phase 1b - green (phase 1 business systems)
United States Business Interstate Highways: usaib
phase 2 - red (continental systems)
Trans-Canada Highway: cantch
United States Numbered Highways: usaus
phase 2a - orange (bannered continental systems)
United States Auxillary Numbered Highways: usausb
phase 3 - brown (state/provincial/etc systems)
New Brunswick Provincial Highways: cannb
Nova Scotia Trunk Routes: cannst
Northwest Territorial Highways: cannt
Ontario King's Highways: canon
Prince Edward Island Provincial Highways: canpe
(Canada Select Provincial Highways: cansph)
Yukon Territorial Highways: canyt
(Alaska State Highways: usaak)
Arizona State Highways: usaaz
(California State Highways: usaca)
Connecticut State Highways: usact
District of Columbia District Highways: usadc
Delaware State Highways: usade
Hawaii State Highways: usahi
(Florida State Highways: usafl)
Iowa State Highways: usaia
Idaho State Highways: usaid
Illinois State Highways: usail
Kansas State Highways: usaks
Kentucky State Highways 1-999: usaky
(Kentucky State Highways 1000-1499: usaky3)
(Kentucky State Highways 1500-1999: usaky4)
(Louisiana State Highways 1-499: usala1)
Maine State Highways: usame
Maryland State Highways: usamd
Massachusetts State Highways: usama
Michigan State Highways: usami
Minnesota State Highways: usamn
Missouri State Highways: usamo
(Montana Primary State Highways: usamt)
Nebraska State Highways: usane
Nevada State Highways: usanv
New Hampshire State Highways: usanh
New Jersey State Highways: usanj
(New Mexico State Highways: usanm)
New York State Highways: usany
North Carolina State Highways: usanc
North Dakota State Highways: usand
Ohio State Highways: usaoh
Oklahoma State Highways: usaok
Oregon State Highways: usaor
Pennsylvania State Highways: usapa
Rhode Island State Highways: usari
(South Carolina State Highways: usasc)
(Utah State Highways: usaut)
(Vermont State Highways: usavt)
Washington State Highways: usawa
West Virginia State Highways: usawv
Wisconsin State Highways: usawi
phase 3a - yellow (state/provincial/national secondary systems)
Nova Scotia Collector Routes: cannsc
With obvious scope for more in each section, addition continents, etc.
When it comes to subregions, Canada, USA, Mexico, UK, Russia and Kazakhstan are split into subregions. France's DOM-TOMs are separate subregions too (and presumably other territories that have ISO-3166 codes), but the regional splits in 'Metropolitan France' aren't made. I'd like to remove Russian/Kazakh subregions and have done so on my files (this will affect about two users). I'd also like the deal with the inconsistent division of the UK, but not (say) Spain, but that would affect too many users to merge all gbn files from eng, sct and wls ones. I personally think that Crimea needs to become a subregion of Ukraine (like the French examples) due to being
de facto part of Russia. I'd also put EUR and NAM super-regions above the region level, which would help with the multiple continents issue.
Quote from: Bickendan on April 06, 2015, 11:06:55 AMAbout those relinquishments... since those cities are supposed to maintain guide banners directing motorists to the next segment (even if they don't, which is usually the case and against the route definitions), I'd argue that we shouldn't break up CA highways that have turnbacks in the middle of the route. Otherwise we'd have to also cut the routes down wherever they overlap with another for consistency (eg, CA 99 along US 50 and I-5 in Sacramento).
I cannot think of anything more frustrating than road numbering by who maintains it (cf France and Norway where their 00s detrunkings caused lots of renumbering, and compare to the UK's detrunking of the same era, which caused zero renumbering!). Even if it isn't explicitly signed through a city, and merely implied, I'd keep it!
QuoteFrom a mapping standpoint, would instead of using two colors for traveled and untravelled, we use a single color and change the line thickness work?
It's harder to see. I'd go with no.
Quote from: english si on April 06, 2015, 12:22:51 PM
Quote from: Bickendan on April 06, 2015, 11:06:55 AMAbout those relinquishments... since those cities are supposed to maintain guide banners directing motorists to the next segment (even if they don't, which is usually the case and against the route definitions), I'd argue that we shouldn't break up CA highways that have turnbacks in the middle of the route. Otherwise we'd have to also cut the routes down wherever they overlap with another for consistency (eg, CA 99 along US 50 and I-5 in Sacramento).
I cannot think of anything more frustrating than road numbering by who maintains it (cf France and Norway where their 00s detrunkings caused lots of renumbering, and compare to the UK's detrunking of the same era, which caused zero renumbering!). Even if it isn't explicitly signed through a city, and merely implied, I'd keep it!
An interesting discussion for later. I'll just note that California is very different from Vermont. Vermont's "town" routes are part of its state highway system, even if maintained and signed differently. California's relinquished segments have been explicitly removed from the state highway system, even if some of the affected communities are for now trying to have it both ways (keeping old state highway signage, while making "improvements" that would not be allowed on state highways).
I don't think it would be inconsistent to preserve implied concurrencies between a state route and another state highway (such as several where CA 1 continues along US 101), while chopping up routes where segments have been removed from the state highway system altogether.
Quote from: english si on April 06, 2015, 12:22:51 PMEven if it isn't explicitly signed through a city, and merely implied, I'd keep it!
I'm guessing there will be a lot of sentiment in favor of this for the new project. We don't want routes that are clearly gone (unless we some time added historical routes, which would certainly be possible) but in those cases where the route is still obvious and for all reasonable purposes is the route, I would be in favor of keeping. I always thought Vermont's town-maintained highways were an obvious case for inclusion, but something like CA 1 would makes sense to me to be included as a continuous route.
One-way alignments: I'm looking at CA 160 in downtown Sac'to and that mess that is RI 15, 114 and others (especially that self-wrong way overlap!)... Keep the general centerline 'trace' or eventually account for the alignment splits? While I personally like the idea of accounting for splits (and it may be needed on transit tracking if/when we look into it), I also realize that it could be a headache to do (moreso than RI's already given us...).
For one-way - that's quite a hassle. It's bad enough that we [have to] do split interstates ;)
Quote from: Bickendan on April 06, 2015, 02:40:28 PM
One-way alignments: I'm looking at CA 160 in downtown Sac'to
Conveniently (or not), that part of CA 160 has been relinquished to the city of Sacramento. That removes from the state highway system the pair of widely-separated one-way alignments within the city, but also breaks the route into two segments, a short one mostly north of city limits, and a longer one south of the city.
Pick your poison, huh?
Quote from: oscar on April 06, 2015, 03:37:28 PM
Quote from: Bickendan on April 06, 2015, 02:40:28 PM
One-way alignments: I'm looking at CA 160 in downtown Sac'to
Conveniently (or not), that part of CA 160 has been relinquished to the city of Sacramento. That removes from the state highway system the pair of widely-separated one-way alignments within the city, but also breaks the route into two segments, a short one mostly north of city limits, and a longer one south of the city.
Pick your poison, huh?
Relinquished portion should remain, per route definition's reliquished segment clause: "For the relinquished former portion of Route 160, the City of Sacramento shall maintain signs directing motorists to the continuation of Route 160."
The fact that CalTrans or the cities are taking down said signs is very, very annoying when they're not supposed to.
Quote from: JimDo I remember someone mentioning that they'd grabbed at least a subset of current .list files?
I have all the current .list files archived.
I suggest, barring a compelling reason, keeping .list files out of Github. Seems to add an unnecessary level of complexity to usage, and would discourage casual travelers.
Quote from: oscarI'll just note that California is very different from Vermont. Vermont's "town" routes are part of its state highway system, even if maintained and signed differently.
Didn't froggie once say, on the old CHM forum, that "One size does not fit all"? I think that applies here. There's an aspect to roadgeeking, and identifying what makes a State Route System, that's not unlike chicken sexing or WW2 plane spotting... Something readily & easily understood by those with experience and knowledge looking at the big picture, that can't be succinctly pinned down in a few words to be easily grokked by newer observers. In Vermont's case, I always thought it appropriate to look at the routes within the framework of local maintenance and Urban Compacts as it exists throughout northern New England...
One thing I hope we can do with the new site is merge highways that are in multiple states properly.
http://cmap.m-plex.com/stat/topstats.php?u=rickmastfan67#ltl
For example, go down to #11 on that list for I-26. There should be a way to include I-26 in Tennessee (and northern NC) since it's part of the same route, and WILL be connected in the future. Heck, maybe we can merge in the NC Future I-26 with it (and show both shields) for the 'top stats' pages. Also on the individual page that I-26 links to (here (http://cmap.m-plex.com/stat/highway.php?u=rickmastfan67&r=nc.i026), and the other part here (http://cmap.m-plex.com/stat/highway.php?u=rickmastfan67&r=tn.i026)). I never agreed with Tim with his reasoning for not somehow combining routes together. I do hope we can rectify this with the new site.
However, I-74 NC could be a problem doing this too. But, there should be some way to have all the Future and Normal segments linked together for the Top Lists.
Or multiple segments. On the current CHM, you have to do math to get an overall percentage on routes like NY 17. Perhaps we could add logic to combine them all into one non-continuous entry, with the old segment names still recognized to maintain backwards compatibility?
Quote from: vdeane on April 07, 2015, 07:10:55 PM
Or multiple segments. On the current CHM, you have to do math to get an overall percentage on routes like NY 17. Perhaps we could add logic to combine them all into one non-continuous entry, with the old segment names still recognized to maintain backwards compatibility?
http://cmap.m-plex.com/stat/highway.php?u=yakra&r=ny.ny017
At the moment, only jim can add stuff to the Github project (or whatever the term is) - that surely needs fixing.
Right, that's definitely not good. I'd like us to be able to gather all of the work everyone's done so far as well as ongoing projects in repositories there. If you'd like to be able to create a repository within our github organization or be able to modify an existing repository within, let me know and I'll add that access. Once we're a bit more organized, I or someone else can create the repositories for the "official" parts of the project that are intended to lead to production versions.
For now, I have three repositories that I think of as work areas: CoreDataProcessing, which so far just has the Python code I wrote to parse .csv and .wpt files last week, and MapGeneration and Web, where I propose we put work toward, well, map generation, and web-facing parts of the project (maybe a new Highway Browser).
But for now, drop me a note about what level of access you want to have within the organization and I'll set it up. I want to make sure that at least two or three people can do pretty much everything so we don't end up back in a bad situation. I'll add people to teams and give teams appropriate levels of access to repositories (and would like to make at least one person, probably two, also an "owner").
do we need the raw data on there, or is it just for the stuff that deals with the data?
In order to process the data it should be there. I would have to guess that it should reside in the CoreDataProcessing repo.
See PM for my requests.
Full access to the repositories - my recommendation would be the head of the project the person setting up the web hosting for sure. Have these been included in your recommendation of number of those with full access?
I'd like a separate repository within the organization for data. I think it would be great if we could start bringing together all of the data we're confident is ready to go. The way I envision it (and please feel free to say so if this doesn't make sense) is that someone working on the data processing part of this would have clones of both of those repositories to be able to do their work. And someone whose focus is on the highway data itself would only clone that one.
I agree that whoever ends up setting up and managing the web hosting would likely get full admin access to the organization's repositories also.
Does anyone want to suggest a different organization for data than what was done in the old CHM project? There, we had a directory with lots of subdirectories: one with all of the .csv files for all of the systems, then a subdirectory for each of those systems with all of that system's .wpt files.
I'd also like us to consider whether it makes sense to have a new .wpt file format without all of the overhead of the full (now old-style, I believe) OSM URL. It makes more sense to me to use this point to have things much simpler: the latitude and longitude and numbers only, followed by a list of one or more waypoint names. I suggest to reverse the order as part of this only to simplify parsing, but it's not a big problem either way (or to remain as is, for that matter). Of course, a new waypoint editor, when available, would be able to work with whatever format we choose.
It would probably be a good idea for people to be able to create branches and push to branches they create. The core crew could test submitted code and merge the branches into the main one.
Quote from: Jim on April 11, 2015, 02:47:55 PM
Does anyone want to suggest a different organization for data than what was done in the old CHM project? There, we had a directory with lots of subdirectories: one with all of the .csv files for all of the systems, then a subdirectory for each of those systems with all of that system's .wpt files.
I found the proliferation of subdirectories, some appearing to be duplicative, horribly confusing to anyone except Tim. When downloading files, I sometimes could not be sure I was grabbing the right ones, and of course we'd have the problem of making sure uploaded (or "pushed") files go to the right place. Simplification of the directory structure would really help.
Quote from: Jim on April 11, 2015, 02:47:55 PM
I'd also like us to consider whether it makes sense to have a new .wpt file format without all of the overhead of the full (now old-style, I believe) OSM URL. It makes more sense to me to use this point to have things much simpler: the latitude and longitude and numbers only, followed by a list of one or more waypoint names. I suggest to reverse the order as part of this only to simplify parsing, but it's not a big problem either way (or to remain as is, for that matter). Of course, a new waypoint editor, when available, would be able to work with whatever format we choose.
So long as there's an easy way to convert from the old format to the new one, in bulk (with thousands of old-style route files out there). We might be forced to do that anyway, if the old-format OSM URLs cease to function.
Quote from: oscar on April 11, 2015, 08:36:56 PM
So long as there's an easy way to convert from the old format to the new one, in bulk (with thousands of old-style route files out there). We might be forced to do that anyway, if the old-format OSM URLs cease to function.
That should be possible using Notepad++. Just open a ton a files in it and do a 'Replace All' for all open documents. Just select the URL parts that are identical in each file and leave the 'Replace with' part completely blank. If needed, I could create a guide to do it w/ images.
Quote from: rickmastfan67 on April 11, 2015, 11:42:42 PM
Quote from: oscar on April 11, 2015, 08:36:56 PM
So long as there's an easy way to convert from the old format to the new one, in bulk (with thousands of old-style route files out there). We might be forced to do that anyway, if the old-format OSM URLs cease to function.
That should be possible using Notepad++. Just open a ton a files in it and do a 'Replace All' for all open documents. Just select the URL parts that are identical in each file and leave the 'Replace with' part completely blank. If needed, I could create a guide to do it w/ images.
Better yet, a nearly-trivial program could do a mass conversion of existing files before they're added to the new system, and that program would be available to anyone who has files not yet on the old CHM site. I'm happy to write and run such a program if and when we agree on a new format.
As a CHM user who has closely followed the threads here and eagerly anticipates the return of the project, I would love to see regular (weekly?) updates about the status of the new site. Nothing too detailed, just something along the lines of "we've got X working, and now we're in the process of doing Y, which may take a few weeks because of...". Maybe here, maybe in a dedicated topic.
Also, it's not been made clear yet whether people with no programming experience would be of use to the project. I'd like to help, but haven't made an account on Github because (a) I don't know what that is and (b) I don't know what skills must be possessed in order to participate there effectively.
Thanks for the work you all have put in so far.
Quote from: vdeane on April 11, 2015, 04:53:58 PM
It would probably be a good idea for people to be able to create branches and push to branches they create. The core crew could test submitted code and merge the branches into the main one.
The best way to do this would probably be to have the repo set up as a fork-and-pull system, where anyone can fork and make their own changes, but any changes to the master system would be made through a pull request to one of the project maintainers. This would allow for crowdsourcing of the data, while still maintaining data consistency by requiring approval of changes.
Quote from: oscar on April 11, 2015, 08:36:56 PM
Quote from: Jim on April 11, 2015, 02:47:55 PM
Does anyone want to suggest a different organization for data than what was done in the old CHM project? There, we had a directory with lots of subdirectories: one with all of the .csv files for all of the systems, then a subdirectory for each of those systems with all of that system's .wpt files.
I found the proliferation of subdirectories, some appearing to be duplicative, horribly confusing to anyone except Tim. When downloading files, I sometimes could not be sure I was grabbing the right ones, and of course we'd have the problem of making sure uploaded (or "pushed") files go to the right place. Simplification of the directory structure would really help.
I agree. I think a more hierarchical structure would be beneficial, as opposed to the somewhat arbitrary phase system in CHM.
Quote from: Purgatory On Wheels on April 19, 2015, 08:12:12 PM
Also, it's not been made clear yet whether people with no programming experience would be of use to the project. I'd like to help, but haven't made an account on Github because (a) I don't know what that is and (b) I don't know what skills must be possessed in order to participate there effectively.
Speaking just for myself (since decisions will likely be made more collaboratively than with CHM), I think we're not at the stage to make team decisions about what to look for in new non-programmer team members, though I expect we'll end up adding some of them. The first priority has to be the programming to set up the new system and the tools needed to help new people contribute. Only once that is well underway would we start thinking about how to expand the team.
I'm a non-programmer, but my extensive travels throughout North America, and especially strong familiarity with Hawaii and the hard-to-map Arctic jurisdictions, helped me get on the CHM team. Other kinds of non-programming expertise would be useful too. But knowing a lot about the highways of regions not covered by current and past CHM veterans (after we figure out who among them will be joining the new project, and in what capacity) would help. Knowledge of, or at least interest in taking on, multiple regions or tasks not specific to highway mapping (such as improving land and water boundaries, or rail or other non-highway systems we might expand into), would also be good since fewer new people would need to be shown the ropes.
Quote from: Thing 342 on April 19, 2015, 08:40:13 PMI think a more hierarchical structure would be beneficial, as opposed to the somewhat arbitrary phase system in CHM.
How do you mean? And the phase system is not how stuff is stored, but what type of highway something is - that it is called 'phase' is arbitrary, but the divisions are still important (though not for storing data - merely rendering it)
As far as I can see, there's three broad options for the data:
1) No subfolders (OK, there might be some splitting Highway Data and Boundary Data, system csvs and behind the scenes stuff like mapviews). Will end up hard to find files as we're looking at over 15k routes.
2) Subfolders per system - the current way. I should point out that there's not many 'duplicates' in terms of folders and they mostly are because Tim has moved the files out of defunct folders but not deleted the folder. There's a few files where overhauls were in the /data folder but hadn't yet been pushed (Yukon, for example) and another few where a route is going to move from grabbag but the final system hasn't been activated. I've fixed these in my rip of the /data folder. This approach would mean finding files by system is infinitely easier than other methods, and new systems wouldn't be messy as all the files would be in one place not getting in the way of existing ones.
3) Subfolders per region/subregion - makes it easier to look for a region's files, though they are already sorted by region by the filename. We maintain files by region. Typically this was how I used to arrange it on my PC, but I've gone with by system now.
----
Can someone please tell me an easy way of batch uploading to Github so the existing data is up there (or at least in a fork of the project)?
From an organizational perspective, I'm a fan of #3.
Quote from: english si on April 20, 2015, 06:38:25 AMCan someone please tell me an easy way of batch uploading to Github so the existing data is up there (or at least in a fork of the project)?
Create the repo on github, clone it onto your computer, add the files to the directory and 'add' them to git (not sure how to batch-add in command-line, but it's easy enough using any of the GUIs), commit, push.
Quote from: Dr Frankenstein on April 20, 2015, 10:47:43 AMCreate the repo on github, clone it onto your computer, add the files to the directory and 'add' them to git (not sure how to batch-add in command-line, but it's easy enough using any of the GUIs), commit,
Done all that, though had to reduce the number of files in each batch, so there's a good ten or so commits.
Quotepush.
Can't seem to do that. The 'publish' (which is the only option I have) fails.
My two cents:
I sort by subregion on my PC:
chm/me, chm/ns, etc...
then chm/me/usai, chm/me/usaus, chm/me/usausb, chm/me/usame, chm/ns/cantch, chm/nb/cannsf, chm/nb/cannst, etc...
This works fine for me since I only work on two North American countries with non-overlapping postal codes. Project-wide, it'd make more sense to do
usa, can, etc...
usa/me, can/ns, etc...
Quote from: english si on April 20, 2015, 12:18:34 PM
Quote from: Dr Frankenstein on April 20, 2015, 10:47:43 AMCreate the repo on github, clone it onto your computer, add the files to the directory and 'add' them to git (not sure how to batch-add in command-line, but it's easy enough using any of the GUIs), commit,
Done all that, though had to reduce the number of files in each batch, so there's a good ten or so commits.Quotepush.
Can't seem to do that. The 'publish' (which is the only option I have) fails.
Are you trying in the TravelMapping/HighwayData repository? If so, I might just need to add some permissions for you. Let's have them in some subdirectory for now (maybe something like "chm_final") and we can then move things into a new hierarchy when we've settled on the organization and as each system is deemed ready to go.
As a side note to this, I feel strongly that we need to continue to do a good job citing our sources used to derive our data.
Quote from: yakra on April 20, 2015, 12:49:53 PM
My two cents:
I sort by subregion on my PC:
chm/me, chm/ns, etc...
then chm/me/usai, chm/me/usaus, chm/me/usausb, chm/me/usame, chm/ns/cantch, chm/nb/cannsf, chm/nb/cannst, etc...
This is another vote for option 3 then?
Burying the subregions in a directory is possible. Another option is USA_AK rather than USA/AK. Mexico, Russia and Kazakhstan are in RUS-MOW or whatever (at least one Mexico code doesn't meet the ISO standards, where they are all four digits)*. The USA_ bit merely puts all USA regions together, rather than strewn across the other regions.
Australia is the only place where we are likely to have a clash, unless we AUS- them (I say no), as there's WA and NT (Western Australia and Northern Territory). WAU and ANT (and SAU even though SA doesn't clash with the US/CAN 2-letter postal system) would fix this, and have the codes all three digits, matching ACT, NSW, QLD, TAS and VIC.
*I propose getting rid of the lot. Certainly for Russia and Kazakhstan, there's not the volume of routes that we're likely to have, nor are the splits needed for route length as the distance is contrasted by sparsity and flatness of the land making the point distances relatively far apart. I don't know about Mexico. I'm not sure that the ISO 3116-2 codes clash with 3-letter 3116-1 country codes, allowing a dropping of MEX- if we keep the subdivisions. And as said elsewhere, I'd rather get rid of the British ones which are odd as there's stronger autonomy elsewhere in Europe (eg Spain) that doesn't translate to subdivisions, but that would muck up .list files of too many people.
Quote from: english si on April 20, 2015, 02:48:52 PM
Australia is the only place where we are likely to have a clash, unless we AUS- them (I say no), as there's WA and NT (Western Australia and Northern Territory). WAU and ANT (and SAU even though SA doesn't clash with the US/CAN 2-letter postal system) would fix this, and have the codes all three digits, matching ACT, NSW, QLD, TAS and VIC.
Brazil and India have states and are pretty large countries, so in theory those could be subdivided, were they ever to be added.
I'm hoping to start making some headway on some part of this project in about a week. I have a little time between the end of my spring classes and the start of my summer class.
I'm thinking I'd like to develop some code that takes all of the data and populates a database, at least something those on the web side will be able to read from as they start working on other parts. So, if you're going to write a new highway browser, compute and report stats pages, or generate maps, what should I make sure is in the database and what form would be convenient? And maybe you would want to comment on my overall thought that the "ingestion" process of the .csv, .wpt, and .list data (or their successors) would do most of the work so the web-facing code would not be computationally intensive on the server or client side.
yakra already wrote a parser that can take all that and put it into a map. The next logical step would be to decide how the database would look and modify the parser (and map? or do that later?) to put the contents in the database.
Quote from: Jim on April 20, 2015, 01:28:53 PMAre you trying in the TravelMapping/HighwayData repository? If so, I might just need to add some permissions for you. Let's have them in some subdirectory for now (maybe something like "chm_final") and we can then move things into a new hierarchy when we've settled on the organization and as each system is deemed ready to go.
If I have permissions, I will be certainly put it in chm_final.
Quote from: vdeane on May 01, 2015, 10:51:49 PM
yakra already wrote a parser that can take all that and put it into a map. The next logical step would be to decide how the database would look and modify the parser (and map? or do that later?) to put the contents in the database.
Hope we can get that code into the repository to use or at least refer to.
Quote from: english si on May 02, 2015, 05:34:37 AM
Quote from: Jim on April 20, 2015, 01:28:53 PMAre you trying in the TravelMapping/HighwayData repository? If so, I might just need to add some permissions for you. Let's have them in some subdirectory for now (maybe something like "chm_final") and we can then move things into a new hierarchy when we've settled on the organization and as each system is deemed ready to go.
If I have permissions, I will be certainly put it in chm_final.
I've created a new group (of which you and I are the only members so far) whose members should be able to write to the TravelMapping/HighwayData repository. See if you can clone, add/commit to your local fork, then push back.
Sheer number of files means multiple commits needed to avoid crashing, but if you look, I've got all non-USA .wpts in a fork.
Should get the USA data in later, and will pull it in (I can't see any option to 'push'. I can 'publish' the fork, and create a pull request, which (presumably) I can respond to)
Quote from: english si on May 02, 2015, 08:54:26 AM
Sheer number of files means multiple commits needed to avoid crashing, but if you look, I've got all non-USA .wpts in a fork.
Should get the USA data in later, and will pull it in (I can't see any option to 'push'. I can 'publish' the fork, and create a pull request, which (presumably) I can respond to)
Great. Still need to figure out the best way to manage things, but I'm happy we'll soon have a complete copy of the last version of the CHM data safe and sound.
I'm curious.. Looking at the waypoint files, there seems to be the OpenStreetMap URL on every line. That's redundant, don't you think? I mean you could like make the data ½ or ⅓ the size, by having the lines have only the coordinates like:
BlaCrkDr +0 43.709994 -79.501424
While I'm here, I think I'd like to join the project. I have a reasonable amount of experience designing databases and developing web applications in Python and PHP, and I could probably also do a bit of web design.
It would be great if you could add me to the GitHub organization; my username is sammdot.
Quote from: sammi on May 02, 2015, 10:05:27 AM
I'm curious.. Looking at the waypoint files, there seems to be the OpenStreetMap URL on every line. That's redundant, don't you think? I mean you could like make the data ½ or ⅓ the size, by having the lines have only the coordinates like:
BlaCrkDr +0 43.709994 -79.501424
We discussed this above. It's easier to parse as <lat> <lon> <name> <alt name> and if we change it, it would be to that.
Size isn't an issue as files are peanuts sizes (biggest one is under 50kB) and most are in the 1k-2k ball park. The actual highway (not boundary) data is 16642 files and 23.3MB (though 74.1MB on my hard disk - no doubt due to the huge amount of files and the 115 folders they are in).
I've merged my fork in. It's done!
Quote from: Jim on May 02, 2015, 07:18:53 AM
Quote from: vdeane on May 01, 2015, 10:51:49 PM
yakra already wrote a parser that can take all that and put it into a map. The next logical step would be to decide how the database would look and modify the parser (and map? or do that later?) to put the contents in the database.
Hope we can get that code into the repository to use or at least refer to.
It's over here: http://205.209.84.174/roads/chm/canvas/
Quote from: vdeane on May 03, 2015, 07:51:36 PM
Quote from: Jim on May 02, 2015, 07:18:53 AM
Quote from: vdeane on May 01, 2015, 10:51:49 PM
yakra already wrote a parser that can take all that and put it into a map. The next logical step would be to decide how the database would look and modify the parser (and map? or do that later?) to put the contents in the database.
Hope we can get that code into the repository to use or at least refer to.
It's over here: http://205.209.84.174/roads/chm/canvas/
This is good. If there are no objections from the authors, I'd like to get that into our repository as reference code.
I propose we go ahead and convert all route (.wpt) files to be included in the project to the
<lat> <lon> <name> [<alt_name>] [<alt_name>] ...
format.
I'm OK with keeping the ".wpt" file extension but maybe we should use something new. I don't think it's the 1980's anymore, so we could use extensions with more than three characters if we want. We could use ".route" or ".waypoint" or ".highway" or something else.
A converter is pretty much trivial to write. If we agree on the above format, I'll put a converter in a few languages up there for anyone to use, and will gladly run it to get us a clean starting point.
The complications I see at the moment:
- The wpt editor would need to be rewritten/updated to use the new format. Any volunteers? I suppose we can't expect the current one to stay around forever anyway.
- What to do about all of those updates various people have made to their own copies and submitted to CHM but which didn't make it into the data we copied over? Get them all brought in now and then convert, or use the last good CHM snapshot as a starting point for the conversion, and all new updates and new systems would have to be converted to the new format before being added?
What did we do when Tim converted from the previous format (.ggm?) to .wpt? I don't recall all of it.
Quote from: froggie on June 02, 2015, 11:02:20 PM
What did we do when Tim converted from the previous format (.ggm?) to .wpt? I don't recall all of it.
I think we just changed the file extension at that time.
Quote from: Jim on June 02, 2015, 09:20:21 PM
I propose we go ahead and convert all route (.wpt) files to be included in the project to the
<lat> <lon> <name> [<alt_name>] [<alt_name>] ...
format.
Sounds good to me.
Quote from: Jim on June 02, 2015, 09:20:21 PM
I'm OK with keeping the ".wpt" file extension but maybe we should use something new. I don't think it's the 1980's anymore, so we could use extensions with more than three characters if we want. We could use ".route" or ".waypoint" or ".highway" or something else.
A different extension would distinguish new-format from old-format files. But .wpt never caused me any problems, even though I would be the most likely to draw them (as a user of WordPerfect, which uses .wpt for some of its files).
Quote from: Jim on June 02, 2015, 09:20:21 PM
The complications I see at the moment:
- The wpt editor would need to be rewritten/updated to use the new format. Any volunteers? I suppose we can't expect the current one to stay around forever anyway.
The other thing is that while existing CHM collaborators have permission to use the old editor, we would want something available to new team members.
Quote from: Jim on June 02, 2015, 09:20:21 PM
- What to do about all of those updates various people have made to their own copies and submitted to CHM but which didn't make it into the data we copied over? Get them all brought in now and then convert, or use the last good CHM snapshot as a starting point for the conversion, and all new updates and new systems would have to be converted to the new format before being added?
I favor the latter approach. That would let me hold off on learning github for awhile, at least until my own travel schedule eases up in late July. It has the more global additional advantage of letting us compare what our system spits out to what CHM displays, to identify any differences that shouldn't be there.
I would also favor holding up on submission of new under-development systems, and updates to existing under-development systems (like my Alaska State Highways set, which has several significant unsubmitted updates and will need some more minor tweaks).
As a first attempt, I'm adding code to my Python program that generates SQL code that will populate a database with all of the information from a subset of the .csv and .wpt files. This part is pretty much independent, other than just a few lines of code, of the .wpt file format and directory organization we've been discussing.
They way I'm thinking of this (as stated previously but now being implemented a bit) is to have a program that takes the entire state of the project (all of the .csv, .wpt, .list or their successor formats) and creates a big SQL file that can be imported into any instance of mysql (probably others but I'm using mysql), which can then in turn be queried by all of our web-facing tools.
Quote from: Jim on June 02, 2015, 09:20:21 PM
- What to do about all of those updates various people have made to their own copies and submitted to CHM but which didn't make it into the data we copied over? Get them all brought in now and then convert, or use the last good CHM snapshot as a starting point for the conversion, and all new updates and new systems would have to be converted to the new format before being added?
Even though I've not been in on the technical decisions, Oscar is right about taking the last good CHM snapshot as the starting point. It would save on a lot of misery to be able to proofread what has been generated and verify that everything is working before moving onward and upward. What's more, since the CHM data has (presumably) been checked for accuracy before, we wouldn't be checking accuracy from scratch but proofreading against a copy (which is easier in my opinion).
I had a post on this that I lost, but yes.
I suggest a folder with it in the <subregion>/<system> layout, which can then be converted.
In fact, if Github wasn't being annoying, that would be up.
I'd like to get a handful of .list files I can use to experiment with populating my first attempt at a database of the travel mapping project data. I've already picked a fairly small subset of the highway data, and I'll put in my own .list. Rather than selecting some random ones, I figured I'd see who might like theirs to be part of initial development and testing. Send CHM username if your .list is up-to-date on the old site, or email your .list directly to me (send private message on this board if you don't have my email).
Quote from: Jim on June 07, 2015, 09:51:19 PM
I'd like to get a handful of .list files I can use to experiment with populating my first attempt at a database of the travel mapping project data. I've already picked a fairly small subset of the highway data, and I'll put in my own .list. Rather than selecting some random ones, I figured I'd see who might like theirs to be part of initial development and testing. Send CHM username if your .list is up-to-date on the old site, or email your .list directly to me (send private message on this board if you don't have my email).
My .list file is on GitHub:
https://github.com/sammdot/my-clinched-highways
Should be way more recent than the one on CHM.
Here's mine: https://github.com/Bickendan/CHM
Are you looking for smaller lists for this test, or a larger one? I ask since both Oscar and I have two of the largest .list files in the project.
Quote from: Jim on June 07, 2015, 09:51:19 PM
I'd like to get a handful of .list files I can use to experiment with populating my first attempt at a database of the travel mapping project data. I've already picked a fairly small subset of the highway data, and I'll put in my own .list. Rather than selecting some random ones, I figured I'd see who might like theirs to be part of initial development and testing. Send CHM username if your .list is up-to-date on the old site, or email your .list directly to me (send private message on this board if you don't have my email).
You can use my old list file, on the CHM server. I'll also send an updated file by email, once my current round of travels are done next weekend, though it will be even more of a "stress test" than the old version (including some bad lines from travels on new roads not in CHM route sets).
Quote from: froggie on June 08, 2015, 08:52:29 AM
Are you looking for smaller lists for this test, or a larger one? I ask since both Oscar and I have two of the largest .list files in the project.
I don't think it should matter much. Mine's also pretty large. Part of this is to get a feel for whether the approach I'm trying has any chance to scale, so having some of the bigger files in there would be good. Mainly, I just want to pick a few people's .list files who might have interest in seeing their data show up in early tests.
I'm hoping in the next few days to have a very basic highway browser up that can read data from the database. Daytime work has to be real work, though, so I make no promises on timing.
My .list isn't super current (and has a few bad lines which I'm leaving in for my purposes), but it is here:
http://cmap.m-plex.com/list/sipes23.list
If you need something more current, I can get it to you.
Here's my current file: http://nysroads.com/files/aaroads/vdeane.list
My list is pretty much up-to-date on the CHM site if you want to use it for testing Jim.
http://cmap.m-plex.com/list/rickmastfan67.list
Si, or anyone else who's willing to fix, I noticed a bunch of the .wpt files (113 by my count) in the chm_final/usausb directory on Github contain an error message instead of the waypoint data. Could you check it out? A spot check shows the files do exist in the CHM data folder, and I can see them in a browser, but when I tried to grab them via wget with a script, I get the same error message. I don't see any problems in any other directory.
Thanks.
I see what's happened. The sheer size of the folder meant that my download-all program couldn't quite get there and started saving the files with error text in instead. Re-downloading the just over 100 files manually.
Given they are all at the end of the folder alphabetically, and usausb is the biggest folder, we should be alright. Other big folders, having checked, are fine.
PS: list file one (https://dl.dropboxusercontent.com/u/84173946/si404.list) and list file two (that simply shows green sign colour in the UK) (https://dl.dropboxusercontent.com/u/84173946/ukprimaryroutes.list)
OK, Github's GUI on my PC won't let me do anything (commit, make a new branch, etc, etc). It's the VA, VT, WA, WI, WV and WY files in usausb that have issues.
I've tried using the shell, but it's gobbledy-gook.
If you still need list files... mine is up to date (well, up to when updates stopped).
http://cmap.m-plex.com/list/drfrankenstein.list
Anyone aware of a .csv or other data file we can get our hands on from the old CHM project that has a list of active and in-development systems with their human readable name equivalents? If not, anyone willing to make one?
Something like this I just knocked up? Obviously the order, colors and fields can be changed to suit our needs and desires.
System;CountryCode;Name;Color;InDev
alba;ALB;Albania Motorways;blue;
auta;AUT;Austria Motorways;blue;
auts;AUT;Austria Expressways;teal;
bela;BEL;Belgium Motorways;blue;
biha;BIH;Bosnia and Herzegovina Motorways;blue;
bgra;BGR;Bulgaria Motorways;blue;
cannsf;CAN;Nova Scotia Provincial Freeways;blue;
canonf;CAN;Ontario Provincial Freeways;blue;
canqca;CAN;Quebec Freeways;blue;
cantch;CAN;Trans-Canada Highway;red;
cannb;CAN;New Brunswick Provincial Highways;brown;
cannst;CAN;Nova Scotia Trunk Routes;brown;
cannsc;CAN;Nova Scotia Collector Routes;yellow;
cannt;CAN;Northwest Territorial Highways;brown;
canon;CAN;Ontario King's Highways;brown;
canpe;CAN;Prince Edward Island Provincial Highways;brown;
canyt;CAN;Yukon Territorial Highways;brown;
chea;CHE;Switzerland Motorways;blue;
cypa;CYP;Cyprus Motorways;blue;
czed;CZE;Czech Republic Motorways;blue;
czer;CZE;Czech Republic Expressways;teal;
deua;DEU;Germany Motorways;blue;
eure;EUR;International E-roads;red;
espa;ESP;Spain National Motorways;blue;
espaca;ESP;Spain Select Autonomous Community Motorways;teal;
fraa;FRA;France Motorways;blue;
mtqa;FRA;Martinique Motorways;blue;
gbnm;GBR;Great Britain Motorways;blue;
gbnam;GBR;Great Britain A Motorways;blue;
gbna;GBR;Great Britain A Roads (Zones 2-5,7);brown;
nirm;GBR;Northern Ireland Motorways;blue;
niram;GBR;Northern Ireland A Motorways;blue;
nira;GBR;Northern Ireland A Roads;brown;
grca;GRC;Greece Motorways;blue;
hrva;HRV;Croatia Motorways;blue;
hunm;HUN;Hungary Motorways;blue;
imna;IMN;Isle of Man A Roads;brown;
irlm;IRL;Ireland Motorways;blue;
irln;IRL;Ireland N Roads;brown;
itaa;ITA;Italy Motorways;blue;
itasm;ITA;Italy Select Named Motorways;teal;
jamt;JAM;Jamaica Motorways;blue;
jeya;JEY;Jersey A Roads;brown;
luxa;LUX;Luxembourg Motorways;blue;
luxb;LUX;Luxembourg Expressways;teal;
mexd;MEX;Mexico Federal Expressways;blue;
mexed;MEX;Mexico State Expressways;teal;
nlda;NLD;Netherlands Motorways;blue;
pola;POL;Poland Motorways;blue;
pols;POL;Poland Expressways;teal;
prta;PRT;Portugal Motorways;blue;
rksr;RKS;Kosovo Motorways;blue;
roua;ROU;Romania Motorways;blue;
svkd;SVK;Slovakia Motorways;blue;
svkr;SVK;Slovakia Expressways;teal;
svna;SVN;Slovenia Motorways;blue;
svnh;SVN;Slovenia Expressways;teal;
turo;TUR;Turkey Motorways;blue;
usai;USA;United States Interstate Highways;blue;
usaib;USA;United States Business Interstate Highways;green;
usaif;USA;United States Future Interstate Highways;teal;
usansf;USA;United States Select Numbered State Freeways;teal;
usasf;USA;United States Select Named Freeways;teal;
usaus;USA;United States Numbered Highways;red;
usausb;USA;United States Auxillary Numbered Highways;magenta;
usaaz;USA;Arizona State Highways;brown;
usact;USA;Connecticut State Highways;brown;
usadc;USA;District of Columbia District Highways;brown;
usade;USA;Delaware State Highways;brown;
usahi;USA;Hawaii State Highways;brown;
usaia;USA;Iowa State Highways;brown;
usaid;USA;Idaho State Highways;brown;
usail;USA;Illinois State Highways;brown;
usaks;USA;Kansas State Highways;brown;
usaky;USA;Kentucky State Highways 1-999;brown;
usame;USA;Maine State Highways;brown;
usamd;USA;Maryland State Highways;brown;
usama;USA;Massachusetts State Highways;brown;
usami;USA;Michigan State Highways;brown;
usamn;USA;Minnesota State Highways;brown;
usamo;USA;Missouri State Highways;brown;
usane;USA;Nebraska State Highways;brown;
usanv;USA;Nevada State Highways;brown;
usanh;USA;New Hampshire State Highways;brown;
usanj;USA;New Jersey State Highways;brown;
usany;USA;New York State Highways;brown;
usanc;USA;North Carolina State Highways;brown;
usand;USA;North Dakota State Highways;brown;
usaoh;USA;Ohio State Highways;brown;
usaok;USA;Oklahoma State Highways;brown;
usaor;USA;Oregon State Highways;brown;
usapa;USA;Pennsylvania State Highways;brown;
usari;USA;Rhode Island State Highways;brown;
usawa;USA;Washington State Highways;brown;
usawv;USA;West Virginia State Highways;brown;
usawi;USA;Wisconsin State Highways;brown;
cannf;CAN;Canada Select Named Freeways;teal;yes
cansph;CAN;Canada Select Provincial Highways;brown;yes
gbna1;GBR;Great Britain A Roads (Zone 1);brown;yes
gbna6;GBR;Great Britain A Roads (Zone 6);brown;yes
gbna8;GBR;Great Britain A Roads (Zone 8);brown;yes
gbna9;GBR;Great Britain A Roads (Zone 9);brown;yes
usaak;USA;Alaska State Highways;brown;yes
usaca;USA;California State Highways;brown;yes
usafl;USA;Florida State Highways;brown;yes
usaky3;USA;Kentucky State Highways 1000-1499;brown;yes
usaky4;USA;Kentucky State Highways 1500-1999;brown;yes
usala1;USA;Louisiana State Highways 1-499;brown;yes
usamt;USA;Montana Primary State Highways;brown;yes
usanm;USA;New Mexico State Highways;brown;yes
usasc;USA;South Carolina State Highways;brown;yes
usaut;USA;Utah State Highways;brown;yes
usavt;USA;Vermont State Highways;brown;yes
Quote from: english si on June 10, 2015, 04:22:16 AM
Something like this I just knocked up?
Looks like it has the essentials to get us started, thanks.
http://cmap.m-plex.com/list/master_son.list is my latest published list file. Note that it has highways systems under development (such as CA and UT) and would produce errors if points are not matched.
I've grabbed a bunch of the .list files from some of those who contacted me to be a part of testing. My program generates log files along the lines of those on the old CHM site, but with what I hope is some additional useful information. For example, I report a note when an old route name is in the .list and I'm instead using the new canonical name.
My spot check of the logs have led to several improvements (better handling of empty lines, lines with extraneous whitespace, etc). I am posting all of the logs from a recent run at
http://www.teresco.org/~terescoj/travelmapping/logs/
If you see anything unexpected there or have any other questions or suggestions, let me know.
One thing I noticed is that some .list files have the "*" entries for former waypoints included in waypoint names. For both those and hidden points starting with "+", I was just matching .list entry waypoints with actual waypoint names with any leading "+" or "*" stripped off. I think it has to be that way for "+" names, since we add a "+" to alternate old names in many cases (see all the +0 and +999 entries in the interstates).
I'm sure there's more to fix, but at least .list files are being parsed and lines are being matched with actual waypoints from actual routes. Progress!
I saw "Note: replacing deprecated route name" in my log file. So, I'll update my list file on my HD so that doesn't show up again in the future. :)
And Jim, might want to add a time code as to when the log file was created like what happens on the CHM site. Lets people know when it was last checked.
I like how "Ignoring line matching highway in inactive system" is now a separate error from "Unknown region/highway combo".
I'm actually in the gradual process of mapping my travels over the last two weeks using your highway browser. I'm liking the simplicity of the URL format since it makes it quicker to go from one highway to another simply by editing the URL rather than hitting back and then having to scroll and click through to other things.
Quote from: Duke87 on June 11, 2015, 01:16:05 AM
I like how "Ignoring line matching highway in inactive system" is now a separate error from "Unknown region/highway combo".
I really like that too.
By the way, I think this has been requested in the original app before, but supporting comments in the file would be a great addition.
Quote from: Dr Frankenstein on June 11, 2015, 10:17:06 AM
By the way, I think this has been requested in the original app before, but supporting comments in the file would be a great addition.
That's simple enough, and I'll add it. I don't see why we can't support comments in .csv and .wpt files too.
Any objection to '#' starting single-line comments in any or all of the project's files? I don't see need for direct multi-line comment support.
That's exactly what I had in mind, provided no routes or waypoints use the '#' character.
CSV is standardized (https://tools.ietf.org/html/rfc4180) with no support for comments, but some parsers stray from the RFC in that regard.
As long as the .list file could support comments :D
I'm happy to report a lot more data processing progress, though none that produces any new visual results. All colocated waypoints are now detected, which will allow me to detect concurrencies efficiently, which is, I think, the next thing on the list. It's also what's needed to support the links to intersecting and continuing routes that are not in the draft highway browser yet. I believe Tim used to have a tolerance to consider very nearby points colocated, but I am requiring exact coordinate matches. If this requirement remains, it means some work to fix up some of the data, but much of that can be done automatically. We'd just need a manual check to make sure no points are merged which were kept separate intentionally.
Tonight's work was also a big win for efficient data structures. The brute force check to find waypoints with matching coordinates was on pace to take a few hours to complete. By coding up and using a quadtree structure to hold the waypoints, the whole program is back down to about a minute of execution time. FYI, the wpt files currently being loaded in contain 365,937 waypoints. So pay attention in your data structures class.
Quote from: Jim on June 12, 2015, 12:09:27 AMSo pay attention in your data structures class.
Indeed.
(As an aside, I'm tired of fixing my colleagues' O(
n) and O(
n²) algorithms at work. All of them have university degrees, including one with an engineer degree. I'm the only one with a community college degree. Pay attention in class, folks!</rant>)
Quote from: Jim on June 12, 2015, 12:09:27 AMSo pay attention in your data structures class.
We never learned about quadtrees in
my data structures class. :) Maybe next year I might.
Quote from: sammi on June 12, 2015, 11:53:35 AM
Quote from: Jim on June 12, 2015, 12:09:27 AMSo pay attention in your data structures class.
We never learned about quadtrees in my data structures class. :) Maybe next year I might.
To be honest, I rarely get to cover quadtrees when I teach data structures or even algorithms. But this very handy example might motivate me to slip them in this fall.
Quote from: Jim on June 12, 2015, 12:09:27 AM
I'm happy to report a lot more data processing progress, though none that produces any new visual results. All colocated waypoints are now detected, which will allow me to detect concurrencies efficiently, which is, I think, the next thing on the list. It's also what's needed to support the links to intersecting and continuing routes that are not in the draft highway browser yet. I believe Tim used to have a tolerance to consider very nearby points colocated, but I am requiring exact coordinate matches. If this requirement remains, it means some work to fix up some of the data, but much of that can be done automatically. We'd just need a manual check to make sure no points are merged which were kept separate intentionally.
Jim, I think we still need to have the offset of .000001 for the linking to other routes. It was needed to keep false multiplexes from being detected. I know that FL-84 & I-75/I-595 are completely separate for a good distance, but need the offset since they almost always intersect the same highways since FL-84 is the service road(s).
http://cmap.m-plex.com/hb/hwymap.php?sys=usafl&rg=all&gr=p&r=fl.fl084
Quote from: Jim on June 12, 2015, 12:09:27 AMI believe Tim used to have a tolerance to consider very nearby points colocated, but I am requiring exact coordinate matches. If this requirement remains, it means some work to fix up some of the data, but much of that can be done automatically. We'd just need a manual check to make sure no points are merged which were kept separate intentionally.
I think Tim's method is the way to go. An exact match was required for a concurrency, while very nearby points meant a simple junction. It comes down to the intentionally-separate points.
Consider TX US183 TX21 (30.027758°, -97.687909°) & TX TX130 461 (30.027758°, -97.687908°). These need to be separate to avoid flagging a concurrency here, as US183 runs along TX130's frontage roads.
But if an exact coordinate match is required, then the point on TX21 will not be detected as a junction with
both routes.
AP-7 and B-30 in Barcelona, ON 401's express and local lanes (if we were to track the two as opposed to one route).
I don't mean those points where we intentionally separate points to avoid a false concurrency should be aligned. I'm talking about those places where a road crosses a state border or where two normal highways intersect in some standard way - the points should be exactly the same. Maybe I'm missing something more subtle here.
Jim - make Near Miss Points an error or something.
There needs to be some leeway for intersecting to deal with local-express setups and similar. Concurrencies are (obviously) exact matches.
Tim's old system worked, other than it didn't tell us where NMPs were.
How many local-express delineations do we have to deal with in this? While it'd be nice to account for them all, I think the only 'valid' one for our purposes is AP-7/B-30 in Barcelona because of the different route numbers (AP-7 express, B-30 local).
RA1-A14 in Italy. There's lots more that possibly might be added.
Many, many, smaller versions exist, for 2 or 3 junctions.
I-91 / CT15
US183 / TX130
A couplefew more TX examples where a state highway is on the frontage roads of a toll road
Maybe I-88/355, I dunno man I didn't do it
Quote from: Jim on June 12, 2015, 03:26:47 PM
I don't mean those points where we intentionally separate points to avoid a false concurrency should be aligned. I'm talking about those places where a road crosses a state border or where two normal highways intersect in some standard way - the points should be exactly the same. Maybe I'm missing something more subtle here.
I think you may be. In my example upthread, TX21 and TX130 are normal highways intersecting in a standard way.
TX TX130 461: (30.027758°, -97.687908°)
TX TX21 US183/130: (30.027758°, -97.687909°)
US 219 and I-90
I-790/NY 49 and I-90
NY 27 and BeltPkwy
Quote from: yakra on June 13, 2015, 01:01:39 AM
Maybe I-88/355, I dunno man I didn't do it
Those are on two separate roadbeds. And there's no interchanges during that parallel section, anyway.
Quote from: yakra on June 13, 2015, 01:01:39 AM
I think you may be. In my example upthread, TX21 and TX130 are normal highways intersecting in a standard way.
TX TX130 461: (30.027758°, -97.687908°)
TX TX21 US183/130: (30.027758°, -97.687909°)
So for this one, and others like it, we're trying to avoid introducing a false concurrency between TX 130 and US 183, correct? Would it be bad to introduce a slightly-moved hidden waypoint between the interchanges? I assume in many cases, the route on the outer roadway has additional intersections anyway.
Mainly, what I want to suggest is that if there's no good reason to have to have points off by a small amount, we should have them in the exact same point. Cases are everywhere, but one that I see is OH 518/OH 644. No reason that I can tell for those to be in different places.
The data used is a bit out of date, but you can see the "near miss" points detected when I turned CHM data into graph form a while back at http://courses.teresco.org/chm/graphs.html in the .nmp links in the rightmost column.
Quote from: Jim on June 13, 2015, 12:06:56 PMMainly, what I want to suggest is that if there's no good reason to have to have points off by a small amount, we should have them in the exact same point.
The only reasons to make intersecting points exactly identical are:
1) anal retentiveness
2) making sure there's a concurrency
3) making it easier to make files by starting with points that intersect that route
The first is literally "no good reason", though the second is (but only applies to specific bits). The third is a time saver (negated by the hassle of keeping points identical - if you move one point to be more accurate because it was slightly off, then you have to do the same in every other route that meets there).
NMP is not a true error (unless a concurrency that should be isn't registered) that causes issues with end users and statistics and so on. But sure, it's a falling short from perfection and ought to be flagged as a potential error.
And certainly, while you have a script for it, I don't want files merged to the average location of the point. All the GB ones come from when, either through peer review or update, I've got a more pinpointed location, and simply not done it on other routes. Averaging out would take a 0m wrong point and a 10m wrong point and make 2x 5m wrong points. If I sort out those NMPs, it will be manually and with the priority of the snipe hunt that it is.
Quote from: english si on June 13, 2015, 01:00:52 PM
The only reasons to make intersecting points exactly identical are
My own reason is that I use the data for academic projects by converting to a graph structure. It's how I justify at least some of the time I put into the project. I have to merge those points to be able to get a lot of the algorithms (like Dijkstra's shortest paths) to produce expected results, but then I sometimes end up merging points that were left separate intentionally. Re-creating/updating graphs as highway data evolves becomes simpler and more accurate if I'm confident that any points that were not exactly the same are that way on purpose. So, bottom line, I have a reason, but not one that anyone else will care about.
Quote from: Jim on June 02, 2015, 08:58:32 PM
Quote from: vdeane on May 03, 2015, 07:51:36 PM
Quote from: Jim on May 02, 2015, 07:18:53 AM
Quote from: vdeane on May 01, 2015, 10:51:49 PM
yakra already wrote a parser that can take all that and put it into a map. The next logical step would be to decide how the database would look and modify the parser (and map? or do that later?) to put the contents in the database.
Hope we can get that code into the repository to use or at least refer to.
It's over here: http://205.209.84.174/roads/chm/canvas/
This is good. If there are no objections from the authors, I'd like to get that into our repository as reference code.
Firstly, a slightly more up-to-date revision of the code:
http://205.209.84.174/roads/chm/canvas/canvas.cpp
No objections, though a few BEWARE!s.
This was only ever really intended to run on my (Linux) system, with my data. Which it does.
Others have had problems. The CSV reading routines don't account for UNIX vs DOS style line feeds. So, if you feed it a file with DOS line feeds, or try to compile and run it in Windows, shit will explode. Sorry about that.
Also, it's not designed to handle a viewport that straddles the 180th meridian. Which may never end up being an issue in practice, but... nota bene.
It reads all the routes from "routes.csv". (Allowing multiple CSVs to be specified on the commandline has been on my to-do list, but a low priority. It works as it is, without too much additional hassle.)
The .list file must be named "list.list".
So, it's a bit rough around the edges. But gets the job done. Really, I view the important part as, not this C++ program (everything will have to be rewritten for the web, in a different language, anyway), but the Javascript that it spits out. Again, far from perfect, but a working proof of concept.
I envision implementing multiplex detection eventually, because:
1.) even if everything is marked as unclinched, plotting, say, US202 on top of ME100 on top of ME17 on top of ME11 makes the line noticeably darker/bolder. Yecch.
2.) drawing stuff in the right color. Example (http://205.209.84.174/roads/chm/canvas/stuff/bangor.htm): Only ME15 is marked as clinched, and as a result doesn't get the dark blue Interstate color.
This would require that things be done quite differently from how they are now. That's a bigger discussion for another time.
Another interesting/worthwhile side discussion is how to cull line segments that fall outside the viewport, in order to keep down both the size of the JavaScript code, and CPU time needed to crunch it.
My program doesn't do (http://205.209.84.174/roads/chm/canvas/stuff/portland.png) any sort of culling. Tim's centermap.php does (http://205.209.84.174/roads/chm/canvas/stuff/centermap.php.png). Can you spot the difference?
Lastly, a link (http://205.209.84.174/roads/chm/canvas/stuff/me-nh-vt-borders.htm) to what I consider the canonical example of the program's output.
Texas, complete with state highways, loops and spurs
http://205.209.84.174/roads/chm/canvas/stuff/tx-final.htm
A bit more boring however in that nothing is marked as clinched.
Quote from: yakra on June 13, 2015, 04:14:22 PM
[Firstly, a slightly more up-to-date revision of the code:
http://205.209.84.174/roads/chm/canvas/canvas.cpp
No objections, though a few BEWARE!s.
It's definitely got a ton of useful stuff there, even though I doubt we'll be able to use any C++. I am confident we can get PHP to output the same kind of Javascript, but getting its data from the database I'm building, and its bounds from a web page (I hope) someone will develop that will pick regions, cities, etc.
Quote from: Jim on June 10, 2015, 10:38:05 PM
I've grabbed a bunch of the .list files from some of those who contacted me to be a part of testing. My program generates log files along the lines of those on the old CHM site, but with what I hope is some additional useful information. For example, I report a note when an old route name is in the .list and I'm instead using the new canonical name.
My spot check of the logs have led to several improvements (better handling of empty lines, lines with extraneous whitespace, etc). I am posting all of the logs from a recent run at
http://www.teresco.org/~terescoj/travelmapping/logs/
If you see anything unexpected there or have any other questions or suggestions, let me know.
Many South Dakota state routes seem not to have made it to the new database, unless that was taken care of after the log was generated.
Most of the other errors in my log are expected, from new routes or route extensions after CHM's data freeze, or in anticipation of the new NM and UT state route sets. Some are puzzling at first glance, but I'll dig deeper once I'm back home, and make any necessary fixes before I submit a new list file.
Quote from: oscar on June 14, 2015, 09:08:19 AMMany South Dakota state routes seem not to have made it to the new database, unless that was taken care of after the log was generated.
Turns out usasd was omitted unintentionally. It's in now, and next update will take care of it.
Thanks.
I have some changes that were to be done in SD, the notes sitting sitting on my desktop. I didn't do them yet.
Jim, would you mind if I got access to your database? I'm developing a web frontend (http://travelmapping.sammdot.ca/) and was hoping to use the data on your site.
Quote from: sammi on June 17, 2015, 10:59:50 PM
Jim, would you mind if I got access to your database? I'm developing a web frontend (http://travelmapping.sammdot.ca/) and was hoping to use the data on your site.
Of course. At this point, since I'm occasionally breaking things, it's probably best to have your own private copy. Several ways we could go with this. I'll send a private message so we can discuss specifics.
I've been working yesterday and today to rewrite Jim's data processing code. In particular, I want to know what we're doing with the .wpt file format. I've written some parsing code for a possible new format (https://github.com/sammdot/TravelMapping-DataProcessing/blob/master/waypoint.py#L24) already, but I'm not quite sure where that format stands.
I also noticed a discrepancy between Jim's log output and mine. A rather large one, too. So I'm actually not sure if my code is doing the right things...
Log file created at: 2015-06-18 22:24:11.973266
Waypoint label(s) not found in line: ON ON10 ValRd RR12
Processed 44 good lines marking 1021 segments traveled.
vs (after changing ValRd to ValBlvd)
Processed 45 good lines marking 868 segments traveled.
Couple things to think about:
1) is the update intended to be one entire database dump? If that's the case, you'll be spending a while doing something else while your ssh finishes its job.
2) Those using python - are your versions similar or the same? I use an older version of linux due to supporting older software and am forced to use one earlier than the one Jim uses. It made a big deal b/c the flush function for strings does not exist in mine.
Quote from: SSOWorld on June 24, 2015, 08:14:57 PM
Couple things to think about:
1) is the update intended to be one entire database dump? If that's the case, you'll be spending a while doing something else while your ssh finishes its job.
2) Those using python - are your versions similar or the same? I use an older version of linux due to supporting older software and am forced to use one earlier than the one Jim uses. It made a big deal b/c the flush function for strings does not exist in mine.
The differences between Python 2.x and Python 3.x are significant. I'm using Python 3, and it's not a good idea to try to back port it, most likely. However, Python 3 has been around a long time now, and I'd be surprised if any Linux still worth running doesn't have it. Typically, it's installed to run as "python3" from the command line.
As mentioned, I have an older distro. It installed Python 3.2.3 when I called for it. It's still older than yours though. the "flush" function is on a newer minor iteration (3.3?), I just commented it out and the code "worked". but I agree we should not backport it unless we absolutely have to.
Quote from: SSOWorld on June 24, 2015, 09:07:12 PM
As mentioned, I have an older distro. It installed Python 3.2.3 when I called for it. It's still older than yours though. the "flush" function is on a newer minor iteration (3.3?), I just commented it out and the code "worked". but I agree we should not backport it unless we absolutely have to.
I hope there's not much that would matter in that case. You're correct that things will work just fine without the flush calls - that's just to be able to see output on the screen before the newline in printed.
This could go here or in either of a few other threads. It's of most interest to those who'll be maintaining highway data.
My first pass at data checks on the highway data are in place. CHM had a list of 18 error checks (plus one "undisclosed"), and those are shown at http://cmap.m-plex.com/tools/datacheck.php (http://cmap.m-plex.com/tools/datacheck.php). I believe I have code that attempts to check for all except #11, which I haven't quite figured out how to do easily.
The current detected errors are in http://www.teresco.org/~terescoj/travelmapping/logs/datacheck.log (http://www.teresco.org/~terescoj/travelmapping/logs/datacheck.log). I am making no attempt to handle false positives yet, and my distances don't have that factor Tim used to improve route length approximations. It's essential we find a way to account for the thousands of FPs we reported in CHM before we worry about fixing these. Other than that, things should probably match the errors shown on CHM's list.
Please let me know if you notice anything that seems wrong about the reported errors.
Quote from: Jim on June 25, 2015, 09:10:38 PM
Please let me know if you notice anything that seems wrong about the reported errors.
Spotted this weird one where the ')' is in the wrong place.
CA CA47 I-710(0 mi)ght not refer to an exit 0
Quote from: rickmastfan67 on June 25, 2015, 10:19:59 PM
Quote from: Jim on June 25, 2015, 09:10:38 PM
Please let me know if you notice anything that seems wrong about the reported errors.
Spotted this weird one where the ')' is in the wrong place.
CA CA47 I-710(0 mi)ght not refer to an exit 0
Thanks - copy and paste mistake and this and others like it should be gone next time I update.
Quotemy distances don't have that factor Tim used to improve route length approximations.
The exact value of the fudge factor is 1.02112.
What exactly does this 'fudge factor' do? What does it stand for?
Quote from: sammi on June 25, 2015, 11:35:34 PM
What exactly does this 'fudge factor' do? What does it stand for?
It stands for freedom.
(Sorry, couldn't resist...*ducks out*)
Quote from: sammi on June 25, 2015, 11:35:34 PM
What exactly does this 'fudge factor' do? What does it stand for?
As Jim said, it's used to improve route length approximations.
It dates back to the very early days of Tim's site, back when it was just "Clinched Interstate Mapping". There were no hidden shaping points back then, and due to the route trace directly connecting adjacent interchanges, even when distantly spaced and separated by a meandering roadway, the calculated mileage came up about 2% short on average from the actual route lengths. So, Tim calculated the fudge factor to multiply the routes' length, and more accurately match the total mileage of the interstate system.
A few years back, there was a little (just a little) discussion about recalibrating the fudge factor due to having more accurate route traces with the inclusion of hidden shaping points. But it wasn't viewed as terribly important, and ultimately never happened.
Quote from: yakra on June 26, 2015, 11:43:11 AM
Quote from: sammi on June 25, 2015, 11:35:34 PM
What exactly does this 'fudge factor' do? What does it stand for?
As Jim said, it's used to improve route length approximations.
It dates back to the very early days of Tim's site, back when it was just "Clinched Interstate Mapping". There were no hidden shaping points back then, and due to the route trace directly connecting adjacent interchanges, even when distantly spaced and separated by a meandering roadway, the calculated mileage came up about 2% short on average from the actual route lengths. So, Tim calculated the fudge factor to multiply the routes' length, and more accurately match the total mileage of the interstate system.
A few years back, there was a little (just a little) discussion about recalibrating the fudge factor due to having more accurate route traces with the inclusion of hidden shaping points. But it wasn't viewed as terribly important, and ultimately never happened.
For now, I think I should use the CHM factor so we can see unintentional discrepancies between old and new stats. If I add that factor into a couple places I think we should see matching numbers.
I think it remains a long-term and low-priority goal to come up with something better, like the option of per-route or even per-segment overrides to account more accurately for the lengths of especially straight or especially curvy routes.
Well, what do you know.. the factor was sitting right there in the JS distance functions I copied from Tim years ago. So route lengths in the draft highway browser should match CHM, and should later match CHM in the mapping overlay viewer once I code up a way to avoid counting segments along concurrencies multiple times.
Quote from: Jim on June 26, 2015, 04:02:58 PMI think it remains a long-term and low-priority goal to come up with something better, like the option of per-route or even per-segment overrides to account more accurately for the lengths of especially straight or especially curvy routes.
If a given segment carries US202, ME4, and ME100, and each route has a different respective scale factor... what then? What is the scale factor of this segment? What is this segment's length?
That, and just the idea of potentially putting that much more work on each collaborator's
place plate when drafting a system, makes me skeptical. I just see headaches.
Quote from: yakra on June 27, 2015, 02:26:09 AM
If a given segment carries US202, ME4, and ME100, and each route has a different respective scale factor... what then? What is the scale factor of this segment? What is this segment's length?
That, and just the idea of potentially putting that much more work on each collaborator's place when drafting a system, makes me skeptical. I just see headaches.
Great point. What's wrong with scrapping the scale factor entirely, then, and instead encouraging more accurate representation of a route's true shape via more hidden shaping points? The most compelling arguments I've read so far were, essentially, that it would be more work to make the initial file, and that processing the larger files would take longer when drawing the maps.
Quote from: mapcat on June 28, 2015, 12:48:10 PM
Great point. What's wrong with scrapping the scale factor entirely, then, and instead encouraging more accurate representation of a route's true shape via more hidden shaping points? The most compelling arguments I've read so far were, essentially, that it would be more work to make the initial file, and that processing the larger files would take longer when drawing the maps.
Not just drawing the maps, but, quoting Tim from back in 2010:
Quote- Some web scripts in the site necessarily have execution times roughly proportional to the number of points to be searched or plotted. Raising the shaping point count by a factor of 5 while raising the number of supported routes will eventually have a crippling effect.
I'm thinking multiplex detection and its O(n log n) complexity. There may be other similar routines too. (Show Intersecting Highways?)
In the same post, Tim went on to outline the diminishing returns of putting more effort into greater shaping point detail:
Quote- Using 5 times as many points as needed -- mostly points that will never ever be used by anyone - generates only a 1-2% improvement in the highway mileage, which is clearly not worth the effort.
- No one comes to the CHM site for hi-resolution maps, so 5 times the effort doesn't pay off there either.
- The use of a modest number of shaping points in addition to the required points has proven to be a more modest amount of work for a more significant improvement in maps and mileages than a subsequent doubling or more of the number of shaping waypoints. This is why shaping points should be based on an intermediate scale: 5mi/10km on Google.
Tim, Sun Dec 06, 2009:
QuoteYes, some of the right-angle turns make big % errors if you skip them, so you should add shaping points there. I pointed out only 2 errors of this sort in the SD set recently, but it's a general point. We weren't this picky when collecting the US highway, but enough of you suggested we bump up the standard, so we try to get the highways a little more accurate now.
The goal I'd like to see is to get lengths of routes correct to within 2% of the correct length, but verifying that goal can be tedious and not worth the effort. So I came up with a more visual goal instead, which Eric mentioned, to approximate it: keep the centerline of the highway within the blue polyline at the 5mi/10km zoom level of Google Maps in the HB. This should make the grid and city-level php maps a little more accurate too where we want the shape correct as well as the total length.
Also note that there is an automatic +2% correction slapped onto all the highway segments and therefore also all the route totals, since in general we underestimate route lengths by our low-res highway data. This correction roughly fixes freeway mileages on average (some are high and some are low, instead of all being low), and I haven't calibrated it any further.
Tim, Wed Dec 09, 2009:
QuoteWell, the 2% accuracy goal (that I wanted but threw out) and the 2% fudge factor are two different things. It's a coincidence that they are the same at the moment. (Actually the fudge factor is 2.112% in my code and should probably be a little lower now that more routes are better traced.)
I do promise not to shrink the width of the blue line in the HB.
If anyone would like to recalibrate the fudge factor, I have some good ideas for doing so, but no motivation to do it myself.
Some great arguments for not being over-detailed universally, sure. And 2% accuracy is outstanding; nobody would argue with that, or even 4 or 5%, I'm sure. But on the minority of routes that have dozens of hairpin turns *and* are quite long, what's the harm of being a little more true to the actual shape of the road? Those examples where the file shows a route being >10% off reality are the ones that seem to warrant re-examining the rules, IMO.
Quote from: mapcat on June 28, 2015, 05:32:39 PM
Some great arguments for not being over-detailed universally, sure. And 2% accuracy is outstanding; nobody would argue with that, or even 4 or 5%, I'm sure. But on the minority of routes that have dozens of hairpin turns *and* are quite long, what's the harm of being a little more true to the actual shape of the road? Those examples where the file shows a route being >10% off reality are the ones that seem to warrant re-examining the rules, IMO.
Even some of the shorter routes with lots of hairpin curves would bloat out a lot to gain better distance accuracy. HI 360, for example, would go from two dozen points to hundreds or even thousands. The distance accuracy would improve a lot percentage-wise, but IMO not worth the effort, even though it would add several miles to my Hawaii mileage (too bad I can't get extra credit for having traveled it more than a dozen times).
Tim's guidance, quoted above, allows latitude for adding a small number of "extra" shaping points to improve distance accuracy without too much more work for the route file drafter and/or the server. I took advantage of that for HI 378, which has several largish switchbacks, not all of which needed shaping points under the usual rules but I included them anyway. Nick did same for his draft UT 261 route file (which not only improved accuracy, but also highlights the famously hairy part of the route in the middle),
I know I started this whole discussion with my comment about possibly adding per-route or per-segment overrides to the factor, but I want to emphasize that it would be something well down the road, when we have all the data and tools we could ever want and have everything in place, and we know the efficiency of all of the processes when they've been tested at scale. I suspect we'll be able to handle more data than the old system did efficiently. I've been trying to design everything to put as much of the computationally intensive things in the site update program (which I envision running maybe once or twice a day, so it could reasonably take a few hours - right now takes about 5 minutes) and into the DB queries (which so far are instantaneous for all intents and purposes) rather than into web-facing PHP or JS code that would run on demand when a page is accessed. As more stats are added and more users are added, something's going to get a lot more expensive, but I hope it's rarely going to be anything that makes individual page accesses (with the stats and maps they generate) too expensive. So I suggest we maintain the CHM guidelines for waypoint density for now, and think about the issue when we know better what the costs would be (in code development time, highway system development time, and compute time).
Quote from: SSOWorld on June 24, 2015, 09:07:12 PM
As mentioned, I have an older distro. It installed Python 3.2.3 when I called for it. It's still older than yours though. the "flush" function is on a newer minor iteration (3.3?), I just commented it out and the code "worked". but I agree we should not backport it unless we absolutely have to.
Not an issue, found another ppa and got the right version installed.
FYI - and if anyone's already started tell me so I stop - I'm starting on the web rendering of yakra's cpp to generate the CHM maps of systems. Prototyping first, fine tune later. yakra - please let me know if you have tinkered more with your code since the last check-in to github. NOTE: instead of pulling data from csvs as yakra did - I'll be using the database.
Will be (obviously) writing it in PHP.
I'll keep you posted to my progress but since it's on my off time, don't expect results this week ;) :awesomeface:
I haven't changed the code since then. It's not really something I'm actively working on anymore.
Quote from: SSOWorld on June 29, 2015, 10:02:28 AM
FYI - and if anyone's already started tell me so I stop - I'm starting on the web rendering of yakra's cpp to generate the CHM maps of systems. Prototyping first, fine tune later. yakra - please let me know if you have tinkered more with your code since the last check-in to github. NOTE: instead of pulling data from csvs as yakra did - I'll be using the database.
Will be (obviously) writing it in PHP.
I'll keep you posted to my progress but since it's on my off time, don't expect results this week ;) :awesomeface:
Excellent! If you need anything different in the DB to support this, let me know. Have you been able to populate your own copy of the DB to experiment with?
Quote from: Jim on June 29, 2015, 02:29:54 PM
Quote from: SSOWorld on June 29, 2015, 10:02:28 AM
FYI - and if anyone's already started tell me so I stop - I'm starting on the web rendering of yakra's cpp to generate the CHM maps of systems. Prototyping first, fine tune later. yakra - please let me know if you have tinkered more with your code since the last check-in to github. NOTE: instead of pulling data from csvs as yakra did - I'll be using the database.
Will be (obviously) writing it in PHP.
I'll keep you posted to my progress but since it's on my off time, don't expect results this week ;) :awesomeface:
Excellent! If you need anything different in the DB to support this, let me know. Have you been able to populate your own copy of the DB to experiment with?
Jim - yes I have. I'll keep you posted
yakra - thanks for the heads up.
Quote from: english si on June 13, 2015, 01:00:52 PM
Quote from: Jim on June 13, 2015, 12:06:56 PMMainly, what I want to suggest is that if there's no good reason to have to have points off by a small amount, we should have them in the exact same point.
The only reasons to make intersecting points exactly identical are:
1) anal retentiveness
2) making sure there's a concurrency
3) making it easier to make files by starting with points that intersect that route
The first is literally "no good reason", though the second is (but only applies to specific bits). The third is a time saver (negated by the hassle of keeping points identical - if you move one point to be more accurate because it was slightly off, then you have to do the same in every other route that meets there).
NMP is not a true error (unless a concurrency that should be isn't registered) that causes issues with end users and statistics and so on. But sure, it's a falling short from perfection and ought to be flagged as a potential error.
And certainly, while you have a script for it, I don't want files merged to the average location of the point. All the GB ones come from when, either through peer review or update, I've got a more pinpointed location, and simply not done it on other routes. Averaging out would take a 0m wrong point and a 10m wrong point and make 2x 5m wrong points. If I sort out those NMPs, it will be manually and with the priority of the snipe hunt that it is.
Whoops. I left a drafted response here on Desktop #1 for weeks now...
Si, I'm glad to see you NMPs as a falling short from perfection [which] ought to be flagged as a potential error.
I always regarded it as best practice to ensure intersecting routes had identical coordinates, worth the anal retentiveness and a little extra hassle.
True, with Tim's existing algorithms, close-enough points still flag an intersecting route; no-harm-no-foul.
One justification I see is future-proofing routes just in case a concurrency comes along in the future.
A real-world example:
ME US1 ME196 @ 43.919319°, -69.953256°
ME ME196 US1 @ 43.919319°, -69.953255°
I just checked this a few minutes ago, and [Popeye] Well blow me down! [/Popeye] -- the points actually have different coords. A perfect example.
(Looks like a case of that mysterious "point drift" that plagued files in the early years. But anyway...)
Intersecting highways are shown and linked just as intended.
BUT! In October 2014 (after the last hwy data update) ME24 was relocated, to follow US1, ME196, and Bypass Dr.
My local, updated ME ME24 has US1_S @ 43.919319°, -69.953256°.
If my local copies of all three files are submitted & ingested into the DB & yadda yadda, the US1/ME24 concurrency will register, but ME24/196 will not.
Having point coords match up from the get-go will avoid having to go back later when a new concurrency comes along and adjust point coords on intersecting routes.
Quote from: yakra on July 03, 2015, 01:56:52 PMHaving point coords match up from the get-go will avoid having to go back later when a new concurrency comes along and adjust point coords on intersecting routes.
Sure, but when some nasty peer-reviewer called Eric comes and asks me to move a point a tiny bit, do I move it on all intersecting routes (often ones that have been peer reviewed and perhaps are active) that were identical or do I simply wait for a concurrency to require the adjustment? ;)
It's clear you feel that moving waypoints so that the concurrency works is bad, so (to play devil's advocate) why should I do it when there's no concurrency to require it?
Also, I had one the other day where the points were identical coordinates (I tripled checked) in Tim's .wpt editor, but they weren't giving the DC error, and I could see they were different. Bizzareness in Arizona (making Historic 66 file, given it is signed)!
Quote from: english si on July 03, 2015, 04:00:47 PM
Sure, but when some nasty peer-reviewer called Eric comes and asks me to move a point a tiny bit, do I move it on all intersecting routes (often ones that have been peer reviewed and perhaps are active) that were identical or do I simply wait for a concurrency to require the adjustment? ;)
The proper procedure would be to silently tell that blighter to get stuffed. :bigass:
QuoteIt's clear you feel that moving waypoints so that the concurrency works is bad, so (to play devil's advocate) why should I do it when there's no concurrency to require it?
Wait, uhh... that bit confuses me. It's not clear. I don't feel that way.
Unless you're joking and it sailed over my head. Stupid Internets.
FWIW, I did end up changing the point coords on ME196 to match the other two files.
It's also just better when you zoom in on the map and everything lines up perfectly. :spin:
I think part of the problem with making fixes in CHM was the fact that there was just one complete copy of the data, and only one person had the authority to make changes to it. Other people had copies of just their own parts, and coordinating changes that affected multiple people meant distributed copies being updated, all kinds of problems that the CHM collaborators understand all too well. I think it will be important to get all those who will be contributing seriously to the highway data to be up to speed on git. We don't want just anyone to be able to commit/push/whatever changes back in, but we do want people to be able to make the changes, and after a review process that we need to come up with, the changes could either be rejected, sent back for further fixing, or integrated into the master copy. I really think we're going to be able to manage changes, especially fixes, much more efficiently. A big part of it is that before if someone made a change that affected usany, for example, it had to go through me. If someone else changed the official copy, I'd have to update my copy. With git, that's all manageable. We merge in the reviewed and approved change, and I pull down the update into my own local working copy.
We'll also be able to have multiple instances of the project very easily. We could for example have a staging server where we can try out a new system (sort of the highway activation equivalent of a "soft opening" of a store) and see how it all looks and work out some problems. Since everything is in public repositories, there's nothing stopping an individual from experimenting with a full copy of the project on their own computer/server/whatever.
If you're following the GitHub repositories (and who isn't?), you might have seen the current data, now in the new directory organization, has been added to the master copy of the HighwayData repository by Si. We can start working on fixing up the current data. If you are and plan to continue to maintain data in some regions, I think it's best to work through GitHub. I'm hoping someone with some experience in it can work with Si and I to come up with the appropriate workflow and instructions on how to make it happen. At the moment, Si and I are the only ones who can pull changes into the master, but I hope we'll have more people able to do so once we have the process down.
Quote from: Jim on July 06, 2015, 09:40:02 PM
If you're following the GitHub repositories (and who isn't?), you might have seen the current data, now in the new directory organization, has been added to the master copy of the HighwayData repository by Si. We can start working on fixing up the current data. If you are and plan to continue to maintain data in some regions, I think it's best to work through GitHub. I'm hoping someone with some experience in it can work with Si and I to come up with the appropriate workflow and instructions on how to make it happen. At the moment, Si and I are the only ones who can pull changes into the master, but I hope we'll have more people able to do so once we have the process down.
I will want to make updates, but that'll take a few weeks until i get back home (now in northern Alberta -- yakra asked me to check out some things there, which I hope to take care of tomorrow) to assemble my updates, and also can get the hang of GitHub. I agree that going through GitHub, rather than by e-mails to Jim or Si, makes the most sense.
Part of the process to be worked out (if not already) is creation of an "updates" page, and procedure to make additions to that page. Would be nice to incorporate CHM's update items through mid-2014, so we have in one place all the changes going into the highway data we're using. But that's not high-priority unless we think the CHM site (with its updates lists) is going to wink out on us.
Fortunately, my travel schedule will ease up (both I and my car are already pretty beat up) as Jim starts his own travels, then gears up for the upcoming academic year. So I'll be in a position to pick up some of the slack on the non-technical side.
Regarding finding clinched segments for a user from the DB so they can be colored differently on maps: I decided that rather than sending an email about this to SSOWorld, I'd post here in case it's useful to others. I'm also very open to suggestions about better ways to do this.
In https://github.com/TravelMapping/Web/blob/master/hbtest/mapview.php (https://github.com/TravelMapping/Web/blob/master/hbtest/mapview.php), it's the code from lines 196 to 234 that's using SQL query results to build the JS arrays that are used for this.
The first part builds an array of waypoints, with the waypoint labels and coordinates that exist in the systems or regions being mapped. Along the way, a second array called newRouteIndices keeps track of the first index into the waypoints array for each route. There are also arrays built there that track the tier and color for each of those routes.
The second part builds an array of segments by "segmentID" which is just a unique identifier for each segment from the DB. The key here is that these segments come out of the DB in the same order as the waypoints. So if the first route being plotted has, say, 27 waypoints, the first 26 segments in the segments array are the 26 segments connecting those 27 waypoints. Then the next route has 11 waypoints, so the next 10 segments entries connect those 11 waypoints, and so on. So even though we only get segmentID out of the DB and save only that number in the segments array, from it, we can find the corresponding waypoints at each end of the segment from the waypoints array.
The third part builds an array of clinched segments, again by segmentID. So when plotting the segments later, we'll check to see if the segmentID is found in the clinched array. If so, plot it with a vivid color, otherwise with the lighter shade. We can do this check efficiently as well, since we know the segmentIDs in clinched (when they exist) are in the same order as in the segments array.
So the PHP here builds all of this, and it's the JS code in https://github.com/TravelMapping/Web/blob/master/hbtest/chmviewerfunc3.js (https://github.com/TravelMapping/Web/blob/master/hbtest/chmviewerfunc3.js) that uses it all to plot the maps. The updateMap function does the work. JS will execute the PHP-constructed function waypointsFromSQL to populate the arrays as described above. The logic in play here to use the waypoints, newRouteIndices, segments, and clinched arrays is in updateMap starting at line 637. I think the logic should be similar for building other kinds of maps, except you won't be making google.maps.Polyline objects, you'll be drawing on your map.
Writing this reminds me of some of the things I hope to enhance on the Google Maps overlay maps implemented in the files described above. I'd like to be able to have ways to highlight a particular route, to be able to click on a segment and see information about it (what route(s) it carries, clinched status, length, endpoint waypoint names, etc.). As I mentioned yesterday, that's on hold for a bit.
What steps need to be taken to use the stats/logs and maps that are currently on Jim's test site and create a web based front end?
My latest log looks great (except for a couple minor errors to fix on the next update) http://www.teresco.org/~terescoj/travelmapping/logs/bejacob.log (http://www.teresco.org/~terescoj/travelmapping/logs/bejacob.log) as do the various maps, such as my home state of Ohio http://www.teresco.org/~terescoj/travelmapping/hbtest/mapview.php?rg=OH&u=bejacob (http://www.teresco.org/~terescoj/travelmapping/hbtest/mapview.php?rg=OH&u=bejacob).
I've looked at the source code from the CHM web pages and only understand a fraction of it. :confused: I'm sure many many folks working on this have excellent web/programming skills and understand exactly what the code that generates a page like this http://cmap.m-plex.com/stat/region.php?u=bejacob&c=usa&rg=oh&du=mi&sort=ra (http://cmap.m-plex.com/stat/region.php?u=bejacob&c=usa&rg=oh&du=mi&sort=ra) does.
How easy would it be to create a page like this http://cmap.m-plex.com/stat/traveler.php (http://cmap.m-plex.com/stat/traveler.php) for the 30+ users that have .list files in the test db?
With the stats/logs available, who is able to take this output and transform it into web pages?
I'm impressed with everything that's happened so far. I don't know how much of it was Jim's work and how much others have contributed, but I thank everyone involved. I've been working with Jim to make sure my maps/list are correct (nearly there). Now I want to see what sort of web interface there will be. I'm sure it will take some time.
I think I speak on behalf of other non-techie folks when I say, keep up the good work. :clap: We're eager to see what comes next.
Brian
Quote from: bejacob on July 16, 2015, 05:31:15 PM
What steps need to be taken to use the stats/logs and maps that are currently on Jim's test site and create a web based front end?
My latest log looks great (except for a couple minor errors to fix on the next update) http://www.teresco.org/~terescoj/travelmapping/logs/bejacob.log (http://www.teresco.org/~terescoj/travelmapping/logs/bejacob.log) as do the various maps, such as my home state of Ohio http://www.teresco.org/~terescoj/travelmapping/hbtest/mapview.php?rg=OH&u=bejacob (http://www.teresco.org/~terescoj/travelmapping/hbtest/mapview.php?rg=OH&u=bejacob).
I've looked at the source code from the CHM web pages and only understand a fraction of it. :confused: I'm sure many many folks working on this have excellent web/programming skills and understand exactly what the code that generates a page like this http://cmap.m-plex.com/stat/region.php?u=bejacob&c=usa&rg=oh&du=mi&sort=ra (http://cmap.m-plex.com/stat/region.php?u=bejacob&c=usa&rg=oh&du=mi&sort=ra) does.
How easy would it be to create a page like this http://cmap.m-plex.com/stat/traveler.php (http://cmap.m-plex.com/stat/traveler.php) for the 30+ users that have .list files in the test db?
With the stats/logs available, who is able to take this output and transform it into web pages?
I'm impressed with everything that's happened so far. I don't know how much of it was Jim's work and how much others have contributed, but I thank everyone involved. I've been working with Jim to make sure my maps/list are correct (nearly there). Now I want to see what sort of web interface there will be. I'm sure it will take some time.
I think I speak on behalf of other non-techie folks when I say, keep up the good work. :clap: We're eager to see what comes next.
Brian
These will be implemented down the road but the priority right now is to get the statistical calculations working and the mapping application functional.
As I mentioned elsewhere, I have a code base started to build the "CHM style" mapping system but that is in the "alpha" stage right now and has many issue that need to be resolved. It draws the lines and that's it. The end result ideally will work - at minimum - like on Tim's site. I hope to get back on that next weekend (out of town this weekend). The maps can be developed in parallel with the stats pages, but the issue is with the current resources, that may not happen yet, but it is on the list.
Thanks for the answer. I know there is a great deal of effort needed to make these things happen. It's nice to know where things stand.
(Hello briefly from Thunder Bay, ON.) One more note: while the information in the log files is or can easily be extended to all of the stats we'd need and more, the logs themselves won't be used for that purpose. Stats pages should query the DB as needed and generate pages on the fly. It shouldn't be hard - relatively straightforward web stuff.
Quote from: Jim on July 17, 2015, 08:39:12 PM
(Hello briefly from Thunder Bay, ON.) One more note: while the information in the log files is or can easily be extended to all of the stats we'd need and more, the logs themselves won't be used for that purpose. Stats pages should query the DB as needed and generate pages on the fly. It shouldn't be hard - relatively straightforward web stuff.
Agreed.
and how's the great white north? :sombrero:
Quote from: SSOWorld on July 17, 2015, 08:43:52 PM
and how's the great white north? :sombrero:
Excellent! And green.
Quote from: Jim on July 17, 2015, 08:48:10 PM
Quote from: SSOWorld on July 17, 2015, 08:43:52 PM
and how's the great white north? :sombrero:
Excellent! And green.
Hope the wildfire smoke isn't drifting in there from Manitoba. It was worst in northern Alberta and Saskatchewan, where the fires were when I was there last week, but much of the smoke was drifting southeast.
In case anyone's interested in seeing the output produced by the Python site update program, the site update script now captures that program's output and places it with the other log files:
http://tm.teresco.org/logs/read_data.log (http://tm.teresco.org/logs/read_data.log)
The numbers at the start of many lines are elapsed times in seconds from the invocation of the program.
I was wondering why a route in St Petersburg had so many co-locations, and then I remembered that Oscar copied that route as a place holder for California state routes.
Speaking of co-locations, two things related to them:
1) NMP and the ability to add FPs would be useful
2) In the Highway Browser, intersecting route links would be fantastic.
PS: Thunder Bay is in the north? I'm only a few miles short of nearly 3.5 degrees further north and I'm in the south of my country. :-P
Quote from: english si on November 25, 2015, 04:54:05 PM
I was wondering why a route in St Petersburg had so many co-locations, and then I remembered that Oscar copied that route as a place holder for California state routes.
That threw me at first, too.
QuoteSpeaking of co-locations, two things related to them:
1) NMP and the ability to add FPs would be useful
2) In the Highway Browser, intersecting route links would be fantastic.
Both definitely in my plans.
Jim: how much of the scripting is being written in Python? Asking because I'm taking a Python class this semester.
Quote from: froggie on November 25, 2015, 11:33:33 PM
Jim: how much of the scripting is being written in Python? Asking because I'm taking a Python class this semester.
All of the data processing to go from .csv and .wpt data into logs, stats, and a mysql DB instance with that information in a form useful for the web side is in Python.
https://github.com/TravelMapping/DataProcessing/blob/master/siteupdate/python-teresco/read_data.py (https://github.com/TravelMapping/DataProcessing/blob/master/siteupdate/python-teresco/read_data.py)