News:

Thanks to everyone for the feedback on what errors you encountered from the forum database changes made in Fall 2023. Let us know if you discover anymore.

Main Menu

Technical/Design/Implementation Discussions (CHM/Travel Mapping)

Started by Jim, April 04, 2015, 09:50:22 PM

Previous topic - Next topic

yakra

My two cents:

I sort by subregion on my PC:
chm/me, chm/ns, etc...
then chm/me/usai, chm/me/usaus, chm/me/usausb, chm/me/usame, chm/ns/cantch, chm/nb/cannsf, chm/nb/cannst, etc...

This works fine for me since I only work on two North American countries with non-overlapping postal codes. Project-wide, it'd make more sense to do
usa, can, etc...
usa/me, can/ns, etc...
"Officer, I'm always careful to drive the speed limit no matter where I am and that's what I was doin'." Said "No, you weren't," she said, "Yes, I was." He said, "Madam, I just clocked you at 22 MPH," and she said "That's the speed limit," he said "No ma'am, that's the route numbah!"  - Gary Crocker


Jim

Quote from: english si on April 20, 2015, 12:18:34 PM
Quote from: Dr Frankenstein on April 20, 2015, 10:47:43 AMCreate the repo on github, clone it onto your computer, add the files to the directory and 'add' them to git (not sure how to batch-add in command-line, but it's easy enough using any of the GUIs), commit,
Done all that, though had to reduce the number of files in each batch, so there's a good ten or so commits.
Quotepush.
Can't seem to do that. The 'publish' (which is the only option I have) fails.

Are you trying in the TravelMapping/HighwayData repository?  If so, I might just need to add some permissions for you.  Let's have them in some subdirectory for now (maybe something like "chm_final") and we can then move things into a new hierarchy when we've settled on the organization and as each system is deemed ready to go.

As a side note to this, I feel strongly that we need to continue to do a good job citing our sources used to derive our data.
Photos I post are my own unless otherwise noted.
Signs: https://www.teresco.org/pics/signs/
Travel Mapping: https://travelmapping.net/user/?u=terescoj
Counties: http://www.mob-rule.com/user/terescoj
Twitter @JimTeresco (roads, travel, skiing, weather, sports)

english si

Quote from: yakra on April 20, 2015, 12:49:53 PM
My two cents:

I sort by subregion on my PC:
chm/me, chm/ns, etc...
then chm/me/usai, chm/me/usaus, chm/me/usausb, chm/me/usame, chm/ns/cantch, chm/nb/cannsf, chm/nb/cannst, etc...
This is another vote for option 3 then?

Burying the subregions in a directory is possible. Another option is USA_AK rather than USA/AK. Mexico, Russia and Kazakhstan are in RUS-MOW or whatever (at least one Mexico code doesn't meet the ISO standards, where they are all four digits)*. The USA_ bit merely puts all USA regions together, rather than strewn across the other regions.

Australia is the only place where we are likely to have a clash, unless we AUS- them (I say no), as there's WA and NT (Western Australia and Northern Territory). WAU and ANT (and SAU even though SA doesn't clash with the US/CAN 2-letter postal system) would fix this, and have the codes all three digits, matching ACT, NSW, QLD, TAS and VIC.

*I propose getting rid of the lot. Certainly for Russia and Kazakhstan, there's not the volume of routes that we're likely to have, nor are the splits needed for route length as the distance is contrasted by sparsity and flatness of the land making the point distances relatively far apart. I don't know about Mexico. I'm not sure that the ISO 3116-2 codes clash with 3-letter 3116-1 country codes, allowing a dropping of MEX- if we keep the subdivisions. And as said elsewhere, I'd rather get rid of the British ones which are odd as there's stronger autonomy elsewhere in Europe (eg Spain) that doesn't translate to subdivisions, but that would muck up .list files of too many people.

rschen7754

Quote from: english si on April 20, 2015, 02:48:52 PM

Australia is the only place where we are likely to have a clash, unless we AUS- them (I say no), as there's WA and NT (Western Australia and Northern Territory). WAU and ANT (and SAU even though SA doesn't clash with the US/CAN 2-letter postal system) would fix this, and have the codes all three digits, matching ACT, NSW, QLD, TAS and VIC.


Brazil and India have states and are pretty large countries, so in theory those could be subdivided, were they ever to be added.

Jim

I'm hoping to start making some headway on some part of this project in about a week.  I have a little time between the end of my spring classes and the start of my summer class. 

I'm thinking I'd like to develop some code that takes all of the data and populates a database, at least something those on the web side will be able to read from as they start working on other parts.  So, if you're going to write a new highway browser, compute and report stats pages, or generate maps, what should I make sure is in the database and what form would be convenient?  And maybe you would want to comment on my overall thought that the "ingestion" process of the .csv, .wpt, and .list data (or their successors) would do most of the work so the web-facing code would not be computationally intensive on the server or client side.
Photos I post are my own unless otherwise noted.
Signs: https://www.teresco.org/pics/signs/
Travel Mapping: https://travelmapping.net/user/?u=terescoj
Counties: http://www.mob-rule.com/user/terescoj
Twitter @JimTeresco (roads, travel, skiing, weather, sports)

vdeane

yakra already wrote a parser that can take all that and put it into a map.  The next logical step would be to decide how the database would look and modify the parser (and map? or do that later?) to put the contents in the database.
Please note: All comments here represent my own personal opinion and do not reflect the official position of NYSDOT or its affiliates.

english si

Quote from: Jim on April 20, 2015, 01:28:53 PMAre you trying in the TravelMapping/HighwayData repository?  If so, I might just need to add some permissions for you.  Let's have them in some subdirectory for now (maybe something like "chm_final") and we can then move things into a new hierarchy when we've settled on the organization and as each system is deemed ready to go.
If I have permissions, I will be certainly put it in chm_final.

Jim

Quote from: vdeane on May 01, 2015, 10:51:49 PM
yakra already wrote a parser that can take all that and put it into a map.  The next logical step would be to decide how the database would look and modify the parser (and map? or do that later?) to put the contents in the database.

Hope we can get that code into the repository to use or at least refer to.
Photos I post are my own unless otherwise noted.
Signs: https://www.teresco.org/pics/signs/
Travel Mapping: https://travelmapping.net/user/?u=terescoj
Counties: http://www.mob-rule.com/user/terescoj
Twitter @JimTeresco (roads, travel, skiing, weather, sports)

Jim

Quote from: english si on May 02, 2015, 05:34:37 AM
Quote from: Jim on April 20, 2015, 01:28:53 PMAre you trying in the TravelMapping/HighwayData repository?  If so, I might just need to add some permissions for you.  Let's have them in some subdirectory for now (maybe something like "chm_final") and we can then move things into a new hierarchy when we've settled on the organization and as each system is deemed ready to go.
If I have permissions, I will be certainly put it in chm_final.

I've created a new group (of which you and I are the only members so far) whose members should be able to write to the TravelMapping/HighwayData repository.  See if you can clone, add/commit to your local fork, then push back.
Photos I post are my own unless otherwise noted.
Signs: https://www.teresco.org/pics/signs/
Travel Mapping: https://travelmapping.net/user/?u=terescoj
Counties: http://www.mob-rule.com/user/terescoj
Twitter @JimTeresco (roads, travel, skiing, weather, sports)

english si

Sheer number of files means multiple commits needed to avoid crashing, but if you look, I've got all non-USA .wpts in a fork.

Should get the USA data in later, and will pull it in (I can't see any option to 'push'. I can 'publish' the fork, and create a pull request, which (presumably) I can respond to)

Jim

Quote from: english si on May 02, 2015, 08:54:26 AM
Sheer number of files means multiple commits needed to avoid crashing, but if you look, I've got all non-USA .wpts in a fork.

Should get the USA data in later, and will pull it in (I can't see any option to 'push'. I can 'publish' the fork, and create a pull request, which (presumably) I can respond to)

Great.  Still need to figure out the best way to manage things, but I'm happy we'll soon have a complete copy of the last version of the CHM data safe and sound.
Photos I post are my own unless otherwise noted.
Signs: https://www.teresco.org/pics/signs/
Travel Mapping: https://travelmapping.net/user/?u=terescoj
Counties: http://www.mob-rule.com/user/terescoj
Twitter @JimTeresco (roads, travel, skiing, weather, sports)

sammi

I'm curious.. Looking at the waypoint files, there seems to be the OpenStreetMap URL on every line. That's redundant, don't you think? I mean you could like make the data ½ or ⅓ the size, by having the lines have only the coordinates like:

    BlaCrkDr +0 43.709994 -79.501424




While I'm here, I think I'd like to join the project. I have a reasonable amount of experience designing databases and developing web applications in Python and PHP, and I could probably also do a bit of web design.

It would be great if you could add me to the GitHub organization; my username is sammdot.

english si

Quote from: sammi on May 02, 2015, 10:05:27 AM
I'm curious.. Looking at the waypoint files, there seems to be the OpenStreetMap URL on every line. That's redundant, don't you think? I mean you could like make the data ½ or ⅓ the size, by having the lines have only the coordinates like:

    BlaCrkDr +0 43.709994 -79.501424
We discussed this above. It's easier to parse as <lat> <lon> <name> <alt name> and if we change it, it would be to that.

Size isn't an issue as files are peanuts sizes (biggest one is under 50kB) and most are in the 1k-2k ball park. The actual highway (not boundary) data is 16642 files and 23.3MB (though 74.1MB on my hard disk - no doubt due to the huge amount of files and the 115 folders they are in).

I've merged my fork in. It's done!

Jim

Photos I post are my own unless otherwise noted.
Signs: https://www.teresco.org/pics/signs/
Travel Mapping: https://travelmapping.net/user/?u=terescoj
Counties: http://www.mob-rule.com/user/terescoj
Twitter @JimTeresco (roads, travel, skiing, weather, sports)

vdeane

Quote from: Jim on May 02, 2015, 07:18:53 AM
Quote from: vdeane on May 01, 2015, 10:51:49 PM
yakra already wrote a parser that can take all that and put it into a map.  The next logical step would be to decide how the database would look and modify the parser (and map? or do that later?) to put the contents in the database.

Hope we can get that code into the repository to use or at least refer to.
It's over here: http://205.209.84.174/roads/chm/canvas/
Please note: All comments here represent my own personal opinion and do not reflect the official position of NYSDOT or its affiliates.

Jim

Quote from: vdeane on May 03, 2015, 07:51:36 PM
Quote from: Jim on May 02, 2015, 07:18:53 AM
Quote from: vdeane on May 01, 2015, 10:51:49 PM
yakra already wrote a parser that can take all that and put it into a map.  The next logical step would be to decide how the database would look and modify the parser (and map? or do that later?) to put the contents in the database.

Hope we can get that code into the repository to use or at least refer to.
It's over here: http://205.209.84.174/roads/chm/canvas/

This is good.  If there are no objections from the authors, I'd like to get that into our repository as reference code.
Photos I post are my own unless otherwise noted.
Signs: https://www.teresco.org/pics/signs/
Travel Mapping: https://travelmapping.net/user/?u=terescoj
Counties: http://www.mob-rule.com/user/terescoj
Twitter @JimTeresco (roads, travel, skiing, weather, sports)

Jim

I propose we go ahead and convert all route (.wpt) files to be included in the project to the

<lat> <lon> <name> [<alt_name>] [<alt_name>] ...

format.

I'm OK with keeping the ".wpt" file extension but maybe we should use something new.  I don't think it's the 1980's anymore, so we could use extensions with more than three characters if we want.  We could use ".route" or ".waypoint" or ".highway" or something else.

A converter is pretty much trivial to write.  If we agree on the above format, I'll put a converter in a few languages up there for anyone to use, and will gladly run it to get us a clean starting point.

The complications I see at the moment:

- The wpt editor would need to be rewritten/updated to use the new format.  Any volunteers?  I suppose we can't expect the current one to stay around forever anyway.

- What to do about all of those updates various people have made to their own copies and submitted to CHM but which didn't make it into the data we copied over?  Get them all brought in now and then convert, or use the last good CHM snapshot as a starting point for the conversion, and all new updates and new systems would have to be converted to the new format before being added?
Photos I post are my own unless otherwise noted.
Signs: https://www.teresco.org/pics/signs/
Travel Mapping: https://travelmapping.net/user/?u=terescoj
Counties: http://www.mob-rule.com/user/terescoj
Twitter @JimTeresco (roads, travel, skiing, weather, sports)

froggie

What did we do when Tim converted from the previous format (.ggm?) to .wpt?  I don't recall all of it.

rickmastfan67

Quote from: froggie on June 02, 2015, 11:02:20 PM
What did we do when Tim converted from the previous format (.ggm?) to .wpt?  I don't recall all of it.

I think we just changed the file extension at that time.

oscar

Quote from: Jim on June 02, 2015, 09:20:21 PM
I propose we go ahead and convert all route (.wpt) files to be included in the project to the

<lat> <lon> <name> [<alt_name>] [<alt_name>] ...

format.

Sounds good to me.

Quote from: Jim on June 02, 2015, 09:20:21 PM
I'm OK with keeping the ".wpt" file extension but maybe we should use something new.  I don't think it's the 1980's anymore, so we could use extensions with more than three characters if we want.  We could use ".route" or ".waypoint" or ".highway" or something else.

A different extension would distinguish new-format from old-format files. But .wpt never caused me any problems, even though I would be the most likely to draw them (as a user of WordPerfect, which uses .wpt for some of its files).

Quote from: Jim on June 02, 2015, 09:20:21 PM
The complications I see at the moment:

- The wpt editor would need to be rewritten/updated to use the new format.  Any volunteers?  I suppose we can't expect the current one to stay around forever anyway.

The other thing is that while existing CHM collaborators have permission to use the old editor, we would want something available to new team members.

Quote from: Jim on June 02, 2015, 09:20:21 PM
- What to do about all of those updates various people have made to their own copies and submitted to CHM but which didn't make it into the data we copied over?  Get them all brought in now and then convert, or use the last good CHM snapshot as a starting point for the conversion, and all new updates and new systems would have to be converted to the new format before being added?

I favor the latter approach. That would let me hold off on learning github for awhile, at least until my own travel schedule eases up in late July. It has the more global additional advantage of letting us compare what our system spits out to what CHM displays, to identify any differences that shouldn't be there.

I would also favor holding up on submission of new under-development systems, and updates to existing under-development systems (like my Alaska State Highways set, which has several significant unsubmitted updates and will need some more minor tweaks).
my Hot Springs and Highways pages, with links to my roads sites:
http://www.alaskaroads.com/home.html

Jim

As a first attempt, I'm adding code to my Python program that generates SQL code that will populate a database with all of the information from a subset of the .csv and .wpt files.  This part is pretty much independent, other than just a few lines of code, of the .wpt file format and directory organization we've been discussing.

They way I'm thinking of this (as stated previously but now being implemented a bit) is to have a program that takes the entire state of the project (all of the .csv, .wpt, .list or their successor formats) and creates a big SQL file that can be imported into any instance of mysql (probably others but I'm using mysql), which can then in turn be queried by all of our web-facing tools.
Photos I post are my own unless otherwise noted.
Signs: https://www.teresco.org/pics/signs/
Travel Mapping: https://travelmapping.net/user/?u=terescoj
Counties: http://www.mob-rule.com/user/terescoj
Twitter @JimTeresco (roads, travel, skiing, weather, sports)

sipes23

Quote from: Jim on June 02, 2015, 09:20:21 PM
- What to do about all of those updates various people have made to their own copies and submitted to CHM but which didn't make it into the data we copied over?  Get them all brought in now and then convert, or use the last good CHM snapshot as a starting point for the conversion, and all new updates and new systems would have to be converted to the new format before being added?

Even though I've not been in on the technical decisions, Oscar is right about taking the last good CHM snapshot as the starting point. It would save on a lot of misery to be able to proofread what has been generated and verify that everything is working before moving onward and upward. What's more, since the CHM data has (presumably) been checked for accuracy before, we wouldn't be checking accuracy from scratch but proofreading against a copy (which is easier in my opinion).

english si

I had a post on this that I lost, but yes.

I suggest a folder with it in the <subregion>/<system> layout, which can then be converted.

In fact, if Github wasn't being annoying, that would be up.

Jim

I'd like to get a handful of .list files I can use to experiment with populating my first attempt at a database of the travel mapping project data.  I've already picked a fairly small subset of the highway data, and I'll put in my own .list.  Rather than selecting some random ones, I figured I'd see who might like theirs to be part of initial development and testing.  Send CHM username if your .list is up-to-date on the old site, or email your .list directly to me (send private message on this board if you don't have my email).

Photos I post are my own unless otherwise noted.
Signs: https://www.teresco.org/pics/signs/
Travel Mapping: https://travelmapping.net/user/?u=terescoj
Counties: http://www.mob-rule.com/user/terescoj
Twitter @JimTeresco (roads, travel, skiing, weather, sports)

sammi

Quote from: Jim on June 07, 2015, 09:51:19 PM
I'd like to get a handful of .list files I can use to experiment with populating my first attempt at a database of the travel mapping project data.  I've already picked a fairly small subset of the highway data, and I'll put in my own .list.  Rather than selecting some random ones, I figured I'd see who might like theirs to be part of initial development and testing.  Send CHM username if your .list is up-to-date on the old site, or email your .list directly to me (send private message on this board if you don't have my email).

My .list file is on GitHub:

    https://github.com/sammdot/my-clinched-highways

Should be way more recent than the one on CHM.



Opinions expressed here on belong solely to the poster and do not represent or reflect the opinions or beliefs of AARoads, its creators and/or associates.