News:

The AARoads Wiki is live! Come check it out!

Main Menu

Stats pages

Started by mapcat, December 15, 2015, 11:00:17 PM

Previous topic - Next topic

mapcat

Hello all,

I'd like to get more involved in the TravelMapping project, and now that I have a few weeks off, I was wondering if anyone would be able to work with me for a few days to help me get comfortable with GitHub. Eventually I'd like to be in a position to take over Ohio and possibly another unclaimed state.

On a related note, I'm also interested in figuring out a way to produce some simple user stats for TM. Unfortunately I don't know how to code anything that could pull the info from the database or output it into a simple text file similar to the current log files. Again, with plenty of time to learn and test, I just need someone willing to offer some basic information on how to get started.

Stats I'm interested in compiling would be similar to those produced by CHM, both for individual users and the entire group.

All traveler stats:
Totals/rank for various systems, like this: http://cmap.m-plex.com/stat/travsystem.php?sys=usai
Totals/rank for various regions, like this: http://cmap.m-plex.com/stat/travregion.php?c=usa&rg=ky&sort=dd

Individual stats:
Summary pages for various systems, like this: http://cmap.m-plex.com/stat/system.php?u=mapcat&sys=usai&du=mi&sort=ra&rpp=2000#th
Summary pages for various regions, like this: http://cmap.m-plex.com/stat/region.php?u=mapcat&c=usa&rg=al&du=mi&sort=ra&rpp=2000#th


Jim

I've been in touch with Dave about this offline.  For anyone interested in helping out with the stats pages, I think most if not all of the stats are already in or could easily be added to the DB.  All of these stats pages are a matter of getting (from a web form/query string parameters) the desired subset of the stats, formulating the appropriate SQL query or queries, and putting them into a PHP page that makes the query and reports the results in HTML.

It's fairly high on my list of priorities to get going on this.  So there's a chance progress could be made by me over the coming break, but most likely it's going to be summer.  So I definitely welcome contributions from anyone who can help make this all a reality sooner.
Photos I post are my own unless otherwise noted.
Signs: https://www.teresco.org/pics/signs/
Travel Mapping: https://travelmapping.net/user/?u=terescoj
Counties: http://www.mob-rule.com/user/terescoj
Twitter @JimTeresco (roads, travel, skiing, weather, sports)

Rothman

I like the stats that are already available through the log, but also look forward to what comes of this effort.
Please note: All comments here represent my own personal opinion and do not reflect the official position(s) of NYSDOT.

vdeane

I can think of a couple reason to look forward to these:
-Able to link to my TravelMapping site from elsewhere just like I could from my CHM site
-Ability to open route pages in the highway browser with my travels already highlighted
Please note: All comments here represent my own personal opinion and do not reflect the official position of NYSDOT or its affiliates.

yakra

Quote from: vdeane on December 16, 2015, 01:22:56 PM
-Ability to open route pages in the highway browser with my travels already highlighted
You can do this now, although you need to manually edit the URL:
http://tm.teresco.org/devel/hb.php?r=ny.ny005trksch&u=yakra
It would be nice though to have a persistent u=username that can follow you around from page to page, or that works on the main hb.php page...
"Officer, I'm always careful to drive the speed limit no matter where I am and that's what I was doin'." Said "No, you weren't," she said, "Yes, I was." He said, "Madam, I just clocked you at 22 MPH," and she said "That's the speed limit," he said "No ma'am, that's the route numbah!"  - Gary Crocker

vdeane

That's why I phrased it as "already highlighted".  Manually editing the URL can get annoying when one is making a large .list file update.
Please note: All comments here represent my own personal opinion and do not reflect the official position of NYSDOT or its affiliates.

bejacob

I'll add my voice to those who want to see a web-based stats page similar to what CHM had. I find the logs to be useful, but the functionality of of the old web interface was quite good. Being able to click through to various details (system, region, user, etc.) was great. I could view my progress and plan future travel. I can still do these things, it just take a bit more effort.

I also think that adding some of the user interface will encourage more folks from the old site to come take a look at what is happening here. The work done so far has been great. It's nice to see regular updates and new systems being activated. Seeing that continue alongside interactive stats will really be exciting.

Thing 342

Now that I have some free time on my hands, I'd like to volunteer to help with this. I'm currently working on a basic concept design for the userpage in my fork of the Web repository. Right now, my goal is to replicate most of the functionality of the existing CHM userpages.

Jim

Quote from: Thing 342 on December 19, 2015, 01:02:58 PM
Now that I have some free time on my hands, I'd like to volunteer to help with this. I'm currently working on a basic concept design for the userpage in my fork of the Web repository. Right now, my goal is to replicate most of the functionality of the existing CHM userpages.

Excellent!  Most if not all of the stats we'd want to display are readily available in the DB.  It would be good to have the option, as we do on the maps, to select either the regular DB instance or the one that includes the preview routes as if they were active.

See also my plans for a short-term solution for those looking for stats and rankings: https://github.com/TravelMapping/DataProcessing/issues/8
Photos I post are my own unless otherwise noted.
Signs: https://www.teresco.org/pics/signs/
Travel Mapping: https://travelmapping.net/user/?u=terescoj
Counties: http://www.mob-rule.com/user/terescoj
Twitter @JimTeresco (roads, travel, skiing, weather, sports)

Jim

I put together a first version of TM stats as csv files.  In http://tm.teresco.org/logs/ (http://tm.teresco.org/devlogs/ if you want preview systems included in stats) there are now a bunch of .csv files.  The file allbyregion.csv lists all travelers and their total clinched mileage in each region.  All travelers have a line in this file.  The files *-all.csv, one for each TM system, has per-system stats by traveler.  Travels are further broken down by region.  These files only have lines for those travelers who have any clinched mileage in a system.

If this sounds interesting to you, please download and check out some of the files and spot check for accuracy.  The idea is that you can load the .csv files into your favorite spreadsheet and sort by various columns to get (in a very basic, but complete, form) user rankings by region overall and within each system.

If these files look OK, it's easy for me to add more stats to these or to create additional files.  The thing I think might be most interesting to me is to add all columns for each connected route to each system-specific csv, or all routes in new region-specific csvs.

These take just a little effort for me to create and add negligible cost to the site update process, so if people find them useful even just to tide you over until a nice web front-end gets developed, I'm happy to expand them.

Important note: these files are not intended to be read in and processed into web pages.  All of the information in these files has been in the DB for months, and DB queries should be used in the development of web-based stats pages.
Photos I post are my own unless otherwise noted.
Signs: https://www.teresco.org/pics/signs/
Travel Mapping: https://travelmapping.net/user/?u=terescoj
Counties: http://www.mob-rule.com/user/terescoj
Twitter @JimTeresco (roads, travel, skiing, weather, sports)

michih

Quote from: Jim on December 19, 2015, 11:26:14 PMAll travelers have a line in this file.

Thanks. Is it possible to exclude non-real users like eroads_2digit, eroads_3digit, eroads_all, eroads_even, eroads_main, eroads_odd et cetera?

btw: Why do these user exist at all?

Jim

Quote from: michih on December 20, 2015, 03:08:14 AM
Quote from: Jim on December 19, 2015, 11:26:14 PMAll travelers have a line in this file.

Thanks. Is it possible to exclude non-real users like eroads_2digit, eroads_3digit, eroads_all, eroads_even, eroads_main, eroads_odd et cetera?

btw: Why do these user exist at all?

My understanding is that they're there to help the developers of those systems find possible errors.  I think it's to help detect missing or unintentional concurrencies but there are probably other purposes.  I'll leave it to those who have added them to explain further if they wish.

It's possible there are other ways to accomplish the purposes of those files, but I'd first need a better understanding of those purposes before proposing and implementing those other ways to accomplish them.
Photos I post are my own unless otherwise noted.
Signs: https://www.teresco.org/pics/signs/
Travel Mapping: https://travelmapping.net/user/?u=terescoj
Counties: http://www.mob-rule.com/user/terescoj
Twitter @JimTeresco (roads, travel, skiing, weather, sports)

Duke87

So here's somethng interesting. Allbyregion.csv shows me with 8939.66 miles in NY. Mapview.php shows me with 9964.32 miles. I see a discrepancy in every region I look at, both for my numbers and for the total miles available. Is this simply a matter of the latter double counting concurrent routes while the former does not? Or is something else going on here?


One other comment - the all by region file shows regions in Mexico as "MEX-" prior to the region code, it would be nice if the same treatment were given to regions in the US and Canada. This would make it easier for North American users to sort/filter out regions in Europe and vice versa. It seems the travels of most users are limited to one continent or the other.
If you always take the same road, you will never see anything new.

Jim

#13
Quote from: Duke87 on December 20, 2015, 11:09:49 AM
So here's somethng interesting. Allbyregion.csv shows me with 8939.66 miles in NY. Mapview.php shows me with 9964.32 miles. I see a discrepancy in every region I look at, both for my numbers and for the total miles available. Is this simply a matter of the latter double counting concurrent routes while the former does not? Or is something else going on here?

Yes, the mapview one is not getting information from the DB.  It shouldn't even be there anymore - it doesn't make any attempt to avoid double counting concurrencies.  The stats in the new .csv files and in the logs that have been there from the start should be accurate.  I haven't been making many fixes to mapview.  It needs some behind-the-scenes things to move from its current home in the "hbtest" directory (which I would like to go away) into the "devel" directory.  Some day when the project is mature enough to call it a production version, all of the tools people are intended to use will get new URLs.

QuoteOne other comment - the all by region file shows regions in Mexico as "MEX-" prior to the region code, it would be nice if the same treatment were given to regions in the US and Canada. This would make it easier for North American users to sort/filter out regions in Europe and vice versa. It seems the travels of most users are limited to one continent or the other.

I expect that once a nice web front end is developed, there will be options to select maps and stats not just by region but also by country, continent, etc. so I would hesitate to change any region codes (which would break everyone's list files if we just did it by a quick renaming).
Photos I post are my own unless otherwise noted.
Signs: https://www.teresco.org/pics/signs/
Travel Mapping: https://travelmapping.net/user/?u=terescoj
Counties: http://www.mob-rule.com/user/terescoj
Twitter @JimTeresco (roads, travel, skiing, weather, sports)

mapcat

Beautiful work, Jim! Thanks so much for putting this together. So far I haven't found any issues with the output.

Quote from: Jim on December 20, 2015, 11:30:20 AM
Quote from: Duke87 on December 20, 2015, 11:09:49 AM
One other comment - the all by region file shows regions in Mexico as "MEX-" prior to the region code, it would be nice if the same treatment were given to regions in the US and Canada. This would make it easier for North American users to sort/filter out regions in Europe and vice versa. It seems the travels of most users are limited to one continent or the other.

I expect that once a nice web front end is developed, there will be options to select maps and stats not just by region but also by country, continent, etc. so I would hesitate to change any region codes (which would break everyone's list files if we just did it by a quick renaming).

How difficult would it be to code just the allbyregion file to output, say, USA-AL when it reads AL from the DB? Same for CAN-AB, etc.

bejacob

Is the intention to start by replicating what the old CHM site had?

I liked being able to drill down on specifics by clicking a system, region, or route. I'm sure as folks work on the web pages, new ideas will emerge. While duplicating exactly what CHM had may not be feasible, starting with something similar makes sense.

Just curious about what ideas are being explored as this part of the project gets started.

Jim

Quote from: bejacob on December 20, 2015, 12:19:55 PM
Is the intention to start by replicating what the old CHM site had?

I liked being able to drill down on specifics by clicking a system, region, or route. I'm sure as folks work on the web pages, new ideas will emerge. While duplicating exactly what CHM had may not be feasible, starting with something similar makes sense.

Just curious about what ideas are being explored as this part of the project gets started.

I don't know of anything CHM did that can't be supported by TM.  Most if not all of the needed information is in the DB, and anything else needed to support other kinds of reports people come up with is likely not to be too hard to add.  I think once we have the first few kinds of stats pages operational, it won't be hard at all to get the rest going.

There was some work on CHM-style maps but that seems to have died out.  To start, any maps would probably use the mapview-style I developed, but I'd be very happy to see other map types supported.
Photos I post are my own unless otherwise noted.
Signs: https://www.teresco.org/pics/signs/
Travel Mapping: https://travelmapping.net/user/?u=terescoj
Counties: http://www.mob-rule.com/user/terescoj
Twitter @JimTeresco (roads, travel, skiing, weather, sports)

Jim

Quote from: mapcat on December 20, 2015, 12:15:45 PM
How difficult would it be to code just the allbyregion file to output, say, USA-AL when it reads AL from the DB? Same for CAN-AB, etc.

At some point a mechanism will be needed to bring groups of regions together into countries, nations, continents, whatever.  But since these csv stats files are really intended just as a stopgap way to get the information out there until we have a better way to access it, I'm not inclined to do anything that doesn't come quickly and easily from the in-memory data I use to generate the DB entries.

Later, if some users are still interested in csv stats in addition to what they can get through our eventual web interface, I think it would be nice to have PHP that would generate csv directly from the DB.
Photos I post are my own unless otherwise noted.
Signs: https://www.teresco.org/pics/signs/
Travel Mapping: https://travelmapping.net/user/?u=terescoj
Counties: http://www.mob-rule.com/user/terescoj
Twitter @JimTeresco (roads, travel, skiing, weather, sports)

Thing 342

Quote from: Jim on December 20, 2015, 01:48:47 PM
Quote from: mapcat on December 20, 2015, 12:15:45 PM
How difficult would it be to code just the allbyregion file to output, say, USA-AL when it reads AL from the DB? Same for CAN-AB, etc.

At some point a mechanism will be needed to bring groups of regions together into countries, nations, continents, whatever.  But since these csv stats files are really intended just as a stopgap way to get the information out there until we have a better way to access it, I'm not inclined to do anything that doesn't come quickly and easily from the in-memory data I use to generate the DB entries.

Later, if some users are still interested in csv stats in addition to what they can get through our eventual web interface, I think it would be nice to have PHP that would generate csv directly from the DB.

An SQL table containing regions to country codes and continents would definitely be helpful. Currently, getting a table of country codes to regions requires quite a bit of SQL-fu.

Jim

Quote from: Thing 342 on December 20, 2015, 04:27:43 PM
An SQL table containing regions to country codes and continents would definitely be helpful. Currently, getting a table of country codes to regions requires quite a bit of SQL-fu.

Definitely seems worth doing.  https://github.com/TravelMapping/HighwayData/issues/272
Photos I post are my own unless otherwise noted.
Signs: https://www.teresco.org/pics/signs/
Travel Mapping: https://travelmapping.net/user/?u=terescoj
Counties: http://www.mob-rule.com/user/terescoj
Twitter @JimTeresco (roads, travel, skiing, weather, sports)

Thing 342

I'd like to give people the ability to demo what I've been working on, but I don't have any free server space. Instead, I have screenshots:






It's not especially pretty, but it's mostly functional. Please let me know what you guys are interested in terms of features, and I'll look into implementing them.

michih

Quote from: Thing 342 on December 23, 2015, 10:04:41 PM
I'd like to give people the ability to demo what I've been working on, but I don't have any free server space. Instead, I have screenshots:
It's not especially pretty, but it's mostly functional. Please let me know what you guys are interested in terms of features, and I'll look into implementing them.

Thanks, looks good :).

- Rows should be numbered 1, 2, 3,... in a new column.
- Columns should have a sorting option (ascending and descending order).
- A filter feature for each column would be fine, minimum for country, region, system, tier, status.
- Maybe a "hide column" feature so that I can reduce the table width, e.g. "status" is not necessary for me because the rows are already colored green/yellow/red.

The screen shots are quite small (also when opening on photobucket.com). I think a better feedback is only possible with a prototype being online and I can check how it works.
For instance, I prefer preselecting the map style (map type and colors for tier 1-5) only once and all stats maps should already have this style on calling.

Rothman

Forgive my ignorance, but what are the tiers?
Please note: All comments here represent my own personal opinion and do not reflect the official position(s) of NYSDOT.

Jim

Quote from: Rothman on December 24, 2015, 08:08:12 AM
Forgive my ignorance, but what are the tiers?

Defined by the second-to-last column in https://github.com/TravelMapping/HighwayData/blob/master/systems.csv, the tiers are used to determine the relative importance of the systems in each country/region.  We currently number 1 through 4, with 1 for things like the U.S. Interstates, and 4 for state highway systems.

They're used in drawing the maps so that segments of higher tier routes are drawn on top of lower tiers.
Photos I post are my own unless otherwise noted.
Signs: https://www.teresco.org/pics/signs/
Travel Mapping: https://travelmapping.net/user/?u=terescoj
Counties: http://www.mob-rule.com/user/terescoj
Twitter @JimTeresco (roads, travel, skiing, weather, sports)

mapcat

Quote from: Thing 342 on December 23, 2015, 10:04:41 PM
I'd like to give people the ability to demo what I've been working on, but I don't have any free server space. Instead, I have screenshots:

It's not especially pretty, but it's mostly functional. Please let me know what you guys are interested in terms of features, and I'll look into implementing them.
These are a great start. Thank you!

In general, I like the format of having the map at the top and the stats below. The menu at the top and links within the stats seem sufficient. Eventually would it be possible to generate a map and stats of the entirety of a single highway across all regions it passes through? For example, an I-64 page showing a map and mileage from Missouri to Virginia.

Quote

The overall figure at the top is helpful, but I personally don't see any benefit to mileage broken down by tiers. Are the columns sortable by clicking on the headers?

Quote

As long as the color coding stays (thanks for doing that!), I don't see a need for the "status" column. Besides that one, I also wouldn't need the System Code or Tier columns. Could users turn them off/on as needed?

Would it be possible to see a total number of traveled and clinched highways for each system in the region?

Quote

For these, I think it would look cleaner if columns 1 & 3 were combined, and column 2 eliminated.



Opinions expressed here on belong solely to the poster and do not represent or reflect the opinions or beliefs of AARoads, its creators and/or associates.