Data hoarders

[bugo]

I am a data hoarder. I have tons of pictures, text files. audio files and other documents saved on my hard drives. Many of these documents are maps and other road-related documents such as meeting minutes, route logs and orders.

When a state highway department posts an updated map on their website, the old map is almost always removed. And when they're removed, you usually can't get them anywhere else unless somebody has saved it and sends it to you. That's why data hoarding is important.

We won't even start on mapping programs like Google Maps that are constantly being updated. It's easy to buy a 2010 paper road atlas, but with mapping software, there are no old editions to archive. This will make it more difficult for future road researchers to try to pin the date of the opening of a road by using maps.

Does anybody else obsessively archive maps and other road-related documents?

[oscar]

Not "obsessive" exactly for me -- more like being a lazy pack rat. But in addition to old road atlases, I keep around (but have not installed on my computer) old versions of Microsloth Streets and Trips. The latest one is from 2013, but I still use it sometimes.

Also, I have old Alaska DOT&PF route logs for the few numbered and signed highways it maintains. The DOT says they're obsolete, but last I checked (long ago) new versions have not been posted online. The new versions are less detailed, and the lost details are still somewhat useful for maintaining Travel Mapping's Alaska route files.

I copied a lot of old Hawaii maps and documents, too. The most interesting instance where that came in handy was a copy of the map showing the temporary highway network on Oahu (Honolulu) set up ahead of World War II, which fell into disuse when the war was over. The original map I copied was stored in a basement, which later was destroyed in a flood. I web-posted my copy of the map, so it's still available for reference.

[Bruce]

While I do keep some offline collections, especially for media that I know won't be saved in a proper fashion, there are places dedicated to archiving online stuff. The Internet Archive operates the Wayback Machine, which has been collecting cached webpages for longer than I have been alive and has trawled many DOT webpages and roadgeek sites. It's incredibly useful if you are able to find a link and timeframe from another mention (e.g. newspapers or internet forums) and can plug it in.

I just hope there's better archiving practices in place for DOTs in this digital era. A few have been consistent with their archiving of online documents (such as Oregon, who forwards their DOT work to the state library's online collections), but many haven't set things up properly.

[Max Rockatansky]

I found hoarding data to be burdensome over the years.  It tended to grind on me more and more as time goes on, enough to the point that I found letting go freeing.  This would be probably the most significant reason I don't have stuff like a Travel Maps or Mob Rule account. 

Now, I did get my photos organized back in 2016.  It was about a four month effort to complete but I somehow managed to break it up into health chunks of time.  I can't imagine would be able to replicate the feat now that I'm married, my wife would be upset at that empty use of hours.

I do have an almost full collection of California Highways & Public Works which were denoted to me due to Gribblenation.  I more or less hung onto a lot of maps (especially NPS) by accidents given I like to occasionally look at them.

[formulanone]

In similar vein, lots of my photos. I take about 25,000-40,000 pics in a year, and save everything to three portable hard drives, in a bit of a grandfather-father-son method. (It's honestly more like Stay-at-home-Father, Son, and the Wayward Son...)

I'd save every file that wasn't a completely indistinguishable image...just in case. And for each image I modified, I'd save the full-sized-modified images and the smaller online version. About six months ago, I stopped saving the full-size "updated" images and just went with the smaller ones, to save time in the process of creation and backing it all up.

It took getting a new camera to finally get me to start culling through the old stuff. (After all, I don't yet feel like buying another new portable hard drive and spending a week to back that all up...again.) These images are now usually 2-3 times the file size from my previous camera, so do I really need 15 similar shots of the same road? Those test shots in the parking lot in the morning? Why did I think those images in total darkness were going to be masterful and sublime? So cleared out about 40-50 GB in stuff like that.

The problem is that I usually don't have as much time right away, and just sock them away in the folder until several months (or even years) later. But I'm getting better at being a little more picky and not worrying if IMG_2345 is almost identical to IMG_2354 and neurotically keeping all twelve of them, and not feeling like the task of preparing them for uploading is an insurmountable one.

[Bruce]

I've got thousands of photos I need to process (aka basic editing and tagging) and upload to Wikimedia Commons before it's too late. I have backups but most are on-site.

[Rothman]

I am old enough now to start wondering where everything will go when I pass away.  I can't imagine my kids preserving my stuff for more than a generation.

[Max Rockatansky]

I know where my highway signs will probably go, I've bought way too many from estate sales to not know that answer.  Regarding my photos, since they aren't in a physical album they likely will be thrown in the trash or be purged from the online sources they are online.  At minimum some of the stuff I've written about might live through the years.  Given I don't have kids' general memory of me will likely fade within a couple decades.  As an example, I noticed US 66 ending at Fletcher Drive is already starting to make the rounds into more normalized circles of road interests. 

[zachary_amaryllis]

I am a data hoarder. I have tons of pictures, text files. audio files and other documents saved on my hard drives. Many of these documents are maps and other road-related documents such as meeting minutes, route logs and orders.

When a state highway department posts an updated map on their website, the old map is almost always removed. And when they're removed, you usually can't get them anywhere else unless somebody has saved it and sends it to you. That's why data hoarding is important.

We won't even start on mapping programs like Google Maps that are constantly being updated. It's easy to buy a 2010 paper road atlas, but with mapping software, there are no old editions to archive. This will make it more difficult for future road researchers to try to pin the date of the opening of a road by using maps.

Does anybody else obsessively archive maps and other road-related documents?

I'm a datahoarder, though it's different data that I hoard.
Every 10 minutes, I (well, the cron job..) grabs pictures from 3 local radio towers. I've been doing this for almost a year now.

I have 3 cameras on the house, looking at different mountainy stuff, that snap a picture once every minute. That's been going on for about 6 months now. (I like to edit them into timelapses).

I have an older machine, that I've repurposed into sorta a NAS, it just sits there and spins the drives. Every last image is saved. No idea what I'll do with them, but it was fun to set up as sort of 'proof of concept'.

[HighwayStar]

I've become more of a data hoarder with time. The biggest contributor to this has been a realization that the mantra of the 90's/2000's when people assumed information on the internet was permanent has turned out to be incorrect. Link rot, site bankruptcies etc. have long proved otherwise. Mind you, I don't care about everything, but generally when I have read an article I like to be able to refer back to it.

A year or so ago I went though all my bookmarks of what I had saved over the years, and in the oldest tranche of data, about 7-8 years old, probabally 40% of the links were rotten. About half of those could be recovered by searching the article title, but I bet a solid 20% were difficult or impossible to recover.

After that experience I have moved to saving articles with citation management software so I can always go back if needed.

Likewise, YouTube in the early years was an excellent repository of TV shows that were otherwise impossible to find, but has significantly declined as an archive platform.

[ZLoth]

This is partially why I built a TrueNAS server which is partially a backup storage server, partially a media server, and partially a file storage server. With eight 8 TB drives in a RAIDZ2 configuration, that gives me effectively 48TB of storage.

Data hoarding? Of course, when you consider I have:

• Backup copies of my online storage providers (OneDrive, Dropbox, Google Drive)
• Electronic copies of every mortgage statement and home improvement expense
• Electronic receipts for tax preparation reasons
• Configured ShareX so that every screen shot is saved into a dated directory (e.g. \\server\screenshots\ShareX\%year\%month)
• Symbolic links so that all of my save game files are on the NAS server instead of my SSD. (I'm looking at you The Witcher: Enhanced Edition)

Yes, it's all backed up to several external physical hard drives. However, some critical and/or non-replaceable items (system backups, photos, personal files, KeePass) are backed up every night to encrypted folders on two different cloud storage providers.

I've become more of a data hoarder with time. The biggest contributor to this has been a realization that the mantra of the 90's/2000's when people assumed information on the internet was permanent has turned out to be incorrect. Link rot, site bankruptcies etc. have long proved otherwise. Mind you, I don't care about everything, but generally when I have read an article I like to be able to refer back to it.

A year or so ago I went though all my bookmarks of what I had saved over the years, and in the oldest tranche of data, about 7-8 years old, probabally 40% of the links were rotten. About half of those could be recovered by searching the article title, but I bet a solid 20% were difficult or impossible to recover.

This is why I still collect physical media, and transfer the CDs, DVDs, and BluRays to my media server even though some folks say "you can just watch it online". For starters, there is some niche material that I have which never made it online. The anime series Planetes (2003) was released on DVD almost two decades ago (publisher no longer exists), was released in Region B on BluRay in 2021, and is not even available on Crunchyroll. Another international dark comedy film 6ixtynin9 (1999) was nominated for a foreign language Oscar, but is long out of print in DVD (bankruptcy). In addition, there have been series and movies which have been notably yanked from the streaming services by the studios for "cost reasons", especially Warner Bros-Discovery but also Disney. Lord knows how many streaming services I would need to subscribe to in order to view what I have on my own private server.

I have moved to saving articles with citation management software so I can always go back if needed.

Got any examples that I can look at?

[Chris]

I've been doing highway history research in Europe, and often the most difficult timespan to find dates from is not from 50 years ago, but the 1995-2010 timeframe. It's surprising how little media archives are from that period. Archived newspapers mostly don't go beyond the 1990s while many websites running before 2010 are completely revamped with lots of data not being available anymore.

Google isn't helping here either. It prioritizes recent news reports over anything else, making it difficult to find even something from more than 5 years ago. The best documentation may be internet forums, but Google has clearly deranked internet forums from its search results. Even discussions running for 10+ years on internet forums can only be found on the 4th or 5th page of results.

However Google still has the most results. An attempt to search on Bing or DuckDuckGo leaves even less relevant results.

[Max Rockatansky]

If we are counting highway research as data hoarding then I've done my fair share for California.  A couple years ago I began inserting images from CHPWs, maps, historic images and other sources into the California blogs on Gribblenation.  I'm closing in on an almost complete set of representative blogs for all the state highways.  Personally I'm more proud of blogs for the more obscure times like the Stockton-Los Angeles Road, Lone Pine-Porterville High Sierra Road and Mineral King Road:

https://www.gribblenation.org/p/golden-state-highways-version-30.html?m=1

[mgk920]

Also the absurdly long copyright protection terms in the USA, with rights holders dumping their stuff in a memory hole if it is somehow 'un-PC' in current standards or for any other reason and then suing the daylights out of any leaks, even if the material is multi-decades old.  My sense is that much of our very culture is being lost for this reason.

Mike

[MikeTheActuary]

I have a bad habit of downloading stuff (not specifically roadgeek) and just not deleting it, or of making redundant backups because I don't take the time to remember whether or not I actually need that particular backup.  Every picture or video on my or my wife's phone automatically gets saved...even the accidental photos, or the photos I take of computer/radio/electronics wiring to better see what's going in typically dark, tight spaces....

That's probably why my NAS is currently complaining about running out of disk space, and why I have a weekend project of upgrading the hard drives in it.

Most of my data hoarding is just my not being able to motivate myself to go through and figure out what I want/need to keep, and what can be purged.  But there has been a time or two that I've wanted to refer to something random I once saw, and searching through my archives has proven pleasantly fruitful.

[1995hoo]

I am old enough now to start wondering where everything will go when I pass away.  I can't imagine my kids preserving my stuff for more than a generation.

I've pondered how to deal with my late father's photos from family vacations over the years (he died in 2019). He usually shot slides and he had a slide projector (that no longer works) of a style that has been discontinued, a system that used cubes instead of the more common carousel, although later on he got a second projector that used a carousel. What that means as a practical matter is that trying to view the slides on a wall or a screen in the traditional manner would be difficult. I'd like to scan them all sometime–in particular, I'd like to compare the pictures from our 1982 family trip to Nova Scotia and Newfoundland with my pictures from my 2008 trip to some of the same places. But I don't really have a good sense for how to approach the project because I would like to find some semi-automated way to do it, like if I could use a slide scanner and stack up a full cube's worth of slides and have the machine scan them all automatically. Ideally, I'd find a slide scanner that can also scan negatives so that I could scan my old negatives from my film camera days. I don't know what sort of thing exists for that sort of project and I know it would be time-consuming either way (maybe a good wintertime project), but the alternative–paying a service to do it–would be way too expensive.

Does anyone know much about slide and negative scanners? I suppose I could go to B&H in New York when I'm up there next week, but I have a feeling going there would be more worthwhile if I had some idea of what I want to look for before I go.

I'm guessing if I pursue this I might set up a separate PC to be the photo storage machine.

[J N Winkler]

I've pondered how to deal with my late father's photos from family vacations over the years (he died in 2019). He usually shot slides and he had a slide projector (that no longer works) of a style that has been discontinued, a system that used cubes instead of the more common carousel, although later on he got a second projector that used a carousel. What that means as a practical matter is that trying to view the slides on a wall or a screen in the traditional manner would be difficult. I'd like to scan them all sometime–in particular, I'd like to compare the pictures from our 1982 family trip to Nova Scotia and Newfoundland with my pictures from my 2008 trip to some of the same places. But I don't really have a good sense for how to approach the project because I would like to find some semi-automated way to do it, like if I could use a slide scanner and stack up a full cube's worth of slides and have the machine scan them all automatically. Ideally, I'd find a slide scanner that can also scan negatives so that I could scan my old negatives from my film camera days. I don't know what sort of thing exists for that sort of project and I know it would be time-consuming either way (maybe a good wintertime project), but the alternative–paying a service to do it–would be way too expensive.

Does anyone know much about slide and negative scanners? I suppose I could go to B&H in New York when I'm up there next week, but I have a feeling going there would be more worthwhile if I had some idea of what I want to look for before I go.

I'm guessing if I pursue this I might set up a separate PC to be the photo storage machine.

I do have some experience with slide scanning, but it is very dated, since I lost the capability about 17 years and three computers ago.

When I was still actively doing it, I used two Benq slide scanners that could handle four slides at a time and could also scan negatives.  Both devices required installation of a SCSI card.  I got the second because the first could not remove dust specks in scans, and I soon realized that capability was essential--otherwise you waste an enormous amount of time per scan Photoshopping out dust motes.  I also discovered that the color balance in scans tilted noticeably toward the red as the scanner warmed up, so I had to put a block of frozen cheese on top in order to ensure consistent results over the course of a session.

Unless the technology has improved considerably, scanning slides is a huge time pit, since you have to manually load the slide carrier and be prepared to change out the slides once the scanner finishes a TWAIN acquisition.  I have never heard of or seen a consumer-grade scanner with the ability to scan a whole slide carousel.

I generally pre-selected a variable number of slides from each roll to scan, so the total time to process might run from just a few minutes to over an hour.  I used a battery-powered slide viewer for looking at slides and eventually burned out the bulb.  Since I could not easily find a replacement, I then had to hold slides up to the light, which is not ideal for evaluating tone and contrast.  Since my slide processor did not number slides, I ran a marker diagonally over the top edge of each row (there were two in each box) so that I could re-file scanned slides in order.

I never came close to processing even half of the approximately 400 rolls I shot.  I do actually still have the slide scanners, and even an old desktop computer that should be able to accept a SCSI card, but I don't know whether I would still be able to find compatible drivers (I was running Windows 98 at the time and the desktop runs Windows XP out of the box).  Frankly, this is not a project I want to undertake now or in the immediately foreseeable future.  When I do eventually tackle it, I want to do some in-depth research into what it would take to obtain professional-grade results (which I did not do, or even have the ability to do, 20 years ago), and possibly invest in new equipment.

[1995hoo]

Thanks for that info. Not encouraging to hear, but I guess that's important to know as well. Maybe I'll swing by B&H Monday night to ask about it, recognizing of course their interest is in upselling me to something more than I really need.

[J N Winkler]

Thanks for that info. Not encouraging to hear, but I guess that's important to know as well. Maybe I'll swing by B&H Monday night to ask about it, recognizing of course their interest is in upselling me to something more than I really need.

It certainly wouldn't hurt to do a preliminary investigation of feasibility.  There is still considerable demand for image capture from legacy materials, and I suspect there have been major improvements in speed and capability over the past couple of decades.

The size of the collection you are working with is also a factor.  I've determined that my parents and paternal grandparents also shot slides, but I would expect that at least 80% of the rolls I might be working with down the line are mine.  For the time being, I have focused more on ensuring that everything is indexed so I can establish context later on.  Every box of my slides has a number, and for most I can pinpoint (at least approximately) the month and year I shot it.

[J N Winkler]

In regard to the general archiving question in the OP, I would estimate that about 95% of what I have (by byte count) consists of document packages for highway construction contracts.  I also gather the same for some rail (especially high-speed) and urban transit projects, though far less extensively and systematically.

For some--not all--of the agencies for which I collect material, I extract pattern-accurate signing sheets, according to criteria that vary from one agency to another but generally call for sign layouts only if the agency in question does not routinely provide sign panel detail and sign elevation sheets.  These days I try to limit the effort I invest in this on an ongoing basis, so for some agencies I have discontinued it, though I still maintain collection in hopes eventually of using computer vision to zero in on signing.

I have about 127 GB worth of signing sheets, sorted by agency.  I have over 25,000 (about 6.25 GB total) for TxDOT alone.  Just in the last month, I crossed the threshold of 20,000 (21.1 GB) for German highway agencies as a whole.

[dlsterner]

I've pondered how to deal with my late father's photos from family vacations over the years (he died in 2019). He usually shot slides and he had a slide projector (that no longer works) of a style that has been discontinued, a system that used cubes instead of the more common carousel, although later on he got a second projector that used a carousel. What that means as a practical matter is that trying to view the slides on a wall or a screen in the traditional manner would be difficult. I'd like to scan them all sometime–in particular, I'd like to compare the pictures from our 1982 family trip to Nova Scotia and Newfoundland with my pictures from my 2008 trip to some of the same places. But I don't really have a good sense for how to approach the project because I would like to find some semi-automated way to do it, like if I could use a slide scanner and stack up a full cube's worth of slides and have the machine scan them all automatically. Ideally, I'd find a slide scanner that can also scan negatives so that I could scan my old negatives from my film camera days. I don't know what sort of thing exists for that sort of project and I know it would be time-consuming either way (maybe a good wintertime project), but the alternative–paying a service to do it–would be way too expensive.

Does anyone know much about slide and negative scanners? I suppose I could go to B&H in New York when I'm up there next week, but I have a feeling going there would be more worthwhile if I had some idea of what I want to look for before I go.

I'm guessing if I pursue this I might set up a separate PC to be the photo storage machine.

About 10 years ago when my mother passed I grabbed the family 35mm slide collection with the intent of digitizing them and saving them to a CD-R or DVD-R.  About 2000 slides from 1957 through 1986 (when my father passed), all meticulously noted with date and place.

I found a scanner on Amazon back then - it was $120-ish - that would scan slides (one at a time) and save them to a SD card which plugged into the scanner.  I could then connect the scanner to my Mac (certainly a PC works too) where it would behave like a USB drive and I could copy the files to my hard drive.  (I did see one that was automated but it was like $1600 - yikes)

I'm not home right now so I don't have the information about the scanner, but I'll check when I get home and edit this post (or post a reply).

Sadly a handful of slides had faded to the point where all the blue and green information was gone, just the red remained.  Maybe it was the brand of film.  Those I then converted to grayscale with a photo editing program.  Salvageable, and better than nothing.

[Dirt Roads]

In regard to the general archiving question in the OP, I would estimate that about 95% of what I have (by byte count) consists of document packages for highway construction contracts.  I also gather the same for some rail (especially high-speed) and urban transit projects, though far less extensively and systematically.

For some--not all--of the agencies for which I collect material, I extract pattern-accurate signing sheets, according to criteria that vary from one agency to another but generally call for sign layouts only if the agency in question does not routinely provide sign panel detail and sign elevation sheets.  These days I try to limit the effort I invest in this on an ongoing basis, so for some agencies I have discontinued it, though I still maintain collection in hopes eventually of using computer vision to zero in on signing.

I have about 127 GB worth of signing sheets, sorted by agency.  I have over 25,000 (about 6.25 GB total) for TxDOT alone.  Just in the last month, I crossed the threshold of 20,000 (21.1 GB) for German highway agencies as a whole.

I've been downloading rail transit planning information since the early days of the Internet.  Some of it was originally stored on 5-1/4 inch floppies and converted to 3-1/2 hard floppies.  I've also got a ton of really old stuff on Zip drives.  Over the years, I've filled up several 4gB hard drives full of data.  But recently I've been on a Road Diet (deleting the old Roadgeek stuff) and will formally begin a Rail Diet next month (after my 10-year safety-and-security protection clause expires).  What is odd is that I've never really needed to look up any old project information, but I'm so forgetful that I often can't understand how I remembered some of the information when asked.  (Which means, I did go back and check several times).  I've got so much stuff that I usually can't find what I'm looking for (cueing up the U2 song).

For those wondering if they should become a Data Hoarder...

The firm that I worked for was increasingly pressed to produce cost estimates for almost everything where people get moved (trains, buses, vans, taxis, moving walkways, elevators, etc.) and we didn't have any semblance of a cost database.  As I started to collect data for a complex project, I realized that we had access to many resources that just needed catalogueing.  If there was an article with a dollar $ sign (or pound £ sign, if you are so inclined), it got collected and added to a lengthy document.  Then I started working backwards in time.  We got really good at cost estimating without having any Professional Cost Estimators on staff (after all, the folks hiring us almost always had them).  But it was an effort you could never stop.  Eventually, I got where the demands of safety certification would not allow me to continue being a cost wizard.  But this Data Hoarding technique would be quite useful in almost any field of business.

[HighwayStar]

I have moved to saving articles with citation management software so I can always go back if needed.

Got any examples that I can look at?

I originally used Mendeley and Zotero. I've abandoned Mendeley however because Elsevier has endeavored to make it hard to transfer the library to other software. Zotero is much better in that respect and has become by sole solution. Main downside has been backing up and synchronizing the library across devices. Its a bit tricky to get it playing nicely with cloud storage.

[HighwayStar]

I am old enough now to start wondering where everything will go when I pass away.  I can't imagine my kids preserving my stuff for more than a generation.

I've pondered how to deal with my late father's photos from family vacations over the years (he died in 2019). He usually shot slides and he had a slide projector (that no longer works) of a style that has been discontinued, a system that used cubes instead of the more common carousel, although later on he got a second projector that used a carousel. What that means as a practical matter is that trying to view the slides on a wall or a screen in the traditional manner would be difficult. I'd like to scan them all sometime–in particular, I'd like to compare the pictures from our 1982 family trip to Nova Scotia and Newfoundland with my pictures from my 2008 trip to some of the same places. But I don't really have a good sense for how to approach the project because I would like to find some semi-automated way to do it, like if I could use a slide scanner and stack up a full cube's worth of slides and have the machine scan them all automatically. Ideally, I'd find a slide scanner that can also scan negatives so that I could scan my old negatives from my film camera days. I don't know what sort of thing exists for that sort of project and I know it would be time-consuming either way (maybe a good wintertime project), but the alternative–paying a service to do it–would be way too expensive.

Does anyone know much about slide and negative scanners? I suppose I could go to B&H in New York when I'm up there next week, but I have a feeling going there would be more worthwhile if I had some idea of what I want to look for before I go.

I'm guessing if I pursue this I might set up a separate PC to be the photo storage machine.

I've done them, using a prosumer flatbed scanner. You can get a nice one with slide scanning capabilities used now days for not a lot of money. Go high end. For example, I bought a CanoScan 9000F for $60 on OfferUp earlier this year. That was ~$900 scanner when new and will do an excellent job on slides.

As others have said, it is a bit time consuming to do right, but the truth is ALL analog to digital conversion is time consuming to do right.

By far the most important lesson learned over the  years is to spend lots of time up front figuring out the workflow. For slides this is something along the lines of scanning at high resolution and bit depth in a lossless format. Also organize the slides by date and sequence (both of which should be on the frames in most cases) and scan in that order and name accordingly to make organization easy.

Before I understood this, I did conversion projects where I scanned at low resolution and in JPEG. I had to redo those. Time spent getting a good workflow that produces results you can live forever with is important.

It will take some time to sit and load the scanner that many times, but if you set it up next to you at the desk its possible to do a fair bit of multitasking and get some of the time back for other things.

[HighwayStar]

It certainly wouldn't hurt to do a preliminary investigation of feasibility.  There is still considerable demand for image capture from legacy materials, and I suspect there have been major improvements in speed and capability over the past couple of decades.

Unfortunately that has generally not been the case for many legacy formats. The best equipment for capture has in many cases actually been discontinued and is only available on the used market at greatly inflated prices. Slides are fortunately an exception to this trend, in that good quality scanners are still being made, but for many other formats that is not the case. VHS for example is a nightmare to capture at this point and was easier to do 15 years ago.

That said, the best way to scan slides is likely going to be a flatbed scanner, loaded a few at a time. Any contraption that someone has come up with to take the whole carousel I would be quite wary of.