Data hoarders

[J N Winkler]

I've been downloading rail transit planning information since the early days of the Internet.  Some of it was originally stored on 5-1/4 inch floppies and converted to 3-1/2 hard floppies.  I've also got a ton of really old stuff on Zip drives.  Over the years, I've filled up several 4gB hard drives full of data.  But recently I've been on a Road Diet (deleting the old Roadgeek stuff) and will formally begin a Rail Diet next month (after my 10-year safety-and-security protection clause expires).  What is odd is that I've never really needed to look up any old project information, but I'm so forgetful that I often can't understand how I remembered some of the information when asked.  (Which means, I did go back and check several times).  I've got so much stuff that I usually can't find what I'm looking for (cueing up the U2 song).

Did you explore the option of simply copying this material over to a mass storage device?  At nominal capacities of 360 KB (5 1/4" floppy), 1.44 MB (3 1/2" floppy), 100 MB (Zip drives), and 4 GB for a handful of mass storage devices, this sounds like a collection that could easily fit on a lightweight 2 TB drive costing well under $100.

For those wondering if they should become a Data Hoarder...

A few lessons I've learned (some high-level, some low-level):

*  Keep things organized

*  Automate organization as much as possible

*  Don't neglect metadata generation

*  Automate metadata generation as much as possible

*  When automating acquisition, use regex-capable tools (even if this means using ports of Unix shell commands in a Windows environment)

*  Work in Unicode (with en-dashes appearing in filenames, pure ASCII is not safe even in 100% US-based contexts)

*  Log source URLs, server addresses, etc. (useful for backtracing documents using the Web Archive)

*  If you use scripts that need revision from time to time, save the old versions

By far the most important lesson learned over the  years is to spend lots of time up front figuring out the workflow. For slides this is something along the lines of scanning at high resolution and bit depth in a lossless format. Also organize the slides by date and sequence (both of which should be on the frames in most cases) and scan in that order and name accordingly to make organization easy.

IMV, the part in boldface cannot be stressed enough.

[Dirt Roads]

I've been downloading rail transit planning information since the early days of the Internet.  Some of it was originally stored on 5-1/4 inch floppies and converted to 3-1/2 hard floppies.  I've also got a ton of really old stuff on Zip drives.  Over the years, I've filled up several 4gB hard drives full of data.  But recently I've been on a Road Diet (deleting the old Roadgeek stuff) and will formally begin a Rail Diet next month (after my 10-year safety-and-security protection clause expires).  What is odd is that I've never really needed to look up any old project information, but I'm so forgetful that I often can't understand how I remembered some of the information when asked.  (Which means, I did go back and check several times).  I've got so much stuff that I usually can't find what I'm looking for (cueing up the U2 song).

Did you explore the option of simply copying this material over to a mass storage device?  At nominal capacities of 360 KB (5 1/4" floppy), 1.44 MB (3 1/2" floppy), 100 MB (Zip drives), and 4 GB for a handful of mass storage devices, this sounds like a collection that could easily fit on a lightweight 2 TB drive costing well under $100.

The rationale for keeping the floppies was that I also maintained the original x386, x486 and early Pentium computers that could still run the older versions of software.  One of the problems was that we worked on too many software platforms that didn't port over from one generation to another.  Many of those software platforms didn't survive past two generations of hardware (about 6 years) and two generations of Windows (about 5 years), overlapping (on average about 7-1/2 years).  Ouch, even MS-Fortran and the original Visual Basic didn't migrate with the newer stuff.  Plus, it was a real pain to reinstall VB 1.0, upgrade to VB 1.1, V.B 1.3, VB 2.0 and VB 3.0 just to find out that the newer Windows add-ons/add-ins would not allow the old programs to operate.  I could have bought a separate new computer and downgraded to the older versions of Windows, but it was much easier to add extra cooling power to the old computers.

Did you explore the option of simply copying this material over to a mass storage device?

Some of the really old stuff has been ported over to the disk drives.  For the record, I maintain an online computer (with normal data) and an offline computer (with private data and proprietary data).  I have two 4GB backup drives, two 2GB backup drives, three 1GB backup drives (one bad), two old offline computers with 1GB drives and two old offline computers with 720MB drives.  At one time, almost all of those drives were full (except for the old Windows Vista machine that crashed and I had to replace its hard drive, and even that one got half-full at one point).  Plus, I still have two really old 350MB plug-in-wall drives (in the days before they could work on USB power).

After the most recent cleanup, I can probably store everything on a 2GB drive.  I do try to alternate the drives when I perform frequent manual backups and annual archivals.  Which leads to your next comment...

A few lessons I've learned (some high-level, some low-level):

*  Keep things organized

*  Automate organization as much as possible

*  Don't neglect metadata generation

*  Automate metadata generation as much as possible

I have a pretty good Windows File architecture for everything.  But sadly, with many years of 100+ hour workweeks, I never had any time to keep the backup storage organized.  Much of the Data Hoarding efforts involved backups-over-backups, backups-of-backups (ouch) and even sometimes archives-of-archives (double ouch).  If you couldn't already tell, I always tried to keep two independent backups such that in the case of another crash of a backup drive I don't lose too much stuff.  After the next cleanup, I shouldn't need to keep two sets of backups.

I've never trusted the automated backup software and metadata software with personal data and proprietary data (too much alien "E.T., phone home" going on inside there).  If it weren't for my wife and I both generating photo albums that need to be consolidated occasionally, I would purchase smaller drives and automate the backup process on both machines and quit worrying about Google collecting a copy of everything.  I still find things that never made it into my archives system.

There's no room here for all of this stuff.  But what is scary about all of this is how much money has been poured into computers, software and backup drives over the course of my lifetime.  Given that I'm a big cheap-O and have probably spent less than a tenth of what everybody else who had personal computers in the 1970s, I can't even imagine what they have spent.

[Scott5114]

I imagine there are still audio/video or film processing companies in most cities that will scan slides and return both the slides and digital image files to you for a fee. My parents used a local company in Oklahoma City to transfer VHS tapes to DVD back in the day. I would be surprised if there wasn't someone offering the same service for slides, especially in the DMV area. Wouldn't hurt to check, at least...the fee would probably be less than the cost of getting your own equipment, never mind the time investment.

[formulanone]

I imagine there are still audio/video or film processing companies in most cities that will scan slides and return both the slides and digital image files to you for a fee. My parents used a local company in Oklahoma City to transfer VHS tapes to DVD back in the day. I would be surprised if there wasn't someone offering the same service for slides, especially in the DMV area. Wouldn't hurt to check, at least...the fee would probably be less than the cost of getting your own equipment, never mind the time investment.

Jim Grey has a pretty good write-up on a slide scanner from 2014. He's a part-time member on these boards.

Converting slides yourself is a slow process; you can buy a slide reader which is effectively a lower-end digital camera with a light source and cover. If you're just sharing them for fun on social media without worrying about the archival image depth and preservation, then go for it. The problem with the cheaper ones seem to be vignetting in the corners and less-than accurate color reproduction, since the light emitted from the slide reader has its own color cast. You also have to carefully fit each one into the reader and may probably want to adjust lighting, colors, and perhaps perform "dust and scratches" clean-up work.

There's higher-end equipment and also professionals who will do the work for you. Like pretty much anything in the sphere of photography, it just depends on how much you want to spend, and how noticeable the differences are valued. Though, someone who is familiar with the slide images is going to know and remember the colors used with a little more care than someone almost entirely emotionally unattached from the process. Slide film was the closest thing to a compact high-definition image and some of the results of 70+year-old slides are still pretty amazing to what most prosumer cameras of the last decade can yield.

I'm surely going inherit hundreds of old family slides from the 1960s-80s one day, but have no idea how I'm going to do it without breaking the bank. I still have a few thousand photos to about 2002 or so that I have yet to feed into a scanner. I want quality, but I also just wonder if anyone's going to even care a few years after I'm gone...on the other hand, maybe the quality will be something to treasure in the future. (Sorry to get sappy there.)

[1995hoo]

I imagine there are still audio/video or film processing companies in most cities that will scan slides and return both the slides and digital image files to you for a fee. My parents used a local company in Oklahoma City to transfer VHS tapes to DVD back in the day. I would be surprised if there wasn't someone offering the same service for slides, especially in the DMV area. Wouldn't hurt to check, at least...the fee would probably be less than the cost of getting your own equipment, never mind the time investment.

I did look into that and from what I found, I'd be looking at at least several thousand. Hence why I was exploring DIY options! I will concede the point formulanone makes about whether anyone will care in the future has crossed my mind. My wife and I don't have any kids and my brother is not married (as far as I know, he doesn't have any kids he knows about, at least...) BUT as it is now, the slides all sit in cabinets in my mother's living room.

I think the point HighwayStar makes about figuring out the best way to run a slide or negative scanner while doing other things is exceptionally important!

[HighwayStar]

I imagine there are still audio/video or film processing companies in most cities that will scan slides and return both the slides and digital image files to you for a fee. My parents used a local company in Oklahoma City to transfer VHS tapes to DVD back in the day. I would be surprised if there wasn't someone offering the same service for slides, especially in the DMV area. Wouldn't hurt to check, at least...the fee would probably be less than the cost of getting your own equipment, never mind the time investment.

I did look into that and from what I found, I'd be looking at at least several thousand. Hence why I was exploring DIY options! I will concede the point formulanone makes about whether anyone will care in the future has crossed my mind. My wife and I don't have any kids and my brother is not married (as far as I know, he doesn't have any kids he knows about, at least...) BUT as it is now, the slides all sit in cabinets in my mother's living room.

I think the point HighwayStar makes about figuring out the best way to run a slide or negative scanner while doing other things is exceptionally important!

Unfortunately the paid conversion services are mostly crap and best avoided.
The problem is to do good quality conversions of any old analog media to digital takes specialized knowledge, attention to detail, and a decent amount of time, along with expensive and sometimes rare equipment. The actual cost to do it correctly will almost always be well above what most people would pay, so the only way to make a business out of it is to hire minimum wage labor that has no clue what they are doing and churn out poor quality conversions using substandard equipment. They get away with it because people invariably attribute poor quality to the "old tape/film/slides/etc." not knowing that they should look far better if done right.
There might be a local service here or there that can do it, but don't even consider giving them your business unless they can give plenty of details about the process, what they do or do not do, equipment used, etc. and pass all those tests. The only places I ever tried that with were evasive, vague, or simply told me that they would not discuss it.
Here is an excellent expose of Legacybox done by VWestlife which should disabuse anyone from ever being tempted by that service. I suspect many others would be a similar story.

Doing it yourself will not only save money but likely result in vastly superior quality.

[bugo]

Here's a small part of my map hoard. This Google Drive folder contains several hundred maps, most in convenient PDF format but a few TIF and even some JPG files are in the folder including state map archives for several states, Arkansas and Oklahoma control section maps (including several that are no longer available on the ArDOT or ODOT websites) and, in the folder Atlases, some road atlases going as far back as 1926.

https://drive.google.com/drive/folders/1DXfHOykUwntSdiO56f4tto4ivXpAh_ep?usp=drive_link