|
Image Management for Photographers: |
The 'dirty little secret' of digital photography is the growing complexity of storing an increasing number of larger and larger images. When I first converted fully to digital in 2000, the state of the art for most of us were 1 MB JPEG files from the 2.7Megapixel D1. And the camera itself was slow so the images didn't pile up so fast. They were easy to store and not that hard to back up to tape cartridges.
Now I'm shooting with 12MP cameras in Raw mode, with up to 8 frames per second of 12MB files. And that's compressed. Real purists are generating images even larger images. Instead of measuring our storage needs in Gigabytes, most of us are now counting in Terabytes. The good news is that disk drives are getting big even faster than camera sensors. But we're in danger of replacing all that time we saved by not having to carefully sleeve and hang slides with the time we spend storing and archiving our digital assets. More importantly if you don't take care of your assets then you are at risk of losing your images permanently. In this installment of our "how to" series we'll introduce you to some of the best techniques we know for storing and archiving your images effectively along with creating reliable backups.
First we need to make sure we all understand the terms we're using. Storage is simply where you put your images for day to day use. Typically a disk drive or perhaps a Network Attached Storage box. Archives are your images that you don't have room or don't want to to store online anymore. Archiving is not the same as backup. Backups are copies of your stored and archived images that are duplicates to be used in the event of file corruption or some other catastrophic failure. A Working Directory, if you use one, is a location on a high speed drive so you can more quickly edit images that you retrieve from your Storage location for processing.
The perfect storage device would be fast, accessible from all your computers without them needing to be on, reliable, and easy to back up. And of course expandable and inexpensive. Unfortunately no one device meets all those requirements. You'll need to prioritize your requirements and decide how much you can afford. Roughly speaking there are several types of storage devices worth considering:
The simplest solution is, as you might expect, just using your computer's existing hard drive(s). If it is large enough or you can add a drive to it that's an okay place to start. Your solution will be limited in size and difficult expand but will have excellent performance since it is right inside your machine (assuming you purchase a reasonable performance 7200rpm or better disk drive). Unfortunately it won't be at all reliable in case of failure so you'll need to be extra careful with your backups.
If you're building a new machine or are handy with upgrading computers you can purchase a RAID (Redundant Array of Inexpensive Disks) controller and install several identical drives to provide a RAID array. This can give you an increase in reliability (if one drive fails then it can be swapped out and rebuilt from the remaining drives, using RAID modes 1 or 5) and possibly an increase in performance as well (if you use a Hybrid RAID mode like 10 or 0+1). However, these arrays are not easy to upgrade. Not only do you need to replace internal drives which is a hassle (drive trays make this much simpler) but upgrading the array requires backing up the array, installing all the new drives, formatting and then restoring the backup.
RAID controllers also aren't cheap, ranging from $100 to $300 for high-performance desktop models. The good news is with 1 Terabyte drives available for $100 (as of November 2008) you could put together 2TB of storage for about $400 ($200 controller plus 3 1TB drives configured as RAID 5 to give you 2TB of reliable storage).
A little more expensive and perhaps awkward looking alternative is to use an external drive or RAID array. You can move the external enclosure separate from your computer--or even use it on a different computer if you need to. Simple enclosures cost from $100 to $200 depending on how many drive bays you need. Enclosures that support RAID are more expensive and can cost from $250 to $400. And of course you need to stock them with hard drives. If they are connected over a high speed interface (USB 2.0 is okay, although Firewire 800 is better and eSata is even better) they can perform essentially as well as internal drives.
The cadillac of external enclosures is the Drobo from Data Robotics. In addition to having an excellent package and support for RAID built-in it features a uniquely simple upgrade path. Just add more drives and the Drobo will reconfigure itself to use them appropriately. Unlike traditional RAID you can also use different capacity drives at the same time. The newest version benchmarks almost as well as a standard drive enclosure so performance is quickly becoming a non-issue.
If you'd prefer to have a solution that is independent of any particular computer a Network Attached Storage device might be for you. Buffalo and Infrant (now owned by Netgear) were the early SOHO pioneers in this space. Their products are essentially external RAID arrays with a network interface. They tend to run Linux or something similar and are accessed through a web-browser. Some of them offer advantages over traditional RAID arrays (ReadyNAS units from Netgear for example support "X-RAID" which allows you to replace the drives in an array with larger drives and have the unit automatically upgrade itself. A feature you might only need once but which will pay for itself in that one use!)
The traditional problem with NAS units has been speed. Despite claims of Gigabit Ethernet interfaces the data transfer rates over a network have never equalled those of local drives. This is an excellent use for a Working Directory--a location where you can copy your images for high-performance editing prior to returning them to your storage device. But the units are getting faster and cheaper so they are quickly becoming a viable alternative. Even Drobo has begun to address this market with their Drobo Share add-on which turns a Drobo into a NAS unit. Pre-built NAS units do provide a simple "one stop" shop for storage you can get to from any computer. One important resulting advantage is that you don't need to have any of your computers on to have your images accessible.
If the cost of a NAS throws you for a loop and you have an old computer lying around you might consider using it as a dedicated image server. As a bonus you could probably also store your music on it. Just add some disk drives like we detail above and share the disks across the network.
Performance of even the best broadband solutions is not currently sufficient for your main image storage for editing if you do any serious photography at all. We do discuss online as an option for backup later on.
Many of us started out trying to simply copy files from one folder to another to make a backup copy. We quickly found this process slow and error prone. Fortunately there are some great and free solutions. My favorite is AllwaySync for Windows although Microsoft's SyncToy is another nice option. There are similar utilities for Mac.
Sync tools like these are quicker because they only copy the changed files, and more reliable because they have better error logging and recovery--for example they'll keep going even after they have a problem with a file. The advantage of using a sync utility instead of a more common backup program is that the result is a fully useable duplicate of the file system. That makes it easy to recover individual files, compare folders, and even use the sync copy in place of your original if you need to in case of some catastrophic failure of your original device.
There are two important and non-obvious tips you'll want to know before you get started with a sync tool:
It used to be practical to backup to tape cartridges. It's still possible, but unless you have thousands to spend on a very high end tape backup system tape capacity has not kept up with the size of our image libraries. My SCSI DAT tape drive, for example, has 40 Gigabyte Cartridges while my image library is close to 2 Terabytes. So it would take 50 tapes if I wanted to back my entire library up to tape. Aside from the several hundred dollars the tapes would cost and the days it would take, if it fails in the middle most backup systems require that you start over. So unless you have a small image library or a huge budget tape isn't very practical.
DVDs have actually jumped ahead of Tape in both ease of use and capacity. With DVD capacity closing in on 50GB they are a potential alternative for backups. I don't think they are a great solution as a primary backup for your entire library (unless you have a small library or the patience to burn a few dozen of them) but they provide a very nice solution for incremental backups of your images from a particular event, trip or time range. Many of our safari participants burn their images to DVD as they travel and keep that backup even when they are home as an emergency backup if something happens to their disk drive and online backups.
Because Network Attached Storage (NAS) devices have their own CPU and Operating System they can be programmed to do some clever things like copy themselves automatically on a regular basis. Most current units can copy their contents either to USB drives attached directly to the unit or to another identical NAS unit elsewhere on your network. This is a really nice and low overhead way to keep a copy. The only caveat is the one we gave above about automatic copies. Be careful you don't accidentally propogate a problem onto your backup device.
Whatever strategies you use, make sure to diversify your backups. Keep them in different places using more than one technology. That limits your risk from a single failure. One important way to diversify is to keep a full set of backups off site. Whether it is at a friend's house or a bank, it helps protect you in case of a serious problem in your home or office--like a fire, theft, earthquake or lightning strike. I use a simple "leapfrog" system where I purchase a new (usually larger) backup drive array every 6 to 12 months and move the previous one to a relative's house in a different town. Obviously for more protection you could create a new offsite backup with greater frequency.
There is another not often mentioned but ugly problem with image storage. All media--disk, CD, DVD, tape included--are susceptible to random image deterioration, often called "bit rot." This can happen whether a CD or Drive is online, offline, or in a vault. You can find out if it has happened by creating a special value called a checksum from the file contents and then later re-creating it and comparing the two. Unfortunately that only tells you that you have a problem and you shouldn't rely on the file but it doesn't help you fix the issue. The simplest way to help mitigate this issue is to keep mulitple independent copies (called Replicas). It is one reason that I don't re-use my annual backup copies. I keep each one, so for a three year old image (for example) I've got 5 different copies (from 3 years ago, 2 years ago, 1 year ago, my local backup and my working copy) at least three of which are permanent and would not be affected by any future corruption of the working image. So the key is to have a system where corruption in your working files doesn't "infect" your backup copies of the file and where you have multiple independent backups so an infection in one doesn't affect all of them.
It seems like hardly a week goes by without an announcement relating to online storage. There are over a dozen solutions which offer to help you keep your valuable files safe online. There are three really significant problems with these solutions for photographers:
First, upload performance. It might literally take months to upload a terabyte of images to a server over even a fast home broadband connection. Some services let you "seed" your online backup by sending them a hard drive loaded with images, but at that point I have to wonder why I wouldn't just put it in a safe deposit box and save the money.
Second, cost. The consumer services (Carbonite, Microsoft, Google, etc.) all have very low storage or bandwidth limits making them unusable for serious photographers. Services which offer larger capacities and higher speeds are very expensive for the storage you get. You're better off keeping a backup drive offsite and updating it periodically--unless you have a serious business need for global online access to your images or for immediate image recovery in the event of a disaster.
Third, longevity. The recent bankruptcy and sudden closure of Digital Railroad points out that many of the firms offering these services may not last as long as your photo career. Many of us have dead-ended boxes lying around our studios from other bankrupt firms which were offering hybrid local+online storage servers. So you're putting your life in someone else's hands when you rely on them to safeguard your most valuable assets.
Right now online backup only makes sense on the two extremes--the very casual photographer with a small image library and a highspeed connection and the very high-end studio with a very real need for guaranteed online access to their images and the budget to fund it. For most advanced amateurs and one or two person professional photo partnerships there isn't a good balance of cost and performance to online solutions yet.
Archiving is the most specialized of the areas we'll cover. Most of us don't really need to archive images. We can keep them all online and then back them up. Archiving is needed when you have too many images to keep them all online and need to store some offline. With disk drives selling for $100 per Terabyte and computers able to use several drives most of us can keep all our images handy on connected drives. But if you can't or don't want to then you have a couple options for archiving:
Archive to offline Hard Drives: The simplest solution to archiving is to simply copy the images to an external hard drive and then take it offline. DigitalPro 5 offers a unique feature where once you've browsed or cataloged a drive it can optionally remember all the images on that drive and let you search for them. Then you'll be reminded of which images you have offline and you can go re-attached the drive.
Archive to DVD: With the advent of Blu-Ray DVDs you can put almost as much on a DVD as on a tape drive. They are large enough for most events or even trips to fit on one or two DVDs. This makes them an excellent choice for image archives. They can of course fail, like any device, so remember to make duplicates--in essence a backup of your archive.
The downside to archiving is that any copy of an image that isn't actively managed and rewritten periodically can physically deteriorate--and you won't know it. So your DVDs or external drive might be slowly going bad someplace but since they are offline you won't realize it until you try to retrieve an image.
--David Cardinal, Pro Shooters LLC, Contents Copyright Cardinal Photo 2009. All Rights Reserved.
Remember you can always download the latest version of DigitalPro, including a 30-day free and fully functional trial, at ProShooters and you can always purchase it from our online distributor, WRP.
If you have feedback on this article, topics you'd like to see covered, or tips you'd like to share with other photographers please feel free to post them to our popular Digital Photographer Forums on nikondigital.org.