Storage and Backup Solutions for Photographers (advice from a software engineer!)
I’ve known and worked with many photographers who have had a number of issues with losing images. Whether it’s out of laziness, tech confusion or a total lack of understanding, it can ruin a photographer’s business entirely if they were to lose a client’s images.
Storage and backup of RAW files should be a very high priority for any photographer.
Years ago, I myself had an issue with a corrupted CF card when I was shooting with only one card slot at a wedding. It was mortifying but thankfully I have always known it was possible and use smaller CF cards for that reason. If one was to get corrupted and you shot the entire day on it, you’d lose nearly everything.
When you work on smaller cards, you minimize your risk and are limiting what shots could get ruined or lost.
I think this post comes mainly from the fact that I am equal parts paranoid and organized. I love having a very firm system that I use for every session and wedding because taking the guesswork out of it makes my life easier. I am also extremely paranoid and while I know my contracts are very tight, I prefer to not have to tell someone that I lost their images.
I am so excited that my husband, Kyle, agreed to get on board with this tech post for photographers!
Kyle is a software engineer and a solutions architect and he specializes in problem solving through automation, custom application development and API integrations among many other things. He on occasion will analyze, monitor and tune enterprise storage for clients to solve problems. Which is kind of what we’re going to talk about… but just on a small business scale.
This post was very difficult for Kyle because I continually asked him to simplify things that in his mind are not able to be simplified. On our first few drafts, we talked about things like RAID configurations, network card throughputs, disk read and write rates, disk and I/O block sizes, max speeds of 802.11ac wireless versus wired gigabit ethernet connections, Amazon AWS S3 storage and disk cache.
If you’re lost, that’s understandable. So was I. But we worked together and talked through everything many times in an attempt to make this post both easier to understand and more helpful.
Kyle’s Suggested Backup and Storage Solutions
Cloud Storage
This doesn’t work for most photographers and may not work for you depending on what/how much you shoot and your internet speed. Cloud storage of RAW files isn’t feasible for me and we do have quite fast upload speeds. However, it is always preferred when used correctly!
NAS System
Network Attached Storage (NAS) is similar to an external hard drive but instead of plugging in to a USB, it’s over your network and it’s a bit smarter. A NAS is a computer system thats sole purpose is storage. Ideally you would set your NAS to have mirrored hard drives. This is what I use!
Direct Attached Storage
Direct Attached Storage (DAS) is the classic external hard drive plugged in using USB, firewire, etc. If you’re going to do this, you still need to have everything in two places. Using one external hard drive is never sufficient.
Kyle’s Top Tips for Storage and Backup
Don’t put all of your eggs in one basket. Basically, don’t ever put all of your backup plans in one solution. You need to assume that at any time something could implode and you and your business would still be absolutely fine.
In the event that something does implode, leaving you with your data in only one location, you absolutely must duplicate it elsewhere ASAP. Don’t ever push this off. It becomes Priority #1.
Hard drives WILL fail. It’s not if, it’s when. Hard drives are like a car. They could run great for 400,000 miles or they might start having issues at 50,000. There’s no way to predict this but eventually, they will stop running all together. Like most things, hard drives have a life-span. You need to know this, understand it and accept it. Do not treat them as a reliable backup / storage solution on their own. This applies to all hard drives - in your laptop, in your desktop computer, in your NAS and your external USB hard drives.
If you have the bandwidth, knowledge and funds - cloud storage solutions are the best (like mentioned above) when used correctly. Always. However, for most photographers, this isn’t practical unless you have fast enough upload internet like gigabit internet. It also can be very pricey and difficult to get “caught up” when you first start using cloud storage.
Editing and exporting things on your local hard drive will always be faster than over a networked drive or USB drive. However, you still need to have those RAWs in two locations. So if you’re able to have the RAWs on your laptop/desktop computer’s hard drive until you export them for the client, that’s ideal for speed. Savannah’s custom solution below explains this!
Know where the bottlenecks are in your system hardware (this could be an entirely separate blog post so this is just glazing the surface of this issue). You could invest a lot money in the fastest solid state drive (SSD) in your network attached storage (NAS) system but most likely you’re connecting to your NAS over your network which will have a maximum speed that is slower than your SSD. So essentially, you could get the same performance out of a more affordable storage solution (a hard drive direct connected or in your NAS).
If you’re using a NAS or a DAS, it can be quicker to transfer your RAW images to them in a ZIP file. Zip up the folder containing all of your images while it’s on your computer and then transfer the large ZIP to your NAS/DAS. It can be quicker to do this one large ZIP than it is to do many small files.
Google is your friend. It can help you find products, someone to help you set up your storage solution, the answer to your one nagging technical question, tutorials, recommendations and more. If you’re lost or confused, start Googling.
Savannah’s Storage and Backup Solution
What is my setup? I have a NAS. It is a small, square computer with four slots for hard drives and the same processing power as say… your cell phone. It is on and running 24/7. It has 4 disk bays (aka slots for hard drives) and two currently have hard drives in them. The two empty slots are for eventual expansion.
QNAP NAS
HGST 4TB Hard Drive
Toshiba 4TB Hard Drive
How is the NAS connected to my computer? I can connect to my NAS via my laptop over the network - which is a collection of devices that can talk to each other. Everything on your home wifi is a part of your home network. Usually, a NAS is directly plugged into a router via ethernet which allows any device that also communicates with that router (for example… your laptop using wifi) to be able to connect to the NAS.
What does my NAS do? My NAS is set up so that I write to the NAS (put files on it, make changes, etc.) and then the NAS applies all of this to both hard drives. The two hard drives are mirrored - each one contains the exact same data. In a very simple way, it’s almost like automating the process of you putting your files on two separate portable hard drives. I do one thing on the NAS and the NAS duplicates that automatically. This is called RAID 1.
Why RAID 1? This duplication allows either drive to fail or any kind of reason and I will still have my data on the other. The NAS will notify me if either one of the hard drives is unhealthy, about to fail, etc. and I can simply replace it and then the NAS will automatically copy over the rest of the data.
But, about those hard drives… If both drives fail at the same time… there’s nothing I can do about that. To attempt to prevent this and decrease the likelihood of this ever happening, it’s important to purchase two identically spec’d hard drives from two different manufactures so that they’re a lot less likely to fail at the same time. Look for hard drives with the same capacity, rpm and same cache but different manufacturers. Example: I have a 4 TB Hitachi and a 4 TB Toshiba both 7200 rpm and 128 MB cache.
What is the catch? You might be thinking, this sounds great! Well, I’m paying for 8 TB of storage however, only 4 is usable since they are exact copies of each other. This however is still more cost effective than cloud storage. I could have the NAS automatically then back up to Backblaze or a similar cloud storage solution but this can be expensive.
Savannah’s Storage and Backup Workflow
I shoot on a camera with dual card slots. While shooting, I’m writing RAW files to both a 128 GB SD card and a variety of small sizes of Compact Flash cards. This goes back to not putting all of your eggs in one basket! It’s important to change out those CF cards fairly frequently. I like 16 and 32 GB CF cards.
I cull from the 128 SD card using Photo Mechanic and import the photos to a brand new Lightroom catalog while moving the RAW files from the SD card to my laptop’s hard drive. I always, always use a new Lightroom Catalog every single job, every single time.
I make a copy of the RAW file folder on my desktop and put that copy on my NAS.
I place the 128 GB SD card into a fireproof/waterproof safe in my office and then I format the CF cards. Unless something goes wrong, I don’t use those CF cards for culling or anything at all really.
After I edit the gallery completely in Lightroom, I export the final JPEG files to my Desktop and then upload them to Shootproof, my gallery delivery and client management service.
I move the folder containing the Lightroom catalog and the edited JPEGS to the NAS (where the RAW files already are). Each session or wedding gets a dated folder on my NAS containing the RAWs, edited final JPEGS and the LR Catalog files. Every job gets a new Lightroom Catalog, every time!
I deliver the final gallery to the client.
I delete the RAW files from my computer and format the SD card that has been in my safe.
Q & A
Question: I think I want a NAS. I’m pretty confident about that. But I have no idea how to go about setting one up or buying everything! How do I actually set one up!?
Kyle: Most of the top brands of NAS systems, (Qnap, Synolgy) are trying to be user friendly. You should be able to purchase the NAS system and disks, insert the disks, plug into your router and then follow a provided setup guide.
Question: What does your "always in at least 2 to places from start to finish” process look like with your RAW files?
Savannah:
128 GB SD Card / Multiple Smaller CF cards
128 GB SD Card / Multiple Smaller CF Cards / Laptop Hard Drive
128 GB SD Card / Multiple Smaller CF Cards / Laptop Hard Drive / NAS
128 GB SD Card / Laptop Hard Drive / NAS
128 GB SD Card / NAS
Question: How long do you keep RAW files?
Savannah: Contractually, I do not promise to keep the RAW files beyond gallery delivery but I do obviously try to as long as I can. At this point, I have all of my RAW files from June 2017 and earlier. At the end of 2019, I plan to delete 2017’s RAWs. My contract states that it is the client’s responsibility to download, backup and preserve their JPEG files and I do offer a few solutions and guidance to help them do this. It also states that their gallery will be online for 6 months following the release date and I will keep the JPEG files for 365 days following gallery release.
Question: I’m so hesitant to wipe my cards even though I have mirrored externals AND cloud backup. Help!
Savannah: You shouldn’t be nervous about this because it sounds like you’re doing everything right. Obviously freak accidents happen and that’s where having a strong contract and a great client relationship are really important. If you’re shooting on dual cards and still nervous, try waiting until about 7 days after you deliver the final gallery to the client and then format the backup card. I know we’re all worried about the client asking about random photos you may have culled out but you have to just be confident in your culling and make sure you have an image selection clause in your contract (stating that the final say is always yours). You’ve got this!!
Question: What is the best way to back everything up so that you can confidently clear your cards?
Kyle: At least two places, all the way through gallery delivery! That’s the only way.
Question: Why and how do you use a new Lightroom Catalog every single time?
Savannah: I know a lot of photographers use the same Lightroom Catalog for everything. They’ll use labeling or collections to sort their jobs/images and just constantly use that same catalog. DONT. DO. THIS. You’re again, putting all of your eggs in one basket. While this doesn’t affect the protection of your RAW files, it does affect your workflow. If your Lightroom Catalog gets corrupted (which is much more likely when it’s dealing with that heavy data load), you’ll have to re-ediit not just the current session or wedding but everything you haven’t delivered so far. If you’re doing this, it’s also almost impossible to outsource your editing. Most services and editors require you to use a new catalog every time. It’s just the smart and safe thing to do! From there, you can use collections to sort things like sneak peeks, blog images, vendor images, etc.
Question: Long term storage of 1+ year, what is the safest option?
Kyle + Savannah: Cloud storage would be the safest and therefore the best but evaluate different companies to see which meet your need and budget most (Dropbox, Backblaze, CrashPlan, AWS, Google Drive). It really depends on what you mean by long-term storage. There is definitely a difference in the way you should store data that you need to access frequently vs. data you just want to offload but can’t get rid of completely. If it’s the first, I would suggest cloud storage or a NAS. If it’s the second, cold storage, I would suggest researching Amazon Glacier or finding your own DAS solution.
As photographers, uploading 128 GB+ of raw files can be really difficult, time consuming and expensive. So you need to weigh how much you’re shooting and if this would even be feasible for you considering your upload speeds and the cost. Even though we have very fast upload speeds, this wouldn’t be feasible for me and that’s why I choose to go the route of the NAS + my extra workflow incorporations.
If you are a photographer and you have questions regarding any of this, Kyle and I would love to help you! Just send an email with your question over to Savannah@SavannahSmithPhotography.com
and we’ll help you as best as we can.