Sonar Workflow 4 - Long-Term Archiving and Storage of Projects

From Sonar

Home Page * Getting Started * Workflow * Tips, Techniques and Tutorials * Errors and Workarounds * Making Music * Composing, Arranging & Songwriting * Optimizing Your DAW * Recording Gear * Included Components * Third-Party Effects * Third-Party Virtual Instruments * Computer Systems and Components * Free Downloads * External SONAR resources

Formats

CWP + Audio Files Folder

It is highly recommended that you select "Use Per-Project Audio Folders" in SONAR's Options → Global → Audio Data page (it's at the very bottom, under Per-Project Audio). This puts all the audio files for a given project into their own directory (er, folder), with the project file (.cwp) in the project folder directly above the audio folder. In other words, this option creates a directory structure like the following:

E:\\MySonarProjects
- Killer_Song_01
  - Audio
    - sound001.wav
    - sound002.wav
    - sound003.wav
    - sound004.wav
  - Killer_Song_01.cwp

- Rocking_Dog
  - Audio
    - sound001.wav
    - sound002.wav
    - sound003.wav
    - sound004.wav
  - Rocking_Dog.cwp

As can be seen from this, per-project audio folders make it quite easy to back up projects -- you just copy the top-level folder of your songs (the Killer_Song_01 and Rocking_Dog folders in this case) somewhere else.

The alternative to per-project audio files is having all the audio files from all your songs put in one massive folder, with file names that give you absolutely no clue as to what song they belong to. There is absolutely no advantage to the "all audio in one folder" approach.

Note that when you back up a project while using per-project audio folders, the audio data is not compressed in any way -- it's in simple .wav" files. This can make for a massive amount of data. To get around this, you can use ZIP, RAR or some other type of compression to drastically reduce file sizes. There is a serious caveat to this, however -- the omnipresent danger of putting everything in one big file, which means that a single corrupted byte can leave you with absolutely nothing from your project.

To get around this there are two solutions. First, you can use an applications such as WinRAR which allow you to compress each file to a single archive. You'll end up with the same number of files in the same places this way, but any corrpution will be partial -- you may lose a file or two to a dying DVD, but not your entire project.

The other solution is to create parity files using QuickPAR, which will allow you to recreate with perfect accuracy entire files or even corrupted parts of a single huge file.

CWB/BUN Bundle Files

Whether you use per-project audio folders or not, you can save all the audio files, as well as the project file, from a given song in one large "Bundle" file, which will have the extension .bun or .cwb.

This approach has the advantage of putting everything you need for a given song (except external programs, such as softsynths and plug-ins) in one single place -- one single file, to be exact. However, a very large number of SONAR users are vehemently opposed to this solution, as bundle files are known for becoming corrupted, and of course face all the problems that come with storing something as one big file.

To get around the dangers of having one huge file, QuickPAR is strongly recommended. As to corruption caused by SONAR itself, either when creating or opening bundle files, this is an open question, and there is no known solution for corruption caused this way.

Therefore, the CWP + Audio Files Folder backup method is highly recommended.

Mixdowns

Along with your SONAR project files, which allow you to continue tracking, mixing, twiddling, and so on, it is a good idea to create and store a mixdown of your project in its current state. This way you will always have some version of your song in complete form, since while your project files may work in every future verison of SONAR ever made, your plug-ins and virtual instruments may become lost, you may forget to install them, or they may become incompatible with future operating systems.

At the very least, having a mixdown will allow you to hear what you created so many years ago and approximate the same sound (if that's what you're after) with different plug-ins and virtual instruments.

And again, to reduce file size you can compress the mixdown file as a ZIP or RAR archive, or you can export it directly as a lossless format such as a FLAC file for even better compression.

Physical Media

Once you've decided on a format to back up to, and after you've hopefully created several hundred MB of parity files with QuickPar, you still have to find somewhere to put all these files. This is a far more complicated issue than many believe, as no storage medium is perfect -- all will fail eventually.

This bears repeating: every single storage medium in existence will, sooner or later, fail.

With that in mind, there are ways to can minimize your risk, the easiest being to make multiple backups (hopfully on different types of media) and store them in different places. If you make 5 backups on bad DVDs, they will all most likely die at more or less the same time. And if you have 5 backups on 5 different types of media that are all stored in your house, a fire will quickly eliminate them all.

Hard Drives

Hard drives are fast, cheap and can hold a vast amount of data. However, when they die they often do so catastrophically, meaning quickly and violently. The upside of this is that files can often be recovered from a crashed hard drive with realtive ease -- if you have several hundred or thousand dollars to pay a professional data recovery service.

Hard drives' tremendous convenience often outweighs these considerations, though, and they are often the go-to backup medium for DAW users.

Onto Two Drives At Once

This is not strictly a backup method, but a way of reducing your chances of losing your current work. It involves using two or more hard drives simultaenously to store the same data, and it is known as RAID. With a mirrored RAID array, if one of your drives dies, the other one keeps on ticking and you can keep on working as if nothing had happened. Then, when you can (hopefully soon after the drive dies) you remove the dead hard drive and replace it with another one of equal or greater capacity, and the RAID software (or hardware, depending on your setup) copies the data from the good drive to the empty new one, and your safe again.

Onto The Same Drive Your Project Is On

To save a mix you are particularly fond of before going for that extra level of mixing perfection, for example, backing up onto the same hard drive you use for your SONAR projects, or a different partition on that same drive, may be a practical option. This is risky -- you will lose everything if the drive fails, and the same may happen if your computer is the victim of a virus, an idiot who knows what "format c:\\" does, a nasty power surge, or physical damage.

Think of this as an makeshift version control system more than a backup solution.

Onto A Different Drive In The Same Computer

Backing up your project onto a different hard drive than your project files are on is a (wee) step up from using the same drive for this purpose. It eliminates hard disk failure as a potential cause of lost backups for all intents and purposes (the odds of two drives failing simultaneously are quite low).

But this method still leaves you vulnerable to viruses, idiots, power surges, physical damage -- in short, anything that can damage your audio project drive can damage your backup drive if it's in the same computer, except simple, spontaneous single-drive failure.

Again, this is more of a kludged versioning system than a true backup solution.

Onto A Different Drive In A Different Computer

Now we're starting to enter into genuine backup territory. Putting your backup files on a different hard drive greatly decreases the chances of something happening to both your working and backup files at the same time.

Unless the second hard drive is physically connected (through a network) or on the same part of the power grid (e.g. your kid's computer, in your house, even if it is not networked with your DAW). In these cases power surges can very well affect both drives at once, and you're up the proverbial creek again. Nasty viruses that spread over networks can be another cause of disaster (and you know your kid's computer is chock full of spyware, trojans, viruses, evil macros, malicious toolbar components and other malware).

Onto An External Hard Drive

This is by far the safest hard drive-based backup method. You store your backups on one or more external hard drives (typically Firewire or USB, though even IDE can be used for this).

With an external hard drive, you plug the drive in, copy over the files to be backed up, unplug the drive, and keep it somewhere safe — hopefully off-premises. Do this with more than one drive and your chances of not losing everything at once increase tremendously.

For long-term storage (archiving), it is recommended to plug your drive in and spin it up anywhere from one to four times a year, to prevent stiction.

Onto a Network Attached Storage (NAS)

This can be a step up in reliability from an external hard drive. I recommend the Thecus N5200, which supports 5 disks in a RAID 6 configuration that can tolerate the loss of 2 different disks without losing data. It has a Gigabit Ethernet interface and is almost as fast as an external hard drive. A full system with 5 250GB disks costs around $1200, which is cheap for the peace of mind that it brings for all critical files and backups. And, don't tell anyone, but you can use it for more than just audio. --Dcastle 07:12, 30 September 2006 (EDT)

DVDs

With the advent of cheap DVD burners and media, many thought that the backup issue had finally been solved by this high-capacity medium -- burn your files onto a DVD and forget about them.

Unfortunately, DVDs are extremely prone to partial or total failure, sometimes after as little as two months. Of course, some can last for many years. The problem is you never know which are the good ones and which are the bad ones.

Some general guidelines for using DVDs as archival media:

Always Back Up To Two Or More DVDs

What's it cost? Maybe 25 cents more, plus 5 minutes of your time? Do it!

Always Record Several Hundred MB of Parity Files Along With Your Project Backups Using QuickPar

This will allow you to recover corrupted files from dying DVDs.

Use a real DVD Tester

The "Verify Disk" functions of many (if not all) DVD burning programs are absolutely and totally useless for verifying burn quality. They simply verify that a disk is readable -- and readable lumps utterly perfect DVDs together with ones that are one mote of dust away from total failure, along with everything in between. A DVD declared "readable" by such a program may fail the next week.

To remedy this, verify every recorded DVD with a true DVD tester, such as Nero CD-DVD Speed's Disc Quality and "ScanDisc" functions. Such programs read every sector on the DVD and tell you exactly how many errors there are in each, using a nice and very readable graph. Fewer than 280 PIE errors and 4 PIF errors per sector are considered a good result.

A series of free CD and DVD testing tools is available here.

Handle Your DVDs Properly

DVDs are even more sensitive to mishandling than CDs, as well as most newborns. Treat them right!

Do not put labels on DVDs. The adhesive can damage the written layer (which, contrary to popular belief, is right below the TOP face of the DVD, the opposite side from which it is written) and anything less than perfect positioning can make the DVD wobble.

Do not write anywhere on a DVD except the clear plastic inner ring. No data is stored there. And only use a special felt-tip marker for this.

Never touch the top or bottom surfaces of a DVD. Scratch the bottom and the laser has trouble reading the data. Scratch the top and you can easily destroy the data physically.

This is a very misunderstood concept, but CDs and DVDs are actually more fragile on the top surface than the bottom because scratching through the lacquer and damaging the underlying reflective coating is unrecoverable.

Store DVDs in a dark place. UV rays can damage them.

Store DVDs in a dry place. Mold can grow on them in damp places, adhesives can begin to deteriorate, and so on.

On these last two topics, NIST says: A temperature of 18°C and 40% RH would be considered suitable for long-term storage. A lower temperature and RH is recommended for extended-term storage.

Store DVDs upright.

If you have to clean a DVD, do it with a lint-free cloth using distilled water, isopropyl alcohol or methanol. Clean from the center hole outwards, never in a circle.

Reverify Your DVDs Periodically

Use the same real DVD testing program periodically to make sure your disks are not deteriorating. If they are, recover your data and re-backup as soon as possible. Doing this every three to six months would not be unreasonable.

Use The Latest Firmware For Your Burner

Unlike firmware for many other devices, DVD burner firmware is almost always benign -- the chances of your frying your drive are next to nil.

DVD firmware consists basically of improved write strategies for different media, and/or entirely new write strategies for new and old media alike. Often, these write strategies also allow faster burning of existing media.

Newer firmware almost always improves burn quality, at least of the types of media each update covers.

Use Quality Media

Logical. Makes sense -- why save a few dollars when the ultimate price may be your demo or album? The problem is that it's impossible to identify quality media in the store, or at home for that matter, unless you go to almost ridiculous extremes.

It's a dirty little secret in the industry that the DVDs you buy from Memorex, Fuji, Imation, and any other brand are almost never made by them. These big brand names, as well as all the small fries, are basically mere resellers, having their labels slapped on DVDs made by other companies. In other words, there is no such thing as a good brand of DVD -- only good manufacturers.

Ergo, there is absolutely no correlation between DVD brand and DVD quality.

Further complicating matters, a given brand (the names you trust!) will typically sell DVDs made by multiple manufacturers -- for example, "Memorex" 8x DVD+Rs sold in packs of 10 jewel boxes, in 25-packs, in 50-packs and in 100-packs may well all be made by different companies, and may have vastly different levels of quality. And you have no way of knowing.

Actually, there is one way. To find out the true manufacturer of a given DVD must use special software that reads DVDs Media ID code, which is stored on every disc. Only then will you know who made your DVD -- and you'll probably be surprised. Ever heard of Daxon, Moser Baer, Interaxia, Nanya Tech, CMC Magnetics, LeadData or Mitsubishi-Kagei? These are some of the real DVD makers.

But keep in mind that more than a few manufacturers fake the Media ID code on their DVDs, putting the code of a more prestigious maker onto their own junk.

To find quality DVD media you have to buy blind, test like crazy, and when you find some that works well you have to run back to the same store and buy a couple hundred dollars worth of the exact same brand of the exact same speed of the exact same type DVD, making sure what you buy has the exact same packaging and format as what you tested (the 100-pack is probably made by a different company than the 10-pack). Oh, and you have to hope it's all from the same lot, because resellers change suppliers all the time.

DigitalFAQ has an excellent guide to who makes whose DVDs, as well as links to software for identifying DVD manufacturers. Not that this is any guarantee, as they themselves would be the first to point out.

The same site also has a very good series of FAQs and guides to other DVD-related issues.

Finally, hardware matters. Some DVDs that burn perfectly in one DVD recorder die a quick death when burned in another. So when you do find that killer media after extensive testing, don't go and tell your friends about the great DVDs you found -- they may well be mediocre or even garbage when put in your friends' burners.

If you want to delve very, very deep into this issue, the US National Institute for Standards and Technology (NIST) has a 50 page study called Care and Handling Guide for the Preservation of CDs and DVDs available for your reading pleasure.

Burn at the Right Speed

Burn your DVDs at the right speed. But what is the right speed? The only way to know is by testing each lot of DVDs on your own computer, unfortunately.

The combination of the burner you use, its current firmware, and the DVDs you're writing to all affect the right write speed. In some cases, burning 8x media at 8x media produces optimal results, and in others it produces a disaster as the dye dies due to the high laser power used to write at fast speeds. In other cases, writing to an 8x DVD at 1x may also produce a dead or dying disc, possibly because the write strategy for that particular media in your particular burner's current firmware is not very dependable.

So again, it comes down to systematic trial and error. Recomendations like "burn at 1x" or "burn at half the media's rated speed" are simply meaningless.

Tape

Tape?

Online Services

Storing your music online is not a common alternative, for several reasons: security, speed and size.

You obviously don't want your demo or latest hit to fall into the wrong hands, and you can never really be sure what happens to them once stored at an online service. Such services may also be hacked. To keep your music from getting into the evil clutches of... whoever... you can encrypt the files first, using PGP or other software.

Upload speed is also a problem, and it is exacerbated by the large file sizes involved in digital audio. But if you do decide to back your projects up online, PC World, PC Mag, Yahoo and Lights.com all have lists of free and paid online storage services.

Archiving Security Software

Whatever format and physical media you choose, there are additional steps you can take to protect your valuable (or invaluable) recordings.

QuickPar

QuickPar is a free utility that scans your files, divides them up into virtual blocks, and creates a series of PAR (parity) files that allow you to recover corrupted files from any type of media. It works on the same principles as RAID.

For example, if you choose to use virtual blocks 1MB in size, create 500MB of QuickPar files, and burn them along with 3.8GB of SONAR projects onto a DVD, you can recover up to 500 corrupted file chunks of up to 1 MB in size (it doesn't matter if the chunk has 1 byte or 1 MB of corruption). In other words, you can save up to 500MB of audio files that have been utterly corrupted by a dying DVD -- files you would never be able to access again otherwise.

This is very, very good insurance. And it works for any type of physical media.

SFV

SFV, or Simple File Verification, scans your files and creates a checksum for them. This allows you to determine if any of them have been corrupted. But unfortunately, it doesn't have the capability to recover corrupt files. For that, use QuickPar.

Lists of SFV utilities for download are available here and here. Most are freeware.

MD5

MD5 does the same thing as SFV, and has the same limitations -- it can tell you your files are corrupt, but it can't do anything about it. Again, for that use QuickPar.

A free MD5 utility is available here.

SMS 14:02, 15 September 2006 (EDT)