Get rid of tape? Inconceivable!

Written by W. Curtis Preston Tuesday, 22 January 2013 23:19

Stephen Manley published a blog post today called "Tape is Alive? Inconceivable!"  To which I have to reply with a quote from Inigo Montoya, "You keep using that word. I do not think it means what you think it means."  I say that because, for me, it's very conceivable that tape continues to play the role that it does in today's IT departments.  Yes, its role is shrinking in the backup space, but it's far from "dead," which is what Stephen's blog post suggests should happen.

He makes several good points as to why tape should be dead by now.  I like and respect Stephen very much, and I'd love to have this discussion over drinks at EMC World or VMworld sometime.  I hope that he and his employer see this post as helping him to understand what people who don't live in the echo chamber of disk think about tape.  

Stephen makes a few good points about disk in his post.  The first point is that the fastest way to recover a disk system is to have a replicated copy standing by ready to go.  Change where you're mounting your primary data and you're up and running.  He's right.  He's also right about snapshots or CDP being the fastest way to recover from logical corruption, and the fastest way to do granular recovery of files or emails.

In my initial post on the LinkedIn discussion that started this whole thing, I make additional "pro-disk" points. First, I say that tape is very bad at what most of us use it for: receiving backups across a network -- especially incremental backups.  I also mention that tape cannot be RAID-protected, where disk can be. I also mention that disk enables deduplication, CDP, near-CDP and replication -- all superior ways to get your data offsite than handing tape to a dude in a truck.  I summarize with the statement that I believe that disk is the best place for day-to-day backups.

But...

Disk has all of the above going for it.  But it doesn't have everything going for it, and that's why tape isn't dead yet -- nor will it be any time soon.

I do have an issue or two with the paragraph in Stephen's post called "Archival Recovery."  First, there is no such thing.  It may seem like semantics, but one does not recover from archives; one retrieves from archives.  If one is using archive software to do their archives, there is no "recover" or "restore" button in the GUI.  There is only "retrieve."  Stephen seems to be hinting at the fact that most people use their backups as archives -- a fact on which he and I agree is bad.  Where we disagree is whether or not moving many-years-old backup data to disk solves anything. My opinion is that the problem is not that the customer has really old backups on tape.  The problem is that they have really old backups.  Doing a retrieval from backups is always going to be a really bad thing (regardless of the media you use) and could potentially cost your company millions of dollars in fines and billions of dollars in lost lawsuits if you're unable to do it quickly enough.   (I'll be making this point again later.)

Cost

Disk is the best thing for backups, but not everyone can afford the best.  Even companies that fill their data centers with deduplicated disk  and the like still tend to use tape somewhere -- mainly for cost reasons.  They put the first 30-90 days on deduped disk, then they put the next six months on tape.  Why?  Because it's cheaper.  If it wasn't cheaper, there would be no reason that they do this.  (This is also the reason why EMC still sells tape libraries -- because people still want to buy them.)

Just to compare cost, at $35 per 1.5 TB tape, storing 20 PB on LTO-5 tapes costs $22K with no compression, or $11K with 2:1 compression.  In contrast, the cheapest disk system I could find (Promise VTrak 32TB unit) would cost me over $12M to store that same amount of data.  Even if got a 20:1 dedupe ratio in software (which very few people get), it would still cost over $600K (plus the cost of the capacity-based dedupe license from my backup software company).

It's also the cheapest way to get data offsite and keep it there.  Making another copy on tape at $.013/GB (current LTO-5 pricing) and paying ~$1/tape/month to Iron Mountain is much cheaper than buying another disk array (deduped or not) and replicating data to it.  The disk array is much more expensive than a tape, and then you need to pay for bandwidth -- and you have to power the equipment providing that bandwidth and power the disks themselves.  The power alone for that equipment will cost more than the Iron Mountain bill for the same amount of data -- and then you have the bill for the bandwidth itself.

Now let's talk about long-term archives.  This is data stored for a long time that doesn't need to be in a library.  It can go on a shelf and that'll be just fine.  Therefore, the only cost for this data is the cost of the media and the cost of cooling/dehumidifying something that doesn't generate heat.  I can put it on a tape and never touch it for 30 years, and it'll be fine (Yes, I'm serious; read the rest of the post).  If I put it on disk, I'm going to need to buy a new disk every five years and copy it.  So, even if the media were the same price (which it most certainly is not), the cost to store it on disk would be six times the cost of storing it on tape.

Unlimited Bandwidth

Never underestimate the bandwidth of a truck.  'Nuf said.  Lousy latency, yes.  But definitely unlimited bandwidth.

Integrity of Initial Write

LTO is two orders of magnitude better at writing bits than enterprise-grade SATA disks, which is what most data protection data is stored on.  The undetectable bit error rate of enterprise SATA is 1:10^15, and LTO is 1:10^17.  That's one undetectable error every 100 TB with SATA disk and one undetectable error every 10 PB with LTO.  (If you want more than that, you can have one error every Exabyte with the Oracle and IBM drives.)  I would also argue that if one error every 10 PB is too much, then you can make two copies -- at a cost an order of magnitude less than doing it on disk.  There's that cost argument again.

Long-term Integrity

As I have previously written, tape is also much better than disk at holding onto data for periods longer than five years.  This is due to the  physics of how disks and tapes are made and operated.  There is a formula (KuV/kt) that I explain in a previous blog post that explains how the bigger your magnetic grains are, the better, and the cooler your device is, the better  The resulting value of this formula gives you an understanding of how well the device will keep its bits in place over long periods of time, and not suffer what is commonly called "bit rot."   This is because disks use significantly smaller magnetic grains than tape, and disks run at very high operating temperatures, where tape is stored in ambient temperatures.  The result is that disk cannot be trusted to hold onto data for more than five years without suffering bit rot.  If you're going to store data longer than five years on disk, you must move it around.  And remember that every time you move it around, you're subject to the lower write integrity of disk.

I know that those who are proponents of disk-based systems will say that because it's on disk you can scan it regularly.  People who say that obviously don't know that you can do the same thing on tape.  Any modern tape drive supports the SCSI verify command that will compare the checksums of the data stored on tape with the actual data.  And modern tape libraries have now worked this into their system, automatically verifying tapes as they have time.

Only optical (i.e. non-magnetic) formats (e.g. BluRay, UDO) do a better job of holding onto data for decades.  Unfortunately they're really expensive. Last I checked, UDO media was 75 times more expensive than tape.

Air Gap [Update: I added this a day after writing the inital post because I forgot to add it]

One thing tape can do that replicated disk systems cannot do is create a gap of air between the protected data and the final copy of its backup.  Give the final tape copy to Iron Mountain and you create a barrier to someone destroying that backup maliciously.  One bad thing about replicated backups is that a malicious sysadmin can delete the primary system, backup system, and replicated backup system with a well-written script.  That's not possible with an air gap.

Device Obsolescence

People that don't like tape also like to bring up device obsolescence.  They say things like "you can't even get a device to read the tape you wrote 10 years ago."  They're wrong.  Even if you completely failed to plan, there is a huge market for older tape drives and you can find any tape drive used in the last 20-30 years on eBay if you have no other choice. (I know because I just did it.)

Second, if you're keeping tapes from twenty-year-old tape drives, you should be keeping the drives.  Duh.  And if those drives aren't working, there are companies that will repair them for you.  No problem, easy peasy.  Device obsolescence is a myth.

Device Life

Suppose you have a misbehaving disk from many years ago.  There are no disk repair companies.  There are only data recovery companies that charge astronomical amounts of money to recover data from that drive.

Now consider what you do if you had a malfunctioning tape, which is odd, because there's not much to malfunction.  I have been able to "repair" all of the physically malfunctioning tapes I have ever experienced (which is only a few out of the hundreds of thousands of tapes I've handled).  The physical structure of a modern tape spool is not that difficult to understand, take apart, and reassemble.

Now consider what happens when your old tape drive malfunctions, which is much more likely.  You know what you do?  Use a different drive!  If you don't have another drive, you can just send the one that's malfunctioning to a repair shop that will cost you far less than what a data recovery company will cost you.  If you're in a hurry, buy another one off eBay and have them rush it to you.  Better yet, always have a spare drive.

Legal Issues

This isn't really a disk-vs-tape issue, but I just had to comment on the customer that Stephen quoted in his blog post as saying, "I'm legally required to store data for 30 years, but I’m not required by law or business to ever recover it. That data is perfect for tape." That may be a statement that amuses someone who works for a disk company, but I find the statement to be both idiotic and irresponsible.  If one is required by law to store data for 30 years, then one is required by law to be able to retrieve that data when asked for it.  This could be a request from a government agency, or an electronic discovery request in a lawsuit.  If you are unable to retrieve that data when you were required to store it, you run afoul of that agency and will be fined or worse.  If you are unable to retrieve the data for an electronic discovery request in a lawsuit, you risk receiving an adverse inference instruction by the judge that will result in you losing the lawsuit.  So whoever said that has no idea what he/she is talking about.

Think I'm exaggerating?  Just ask Morgan Stanley, who up until the mid 00's used their backups as archives.  The SEC asked them for a bunch of emails, and their inability to retrieve those emails resulted in a $15M fine.  They also had a little over 1400 backup tapes that they needed months of time to be able to pull emails off of to satisfy an electronic discovery request from a major lawsuit from Coleman Holdings in 2005.  (They needed this time because they stored the data via backup software, not archive software.)  The judge said "archive searches are quick and inexpensive. They do not cost 'hundred of thousands of dollars' or 'take several months.'"  (He obviously had never tried to retrieve emails off of backup tapes.)  He issued an adverse inference instruction to the jury that said that this was a ploy by Morgan Stanley to hide emails, and that they should take that into consideration in the verdict.  They did, and Morgan Stanley lost the case and Coleman Holdings was given a $1.57B judgment.

Doing a retrieval for a lawsuit or a government agency request is a piece of cake -- regardless of the medium you use -- if you use archive software.  If you're use backup software to store data for many years, it won't matter what medium you use either -- retrieval will take forever.  (I do feel it important to mention that there is one product I know that will truly help you in this case, and that's Index Engines. It's a brute-force approach, but it's manageable.  They support disk and tape.)

Summary

Why isn't tape dead?  Because there are plenty of things that it is better at than disk.  Yes, there are plenty of things that disk is better at than tape.  But move all of today's production, backup, and archive data to disk?  Inconceivable!

Love my Mac & iPad; Starting to hate Apple.

Written by W. Curtis Preston Tuesday, 04 December 2012 20:11

Keep it up, Apple, and I'm going back to Windows.

I was a Windows customer for many years.  Despite running virus/malware protection and being pretty good at doing the right things security-wise, I had to completely rebuild Windows at least once a year -- and it usually happened when I really didn't have the time for it.  It happened one too many times and said, "that's it," and I bought my first MacBook Pro. (The last Windows OS I ran on bare metal was Windows XP.)

I made the conversion to MacOS about 4+ years ago.  During all this time, I have never -- never -- had to rebuild MacOS. When I get a new Mac, I just use Time Machine to move the OS, apps, and data to the new machine.  When a new version of the OS comes out, I just push a button and it upgrades itself.  I cannot say enough nice things about how much easier it is to have a Mac than a Windows box.  (I just got an email today of a Windows user complaining about what he was told about transferring his apps and user data to his new Windows8 machine.  He was told that it wasn't possible.)

My first Mac was a used MacBook Pro for roughly $600, for which I promptly got more RAM and a bigger disk drive.  I liked it.  I soon bought a brand new MacBook Pro with a 500 GB SSD drive, making it cost much more than it would have otherwise.  (In hindsight, I should've bought the cheapest one I could buy and then upgrade the things I didn't like.)  It wasn't that long before I realized that I hadn't put enough RAM in it, so I did.  (I didn't account for the amount of RAM that Parallels would take.) 

My company's second Mac was an iMac. After we started doing video editing on that, we decided to max out its RAM.  Another MacBook Pro had more RAM installed in it because Lion wanted more than Snow Leopard, and on another MacBook Pro we replaced the built-in hard drive with an SSD unit and upgraded its RAM.  We are still using that original MacBook Pro and it works fine -- because we upgraded to more RAM and a better disk -- because we could. It's what people that know how to use computers do -- they upgrade or repair the little parts in them to make them better.

The first expensive application we bought (besides Microsoft Office) was Final Cut Pro 7, and I bought it at Fry's Electronics -- an authorized reseller of Apple products.  I somehow managed to pay $1000 for a piece of software that Apple was going to replace in just a few days with a completely different product.  Not an upgrade, mind you, a complete ground-up rework of that product.  Again, anyone who followed that world knows what's coming next.  I wish I had known at the time.

First, Apple ruins Final Cut Pro

For those who don't follow the professional video editing space, Final Cut Pro was the industry standard for a long time.  Other products eventually passed it up in functionality and speed, but a lot of people hung onto Final Cut Pro 7 anyway because (A) they knew it already and (B) it worked with all their existing and past project files.  They waited for years for a 64-bit upgrade to Final Cut Pro 7. 

Apple responded by coming out with Final Cut Pro X, a product that was closer in functionality to iMovie than Final Cut Pro  -- and couldn't open Final Cut Pro 7 projects.  (In case you missed that, the two reasons that people were holding onto Final Cut Pro 7 were gone.  They didn't know how to use the new product because it was night and day a different product, and it couldn't open the old product's projects.)  FCP X was missing literally dozens of features that were important to the pro editing community.  (They have since replaced a lot of those missing features, but not all of them.) And the day they started selling FCP X, they stopped selling FCP 7.  Without going into the details, suffice it to say that there was a mass exodus and Adobe and Avid both had a very good year.  (Both products offered, and may still be offering big discounts to FCP customers that wanted to jump ship.)

But what really killed me is what happened to me personally. I thought that while Apple was addressing the concerns that many had with FCP X, I'd continue using FCP 7.  So I called them to pay for commercial support for FCP 7 so I could call and ask stupid questions -- of which I had many -- as I was learning to use the product.  Their response was to say that support for FCP 7 was unavailable.  I couldn't pay them to take my calls on FCP 7. What?

So here I am with a piece of software that I just paid $1000 for and I can't get any help from the company that just sold it to me.  I can't return it to Fry's because it's open software.  I can't return it to Apple because I bought it at Fry's.  I asked Apple to give me a free copy of FCP X to ease the pain and they told me they'd look into it and then slowly stopped returning my emails.  Thanks a bunch, Apple.  (Hey Apple: If you're reading this, it's never too late to make an apology & give me that free copy of FCP X.)

Apple ruins the MacBook Pro

Have you seen the new MBP?  Cool, huh?  Did you know that if you want the one with the Retina display, you'd be getting the least upgradeable, least repairable laptop in history?  That's what iFixit had to say after they tore down then 15" and 13" MBPs.  You won't be able to upgrade the RAM because it's soldered to the motherboard.  You'll have to replace the entire top just to replace the screen -- because Apple fused the two together.

When I mention this to Apple fans and employees, what I get is, "well it's just like the iPad!"  You're right.  The 15-inch MacBook Pro is a $2200 iPad.  This means that they can do things like they do in the iPad where they charge you hundreds of dollars to go from a 16 GB SSD chip to a 64 GB SSD chip, although the actual difference in cost is a fraction of that.  Except now we're not talking hundreds of dollars -- we're talking thousands.  This means that you'll be forced to buy the most expensive one you can afford because if you do like I did and underestimate how much RAM you'll need, you'll be screwed.  (It costs $200 more to go from an 8GB version to a 16GB version, despite the fact that buying that same RAM directly from Crucial will cost you $30 more -- not $200.)

Apple's response is also that they'll let the market decide.  You can have the MBP with the Retina Display and no possibility of upgrade or the MBP without the Retina Display and the ability to upgrade.

First, I want to say that that's not a fair fight.  Second, can you please show me on the Apple website where they show any difference between the two MBPs other than CPU speed and the display?  Everyone is going to buy the cheaper laptop with the cooler display, validating Apple's theory that you'll buy whatever they tell you to buy. (Update: If you do order one of the Retina laptops, it does say in the memory and hard drive sections, "Please note that the memory is built into the computer, so if you think you may need more memory in the future, it is important to upgrade at the time of purchase." But I don't think the average schmo is going to know what that means.)

Apple Ruins the iMac

I just found out today that they did the same thing they did above, but with the iMac.  And they did this to make the iMac thinner.  My first question is why the heck did the iMac need to be thinner?  There's already a giant empty chunk of air behind my current iMac because it's so stinking thin already.  What exactly are they accomplishing by making it thinner?

One of the coolest things about the old iMac was how easy it was to upgrade the RAM.  There was a special door on the bottom to add more RAM.  Two screws and you're in like Flynn.  Now it's almost as bad as the MacBook Pros, according to the folks over at iFix it.  First, they removed the optical drive.  Great, just like FCP. They made it better by removing features!  Their tear down analysis includes sentences like the following:

• "To our dismay, we're forced to break out our heat gun and guitar picks to get past the adhesive holding the display down."
• "Repair faux pas alert! To save space and eliminate the gap between the glass and the pixels, Apple opted to fuse the front glass and the LCD. This means that if you want to replace one, you'll have to replace both."
• "Putting things back together will require peeling off and replacing all of the original adhesive, which will be a major pain for repairers."
• "The speakers may look simple, but removing them is nerve-wracking. For seemingly no reason other than to push our buttons, Apple has added a barb to the bottom of the speaker assemblies that makes them harder-than-necessary to remove."
• "Good news: The iMac's RAM is "user-replaceable." Bad news: You have to unglue your screen and remove the logic board in order to do so. This is just barely less-terrible than having soldered RAM that's completely non-removable."

It is obvious to me that Apple doesn't care at all about upgradeability and repairabiity.  Because otherwise they wouldn't design a system that requires ungluing a display just to upgrade the RAM!  How ridiculous is that?  And they did all this to make something thinner that totally didn't need to be thinner.  This isn't a laptop.  There is absolutely no benefit to making it thinner.  You should have left well enough alone.

Will they screw up the Mac Pro, too?

I have it on good authority that they are also doing a major redesign of the Mac Pro (the tower config).  This is why we have waited to replace our iMac w/a Mac Pro, even though the video editing process could totally use the juice.  But now I'm scared that they'll come out with another non-repairable product.

Keep it up, Apple, and I'm gone

Mac OS may be better than Windows in some ways, but it also comes with a lot of downsides.  I continually get sick of not being able to integrate my Office suite with many of today's cool cloud applications, for example.  I still have to run a copy of Windows in Parallels so I can use Visio and Dragon Naturally Speaking. 

You are proving to me that you do not want intelligent people as your customers.  You don't want people that try to extend the life of their devices by adding a little more RAM or a faster disk drive.  You want people that will go "ooh" and "ahh" when you release a thinner iMac, and never ask how you did that, or that don't care that they now have to pay extra for a DVD drive that still isn't Blu-Ray.

Like I said when I started this blog post.  I like my Mac.  I love my new iPad Mine, but I am really starting to hate Apple.

Backing Up The Cloud

Written by W. Curtis Preston Monday, 03 December 2012 22:31

"The Cloud" has changed the way I do business, but I'm not always sure how I should back up the data I have "up there."  So I thought I'd write a blog post about my research to address this hole in our plan.

Truth in IT, Inc. is run almost entirely in the cloud.  We have a few MacBooks and one iMac & a little bit of storage where we do our video editing of our educational & editorial content, as well as our ridiculous music video parodies.  But that's it.  Everything else is "out there" somewhere.  We use all of the following:

• Salesforce.com: CRM
• Liquidweb.com: Managed web hosting
• Virtualpbx.com: Phone System
• Sherweb: Hosted Exchange Services
• Quickbooks Online: Online bookkeeping & payroll
• Q-Commission: Online commission management (talks to Salesforce & Quickbooks)
• Act-On: Marketing automation system
• iCloud: Syncs & stores data from mobile devices
• File synchronization system with history*
• A cloud backup service for our laptops*

We have data in salesforce that is nowhere else, and the same is true of our web servers, email servers, & laptops.  Did you know that using salesforce's backups to recover data that you deleted is not included in your contract, and that if you need them to recover your data (due to your error) it will cost at least $10,000?!?!?!

I want my own backups of my data in the cloud.  I don't think we're alone in this regard. I therefore took a look at what our options are.  The process was interesting.  The following is a copy of an actual chat session I had with one of our providers:

curtis preston says:    
one question i've wondered about is how people back up the email that is hosted with you
<support person> says:    
You mean when they lose it?
curtis preston says:    
Let me put it plainly/bluntly: Scenario is you do something wrong and the exchange server i'm hosted on dies and your backups are bad.  What can I do in advance to prepare for that?
<support person> says:    
Well, when using Outlook there is always a copy on the computer that is made that could be used
<support person> says:    
And to be extra-sure you can create backups from time to time
<support person> says:    
but we have a 7 days of backup on a server so the chance both the main server and the backup cannot be backup is pretty low
<support person> says:    
Everything is really well backup here you don't have to worry

And that pretty much sums up the attitude of most of the vendors, "We've got it. Dont worry. That's the whole reason you went to the cloud!" Here's my problem with that.  Maybe they do have it; maybe they don't.  If it turns out they don't know how do IT, there's a good chance they also don't know how to configure a backup system.  I'd like to have my own copy in someone else's system and I don't mind paying for the privilege.  It turned out that all but hosted Exchange had what I would consider a decent answer.  (As far as I can tell, it's not the fault of our provider; Multi-tenant Exchange has some things ripped out of it that create this problem.)

Backups for cloud apps

There are actually a lot of solutions out there to back up cloud applications.  Here's what I found:

• Salesforce can be automatically and regularly backed up via backupify.com, asigra.com, or ownbackup.com.
• Gmail & Google Apps can be backed up via backupify.com.
• Quickbooks Online can be backed up OE Companion. 
• Hosted servers or virtual servers can be backed up via any cloud backup service that supports the operating system that you're using. 
• Laptops and desktops can also easily be backed up by most cloud backup services. 
• If you're using a file synchronization service, those files will also be backed up via whatever you choose for your backup solution for your laptops & desktops.
• Offline copes of Outlook data can be used to restore lost Exchange data, but it seems clunky, and you need to make the offline copy manually.

Does the lack of backups for the cloud serve as a barrier to the cloud for you or your company?  Or are you in the cloud and you have the same worries as me?  Is there a particular app that worries you?  Tell me about it in the comment section.

*I don't give the name of either of these for various reasons.

One RTO/RPO or two? (Onsite & Offsite)

Written by W. Curtis Preston Friday, 30 November 2012 22:45

Disaster recovery experts do not agree whether you should have one-and-only-one recovery time objective (RTO) and recovery point objective (RPO) for each applicaition, or two of them.  What am I talking about?  Let me explain.

What are RTO and RPO, you ask? RTO is the amount of time it should take to restore your data and return the application to a ready state (e.g. "This server must be up w/in four hours").  RPO is the amount of data you can afford to lose (e.g. "You must restore this app to within one hour of when the outage occurred").

Please note that no one is suggesting you have one RTO/RPO for your entire site. What we're talking about is whether or not each application should have one RTO/RPO or two.  We're also not talking about whether or not to have different values for RTO and RPO (e.g. 12-hour RPO and 4-hour RTO).  Most people do that.

In defense of two RTOs/RPOs (for each app)

If you lose a building (e.g via a bomb blast or major fire) or a campus (e.g. via an earthquake or tsunami) it's going to take a lot longer to get up and running than if you just have a triple-disk failure in a RAID6 array.  In addition, you might have an onsite solution that gets you a nice RPO or RTO as long as the building is still intact.  But when the building ceases to exist, most people are just left to their latest backup tape they sent to Iron Mountain.  This is why most people feel it's acceptable to have two RTOs/RPOs: one for onsite "disasters" and another for true, site-wide disasters.

In defense of one RTO/RPO (for each app)

It is an absolute fact that RTOs and RPOs should be based on the needs of the business unit that is using any given application.  Those who feel that there can only be one RTO/RPO say that the business can either be down for a day or it can't (24-hour RTO).  It can either lose a day of data or it can't (24-hour RPO). If they can only afford to be down for one hour (1-hour RTO), it shouldn't matter what the cause of the outage is -- they can't afford one longer than an hour.

I'm with the first team

While I agree with the second team that the business can either afford (or not) a certain amount of downtime and/or data loss, I also understand that backup and disaster recovery solutions come with a cost.  The shorter the RTO & RPO, the greater the cost.  In addition, solutions that are built to survive the loss of a datacenter or campus are more expensive than those that are built to survive a simple disk or server outage.  They cost more in terms of the software and hardware to make it possible -- and especially in terms of the bandwidth required to satisfy an aggressive RTO or RPO.  You can't do an RPO of less than 24-36 hours with trucks; you have to do it with replication.

This is how it plays out in my head.  Let's say a given business unit says that one hour of downtime costs $1M.  This is after considering all of the factors, including loss of revenue and damage to the brand, etc.  So they say they decide that they can't afford more than one hour of downtime.  No problem.  Now we go and design a solution to meet a 1-hour RTO.  Now suppose that the solution to satisfy that one-hour RTO costs $10M.  After hearing this, the IT department looks at alternatives, and it finds out that we can do a 12-hour RTO for $100K and a 6-hour RTO for $2M.

So for $10M, we are assured that we will lose only $1M in an outage.  For $2M we can have a 6-hour RTO, and for $100K we can have a 12-hour RTO.  That means that a severe outage would cost me $10M-11M ($10M + 1 hour of downtime at $1M), or $6M-$12M ($6M + $6M in downtime), or $100K-$12M ($100K + 12 hours of downtime).  A gambler would say that you're looking at definitely losing (spending) $10M, $6M, or $100K and possibly losing $1M, $6M or $12M.  I would probably take option two or three -- probably three.  I'd then put $9.9M I saved and make it work for me, and hopefully I'll make more for the company with that $9.9M than the amount we will lose ($12M) if we have a major outage.

Now what if I told you that I could also give you an onsite 1-hour RTO for another $10K.  Wouldn't you want to spend another $10K to prevent a loss greater than $1M, knowing full well that this solution will only work if the datacenter remains intact?  Of course you would.

So we'll have a 12-hour RTO for a true disaster that takes out my datacenter, but we'll have a 1-hour RTO as long as the outage is local and doesn't take out the entire datacenter.

Guess what.  You just agreed to have two RTOs.  (All the same logic applies to RPOs, by the way.)

If everything cost the same, then I'd agree that each application should have one -- and only one -- RTO and RPO.  However, things do not cost the same.  That's why I'm a firm believer in having two complete different sets of RTOs and RPOs.  You have one that you will live up to in most situations (e.g. dead disk array) and another that you hope you never have to live up to (loss of an entire building or campus).

What do you think?  Weigh in on this in the comments section.

Amazon Glacier changes the game

Written by W. Curtis Preston Friday, 24 August 2012 04:45

 

Zdnet confused about Amazon Glacier pricing

Written by W. Curtis Preston Friday, 24 August 2012 04:45

 

Dell buys a BUNCH of software products

Written by W. Curtis Preston Thursday, 05 July 2012 04:37

Dell continues its global domination plans via more acquisitions. Last year, they acquired Ocarina & Compellant.  This year they've acquired Wyse (they're still in business?) and Sonicwall. 

But now they've entered into my world.  They have announced that they are acquiring Appassure and Quest Software.  For those of you who haven't been following these companies, Appassure is a near-CDP backup software product with quite a bit of success.  Quest has a bunch of products, but among them is NetVault (a traditional network backup software product) and vRanger (a purpose-built backup product for virtualization).

The Appassure acquisition is a very solid one.  I have personally watched that company increase their market share daily in a very aggressive way.  The near-CDP story plays very well both when backing up physical servers and even better when backing up virtual servers.  It's not a small deal that they can recover a server in minutes.  It's interesting that Appassure has been doing so well, given the difficulty that other CDP and CDP-like products have had.

For those unfamiliar with Quest's history, the two backup products they have are the result of relatively recent acquisitions.  NetVault is a general-purpose network backup product that has been around a while, but has never garnered much market share.  I know they've been working to bring it up to speed with other products in the space, and have definitely increased marketing activities for it compared to its previous owner, Bakbone. vRanger was the king of the mountain in virtualization backup at one point, but they seem to have been out-marketed by Veeam lately -- but they're not taking that lying down.  Just look at product manager John Maxwell's comments on my last blog post to see that! Perhaps having Dell's marketing budget behind these products will finally get them the attention they are looking for.

Dell's challenge will be similar to Quest's: integrating all of these products into a single coherent product line.  This is a challenge they already know well.

Yesterday (Backup Version) A Parody Music Video

Written by W. Curtis Preston Tuesday, 03 July 2012 17:17

Finally the world can see what we've been showing in our Backup Central Live! shows.  This video is the first music parody song we produced, and the first music video that we did.  I hope you enjoy it.

Yesterday (Backup Version) from TrueBit.tv on Vimeo.

The musician, singer and actor for the video is none other than Cameron Romney, a talented young man I met in my daughters' show choir. He's 16 yo and did all the instruments, vocals, and sound mixing for this song. In case you're curious, yes, he's related to Mitt Romney. 1st cousin, twice removed, or something like that. This is the first in a series of these videos that I'll be publishing on TrueBit.TV. Make sure to check out our educational videos at TrueBit.TV.

Are purpose-built VMware backup apps a good idea?

Written by W. Curtis Preston Wednesday, 27 June 2012 13:37

When I'm talking about backing up of virtual servers & desktops, I always get asked "what do you think about Veeam/vRanger/PHD Virtual/VMPro?  Should I buy that instead of NBU/TSM/CV/BE/AS/DP/etc's VADP agent?"

The first thing I can say is that it has definitely become a trend to install a purpose-built backup app for VMware/Hyper-V.  On one hand, it's hard to argue with success.  People who have moved to such products have often found their backups much easier than they were before.  On the other hand, since most of them are moving from the agent-in-the-guest approach, anything would be better than that.  Some of them are also moving from their attempt to use Very Crappy Backup (VCB).  That's another one that is not hard to compete with. 

These purpose-built products do have some really awesome features.  For one, they often work around the limitation that VMware creates by using the VSS_BT_COPY backup type.  They do something to make sure that the transaction logs in guests get truncated.  Some of them also have some really interesting features of being able to run a guest directly from a backup, which leads to all sorts of interesting recovery possibilities.

My concern is that many of these products are missing what I would consider to be core functionality for a centralized backup product.  When I look closely at these products, they tend to be missing one or all of the features listed below.  What's worse, when these shortcomings are pointed out, some of their representatives look at you like "why would you want that?" I've been doing nothing but backups for almost 20 years, and I think any decent backup product should have all of these features:

Support for more than one platform

Very few shops are 100% virtualized.  If you can't also backup the few physical machines their environment, you force them to also run some other product to back those up.  More backup apps means more confusion and more confusion with each failed backup and restore.  It's simply math.

Central management

It's very common to require more than one backup server to handle a given environment.  Do you have some sort of centralized management that allows you to see all of your backup servers and manage them all from one place? 

Copies

Backup functionality #1 is to support the copying of the data from the backup source to the destination, and then to another destination.  Many of these purpose-built backup tools are really good at the first step, but have absolutely nothing for the second step.  They tell you that you can run another backup to another destination, which is not good for a number of reasons.  Or they tell you that you can buy a 3rd-party dedupe product that can replicate your backups for you.  I'm sorry, this should be in your product.  You should not require me to buy other people's products to do what your product should do on its own.  (The lack of this feature is why so many people buy inexpensive products like Backup Exec to back up the datastores created by these products.)

Config backup

If you have a database that stores configuration and/or history information about your product, it should be built into your product to back up that database.  Period.  Telling someone they can run a cron job just doesn't cut it.  And if how/when to run that cron job isn't even in your documentation, shame on you.

Tape support

This is at the end of the list for a reason.  I'll admit that this is the least important on the list, but I do believe that a backup product should have the ability to copy to tape for long term storage purposes.  Not everyone can store their backup on disk as long as they need to.  Again, many people use Backup Exec to handle this, but I believe it should be built into any backup product.

My position above has made me very unpopular with some of the purpose-built folks -- some have been very upset with me. That doesn't change the fact that the above features should be considered table stakes for any backup application.  You could argue that tape support isn't table stakes anymore, but I disagree.  That is, it is still table stakes if you want to be considered a full-fledged backup app.

Let's see how things go.

Don't be a Snail: Stop Changing Your Backup Software

Written by W. Curtis Preston Thursday, 10 May 2012 17:10

Sometimes I walk to work in the mornings (I live just over 2 miles from the office), and if I leave early enough I walk down this particular sidewalk that has just been sprinkled with water.  On each side of this sidewalk is grass, and watered grass here in Southern California tends to mean that snails will be there. The snails take advantage of the wet, cool sidewalk (and the fact that the sun isn't overhead), and they decide to cross from one bunch of grass to another bunch of grass. 

On any given morning there will be several hundred snails crossing from one side to another.  200 or so will be crossing from the left to the right, and 200 or so will be crossing from right to left.  You know, 'cause the grass on the other side is, well... you know.  The funniest one I saw was a snail that had made it 95% of the way from one side to the other, and then changed his mind and turned around.

I got to thinking about backup software (as one does), and all of the people I know that are moving from product A to product B.  Then a bunch of other people that are moving from product C to product A, while others are moving from product B to product C -- and they all think that this will make their lives soooooo much better.

I've done hundreds of backup assessments over the years.  I can only think of one or two where the gist of the recommendation was, "Your backup software sucks.  You should change it."

I can, however, think of many, many times where the problem was "you're not streaming your tape drives," or "you're manually specifying an include list and you should use the auto-selection feature," or "you're making too many full backups," or "you're using the scheduler in a way that it wasn't designed to work," and on and on and on.

Changing backup products is one of the riskiest things you can do to your backup environment.  The learning curve of the new backup product is almost definitely going to reduce your recoverability for a significant period of time.

What would be much better is to bring in an expert in that product for a few weeks and have him/her tell you how best to use the product you already have.  The learning curve is much easier, the cost is much lower, and the period of instability will be much shorter.

Don't be a snail.  Learn what the grass on your side of the sidewalk really tastes like before you start crossing the sidewalk.  Remember that some snails die along the way.