Redundancy = Resiliency

Recently I learned that one of my former employers was hit by ransomware, locking out access to a file and SQL server. They are one of a couple of higher education sites in the same state system that’s been hit by attacks over the last year, in a period when higher education domains have been increasingly targeted. In 2021-2022 I’ve talked to the leadership of several of these institutions, and they’re all on pins and needles; afraid that they’ll be the next victim. Now one of them is.

Building a solid defense against ransomware is a multi-pronged effort, and is a part of one’s overall security posture. You first have to convince the organization (CFO) that additional resources spent on information security are necessary, but there is some low-hanging fruit an IT department can do to mitigate or minimize damage before an attack happens. One set of approaches is described in my earlier post on Zero Trust security. The other is adopting an approach toward backups where you don’t rely on just ONE backup that’s also on the LAN.

A few years back I had a client that was hit by ransomware. The owner called and began to describe the “funny thing” that was happening, where all the file server icons were gradually changing. By this time other than pulling the CAT6 cable there was little that could be done, and before long the owner got “the email” asking for a certain amount via bitcoin to decrypt all their files.

However, we didn’t panic because we had (1) a backup service that backed up everything to a remote cloud-based disk array, and (2) a couple of external backups that were regularly rotated offsite, detached from the server. Both of these backup approaches utilized different methods that allowed us to do a Bare-Metal Restore, which we used to put things back overnight with little or no loss of data.

When servers and datacenters were smaller, we would use a variety of ever-larger tape media (DAT, LTO, DLT-S, etc.) to have backup approaches that were “out of band” — tapes once written cannot easily be erased or tampered with, particularly if one segments admin permissions properly: don’t use the same credentials for backup admin that you use for sysadmin functions, and don’t save them to the box! It wasn’t long ago that part of my job involved getting tapes to offsite storage in a bank safe deposit box. In order to go past the space limitations of tape, we had versions of tape “jukeboxes” from Dell and others that would contain a carousel of tapes. As our appetite for storage continued to grow, we moved to hybrid approaches: nightly disk-to-disk backups, followed by weekly or bi-weekly tape transfers. Sure it took longer, but we rejoiced then having not one, but TWO copies of backup media!

Two copies minimum is the least we can do, and don’t forget to occasionally do a restore when it’s not a crisis. There is nothing like attempting a restore only to feel your stomach fall to the floor when a trusted backup fails.

So to truly protect mission-critical resources it is wise to employ a variety of approaches for backups:

• Since servers have become virtualized, Hyper-V and esxi/VMware give one the ability to copy VMs between hypervisors, and along with that, the ability to roll back changes to an earlier point in time…
• With Hyper-V, the so-called “cold backup” method to Azure is a very affordable way to have servers safeguarded beyond one’s LAN. Those cold servers can just sit there in the cloud, accepting changes all the time, until that moment you flip it to on (make it hot in Azure) or promote it to being the authoritative copy, pushed down to the LAN for recovery…
• In addition, SQL servers can also have their DBs compressed, backed up, and discretely file-copied to almost any location. We used an otherwise too-small (and free) file backup from our ISP to stash a nightly copy…
• File servers can be backed up, disk-to-disk, and copies of those datasets moved or disconnected from the network for safety, like before.

So in such a scenario, each mission-critical resource can have as many as 4 backup copies. This does not include utilizing Microsoft 365: where one can move team file server data to SharePoint, and sync personal user files to OneDrive. This can provide even more redundancy of your data, and remove it from places where bad guys can try to take it away from you.