An incident earlier this week made me review the backup systems in place for my wife’s company (iHelm Enterprises).

One of our clients doesn’t exclusively use us, and they ran into a major issue when another hosting provider lost a server—with no backups available. Worse still, the client didn’t regularly back up their own data, assuming that the hosting provider would have backups—which, of course, they didn’t.

Our Backup Strategy—And Its Problems

We’ve always been a bit unusual in that we routinely perform backups, but we don’t always do them daily. Our system has been in place for a while, but the process is… well, let’s say, a bit unorthodox:

  1. Server 1 performs its backup by transferring to Server 2.
  2. Server 2 then transfers both its own backup and Server 1’s backup to the next server.
  3. This continues until the final server, where the files are placed in a repository that only this server can access.
  4. A third-party provider then performs off-site backups from this repository.

After hearing about our client’s catastrophic data loss, I decided to test our backup system—just to be sure. And here’s what I found:

  • (a) Our backup repository was completely full, and I hadn’t received any notifications about it.
  • (b) Getting files off the backup system was a nightmare. Some files were huge, and the server that had access to the repository didn’t have enough space to handle large transfers.
  • (c) Some of the servers weren’t properly notifying me of backup failures, so I had no idea whether some files were complete or corrupted.

The New Backup System

Clearly, this wasn’t ideal, so I decided to rethink our approach.

Now, our servers transmit backups directly to our office computers. One of these computers is always on (24/7). This machine then uses Windows file sharing (yes, despite being a Linux machine!) to copy the backups to a new Network Attached Storage (NAS) device, which is stored securely in the office.

Immediate Benefits We’ve Seen

  • We can easily check what files are backed up on the NAS.
  • Restoring data is much simpler—no more complex file transfers!
  • Each server has a dedicated backup schedule, so if something goes wrong, we only need to check that specific server and re-run the backup for just that machine.

The Takeaway: Always Check Your Backups!

If you provide backups for clients or even just back up your own work, take some time to review your backup system every now and then.

Our system had been in place for over a year, and while it worked at the time, changes in technology (like the affordability of NAS devices) meant that better solutions were available.

From now on, my new mantra is:
Backup – Backup – Backup – Check – Review – Backup – Backup – Backup – Check – Review!