Backup procedures

[Aspman]

Morning all,

We carried out a spot check on our infrastructure team last week to make sure they were carrying out the backups properly. We found that they had a had a balls up last month and been a bit lax in their record keeping.

On checking their backup procedure document (in order to beat them with it) we found it to be a bit lacking to say the least.

I've got to provide a report on what appeared to go wrong last month and provide recommendations for them to redraft the procedure.
I've written my own procedures for others in the past but always either for myself or for computer illiterate staff not for a bunch of MCSE/CCNA folk who should know better.

I've put my initial suggestions below but what have I missed out/overlooked and is their a better way of putting these things to the team so they won't be too pissed off?

A full breakdown of the 3 stage tape rotation system
The location of each tape in the backup series
A full description of the off site tape storage rotation
A detailed method for resolving tape backup failures
A description of what information is to be logged and by whom in the event of backup failures
A process of incident escalation in the event of continued backup failures


The procedure should be usable by an external contractor who has not worked within the organisation before.

[Egaladeist]

Hi Aspman,

This might give you a few extra ideas...

http://backup.knowledgestorm.com/ksb...es/1/index.jsp

Eg

[kr5kernel]

I think the main thing with authoring a good tape backup procedure is the ability to verify data and then the means of data retrieval, I worked a place one time where their script was off and data was not being written correctly. If they never test the integrity of the back up as well as proper means of data recovery, the backup is useless.

[KorpDeath]

Without a doubt I'd do monthly random test restores to be sure the data is where it's supposed to be. The process of actually backing up the data is only a quarter of the job. Anyone can blindly backup data, even MCSEs.

And as far as your comment about how they should know better.... you should know better than to think someone else should know better... did that make sense?

[RoadClosed]

even MCSEs

LMAO. They are pretty good at sliding a tape into a drive and watching the blinking light. You can also use an A+ but they may need some remedial training.

[kr5kernel]

LOL, the customer service aspect of A+ comes in real handy with backup procedures....

[Aspman]

Fair comments, lots of letters after your name doesn't stop you from being lazy & complacent.

Good call about writing random test restores into the procedure.

[phishphreek]

I didn't see this thread at first, so I'm gonna reply now.

I'm responsible for the backups where I work.

I have a 10 tape rotation where I keep data for at leat two weeks on tape.
I have a nightly of imporant data (file servers, domain email database, individual email boxes, etc.)

The reason I do our email server twice is because I have the ability to restore specific mailboxes down to the message. Or if the mail server dies... I can restore the whole thing.

I do a nightly full backup of the file servers to keep the rotation simple. If I were to do incremental, it'd be a pain keeping track of what is where... so I just backup it all.

I do incremental nightly backups of the file servers to a dedicated NAS (network attached storage) for quick restoration over the network. This is tested all the time. Users have a bad habit of deleting their files. All I can say about this is... NAS rules! Backups and restores are so quick and easy. They've saved a lot of people's ass countless times and gotten me pently of "thank you sooooo much!" lunches.

At the end of the month, the last full backup is taken out of the rotation and kept at the remote site. At the end of the year, there is a full backup that is taken out of the rotation and kept at the remote site. At the end of the year, I put all the montly backups back in rotation but keep my year end out of rotation.

The tapes are sent offsite and stored in a fireproof vault.
(Yes... a real walk in need two combos and 2 keys and alarm codes to enter vault)

Those tapes sent offsite are tested on a bimonthly basis as part of our disaster recovery testing.
Audit loves to "surprise" us with disaster recovery tests. Its no surprise... we have everything ready for them at a moments notice.

Everything is logged with date, tape serial number, report of backup status, machines backed up, when and where it was sent offsite. At the remote site I also have a log where the tapes are logged as received at the remote site and also log when it leave the remote site to go back into rotation.

I backup everything so much that there is no way that I'd be stuck explaining why I can't perform a restore on any of my mission critical systems at a moments notice.

[scratchONtheBOX]

JUST TO REFLECT ABOUT BACKUP

I remember my second job in an I.T. Provider for telecommunication companies back in my country. Being a Senior Technical Support for the organization, we are tasked in doing the BACKUP of our system including the database as well as the disaster recovery routine in the DATA CENTER. Transactions through the Server were very critical, since we serve customers nationwide 24/7. For the logs, we have automatic as well as manual logs for the job. Once the job is finished, a program (customized by the company’s programmers) will automatically save the logs and send an e-mail report (via intranet) to corresponding team tasks to audit it in a routine basis. We are still doing manual logs in the data sheet as a way of backup log. We are using HP Tape (forgot the model?) and most of the Servers were either Windows 2000 Server or Windows 2000 Advance Server. The system functions as a 3 ONSITE SQL SERVERS (1 Main Server and 2 Backup Servers). Other Server functions as RAS, PROXY, WEB and PRODUCTION (operations).

1) DAILY BACKUP - The Backup Procedure however, is given a schedule time wherein clients may not be allowed to connect in 15 minutes max. It is being done between 10:30PM-11:00PM nightly. Data includes the SQL Database (after closing daily transactions for all the customers nationwide). Automatic (scheduled) Tape BACKUP as well as mirror Server BACKUP is being done daily for the UPDATED SQL DATA. Each tape each day (labelled MON, TUE...).
2) WEEKLY BACKUP – The next level of BACKUP assigned to TECHNICAL SUPPORT GROUP is the weekly (weekend) BACKUP. ALL OF THE WEEK’S TRANSACTION AND UPDATED DATA will be Automatically (scheduled) saved in the Tape BACKUP as well as mirror Server BACKUP.
3) MONTHLY & YEARLY BACKUP – As to the sensitivity of the DATA as well as the SYSTEM itself (including reports, e-mails, others), this task falls to the NETWORK ADMINISTRATOR and the DATABASE ADMINISTRATOR accompanied by a SENIOR PROGRAMMER.
4) Disaster Recovery is also being simulated on a MONTHLY basis to ensure that the system is prepared for it.

Each backup, once finished is being brought to the Office of the Chief Operating Officer (COO) and being kept there securely (a separate office from the DATA CENTER).

The company had also planned to hire a third-party DATA CENTER to provide IMMEDIATE MIRROR OFFSITE SERVER once the ALREADY-REDUNDANT DATA CENTER SYSTEM FAILS TO FUNCTION. Unfortunately, for the sake of learning more about OFFSITE REDUNDANT SERVER, during those proposal times when I was fortunate to leave the company and accept a job here abroad.

One thing to mention about BACKUPS, tape was sensitive specifically with its labels, since in scheduling, we use the tapes customized label to identify with each schedule. It is the job of the TECH SUPP to check the tape mounted (for next schedule) before logging out the SERVER ROOM (BTW, our Servers were kept in a separate computer room properly monitored by 2 CAMERAS directly monitored and recorder from the remote office of the COO. Even the monitoring has a schedule and tapes of the camera were being kept together with the BACKUP tapes). For added security, the BUILDING has an authentication system (using swipe cards) for entering the building premises. Each employee is provided a card for the main entrance of the floor. Our own system has also a separate swipe card (ID) before entering the DATA CENTER. Every ENTYR-EXIT made by the card is being logged as well in the system.

In the event that Disaster happens, a complete guide to the RECOVERY of the system is a must-know-how for each member of the BACKUP group. A report would follow about the recovery and a brainstorming to be included in the earliest possible meeting together with the concerned group. Warranting an improvement in the system or procedure, as soon as justified and approved, implementation took place.

Lastly, I could add that it is also important to define the MAXIMUM RETENTION PERIOD wherein the DATA should be kept for security purposes. And backup tapes sometimes worn-out, you have to consider it as well.

Yo!

[nihil]

/ slightly off topic

In my experience, if you have a problem with a procedure such as backing up, this may well extend beyond your short term security backups.

think the main thing with authoring a good tape backup procedure is the ability to verify data and then the means of data retrieval, I worked a place one time where their script was off and data was not being written correctly. If they never test the integrity of the back up as well as proper means of data recovery, the backup is useless.

This is an excellent point, and key to the whole issue. Many years ago we were doing a major upgrade to an IBM mid-range machine. We bought brand new "top quality" media and made two sets of backups. When we tested them, neither would have worked because there were some "dud" media items in both sets..............even though the stuff was fresh out of the box

Lastly, I could add that it is also important to define the MAXIMUM RETENTION PERIOD wherein the DATA should be kept for security purposes. And backup tapes sometimes worn-out, you have to consider it as well.

If I were going to be pedantic, I would say that this should be FIRST.............define your requirements then design the processes and procedures to satisfy them.

I once worked in an environment where we received a transactions tape each week which we processed and paid the supplier. Our system would reject claims that did not belong to us.

Now, the supplier only kept records for three months, and the detail for one week, and I discovered that the people who should have paid could not be billed, because of the missing detail.

Now they used to moan about me retaining their tapes (cost about $700 per year) but soon went quiet when I designed a little system, entered my retained data, and put $235,000 on their bottom line. For some strange reason their EVP Finance trusted me and my development team more than he did his entire IT department

So, my message is: perhaps you should look at your entire organisation's data retention and recovery requirements, as you may have problems elswhere that no-one knows about?

Cheers