Backup procedures
Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: Backup procedures

  1. #1
    Frustrated Mad Scientist
    Join Date
    Dec 2004
    Posts
    1,152

    Question Backup procedures

    Morning all,

    We carried out a spot check on our infrastructure team last week to make sure they were carrying out the backups properly. We found that they had a had a balls up last month and been a bit lax in their record keeping.

    On checking their backup procedure document (in order to beat them with it) we found it to be a bit lacking to say the least.

    I've got to provide a report on what appeared to go wrong last month and provide recommendations for them to redraft the procedure.
    I've written my own procedures for others in the past but always either for myself or for computer illiterate staff not for a bunch of MCSE/CCNA folk who should know better.

    I've put my initial suggestions below but what have I missed out/overlooked and is their a better way of putting these things to the team so they won't be too pissed off?

    A full breakdown of the 3 stage tape rotation system
    The location of each tape in the backup series
    A full description of the off site tape storage rotation
    A detailed method for resolving tape backup failures
    A description of what information is to be logged and by whom in the event of backup failures
    A process of incident escalation in the event of continued backup failures


    The procedure should be usable by an external contractor who has not worked within the organisation before.

  2. #2
    T̙͓̞̣̯ͦͭͅͅȂͧͭͧ̏̈͏̖̖Z̿ ͆̎̄
    Join Date
    Dec 2004
    Posts
    3,171
    Hi Aspman,

    This might give you a few extra ideas...

    http://backup.knowledgestorm.com/ksb...es/1/index.jsp

    Eg

  3. #3
    Senior Member kr5kernel's Avatar
    Join Date
    Mar 2004
    Posts
    347
    I think the main thing with authoring a good tape backup procedure is the ability to verify data and then the means of data retrieval, I worked a place one time where their script was off and data was not being written correctly. If they never test the integrity of the back up as well as proper means of data recovery, the backup is useless.
    kr5kernel
    (kr5kernel at hotmail dot com)
    Linux: Making Penguins Cool Since 1994.

  4. #4
    Priapistic Monk KorpDeath's Avatar
    Join Date
    Dec 2001
    Posts
    2,628
    Without a doubt I'd do monthly random test restores to be sure the data is where it's supposed to be. The process of actually backing up the data is only a quarter of the job. Anyone can blindly backup data, even MCSEs.

    And as far as your comment about how they should know better.... you should know better than to think someone else should know better... did that make sense?
    Mankind have a great aversion to intellectual labor; but even supposing knowledge to be easily attainable, more people would be content to be ignorant than would take even a little trouble to acquire it.
    - Samuel Johnson

  5. #5
    Senior Member RoadClosed's Avatar
    Join Date
    Jun 2003
    Posts
    3,834
    even MCSEs
    LMAO. They are pretty good at sliding a tape into a drive and watching the blinking light. You can also use an A+ but they may need some remedial training.
    West of House
    You are standing in an open field west of a white house, with a boarded front door.
    There is a small mailbox here.

  6. #6
    Senior Member kr5kernel's Avatar
    Join Date
    Mar 2004
    Posts
    347
    LOL, the customer service aspect of A+ comes in real handy with backup procedures....
    kr5kernel
    (kr5kernel at hotmail dot com)
    Linux: Making Penguins Cool Since 1994.

  7. #7
    Frustrated Mad Scientist
    Join Date
    Dec 2004
    Posts
    1,152
    Fair comments, lots of letters after your name doesn't stop you from being lazy & complacent.

    Good call about writing random test restores into the procedure.

  8. #8
    AO übergeek phishphreek's Avatar
    Join Date
    Jan 2002
    Posts
    4,325
    I didn't see this thread at first, so I'm gonna reply now.

    I'm responsible for the backups where I work.

    I have a 10 tape rotation where I keep data for at leat two weeks on tape.
    I have a nightly of imporant data (file servers, domain email database, individual email boxes, etc.)

    The reason I do our email server twice is because I have the ability to restore specific mailboxes down to the message. Or if the mail server dies... I can restore the whole thing.

    I do a nightly full backup of the file servers to keep the rotation simple. If I were to do incremental, it'd be a pain keeping track of what is where... so I just backup it all.

    I do incremental nightly backups of the file servers to a dedicated NAS (network attached storage) for quick restoration over the network. This is tested all the time. Users have a bad habit of deleting their files. All I can say about this is... NAS rules! Backups and restores are so quick and easy. They've saved a lot of people's ass countless times and gotten me pently of "thank you sooooo much!" lunches.

    At the end of the month, the last full backup is taken out of the rotation and kept at the remote site. At the end of the year, there is a full backup that is taken out of the rotation and kept at the remote site. At the end of the year, I put all the montly backups back in rotation but keep my year end out of rotation.

    The tapes are sent offsite and stored in a fireproof vault.
    (Yes... a real walk in need two combos and 2 keys and alarm codes to enter vault)

    Those tapes sent offsite are tested on a bimonthly basis as part of our disaster recovery testing.
    Audit loves to "surprise" us with disaster recovery tests. Its no surprise... we have everything ready for them at a moments notice.

    Everything is logged with date, tape serial number, report of backup status, machines backed up, when and where it was sent offsite. At the remote site I also have a log where the tapes are logged as received at the remote site and also log when it leave the remote site to go back into rotation.

    I backup everything so much that there is no way that I'd be stuck explaining why I can't perform a restore on any of my mission critical systems at a moments notice.
    Quitmzilla is a firefox extension that gives you stats on how long you have quit smoking, how much money you\'ve saved, how much you haven\'t smoked and recent milestones. Very helpful for people who quit smoking and used to smoke at their computers... Helps out with the urges.

  9. #9
    Senior Member
    Join Date
    Jan 2005
    Posts
    217

    Arrow JUST TO REFLECT ABOUT BACKUP

    JUST TO REFLECT ABOUT BACKUP

    I remember my second job in an I.T. Provider for telecommunication companies back in my country. Being a Senior Technical Support for the organization, we are tasked in doing the BACKUP of our system including the database as well as the disaster recovery routine in the DATA CENTER. Transactions through the Server were very critical, since we serve customers nationwide 24/7. For the logs, we have automatic as well as manual logs for the job. Once the job is finished, a program (customized by the company’s programmers) will automatically save the logs and send an e-mail report (via intranet) to corresponding team tasks to audit it in a routine basis. We are still doing manual logs in the data sheet as a way of backup log. We are using HP Tape (forgot the model?) and most of the Servers were either Windows 2000 Server or Windows 2000 Advance Server. The system functions as a 3 ONSITE SQL SERVERS (1 Main Server and 2 Backup Servers). Other Server functions as RAS, PROXY, WEB and PRODUCTION (operations).

    1) DAILY BACKUP - The Backup Procedure however, is given a schedule time wherein clients may not be allowed to connect in 15 minutes max. It is being done between 10:30PM-11:00PM nightly. Data includes the SQL Database (after closing daily transactions for all the customers nationwide). Automatic (scheduled) Tape BACKUP as well as mirror Server BACKUP is being done daily for the UPDATED SQL DATA. Each tape each day (labelled MON, TUE...).
    2) WEEKLY BACKUP – The next level of BACKUP assigned to TECHNICAL SUPPORT GROUP is the weekly (weekend) BACKUP. ALL OF THE WEEK’S TRANSACTION AND UPDATED DATA will be Automatically (scheduled) saved in the Tape BACKUP as well as mirror Server BACKUP.
    3) MONTHLY & YEARLY BACKUP – As to the sensitivity of the DATA as well as the SYSTEM itself (including reports, e-mails, others), this task falls to the NETWORK ADMINISTRATOR and the DATABASE ADMINISTRATOR accompanied by a SENIOR PROGRAMMER.
    4) Disaster Recovery is also being simulated on a MONTHLY basis to ensure that the system is prepared for it.

    Each backup, once finished is being brought to the Office of the Chief Operating Officer (COO) and being kept there securely (a separate office from the DATA CENTER).

    The company had also planned to hire a third-party DATA CENTER to provide IMMEDIATE MIRROR OFFSITE SERVER once the ALREADY-REDUNDANT DATA CENTER SYSTEM FAILS TO FUNCTION. Unfortunately, for the sake of learning more about OFFSITE REDUNDANT SERVER, during those proposal times when I was fortunate to leave the company and accept a job here abroad.

    One thing to mention about BACKUPS, tape was sensitive specifically with its labels, since in scheduling, we use the tapes customized label to identify with each schedule. It is the job of the TECH SUPP to check the tape mounted (for next schedule) before logging out the SERVER ROOM (BTW, our Servers were kept in a separate computer room properly monitored by 2 CAMERAS directly monitored and recorder from the remote office of the COO. Even the monitoring has a schedule and tapes of the camera were being kept together with the BACKUP tapes). For added security, the BUILDING has an authentication system (using swipe cards) for entering the building premises. Each employee is provided a card for the main entrance of the floor. Our own system has also a separate swipe card (ID) before entering the DATA CENTER. Every ENTYR-EXIT made by the card is being logged as well in the system.

    In the event that Disaster happens, a complete guide to the RECOVERY of the system is a must-know-how for each member of the BACKUP group. A report would follow about the recovery and a brainstorming to be included in the earliest possible meeting together with the concerned group. Warranting an improvement in the system or procedure, as soon as justified and approved, implementation took place.

    Lastly, I could add that it is also important to define the MAXIMUM RETENTION PERIOD wherein the DATA should be kept for security purposes. And backup tapes sometimes worn-out, you have to consider it as well.

    Yo!
    \"Life without FREEDOM is no life at all\". - William Wallace
    MyhomE MyboX StealtH (loop n. see loop.)
    http://www.geocities.com/sebeneleben/SOTBMulti.gif

  10. #10
    Senior Member nihil's Avatar
    Join Date
    Jul 2003
    Location
    United Kingdom: Bridlington
    Posts
    17,190
    / slightly off topic

    In my experience, if you have a problem with a procedure such as backing up, this may well extend beyond your short term security backups.

    think the main thing with authoring a good tape backup procedure is the ability to verify data and then the means of data retrieval, I worked a place one time where their script was off and data was not being written correctly. If they never test the integrity of the back up as well as proper means of data recovery, the backup is useless.
    This is an excellent point, and key to the whole issue. Many years ago we were doing a major upgrade to an IBM mid-range machine. We bought brand new "top quality" media and made two sets of backups. When we tested them, neither would have worked because there were some "dud" media items in both sets..............even though the stuff was fresh out of the box

    Lastly, I could add that it is also important to define the MAXIMUM RETENTION PERIOD wherein the DATA should be kept for security purposes. And backup tapes sometimes worn-out, you have to consider it as well.
    If I were going to be pedantic, I would say that this should be FIRST.............define your requirements then design the processes and procedures to satisfy them.

    I once worked in an environment where we received a transactions tape each week which we processed and paid the supplier. Our system would reject claims that did not belong to us.

    Now, the supplier only kept records for three months, and the detail for one week, and I discovered that the people who should have paid could not be billed, because of the missing detail.

    Now they used to moan about me retaining their tapes (cost about $700 per year) but soon went quiet when I designed a little system, entered my retained data, and put $235,000 on their bottom line. For some strange reason their EVP Finance trusted me and my development team more than he did his entire IT department

    So, my message is: perhaps you should look at your entire organisation's data retention and recovery requirements, as you may have problems elswhere that no-one knows about?

    Cheers

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •