Archiving filesystems

This procedure is used to create archives of entire filesystems or fractions thereof. It can be used by system administrators or by normal users. Of course, you must have read access to the files you are archiving.

DVAR works by breaking up the target directory tree into compressed tar archives (.tgz files) of several hundred MB each. DVAR then collects the .tgz files into DVD-sized ISO images, ready to burn onto DVDs.

A complete table table of contents is included on each DVD in the set. The TOC will have an entry for the full path to each file that is archived. For each entry, it indicates on which DVD it is located, and within the DVD, which .tgz file. So it is fairly easy to search for and recover files.

New in 2008.09 version

Can run in user mode from workstations
Better labels, and users can modify them (with care).
(Run step 1 only, follow instruction to modify label file, then resume, starting with step 2).

Usage

    tcsh (normal for users):
      setup robot
    bash (normal for sysadms):
      alias DVAR=/srv/local/backup/BNL/CSBarc.01T.ds1/DVAR
    
    usage:
      DVAR -help -- list valid steps
      DVAR [-start N -jid JID] [-end M] root
  	[-start N] -- start with step# N. Must specify JID.
  	[-jid JID] -- Use JID as the job ID.
  	[-end M]   -- go thru step# M.
      
      #DVAR -help
      Step#  Description
        1    Create .spec files for .tgz archives and create label text.
        2    Use .spec to create .tgz archives (takes a long time)
        3    Create .2dvd files which group .tgz archives onto DVDs
        4    Create 0contents listing all files in all .tgz on all DVDs
        5    Using .2dvd, create ISO images from .tgz (takes a long time)
        6    Create PDF label files and XML rimage production files
        7    Print instructions for burning DVDs

At completion of step 6, DVAR will print instructions to initiate the actual burning of the DVDs. Log onto a gigabit core workstation as user (not root). Where $jid is a JobID which DVAR creates:

    alias burnit /srv/local/backup/BNL/2.1.0/burnDVD.py"
    burnit /srv/images/0/root/$jid-??.xml"

Examples

     ds1# alias DVAR /srv/local/backup/BNL/CSBarc.01T/DVAR
     ds1# DVAR /snap/people/day.0
     ds1# DVAR -end 4 /snap/ljr/0/day.0 
     ds1# DVAR -start 5 -jid ljr_0_20060817_1325 /snap/ljr/0/day.0

Notes and caveats

Design documents are in /srv/local/backup/BNL/CSBarc/0readme.*
DVAR defines temporary directories $wdir and $tdir as being subdirectories of /bck/bck/DVARtemp/. Only root can write there. /etc/cron.daily/tmpwatch on ds1 cleans files more than 48 hours old.
A listing of all files in every disk in the archive is in the file 0contents.JobID. A copy is on every disk of the archive, and in $wdir, which DVAR assigns to /bck/bck/DVARtemp/wdir/.
After running DVAR, it is still necessary actually to burn the DVDs. This can be done by root from ds1, or by any user from a java-enabled workstation. To monitor progress, burnit must be able to open a window on the initiating workstation.
DVAR can be run overnight. The actual burning procedure can be run the next day when it can be monitored. However, make sure running DVAR does not overlap with the nightly snapshot backup.
Neither the DVAR wrapper, nor the individual steps, are robust.
I believe that all files in the requested tree are archived (unless they are larger than a DVD). But it is possible that there will be errors in directory ownership or permission if the entire archive is restored. Also, files whose full pathname exceeds 255 characters may cause problems.
When archiving /snap filesystems, be sure to specify only day.0. Otherwise, multiple copies will be archived.

Last Modified: Monday, 23-Mar-2009 14:26:04 EDT

RC Home | Search | Table of Contents | General Information

Richards Center (www.rc.yale.edu) at Yale University (www.yale.edu)
Contact: webmaster_at_ben^rc^yale^edu