Home | Site Map | Facilities | User Guide | Schedule | Forms | Research | News | Staff | Links
[Up]
Computing Environment
Account Request
Change Password
DVD Backup
Remote Desktop

Data Backup

Table of Contents

Back to Macromolecular Crystallography Computer Environment

Overview

This page lists the backup options SSRL supports and how to use them. If you feel there are other technologies that will be useful, please mention this in your "End Of Run Summary" and we will take your suggestions into consideration.

Supported Backup Options

  • Network based backups
  • Use file transfer commands like scp, sftp or rsync to transfer the data to another computer.
    • Local network transfer
    • Connect your own laptop or NAS device to our local 100Mbit/s visitor network.
      It is fast and guaranteed to be in a readable form back at your home lab.
    • Remote network transfer
    • Reliable and convenient if your home lab has a fast network connection to SSRL.
  • DVD creation via web interface
  • Submit backup jobs using a web interface to generate DVDs for mail delivery.
  • Externally attached disk device, FireWire(IEEE1394) or USB.
  • As of the 2006-2007 user run, all beamline computers are running RedHat Entrpise Linux 4. This makes it possible to use USB disks as well as FireWire disks. It also removes the 128GB restriction for using Windows FAT32 partions.
    Please read the instructions.

    • Back to TOC

    Transfer data over network

    Data can be transferred to another computer with any file transfer program that uses a secure transport protocol. Examples of such prgrams are scp, sftp and rsync. Technically, any computer can be used for transferring files, but to avoid interference with data collection and data processing we recommend that you use smbcopy.slac.stanford.edu, which is on a gigabit network link and is dedicated to doing file transfers.

    Some examples:

    scp lyso_*.img user@my_lab.edu:remote_data_dir/

    Recursively and with compression:

    scp -r -C my_image_dir user@my_lab.edu:remote_data_dir/

    Another useful command is rsync. It compares the directory content on two different locations and only copies what is needed to synchronize the two.

    rsync -auvP -e ssh my_data_dir user@my_lab.edu:remote_data_dir/

    The first time you issue the command, everything in my_data_dir will be copied to remote_data_dir. If you repeat the command after collecting some more data only new or modified files will be transferred. Rsync works great for keeping two local directories in sync as well. See the section on using external disks below.

    Make sure that you have enough disk space to store all your data at the target location before you start the file transfer. To get the size of a directory in kilobytes use the command:

    du -sk dir_name

    You might want to compress your data before transferring to reduce the time. Especially files from analysis directory compress well as do the image files from ADSC and mar CCD detectors. Files from the mar345 image plate scanner are already compressed.

    Commonly used compression programs are gzip and bzip2. The latter compress a little bit better at the expense of longer execution time. To compress all image files in your current directory type:

    gzip *.img

    Client software

    There are a lot of software available for doing network file transfers. For Unix/Linux, the command line tools like scp and rsync are documented here. There is a GTK based GUI application, gftp, that is easy to use. Some good freeware clients for Microsoft Windows worth trying are WinSCP and FileZilla. There are also many commercial clients for the Windows platform.

    Back to TOC

    Backing up to locally connected disk device, FireWire (IEEE 1394, i.LINK) or USB.

    Overview

    Backing up directly to a FireWire or USB connected hard disk is a very fast and convenient way to store data. It is relatively secure, provided that you have a second backup of the data as well. Make sure that it is not your only backup of your data.

    We support the use of two Linux file system types, ext2 and ext3, as well as Windows FAT32 file system. Please note that neither Windows NTFS nor Mac HFS can be used.

    Important issues

    If you use a FAT32 formatted file system there area a few important issues you need to know about. For data integrity we recommend that you a Linux file system. But the portability of FAT32 makes it very attractive.

    • Filenames on FAT32 are case insensitive. (By design)

    • Files/directories that only differ by case will be overwritten by the latter version when copying from UNIX to a FAT32 files system. Make sure you name files accordingly.

    Connecting the drive

    The Linux workstations at each beamline are configured to allow connecting an external hard disk via FireWire or USB. A cable labeled "FireWire" comes out of the console panel close to the monitor. The tool board has a spare FireWire and a USB cable that can be used with the workstations at the desk.

    Power up your disk and connect the cable, wait ~10s to let the operating system detect your disk and create appropriate mount points. To verify that your disk has been detected type the following command at a shell prompt:

    ls /media

    If your the disk is connected via FireWire you should see a directory named:

    /media/ieee1394disk

    If the disk is connected via USB there should a direcory named:

    /media/usbdisk

    If the partion has been labeled, i.e. been given a name at the time of formatting the disk (some manufaturers like to put their company name as disk label), then the mountpoint will be the 'label name' regardless of the type of connection used. Example: If the disk has been labled "SEAGATE", the mount point will be:

    /media/SEAGATE

    If the mount point does not appear after connecting the disk, first try disconnecting the cable and wait 15 seconds. Then connect the disk again. Don't power the disk down. If you still have problems, contact support staff.

    If the disk has more than one partition or if you connect more than one disk the naming of the mount point will appear as listed in this table, up to a maximum of ten disks/partitions.

    Disk or partition number FireWire mount point USB mount point
    First /media/ieee1394disk /media/usbdisk
    Second /media/ieee1394disk1 /media/usbdisk1
    Third /media/ieee1394disk2 /media/usbdisk2
    ... /media/ieee1394disk... /media/usbdisk...
    Tenth /media/ieee1394disk9 /media/usbdisk9
    Filesystem labeled "MY_DISK" /media/MY_DISK /media/MY_DISK

    To mount the disk, use the command:

    mount /media/ieee1394disk

    or

    mount /media/usbdisk

    Transferring your files

    To copy your files over to the FireWire disk you can use one of the following commands.

    cp -auv /data/username/my_data_dir /media/ieee1394disk/SSRL/

    rsync -auvP /data/username/my_data_dir /media/ieee1394disk/SSRL/

    Both of these commands will recursively copy all files in the specified directory, symbolic links will not be followed, and links will be preserved as links. If the command is used a second time, only new or modified files will be copied. If you don't want to have the listing scrolling down the screen, remove the "v" parameter from the command line.

    Disconnecting the drive

    Before disconnecting the hard drive, be sure to unmount the file system. There is a real risk of corrupting the file system if disconnecting the cable while writing. To unmount type:

    umount /media/ieee1394disk

    You may get the following error message when unmounting the drive:

    umount: /media/ieee1394disk: device is busy

    The cause of this error is almost always a process which "current working directory" is on the mounted filesystem. Go through your different terminal windows and type pwd to find the offending process.

    File permissions

    When mounting a FAT32 file system, all files, old and new will have the ownership of the account that issues the mount command. They will be world readable and only writable by the owner.

    On a Linux file system, ext2 and ext3, files will keep their ownership and permissions. This implies that before you bring your disk here for the first time, you must create a directory on the file system that is either owned by the same UID as your PX computer account, or there has to be a directory that is world writable. New files will have the same ownership and permissions as the account issuing the mount command.

    Back to TOC