Data Backup

Table of Contents

• Overview
  • What we support
• Transfer of data over the network
• Backup to external Disk device
  • Important Issues
  • Connect the drive
  • Disconnect the drive
  • Transfer the files
  • File permissions

Back to Macromolecular Crystallography Computer Environment

Overview

This page lists the backup options SSRL supports and how to use them. If you feel there are other technologies that will be useful, please mention this in your "End Of Run Summary" and we will take your suggestions into consideration.

Supported Backup Options

• Network based backups
• Use file transfer commands like scp, sftp or rsync to transfer the data to another computer.
  • Local network transfer
  • Connect your own laptop or NAS device to our local 100Mbit/s visitor network.
    It is relatively fast and it is guaranteed to be in a readable format when you are back at your home lab.
  • Remote network transfer
  • Reliable and convenient if your home lab has a fast network connection to SSRL. It will use Internet 2 if possible.
• Externally attached disk device, FireWire(IEEE1394) or USB.
• If you are on-site, this is the by far the fastest way to backup your data.

Back to TOC

Transfer data over the network

Data can be transferred to another computer with any file transfer program that uses a secure transport protocol. Examples of such programs are scp, sftp and rsync. Technically, any computer can be used for transferring files, but to avoid interference with data collection and data processing we recommend that you use smbcopy.slac.stanford.edu, which is on a gigabit network link and is dedicated to doing file transfers.

Some examples:

Recursively and with compression:

Another useful command is rsync. It compares the directory content on two different locations and only copies what is needed to synchronize the two.

The first time you issue the command, everything in my_data_dir will be copied to remote_data_dir. If you repeat the command after collecting some more data only new or modified files will be transferred. Rsync works great for keeping two local directories in sync as well. See the section on using external disks below.

Make sure that you have enough disk space to store all your data at the target location before you start the file transfer. To get the size of a directory in kilobytes use the command:

You might want to compress your data before transferring to reduce the time. Especially files from analysis directory compress well, as do the image files from ADSC and mar CCD detectors. Files from the Pilatus detector in cbf format and files from the mar345 image plate scanner are already compressed.

Commonly used compression programs are gzip and bzip2. The latter compress a little bit better at the expense of longer execution time. To compress all image files in your current directory type:

Client software

There are a lot of software available for doing network file transfers. For Unix/Linux, the command line tools like scp and rsync are documented here, sftp is an interactive commandline tool much like old ftp. There is a GTK based GUI application, gftp, that is easy to use. Some good freeware clients for Microsoft Windows worth trying are WinSCP and FileZilla. There are also many commercial clients for the Windows platform.

Back to TOC

Backing up to locally connected disk device, FireWire (IEEE 1394) or USB.

Overview

Backing up directly to a FireWire or USB connected hard disk is a very fast and convenient way to store data. It is relatively secure, provided that you have a second backup of the data as well. Make sure that it is not your only backup of your data.

We support the use of the de facto standard Linux file systems ext2/ext3/ext4 or xfs, as well as Windows FAT32 file system. Please note that neither Windows NTFS nor Mac HFS can be used.

Important issues

If you use a FAT32 formatted file system there area a few important issues you need to know about. For data integrity we recommend that you use a Linux file system. But the portability of FAT32 makes it very attractive.

• File names on FAT32 are case insensitive. (By design)

• File or directory names that only differ by case will be overwritten by the latter version when copying from UNIX to a FAT32 files system. Make sure you name files accordingly.

Connecting the drive

The Linux workstations at each beamline are configured to allow connecting an external hard disk via FireWire or USB. A cable labeled "FireWire" comes out of the console panel close to the monitor. The tool board has a spare FireWire and a USB cable that can be used with the workstations at the desk.

Power up your disk and connect the cable, wait ~10s to let the operating system detect your disk and create appropriate mount points. The Window Manager will create an icon on the desktop and in the File Manager for each file system (also called a volume) on the disk.

The name of the icon is same as the volume label if one exists, "FAT32_VOL" in the screenshot.

An unlabeled volume will get an icon name that relates to the size of the volume. E.g. "103G Volume" in the screenshot.

To mount the volume, right click on the icon and select "Mount Volume" from the drop down menu.

The mount point for the labeled volume is /media/[Label], in this case the mount point would be

The mount point for an unlabeled volume uses the following pattern, depending on how many unlabled volumes there are.

If the desktop icon does not appear after connecting the disk, try disconnecting the cable and wait 15 seconds. Then connect the disk again. If you still have problems, contact support staff.

Disconnecting the drive

Before disconnecting the hard drive, be sure to unmount all volumes on the disk. There is a real risk of corrupting the file systems if disconnecting the cable while writing.

To unmount, right click on the desktop icon for the mounted disk and select "Unmount Volume" from the drop down menu.

You may get this error message when trying to unmount the volume.

The cause of this error is almost always a process which "current working directory" is on the mounted filesystem. Go through your different terminal windows and type pwd to find the offending process.

Transferring your files

To copy your files over to the FireWire disk you can use one of the following commands.

Both of these commands will recursively copy all files in the specified directory, symbolic links will not be followed, and links will be preserved as links. If the command is used a second time, only new or modified files will be copied. If you don't want to have the listing scrolling down the screen, remove the "v" parameter from the command line.

Important! If the external disk has a Windows FAT file system you must modify the rsync parameters like this to get reliable updates.

File permissions

When mounting a FAT32 file system, all files, old and new will have the ownership of the account that issues the mount command. They will be world readable and only writable by the owner.

On a Linux file system, pre existing files will retain their ownership and permissions. This implies that before you bring your disk to SSRL, you must create a directory on the file system that is either owned by the same UID as your SSRL computer account, or there has to be a directory that is world writable. New files will have the same ownership and permissions as the account issuing the mount command. Note that to change ownership of the disk's top directory, root privileges is generally needed.

Back to TOC