Table of Contents
This page lists the backup options SSRL supports and how to use them. If you feel there are other technologies that will be useful, please mention this in your "End Of Run Summary" and we will take your suggestions into consideration.
Data can be transferred to another computer with any file transfer program that uses a secure transport protocol. Examples of such programs are scp, sftp and rsync. Technically, any computer can be used for transferring files, but to avoid interference with data collection and data processing we recommend that you use smbcopy.slac.stanford.edu, which is on a gigabit network link and is dedicated to doing file transfers.
Some examples:scp lyso_*.img user@my_lab.edu:remote_data_dir/
Recursively and with compression:scp -r -C my_image_dir user@my_lab.edu:remote_data_dir/
Another useful command is rsync. It compares the directory content on two different locations and only copies what is needed to synchronize the two.rsync -auvP my_data_dir user@my_lab.edu:remote_data_dir/
The first time you issue the command, everything in my_data_dir will be copied to remote_data_dir. If you repeat the command after collecting some more data only new or modified files will be transferred. Rsync works great for keeping two local directories in sync as well. See the section on using external disks below.
Make sure that you have enough disk space to store all your data at the target location before you start the file transfer. To get the size of a directory in kilobytes use the command:du -sk dir_name
You might want to compress your data before transferring to reduce the time. Especially files from analysis directory compress well, as do the image files from ADSC and mar CCD detectors. Files from the Pilatus detector in cbf format and files from the mar345 image plate scanner are already compressed.
Commonly used compression programs are gzip and bzip2. The latter compress a little bit better at the expense of longer execution time. To compress all image files in your current directory type:gzip *.img
There are a lot of software available for doing network file transfers. For Unix/Linux, the command line tools like scp and rsync are documented here, sftp is an interactive commandline tool much like old ftp. There is a GTK based GUI application, gftp, that is easy to use. Some good freeware clients for Microsoft Windows worth trying are WinSCP and FileZilla. There are also many commercial clients for the Windows platform.
Backing up to locally connected disk device, FireWire (IEEE 1394) or USB.
Backing up directly to a FireWire or USB connected hard disk is a very fast and convenient way to store data. It is relatively secure, provided that you have a second backup of the data as well. Make sure that it is not your only backup of your data.
We support the use of the de facto standard Linux file systems ext2/ext3/ext4 or xfs, as well as Windows FAT32 file system. Please note that neither Windows NTFS nor Mac HFS can be used.
If you use a FAT32 formatted file system there area a few important issues you need to know about. For data integrity we recommend that you use a Linux file system. But the portability of FAT32 makes it very attractive.
The Linux workstations at each beamline are configured to allow connecting an external hard disk via FireWire or USB. A cable labeled "FireWire" comes out of the console panel close to the monitor. The tool board has a spare FireWire and a USB cable that can be used with the workstations at the desk.
Power up your disk and connect the cable, wait ~10s to let the operating system detect your disk and create appropriate mount points. The Window Manager will create an icon on the desktop and in the File Manager for each file system (also called a volume) on the disk.
The name of the icon is same as the volume label if one exists, "FAT32_VOL" in the screenshot.
An unlabeled volume will get an icon name that relates to the size of the volume. E.g. "103G Volume" in the screenshot.
To mount the volume, right click on the icon and select "Mount Volume" from the drop down menu.
The mount point for the labeled volume is /media/[Label], in this case the mount point would be/media/FAT32_VOL
The mount point for an unlabeled volume uses the following pattern, depending on how many unlabled volumes there are./media/disk
If the desktop icon does not appear after connecting the disk, try disconnecting the cable and wait 15 seconds. Then connect the disk again. If you still have problems, contact support staff.
Before disconnecting the hard drive, be sure to unmount all volumes on the disk. There is a real risk of corrupting the file systems if disconnecting the cable while writing.
To unmount, right click on the desktop icon for the mounted disk and select "Unmount Volume" from the drop down menu.
You may get this error message when trying to unmount the volume.
The cause of this error is almost always a process which "current working directory" is on the mounted filesystem. Go through your different terminal windows and type pwd to find the offending process.
To copy your files over to the FireWire disk you can use one of the following commands.cp -auv /data/username/my_data_dir /media/disk/SSRL/
rsync -auvP /data/username/my_data_dir /media/disk/SSRL/
Both of these commands will recursively copy all files in the specified directory, symbolic links will not be followed, and links will be preserved as links. If the command is used a second time, only new or modified files will be copied. If you don't want to have the listing scrolling down the screen, remove the "v" parameter from the command line.
Important! If the external disk has a Windows FAT file system you must modify the rsync parameters like this to get reliable updates.rsync -rltuvP --modify-window=1 /data/username/my_data_dir /media/disk/SSRL/
When mounting a FAT32 file system, all files, old and new will have the ownership of the account that issues the mount command. They will be world readable and only writable by the owner.
On a Linux file system, pre existing files will retain their ownership and permissions. This implies that before you bring your disk to SSRL, you must create a directory on the file system that is either owned by the same UID as your SSRL computer account, or there has to be a directory that is world writable. New files will have the same ownership and permissions as the account issuing the mount command. Note that to change ownership of the disk's top directory, root privileges is generally needed.
Webmaster Content questions: Thomas
|Last modified:Thursday, 27-Oct-2016 18:27:46 PDT.|