Back to TOC
Transfer data
over network
Data can be transferred to another computer with any file
transfer program that uses a secure transport protocol. Examples
of such programs are scp, sftp and
rsync. Technically, any computer can be used for
transferring files, but to avoid interference with data
collection and data processing we recommend that you use
smbcopy.slac.stanford.edu, which is on a
gigabit network link and is dedicated to doing file
transfers.
Some examples:
scp lyso_*.img
user@my_lab.edu:remote_data_dir/
Recursively and with compression:
scp -r -C
my_image_dir user@my_lab.edu:remote_data_dir/
Another useful command is rsync. It compares the
directory content on two different locations and only copies what
is needed to synchronize the two.
rsync -auvP my_data_dir
user@my_lab.edu:remote_data_dir/
The first time you issue the command, everything in
my_data_dir will be copied to remote_data_dir. If you repeat the
command after collecting some more data only new or modified
files will be transferred. Rsync works great for keeping two
local directories in sync as well. See the section on using
external disks below.
Make sure that you have enough disk space to store all your
data at the target location before you start the file transfer.
To get the size of a directory in kilobytes use the
command:
du -sk dir_name
You might want to compress your data before transferring to
reduce the time. Especially files from analysis directory
compress well as do the image files from ADSC and mar CCD
detectors. Files from the mar345 image plate scanner are already
compressed.
Commonly used compression programs are gzip and
bzip2. The latter compress a little bit better at the
expense of longer execution time. To compress all image files in
your current directory type:
gzip *.img
Client software
There are a lot of software available for doing network file
transfers. For Unix/Linux, the command line tools like
scp and rsync are documented here,
sftp is an interactive commandline tool much like old
ftp. There is a GTK based GUI application,
gftp, that is easy to use. Some good freeware clients
for Microsoft Windows worth trying are WinSCP and FileZilla. There are also
many commercial clients for the Windows platform.
Back to TOC
Backing up to locally connected disk device, FireWire (IEEE
1394) or USB.
Overview
Backing up directly to a FireWire or USB connected hard disk
is a very fast and convenient way to store data. It is relatively
secure, provided that you have a second backup of the data as
well. Make sure that it is not your only backup of your data.
We support the use of the de facto standard Linux file system
ext2/ext3, as well as Windows FAT32 file system.
Please note that neither Windows NTFS nor Mac HFS
can be used.
Important issues
If you use a FAT32 formatted file system there area a few
important issues you need to know about. For data integrity we
recommend that you use a Linux file system. But the portability
of FAT32 makes it very attractive.
Connecting the drive
The Linux workstations at each beamline are configured to
allow connecting an external hard disk via FireWire or USB. A
cable labeled "FireWire" comes out of the console panel
close to the monitor. The tool board has a spare FireWire and a
USB cable that can be used with the workstations at the desk.
Power up your disk
and connect the cable, wait ~10s to let the operating system
detect your disk and create appropriate mount points. The Window
Manager will create an icon on the desktop and in the File
Manager for each file system (also called a volume) on the
disk.
The name of the icon is same as the volume label if one
exists, "FAT32_VOL" in the screenshot.
An unlabeled volume will get an icon name that relates to the
size of the volume. E.g. "103G Volume" in
the screenshot.
To mount the volume, right click on the icon and select
"Mount Volume" from the drop down menu.
The mount point for the labeled volume is
/media/[Label], in this case the mount point would
be
/media/FAT32_VOL
The mount point for an unlabeled volume uses the following
pattern, depending on how many unlabled volumes there
are.
/media/disk
/media/disk2
...
If the desktop icon does not appear after connecting the disk,
try disconnecting the cable and wait 15 seconds. Then
connect the disk again. If you still have problems, contact
support staff.
Disconnecting the drive
Before disconnecting the
hard drive, be sure to unmount all volumes on the disk. There is a real
risk of corrupting the file systems if disconnecting the cable
while writing.
To unmount, right click on the desktop icon for the mounted
disk and select "Unmount Volume" from the drop
down menu.
You may get this
error message when trying to unmount the volume.
The cause of this error is almost always a process which
"current working directory" is on the mounted
filesystem. Go through your different terminal windows and type
pwd to find the offending process.
Transferring your files
To copy your files over to the FireWire disk you can use one
of the following commands.
cp -auv
/data/username/my_data_dir /media/disk/SSRL/
rsync -auvP /data/username/my_data_dir
/media/disk/SSRL/
Both of these commands will recursively copy all files in the
specified directory, symbolic links will not be followed, and
links will be preserved as links. If the command is used a second
time, only new or modified files will be copied. If you don't
want to have the listing scrolling down the screen, remove the
"v" parameter from the command line.
Important! If the
external disk has a Windows FAT file system you must modify the
rsync parameters like this to get reliable updates.
rsync
-rltuvP --modify-window=1 /data/username/my_data_dir
/media/disk/SSRL/
File permissions
When mounting a FAT32 file system, all files, old and new will
have the ownership of the account that issues the mount command.
They will be world readable and only writable by the owner.
On a Linux file system, ext2 and ext3, pre existing files will
retain their ownership and permissions. This implies that before
you bring your disk to SSRL, you must create a directory on the
file system that is either owned by the same UID as your SSRL
computer account, or there has to be a directory that is world
writable. New files will have the same ownership and permissions
as the account issuing the mount command. Note that to change
ownership of the disk's top directory, root privileges is
generally needed.
Back to TOC
