Back to TOC
Transfer data
over network
Data can be transferred to another computer with any file
transfer program that uses a secure transport protocol. Examples
of such programs are scp, sftp and
rsync. Technically, any computer can be used for
transferring files, but to avoid interference with data
collection and data processing we recommend that you use
smbcopy.slac.stanford.edu, which is on a
gigabit network link and is dedicated to doing file
transfers.
Some examples:
scp lyso_*.img
user@my_lab.edu:remote_data_dir/
Recursively and with compression:
scp -r -C
my_image_dir user@my_lab.edu:remote_data_dir/
Another useful command is rsync. It compares the
directory content on two different locations and only copies what
is needed to synchronize the two.
rsync -auvP
my_data_dir user@my_lab.edu:remote_data_dir/
The first time you issue the command, everything in
my_data_dir will be copied to remote_data_dir. If you repeat the
command after collecting some more data only new or modified
files will be transferred. Rsync works great for keeping two
local directories in sync as well. See the section on using
external disks below.
Make sure that you have enough disk space to store all your
data at the target location before you start the file transfer.
To get the size of a directory in kilobytes use the
command:
du -sk dir_name
You might want to compress your data before transferring to
reduce the time. Especially files from analysis directory
compress well as do the image files from ADSC and mar CCD
detectors. Files from the mar345 image plate scanner are already
compressed.
Commonly used compression programs are gzip and
bzip2. The latter compress a little bit better at the
expense of longer execution time. To compress all image files in
your current directory type:
gzip *.img
Client software
There are a lot of software available for doing network file
transfers. For Unix/Linux, the command line tools like
scp and rsync are documented here. There is
a GTK based GUI application, gftp, that is easy to
use. Some good freeware clients for Microsoft Windows worth
trying are WinSCP and FileZilla. There are also
many commercial clients for the Windows platform.
Back to TOC
Backing up to locally connected disk device, FireWire (IEEE
1394) or USB.
Overview
Backing up directly to a FireWire or USB connected hard disk
is a very fast and convenient way to store data. It is relatively
secure, provided that you have a second backup of the data as
well. Make sure that it is not your only backup of your data.
We support the use of two Linux file system types, ext2
and ext3, as well as Windows FAT32 file system.
Please note that neither Windows NTFS nor Mac HFS
can be used.
Important issues
If you use a FAT32 formatted file system there area a few
important issues you need to know about. For data integrity we
recommend that you a Linux file system. But the portability of
FAT32 makes it very attractive.
Connecting the drive
The Linux workstations at each beamline are configured to
allow connecting an external hard disk via FireWire or USB. A
cable labeled "FireWire" comes out of the console panel
close to the monitor. The tool board has a spare FireWire and a
USB cable that can be used with the workstations at the desk.
Power up your disk and connect the cable, wait ~10s to let the
operating system detect your disk and create appropriate mount
points. To verify that your disk has been detected type the
following command at a shell prompt:
ls /media
If your the disk is connected via FireWire you should see a
directory named:
/media/ieee1394disk
If the disk is connected via USB there should a directory
named:
/media/usbdisk
If the partition has been labeled, i.e. been given a name at
the time of formatting the disk (some manufacturers like to put
their company name as disk label), then the mount point will be
the 'label name' regardless of the type of connection
used. Example: If the disk has been labeled "SEAGATE",
the mount point will be:
/media/SEAGATE
If the mount point does not appear after connecting the disk,
first try disconnecting the cable and wait 15 seconds. Then
connect the disk again. Don't power the disk down. If you
still have problems, contact support staff.
If the disk has more than one partition or if you connect more
than one disk the naming of the mount point will appear as listed
in this table, up to a maximum of ten disks/partitions.
| Disk or partition
number |
FireWire mount
point |
USB mount
point |
| First |
/media/ieee1394disk |
/media/usbdisk |
| Second |
/media/ieee1394disk1 |
/media/usbdisk1 |
| Third |
/media/ieee1394disk2 |
/media/usbdisk2 |
| ... |
/media/ieee1394disk... |
/media/usbdisk... |
| Tenth |
/media/ieee1394disk9 |
/media/usbdisk9 |
| Filesystem labeled "MY_DISK" |
/media/MY_DISK |
/media/MY_DISK |
|
To mount the disk, use the command:
mount
/media/ieee1394disk
or
mount /media/usbdisk
Transferring your files
To copy your files over to the FireWire disk you can use one
of the following commands.
cp -auv
/data/username/my_data_dir /media/ieee1394disk/SSRL/
rsync -auvP /data/username/my_data_dir
/media/ieee1394disk/SSRL/
Both of these commands will recursively copy all files in the
specified directory, symbolic links will not be followed, and
links will be preserved as links. If the command is used a second
time, only new or modified files will be copied. If you don't
want to have the listing scrolling down the screen, remove the
"v" parameter from the command line.
Important! If the
external disk has a Windows FAT file system you must modify the
rsync parameters like this to get reliable updates.
rsync
-rltuvP --modify-window=1 /data/username/my_data_dir
/media/ieee1394disk/SSRL/
Disconnecting the drive
Before disconnecting the hard drive, be sure to unmount the
file system. There is a real risk of corrupting the file system
if disconnecting the cable while writing. To unmount
type:
umount /media/ieee1394disk
You may get the following error message when unmounting the
drive:
umount: /media/ieee1394disk: device is busy
The cause of this error is almost always a process which
"current working directory" is on the mounted
filesystem. Go through your different terminal windows and type
pwd to find the offending process.
File permissions
When mounting a FAT32 file system, all files, old and new will
have the ownership of the account that issues the mount command.
They will be world readable and only writable by the owner.
On a Linux file system, ext2 and ext3, files will keep their
ownership and permissions. This implies that before you bring
your disk here for the first time, you must create a directory on
the file system that is either owned by the same UID as your PX
computer account, or there has to be a directory that is world
writable. New files will have the same ownership and permissions
as the account issuing the mount command.
Back to TOC
