Next: Referencing SSRL
Up: User's Guide to Macromolecular Crystallography Experiments
Previous: Using the SSRL Automated Mounting (SAM)
Subsections


Data collection and processing

The following sections describe how to carry out an experiment at from sample screening to full data collection making use of the integrated data collection and analysis environment at SSRL.

For additional help setting up data collection, please consult the Blu-Ice documentation.


Automated crystal screening

The high throughput screening system implemented at SSRL makes it possible to automatically collect and analyze test images and fully characterize the sample in a semi- or fully automated fashion.

To set up automated crystal screening follow the steps below or watch the video tutorials ''Uploading the Sample Spreadsheet'' and ''Automated screening from Blu-Ice''.

  1. Upload a spreadsheet containing sample information to http://smb.slac.stanford.edu/crystal-server as described in the SAM guide . You can do this any time before or during beamtime.

    Note: If no spreadsheet is available, sample characterization must be carried out for each sample manually or with the Web-Ice Autoindexing interface (the latter will keep a permanent record of the results which can be accessed by the user).

  2. Once you are enabled to use the beamline, assign the spreadsheet to the cassette or Unipuck position in the beamline dewar (if you are a remote user, the beamline support staff will tell you the correct position or assign it for you).

  3. Start Blu-Ice and select the Screening Tab. Select the samples you wish to screen, the input directory and image collection parameters as described in the Blu-Ice manual. Some tips:

    • Collect two images at different phi orientations in order to trigger automated autoindexing (check the first two ''Collect Image'' entries in the Action Sequence list of the Blu-Ice screening tab). If you collect only one image, the spots will be analyzed and assigned a score, but autoindex will not be carried out.

      Important: The images must be collected to your area in the /data disk. Please do not try to use a specific data disk (eg /data1 , /data2 , etc.) in the directory path. Use /data/''your-id'', or else the software may fail to write the image to the disk.

    • You may pause the screening after the loop has been centered. This allows you to adjust the sample centering using the click to center option. This is only recommended if you use loops much larger than the crystals (which is not a good idea). It is faster to select a large beam size in the Blu-Ice hutch tab and screen all the samples without pauses.

    • It is possible to pause the data collection after the test images have been collected - this will trigger automated strategy calculation for this sample following autoindex. This option is not recommended, unless you are very sure that you want to collect data from this particular sample (e.g., if the samples have already been screened). It is more time efficient to screen all samples automatically and then remount the best ones for data collection.

    Tip: It is possible to use automated sample characterization with the Blu-Ice screening tab without using the robot (in manual mode). The software will prompt you to change the sample in this case

Selecting samples for data collection

A summary of the autoindexing results (symmetry, resolution and mosaicity estimates and a score) will be written to the sample list information displayed in Blu-Ice and Web-Ice shortly after the images have been collected. This information can be used to help select the best samples for collection of a complete data set. Follow the instructions below or consult the Video Tutorial

To see the screening results in the Blu-Ice screening tab, make sure that you use the ''Results''view to see these columns.

You can inspect the results in detail with Web-Ice:

  1. Use the Web button above the sample information list: Click on the arrow to open the drop down menu, and select Web-Ice. Blu-Ice will open a new browser window (if you do not see the browser window, look for minimized or hidden browser windows). You will be directed to the Web-Ice Screening tab.

  2. In the Web-Ice Cassette Summary page sort the samples by score, mosaicity, resolution, or rmsd by clicking on the title of the corresponding columns.

  3. Select the sample you want to inspect and click on Cassette Details (in the gray navigation toolbar). This will let you see the analyzed image, spot statistics, crystal JPEG and autoindexing results for that particular sample.

    Important: Do not relay blindly on the score or other image statistics as a means of selecting the best crystal - always inspect the results displayed in the Cassette Details page.

    Figure 35: Cassette details navigation in Web-Ice Screening tab: Header displays the image header; Spot Statistics displays the results of image analysis before autoindexing; Crystal image shows a camera shot of the crystal; Autoindex shows the autoindex results and image score; Details is a directory browser and image display tool to inspect log and output files.
    Image datacol-cassette-details

Setting up and starting data collection

Once the optimal sample for data collection has been selected, monochromatic (non-anomalous), and simple MAD and SAD experiments from a single crystal can be set up in a very easy way using Web-Ice. Follow the steps below or look at the Video Tutorial

  1. Mount the sample (either manually or using the robot. If you use the robot, select the sample in the sample information list in the Blu-Ice Screening tab and select the Stop immediately following Loop Alignment in the Action sequence widget; then click the Start button.

  2. Move the sample camera zoom to High, and adjust the sample centering if necessary. If the box defining the beam is much larger than the crystal, adjust the beam size

  3. Go back to the Blu-Ice screening tab and start Web-Ice - or go to the URL
    http://smb.slac.stanford.edu/webice/ and log in using you Unix user ID and password.

  4. Once in Web-Ice select the Autoindex Tab.

  5. Select New run from the gray toolbar menu in the Autoindex tab. Enter a unique run name (the software will ask you to try again if the run name has already been used under your user name)

    Figure 36: How to generate a new run in Web-Ice.
    Image datacol-make-newrun

  6. Select the option to collect 2 images and autoindex. If required, by the software, select the beamline from the drop down menu on the right of the gray navigation toolbar.

    Figure 37: Selecting a beamline in Web-Ice.
    Image datacol-beamline-select

  7. The program will take you through all the steps to set up the test image collection and (for MAD and SAD) fluorescence scan. Some tips:

    • Use the screening results to set up the optimal parameters. If you want to increase the resolution , remember to increase the exposure time too; however, it is important you do not overload spots, this can cause a problem with the exposure time estimate.

    • If you know that your crystals consistently index in a higher symmetry than the correct one (e.g., a monoclinic crystal with the $\beta$ angle close to 90 degrees), supply the correct Laue group and cell.

  8. Once you have started the autoindex/strategy run, you can monitor progress by looking at the Beamline log and Autoindex Log in the Log page in Web-Ice. The Setup page also prints messages as the results become available. Inspect the autoindexing results in the Autoindex Summary page, image integration results in Solutions page, and (very important!) look at the images in the Predictions page. For MAD and optimized SAD experiments, examine the Scan. If everything looks fine, look at the Strategy page.

    Important: Never collect data without inspecting the test diffraction images and the predicted pattern

  9. If the crystal space group is well known, make sure that the strategy for the correct Laue symmetry is selected (remember that if the Laue symmetry is not declared when setting up the run, the strategy selected by default corresponds to the lowest symmetry for the best autoindexing solution).

  10. If the predicted resolution is lower than the target resolution, try recollecting the test images with a longer exposure time. Do this by clicking on the Recollect button; the program will display a page where you can edit the data collection parameters. Note: Increasing the exposure time per image will increase the radiation dose by the same amount. E.g., if the estimated dose is 1.5e7 Gy, you will reach the limit by doubling the exposure time. Always verify that the dose limit is not exceeded unless you have additional good quality crystals.

  11. If the predicted resolution is higher than the target, recollect the test images by clicking on the Recollect button. This will not only test if the diffraction limit estimated by the software is correct, but will also recalculate the correct oscillation angle per image and optimal exposure time, both of which are dependent on the resolution. Use the initial exposure time (used to collect the first test images) and the new sample to detector distance recommended by the software.

    Figure 38: Results menu for a Selected Run in the Web-Ice autoindex tab.
    Image datacol-inspect-results

  12. Once you have obtained a satisfactory strategy (see the notes below), start data collection from by clicking on the Collect button. This will simultaneously export the data collection to Blu-Ice AND start the collection. If you wish to make additional adjustments (e.g. enabling dose mode), you can Export the strategy to Blu-Ice. Like the Collect button, this will create a run in Blu-Ice, but will not initiate collection. This will provide you the chance to edit the experimental parameters in Blu-Ice; once this is done, start the data collection as described in the Blu-Ice documentation

  13. Pausing and interrupting the data collection can be done at any time from Blu-Ice.

Notes on strategy for monochromatic and high resolution experiments

For monochromatic experiments the priority is to maximize unique completeness. If the estimated dose for the experiment is low enough, consider collecting additional data beyond the starting or ending phi to increase data redundancy.

Completeness for the low resolution shells is important, so decrease the exposure time or use additional attenuation if the strategy page displays a warning about overloaded spots in one or both of the images.

If ultra-high resolution data are required and you are on a beamline equipped with a CCD detector (e.g. BL14-1) it may not be possible to collect to the resolution limit without overloading the low resolution reflections. In this case, collect an additional low-resolution pass:

  • Run Web-Ice to calculate the strategy for the high resolution pass. Once this is done, export the strategy to Blu-Ice. This will create a data collection run in Blu-Ice but will not initiate data collection.

  • Examine the high resolution test images to determine the resolution $d$ at which no overloaded reflections are present.

  • Run Web-Ice again to determine the strategy for the low resolution pass. The target resolution for this pass should be 1/2 Å above the resolution $d$ mentioned in the above step. This will allow the data sets to be scaled together. Export the low pass strategy to Blu-Ice.

  • Start data collection from Blu-Ice. To ensure proper measurement of the critical low resolution reflections, it is recommended to collect the low resolution pass first (if the high resolution data are collected first and the crystal degrades, proper scaling of the two passes may be difficult).

A low pass may not be necessary when collecting thin phi sliced data on the high dynamic range Pilatus detector, but check the reflections at low resolution.

Notes on strategy for MAD and SAD experiments

For MAD and SAD experiments is it very important to limit the dose received by the crystal during the experiment. The Web-Ice strategy already incorporates some mitigation procedures (e.g., data collection in wedges, use of two wavelengths for MAD experiments). If the Web-Ice strategy still results in a dose exceeding or at the limit given by the software, consider decreasing the exposure time. This can be done manually either after exporting the data collection parameters to Blu-Ice, or in the edit window Web-Ice displays prior to initiating data collection. The dose is proportional to exposure time, so reducing the exposure time by half will reduce the dose by half at the only expense of a slightly lower data resolution.

For certain space groups and crystal orientations, it is also possible to reduce the absorbed dose by selecting the phi range to maximize unique data set completeness (maximizing Bijvoet pair completeness is the default strategy for MAD and SAD). This usually works for MAD data with a medium to strong anomalous signal.

In unfavorable cases (very small, weakly diffracting crystals) it may be impossible to collect a data set without inflicting serious radiation damage to the crystal. In this case, several crystal will most likely be required for successful structure solution.

Low signal experiments

If the expected anomalous signal is very low (less that 1 % of the average reflection intensity), collection of additional redundancy may be required for structure solution. This can be done by manually adjusting the ending phi before proceeding with data collection in Blu-Ice or Web-Ice. As mentioned above, the exposure time may also require adjusting to avoid excessive irradiation of the crystal.

Multicrystal data collection strategy

On the most intense beamlines, particularly BL12-2, it is possible to collect data from small crystals of dimensions less than 20 microns. However, because diffracted intensity decreases with the crystal volume faster than the deposited dose, use of such small crystals often prevents being able to collect a full data set from a single crystal before radiation damage severely affects the data. This is also often the case for data collection at above cryo and room temperatures, were often crystals last about a factor of 100 less than at cryo. In this cases, it is necessary to stitch a data set from data collected from different crystals. For high symmetry space groups it is often possible to obtain good completeness by starting data collection on a random orientation for each crystal. For low symmetry or scarce samples, it is useful to determine a data collection strategy for each crystal that maximizes the total completeness. This can be achieved with the collection strategy tool available in Web-ice.

Some tips for multicrystal experiments:

  • It is a good idea to estimate the dose before starting data collection so that you have an idea of how long the crystal will last. Collecting a low dose low resolution data set from a single crystal may also give you some idea of the total lifetime of the crystals in the beam and help you plan the experiment.

  • Do not overexpose the crystal: When aiming for high resolution, it is tempting to use very long exposure times to measure very faint spots at 2 /AA/ in test shots...only to find that not only have those spots vanished after collecting the first image, but you cannot even merge those single shots together. In general, slightly lower resolution data are better than no data.

  • While collecting data, monitor the quality of incoming diffraction images using the Collect Tab strip chart. This will tell you when your crystal begins to deteriorate, so that you can determine when to stop data collection or which images to exclude from processing.

  • To maximize the lifetime of the crystal, always expose as much volume as possible: Fully bathing large crystals in the beam at room temperature will result in a longer lifetime (or give higher resolution data for the same total exposure time) than making the beam small and shooting several data sets on different parts of the crystal.

  • It is important to process the data on the fly. If there is a large variation of the unit cell or other lack of isomorphism between different crystals, you may need to collect data from more samples than anticipated to achieve a quality data set. A program such as ``pointless'' or ``sortmtz'' can be used to combine the reflections in a single file previous to scaling. When using autoxds, use the name_xds.mtz files as input to obtain a single file. Note that pointless can also be used to ensure that the indexing for the different crystals is consistent in the space groups where this can be an issue. The program xia2 has a very effective algorithm for processing and merging multiple datasets; xia2.multiplex can also be used to refine the combined dataset and to detect non-isomorphism.

  • Autoindexing only a few degrees of data may not always be reliable. It is useful to provide the indexing software with the symmetry and unit cell if known. When using the autoxds script or xia2 this can be done from the command line.

  • Similarly, pointless can misidentify the space group in an incomplete data set, as there may not be enough reflections to fully characterize all symmetry elements presents in the crystal. If this becomes a problem, the option ``-c'' will force the program to use the input space group.


Data processing

Data processing environment

The following directories are automatically created the first time you log in to a SSRL px computer (these directories are accessible from all computers):

  • /data/username This directory is used for storing images. We recommend that you also use this directory for data processing.

    We also recommend creating subdirectories in your data directory for each data set you collect.

  • /data/username/templates This directory is a symbolic link to the directory containing the SSRL-specific data processing input and shell files. You may click here to view or download these files from the web.

Use the remote data processing servers (pxproc01 - pxproc24) to process the data; from the beamline workstations, use the SSRL menu option of the Xfce panel in the Linux beamline computers and the remote Unix desktop. You can also right-click on the Linux Xfce desktop and select Data Processing from the desktop menu. Clicking on Select Least Loaded displays the load. Avoid using a computer is the load is close to the total number of CPUs (displayed next to the computer name).

Note that, since 2017, Web-Ice attempts automated data processing after a data set has been collected. The purpose of this tool is to provide quick feedback of the data quality. Please consult the Web-Ice data processing help.

For more information about the beamline machines, consult the web document
http://smb.slac.stanford.edu/facilities/computing/

Figure 39: Logging to the data processing servers from the Linux Xfce panel.
Image toolbar

Figure 40: System load window
Image least-load

Data processing documentation

Commonly used software packages for data processing are available at the macromolecular crystallography beamlines. If you are unfamiliar with a particular application, consult the relevant documentation:

For a complete list of supported and unsupported software installed in the SSRL computers, see
http://smb.slac.stanford.edu/facilities/software/xtal_software