A Quick XDS Tutorial for SSRL

Table of Contents:

• Introduction
• Using the SSRL "autoxds" script
• Using the detector-specific .INP files
• Indexing and Integration
• Processing data with kappa offset
• Scaling
• Output Conversion
• Output for the SSRL "solve_structure" script
• Image View
• Output Files
• Useful Tips
• Useful Links

Back to Software Facilities Page

Introduction

X-ray Detector Software (XDS) processes single-crystal diffraction data. This site provides details necessary to run XDS at SSRL. Detailed program documentation is available online

XDS package can process diffraction images collected with various types of detectors. There are two ways to use the program package at SSRL, either (A) using the "autoxds" script developed here at SSRL or (B) copying and editing specific .INP as outlined below.

Using the "autoxds" script

The following steps are required to run XDS using the script.

1. You do not necessarily need to create a subdirectory if you would like to have the XDS output collected in one directory - the "autoxds" script will do this for you. You can either move to the directory where you have collected your images or a different directory of your choosing.
2. Run autoxds. You must supply the first image in the data set as input - e.g. autoxds thau_c3_1_1_001.img. The script will extract all the required information to run XDS from the image header and generate and run all the input scripts for xds, xscale and xdsconv. Note that the input script can be edited and run "manually" as explained in the sections below.
3. By default, the script will attempt to autoindex, integrate and scale all the images sharing the same root name as the input file. If you want to restrict the number of processed images, supply the number of the last image. For instance, to process images 1 to 90 in the previous example, type: autoxds thau_c3_1_1_001.img -last 90.
4. Type autoxds -help to obtain a list of other options.
5. If indexing and integration runs smoothly, you should end up with an output data file in .mtz format containing the reflection amplitudes along with all the log files (with extension .LP) for inspection of the processing and scaling results.
6. If you wish to obtain an output file in a different format, edit the XDSCONV.INP file in the results directory. Edit the line "OUTPUT_FILE= XDS_CONV.HKL CCP4_F". Replace CCP4_F with one of the other options (CCP4_I, CNS or SHELX). Save the file, and type xdsconv in the same directory.

Occasionally the xds indexing fails. The script will print the following output:

In most cases, XDS has probably found a correct autoindexing solution. You can determine this by looking at the end of file "IDXREF.LP" to get a listing of the possible indexing solutions. An example of the listing in the IDXREF.LP file is given below.

The asterisk on the left side indicates the possible choices of the unit cell and a "QUALITY OF FIT" is given for each possibility. The correct cell is typically the last of the asterisked solutions. If the solution appears correct, follow the following steps:

1. Edit the XDS.INP generated by autoxds. In our example, this file will be in the directory thau_c3_1_1_xds.
2. Look for the line "JOB= XYCORR INIT COLSPOT IDXREF" and replace it with "JOB= DEFPIX INTEGRATE CORRECT". Save the file with the same name
3. Run the integration manually, by typing xds in the same directory as the XDS.INP file. When the job is finished, type xscale and xdsconv.

If the autoindexing appears to be incorrect, you may need to use more images. Refer to the troubleshooting section below.

Using the detector-specific .INP files

SSRL provides the detector specific input files and the user needs to make very few changes. Since the indexing and integrating process is slow, it may be better to start the XDS job after the data collection. We have seen advantages in using XDS for the processing of poor quality diffraction images.

We need to run three programs xds, xscale, and xdsconv to get the final reflection file to the user specified format. xds performs autoindexing and integration, xscale is for merging and scaling, and xdsconv for converting the reflection files to the user specified formats.

The XDS package also provides a program, VIEW, for visualizing diffraction images as well as control images produced by the "XDS" package.

Indexing and Integration

The XDS program can be started from any of the data processing machines by typing xds (Program name "xds" is case sensitive).

Following steps are required to run "xds". The "Files" section of the online documentaion provides details of all the input and output files.

1. Create a subdirectory if you would like to have the XDS output collected in one directory. Move to the directory where you would like to run XDS.
2. Make a symbolic link to the diffraction data image directory using "ln -s path_to_image_directory images" command. Or you could just store the image files in an "images" directory under the XDS processing directory.
3. Copy the detector specific input files (eg., XDS_Q315.INP) from the template directory (/data/user_login_name/templates/xds). Rename this file as XDS.INP (Note: This is case sensitive and please use upper case for the name).
4. Go to the "User Input" section of the XDS input file and input values for your specific experiments given below. Please note that these values are at different locations in the XDS input file provided with the XDS package. The "keywords" have to be in upper case.
• Beam Center represented by ORGX and ORGY keywords. Unlike XDS and mosflm programs, XDS requires the beam center values in pixels. To get the pixel value divide the current beam center values (mm) by the "PIXEL_SIZE" from the image header. Since SSRL detectors are always centered, users need to change it only during a data collection with a detector offset.
• DETECTOR_DISTANCE (mm).
• OSCILLATION_RANGE or delta_phi (degrees).
• X-RAY_WAVELENGTH (Angstrom)

Values for the above three parameters are in the image header.

• SPACE_GROUP_NUMBER (xds uses only space group numbers. Change it to "0" for auto indexing.
• UNIT_CELL_CONSTANTS (if the SPACE_GROUP_NUMBER is known).
• NAME_TEMPLATE_OF_DATA_FRAMES (eg., ./images/myoWI3_1_???.img. The ADSC CCD-detector image data format is SMV). XDS package can't read lines longer than 80 characters. Therefore, use link statement (ln -s) to specify long path names.
• DATA_RANGE (number of frames. eg., 1 200)
• SPOT_RANGE (This specifies the range of images for finding spots for auto indexing. Spots from multiple ranges (up to 10) could be selected by inserting additional "SPOT_RANGE" commands. The default mode uses images specified in the "DATA RANGE".)
• FRIEDEL'S_LAW=FALSE !Default is TRUE (The "FRIEDEL'S_LAW" should be "FALSE" to generate a strategy to maximize Bijvoet reflection pairs (this is important for anomalous data collection).
• JOB (where you tell the program to what to do. The various keywords are XYCORR (calculates lookup tables of spatial corrections for each detector pixel), INIT (determines three lookup tables required for the processing steps), COLSPOT (locates strong spots and saves their centroids), IDXREF (perform indexing using the parameters in XDS.INP file and the centroids of the spots), DEFPIX (detects masked regions or untrusted pixels), XPLAN (for running strategy), INTEGRATE (determines the intensity of each reflection), and CORRECT (accepts good reflections from the integrated .HKL file). The default mode runs all of the programs. However, the program running sequence can be modified by selecting only the required keywords. For example, if the reindexing program (IDXREF) fails, we could change the keywords (remove XYCORR, INIT and COLSPOT) to start the program from the reindexing stage. The log files from the individual programs are labeled with an extension LP (XYCORR.LP, IDXREF.LP, etc.).
5. Type xds to start the program. It will use the parametrs provided in the XDS.INP file. Depending on the information provided in the "JOB" section, program will run autoindexing, scaling, or complete processing.

Processing data with kappa offset

Data sets collected with a kappa offset and rotation along phi can be processed by providing the direction cosines of the rotation axis. The direction cosines depends on the omega angle. All SSRL beamlines except beamline 7-1 use an omega angle of 270 degree. The omega angle for beamline 7-1 is 0 degree. Pete Dunten derived the following expression for the direction cosines for omega 0 and 270 degree.

Direction cosines of the rotation axis for omega 0 degree.

Direction cosines of the rotation axis for omega 270 degree.

The new direction cosines can be incorporated into xds by editing the "XDS.INP" file and replacing the three values for the "ROTATION_AXIS" with the calculated parameters. The default values for the "ROTATION_AXIS" are 1.0 0.0 0.0 (ROTATION_AXIS= 1.0 0.0 0.0).

Troubleshooting

In difficult cases, the xds indexing usually comes with an error "!!! ERROR !!! INSUFFICIENT PERCENTAGE (< 70%) OF INDEXED REFLECTIONS " (look at the end of file "IDXREF.LP"). Following things could be tried.

Scaling

The XSCALE.INP file contains parameters that control the action of xscale. Type xscale at the prompt to start the program. The "template" directory has XSCALE_MAD.INP and XSCALE.INP files for MAD data and monochromatic data, respectively. Knowledge of the following terms will be of help. The detailed information about each parameter is available at the xscale parameters site. (Note: You can also run "xscale" using the xds program by changing the JOB definition to CORRECT in the "XDS.INP" file. The output logfile will be automatically directed to the CORRECT.LP file.)

The integration performed in the default triclinic space group can be easily changed to the correct space group using correct cell from indexing output file "IDXREF.LP". You no longer need to give the transformation matrix which in the older version of the program was given in the Bravais lattice list (see the example of the IDXREF.LP file given above). You simply need to give the SPACE_GROUP_NUMBER and the UNIT_CELL_CONSTANTS. You can comment out the "REIDX=" line in XSCALE.INP

Output Conversion

Allows conversion of the output reflection to various user specified formats (SHELX, CNS, CCP4_F and CCP4_I). Type xdsconv at the prompt to start the program. Its output is controlled by the "XDSCONV.INP" file. The detailed information about each parameter is available at the xdsconv parameters site.

If you would like to use the "xds" output reflection file with the SSRL MAD Scripts, please use the "xds_combat_ccp4.com" script in the template area. This script reads the output reflection file from "XSCALE" and generates an mtz file in the scala format.

Brief summary of the various formats.

The following formats produces reflection file suitable for running the f2mtz program to create the corresponding mtz file. They also produce F2MTZ.INP file to run the f2mtz or cad programs.

• h,k,l,F,Sigma(F),i (if FRIEDEL'S_LAW=TRUE)
• h,k,l,F,Sigma (F),DF,Sigma (DF),isym,i (if FRIEDEL'S_LAW=FALSE)

• h,k,l,IMEAN,SIGIMEAN,i (if FRIEDEL'S_LAW=TRUE)
• h,k,l,IMEAN,SIGIMEAN,I(+),SIGI(+),I(-),SIGI(-),i (if FRIEDEL'S_LAW=FALSE)

Output for the SSRL "solve_structure" script

Running the "xds2solve_structure.pl" script in the template directory generates "xds2scalepack.com" and "scala_xds.com" script files. The xds2scalepack.com script will create a reflection file (*.sca) in the scalepack "nomerge original index" format. This reflection is ready for the "solve_structure_denzo.com" script. The "scala_xds.com" script generates log files similar to the scala log files.

Commands to run the scripts

perl xds2solve_structure.pl < xscale_out.ahkl

xscale_out.ahkl is the reflection file generated by "xscale" program. Please set the "MERGE=FALSE" flag while generate the xscale reflection file.

csh < xds2scalepack.com > xds2scalepack.out

Note: The generated scalepack file may show "***" in the intensity column if you have very large intensity for that reflection. If this happens, reduce the value of "scale" in the "xds2scalepack.com" file and rerun this script.

Image View

The "VIEW" command is for displaying diffraction images or images produced by the the XDS package (Note: This command is case sensitive and please use upper case. Lower case "view" refers to the /bin/view program.). To display an image, type VIEW imagename (VIEW xxx_001.img or VIEW BKGINIT.pck). Type "VIEW -h" to dsplay the available options and "VIEW -hv | more" to see a short manual. The alpha unix version of the program displays only the top left quadrant of the image. However, moving the mouse around will show the other parts of the image.

Display of diffraction images will require inputting few detector specific parameters given below. Their values can be obatined from the image header.

1. Number of points in x (fast index) and Number of points in y (slow index). These are the "SIZE1" and "SIZE2" values in the image header.
2. Number of bytes per value. This is the "DIM" value in the header.
3. Offset (header bytes). This corresponds to the "HEADER_BYTES".
4. Big- or Little-endian format. This is the "BYTE_ORDER".

These values for a typical Q315 image are:

1. "Number of points in x (fast index)" and "Number of points in y (slow index)". 3072 and 3072.
2. "Number of bytes per value". 2.
3. "Offset (header bytes)". 512.
4. "Big- or Little-endian format". b (for big_endian).

Output Files

A short explanation of the output files generated by various programs are given below. The log files are written with the program name and a "LP" extension (XDS package generally use upper case letters for file names.). Detailed information about the output files are available online.

• XYCORR.LP. This is the output from the XYCORR program and provides few detector specific informations.
• INIT.LP. This provides information about the "Detector_gain" and "Detector_background" calculations.
• COLSPOT.LP. Details of the number of spots collected for indexing. The first few lines gives the input parameters used for picking of the spots.
• SPOT.XDS. Ascii file with the listing of the spots.
• IDXREF.LP. Output from the indexing program. The information about various Bravais lattices including the quality of each solution are given at the end of ths file.
• XPARM.XDS. The orintation matrix file.
• DEFPIX.LP. Provides information about the "trusted" and "untrusted" regions of the detector.
• XPLAN.LP. Data collection strategy information.
• INTEGRATE.LP. Output file from integration. The number of reflections written to the reflection file is given at the end of the file.
• INTEGRATE.HKL. The reflection file output. First few lines at the beginning provides details of the parameters used for the integration.
• CORRECT.LP. The scaling output file. Search for 'R-FACTOR' to see the R-factor and related information. The final statistics including R-factor, number of observations, unique reflections, etc. are given towards the end of the file (Tip: Search for XDS_ASCII.HKL and above informations come just before this). If you are scaling two or more data sets (eg. MAD), look at the "CORRELATION" between two data sets to get an idea about relative scaling.
• XDS_ASCII.HKL. The scaled reflection file. The first few lines gives details of the parameters used for scaling.
• GXPARM.XDS. The final refined cell parameters and orientation. This file is similar to the "XPARM.XDS" file. If we get a better solution from the refinement in the "CORRECT" step, we could replace the XPARM.XDS file by GXPARM.XDS file and rerun "INTEGRATE" and "CORRECT".

Useful Tips

Knowledge of the following parameters will help with fine tuning the output of the various programs in the XDS package. These parameters are in the XDS.INP files and they are listed in the order in which they appear in this input file. Unless otherwise specified, SSRL specific input file uses defalut values for most of these parameters (a "!" sign before the parameter shows that it is commented out and it will use its default value). Please go to the various sections of the input file to change their values.

• TRUSTED_REGION. Inner and outer relative radii limiting trusted detector region. For a square detector: outer radii is ~1.2 for half corner, 1.4 for full CCD, and 1.0 for detector edge.
• UNTRUSTED_RECTANGLE. To remove a rectangle area from the trusted detector plane (values needs to be in pixels).
• ROTATION_AXIS. Direction cosines on the rotation axis with respect to the laboratory system. The default value "0.0 1.0 0.0" shows rotation axis pointing to the laboratory y-axis. When looking along the axis, the crystal would rotate clockwise during the data collection.
• INCIDENT_BEAM_DIRECTION. x,y,z components of te incident beam direction with respect to the laboratory coordinate system. The dafult value (0.0 0.0 1.0) points the incident direction along the laboratory z-axis.
• FRACTION_OF_POLARIZATION. Fraction of polarization of direct beam in a plane specified by its normal. This value is 0.92 for SSRL. If omitted, the beam is assumed to be unpolarized.
• POLARIZATION_PLANE_NORMAL. x,y,z components of the polarization plane normal with respect to the laboratory coordinate system.
• AIR. Fraction of intensity loss per mm due to air absorption. Each reflection intensity is multiplied by a factor during the "CORRECT" step.
• RESOLUTION_SHELLS. Resolution shell limits in angstrom. Only the high resolution limit of each shell is given. Used by the strategy program (XPLAN) to report completeness of the various hypothetical data sets as a function of resolution.
• STARTING_ANGLES_OF_SPINDLE_ROTATION. This value consists of number triple that specifies first, last and increment of the various starting spindle angles for the crystal rotation to be searched by XPLAN for pinpointing an optimal data collection strategy.
• TOTAL_SPINDLE_ROTATION_RANGES. The parameter (a number triple that specifies minimum size, maximum size, and size increment or degrees of total rotation about the spindle) characterizes a grid of data sets of varying sizes for analysis by XPLAN.
• REFERENCE_DATA_SET. We can specify file name of previously measured data from the same crystal form. This data will be used by XPLAN (uses the old data to tell the user by what strategy a maximum of new data could be collected) and CORRECT (uses the old data for local scaling and comparison with the current data set).
• BACKGROUND_RANGE. Numbers of first and last data image for determining the initial background.
• INDEX_ORIGIN. For specifying the orgin reflection.
• INDEX_ERROR, INDEX_MAGNITUDE, and INDEX_QUALITY. Parameters used by IDXREF. The default values hardly need any changes.
• SEPMIN, CLUSTER_RADIUS, MAXIMUM_ERROR_OF_SPOT_POSITION. Related to the difference vector and location of a diffraction peak. These values hardly require any changes. Used by the IDXREF program.
• VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS. A pair of numbers that are used to define untrusted detector pixels, which may result from shading parts of the detector (eg. shadow from a cryojet). The values outside the range are treated unreliable. The default values for this parameter are 6000 and 30000.
• INCLUDE_RESOLUTION_RANGE. Resolution range to accept a reflection. The default range is "20.0 0.0". The SSRL input files have a range of 30.0 to 0.0.
• EXCLUDE_RESOLUTION_RANGE. Resolution range for excluding reflections. This allows removal of reflections from/in the ice-rings.
• WFAC1. Parameter for recognizing reflections that are incompatible with the observed intensities of symmetry related ones (hardly needs any change).
• BEAM_DIVERGENCE, BEAM_DIVERGENCE_E.S.D., REFLECTING_RANGE, REFLECTING_RANGE_E.S.D.. These values are connected with beam divergence and mosaicity. If these values are left unspecified, they will be determined automatically from the data images. (Note: Slightly larger BEAM_DIVERGENCE value will include some background pixels around each spot. BEAM_DIVERGENCE = arctan(spot diameter/DETECTOR_DISTANCE))
• NUMBER_OF_PROFILE_GRID_POINTS_ALONG_ALPHA/BETA and NUMBER_OF_PROFILE_GRID_POINTS_ALONG_GAMMA. These parameter values define the number of sampling points used for representing reflection profiles. The default value is 9 for both parameters.
• CUT. Cut-off value defining the integration region for profile fitting. The default value (2.0) implies that grid points in the reflection profile less than 2% of the maximum are not used for integration.
• MINPK. Defines the minimum required percentage of observed reflection intensity. If less than MINPK % is observed, the reflection will be discarded. The default value of 75.0 and hardly needs to be changed.
• DELPHI. This parameter allows control of the profile and scaling factors. The number of profiles is approximately equal to 9 * Total rotation range covered by data set/DELPHI. The default value is 5.0 and if there are too few strong reflections, it may be useful to try larger value for DELPHI.
• PATCH_SHUTTER_PROBLEM. If there are deviations of the observed distribution of the mean intensity from the expected uniform distribution, this could indicate shutter problems. The "PATCH_SHUTTER_PROBLEM" parameter provides a way to patch this hardware problem by an appropriate correction factor applied to the reflection intensities.
• STRICT_ABSORPTION_CORRECTION. This parameter controls the calculation of the absorption correction factors in the CORRECT step of XDS. The default value is "TRUE". If STRICT_ABSORPTION_CORRECTION=FALSE, Friedel-pairs are treated as symmetry-equivalent reflections in the calculation of the absorption correction factors. In the presence of anomalous scattering effects this could lead to an underestimate of the anomalous differences.
• STRONG_PIXEL. A 'strong' pixel to be included in a spot must exceed the background by more than the given multiple of standard deviations. The default value is 3.0.
• MAXIMUM_NUMBER_OF_STRONG_PIXELS. This value provides an approximate upper limit for the total number of 'strong' pixels in all of the images scanned by COLSPOT. If this number exceeds, COLSPOT will automatically raise the threshold used for classifying a 'strong' pixel. Default value is 1500000.
• SPOT_MAXIMUM-CENTROID. For eliminating spots whose location of the maximum deviates by more than the specified parameter value from the centroid of the spot (pixel units). The default value is 2.0.
• MINIMUM_NUMBER_OF_PIXELS_IN_A_SPOT. To supress spurious, isolated 'strong' pixels (eg. zingers) from entering the spot list generated by "COLSPOT". The default value is 6.
• NBX and NBY. A box of size (2*NBX+1)*(2*NBY+1) is centered in succession at each pixel of the images specified for background determination and the pixel variation within the box is determined. The results are used to estimate the expected variation in a data image in the absence of any spot and saved in the look-up table GAIN.pck. The default value both parameter is 3.
• BACKGROUND_PIXEL. An image pixel belongs to the background region if the variation in the pixel contents of neighboring pixels (region defined by NBX= and NBY=) does not exceed the specified number of standard deviations. The default value is 6.0. This implies that a pixel at IX, IY is in the background region, if the variation in contents of pixels within the region IX-NBX,IX+NBX; IY-NBY,IY+NBY does not exceed the expected variation by more than a factor of 6.
• SIGNAL_PIXEL. The pixel contents must exceed the background by more than the specified value of standard deviations to be included in the calculation of the Bragg-peak centroid. Default is 3.0.
• REFINE(IDXREF). Specifies the parameters to be refined during indexing (IDXREF program). If the default value "ALL" is specified or the parameter is commented out, detector distance, beam direction, rotation axis, unit cell orientation, and cell constants will be refined in the IDXREF step of XDS. A value of "BEAM AXIS ORIENTATION CELL" refines incident beam direction, rotation axis, unit cell orientation and cell constants (but not the detector distance) during the indexing step.
• REFINE(INTEGRATE). Specifies the parameters to be refined during integration (INTEGRATE program). The default parameter value is "DISTANCE BEAM ORIENTATION CELL".
• REFINE(CORRECT). Specifies the parameters to be refined during scaling (CORRECT program). The default value is "ALL" and it refines detector distance, beam direction, rotation axis, unit cell orientation, and cell constants.

Useful Links

• The XDS wiki page "XDSwiki" provides explanations and hints for the successful use of XDS.