Background: I am trying to get an overview of the data-processing pipeline for solving protein structures by x-ray crystallography, and in a very simple way understand what mathematical procedures are conducted at what step of the processing pipeline. I find data-processing a bit confusing as both textbooks and information on the internet seem to either give very basic explanations or way to technical (i.e. mathematical) explanation (and crystallography also seems to be full of confusing jargon terminology). Or alternatively, the information is focused on “clicking” you way through technical programs to produce an electron density map (which is great!), but without really explaining what the main steps of data processing consists of i.e. why you are doing them.

I would appreciate some clarification/confirmation that I have understood the overall process correctly. I can "click my way through" programs such as XDS/ Aimless/ Pointless/ Ctruncate/ MolRep/ Buccaneer and then finally work in coot for model in the electron density map. However, I don’t really understand the steps (in the data processing), a lot of it is simply just "click this" then "click that".

Specification of question: Everything up to the point of getting a protein diffraction pattern is okay, and everything after the electron density map is generated - is also okay. It is specifically the part between diffraction pattern and electron density map that is a bit confusing (also in terms of terminally used to explain the processing).

My perhaps error-prone understanding so far:

  1. The diffraction pattern (collected at 360 degrees) of a protein crystal represents the reciprocal lattice.

The reciprocal lattice is an abstract concept for mathematical calculations, and does not represent the real crystal which exists in what is referred to as “direct lattice”, “crystal lattice” or “real space” (they are all the same thing?). The reciprocal lattice exists in reciprocal space (which is also referred to as k-space in physics?).

  1. We can index the spots of the reciprocal lattice to generate miller indices(hkl) for each spot.

By indexing the intensities of the spots on the reciprocal lattice, we can determine the amplitude of the waves that diffracted from the crystal, but they do not contain any information about the phases of the wave relative to each other. This is the part where braggs law is used (i.e. a part of indexing?)

  1. We then scale and merge reflections

Merging is grouping together identical reflections which increases the multiplicity of the data set (i.e. a good thing for statistics). Scaling is a bit abstract, and I know it is done in programs such as XDS during indexing (or right after indexing) and it somehow places all the reflections on a common scale.

  1. We need to determine the structure factors (F) that contain information of the phases (or is this done at an earlier step, or at several steps of the process?)

This has to be done my molecular replacement or other methods (i.e. to solve the phase problem)

Assuming we use molecular replacement to solve the phase problem, are we then using the Patterson method of solving the structure?

  1. After determining the phases, we can generate an electron density map is in “direct space/real space” by performing an inverse Fourier transform from the frequency domain to the time domain (f(t))? (or is it opposite, i.e. “normal” Fourier transform?)
  • $\begingroup$ There is software to do all the analysis you mention, look for CCP4 and / or Coot and associated methods used. The first step is to estimate the spot intensity on the detector; not a trivial problem as it happens as spots can have incomplete shapes. Once this is done then FFT methods are used to work out what the crystal alignment was and so find space group and hkl. $\endgroup$
    – porphyrin
    Jan 31, 2020 at 13:49
  • 2
    $\begingroup$ Thanks, yes I know how to use coot and also how to generally go through XDS/Aimless/Pointless/Ctruncate/MolRep/Bucaneer and then coot. However, I dont really understand what I am doing and why, a lot of is is just "click this" then "click that". This is why I asked the question above. I will add this statement to my question. $\endgroup$ Jan 31, 2020 at 13:53
  • $\begingroup$ @CuriousTree, Glad to see that someone is actually thinking rather than clicking buttons. It will be better if you talk to someone in person who is an expert in Xray domain (I mean not the users but actually people who know the practical aspects +theory). $\endgroup$
    – AChem
    Jan 31, 2020 at 15:33
  • $\begingroup$ Have a look at 'Crystallography made Crystal Clear' by G. Rhodes publ. Academic Press which covers most of what you are asking as it specifically covers macromolecular models. (btw, if you work things through note that eqns 5.21/22 seem to have have typos) $\endgroup$
    – porphyrin
    Feb 1, 2020 at 11:26
  • $\begingroup$ Thanks! I think I have seen that book around. Ill have a look. $\endgroup$ Feb 1, 2020 at 13:44