The limit of detection ($LOD$) for an analytical method can be derived from its calibration curve using the following equation:

$$ LOD = {\dfrac{(3.3 \cdot {\sigma_ \mathrm{resids}})} {m}} \tag{1} $$ [see reference at bottom]


  • $ \sigma_\mathrm{resids} $ is the standard deviation of the residuals of the regression fit
  • $m$ is the slope of the regression curve.

Would the above equation still apply if I were to apply it to a calibration curve generated through linear regression of $\ln(y)$ on $\ln(x)$?

Here's an example data set that I generated (artifically - hence apparent rounding errors) to reflect what would be seen in some electrospray ionisation-mass spectrometry data (non-linear and heteroscedastic):

$$\begin{array}{|c|c|c|c|} \hline \text{Calibrant} & x \ (\pu{mg/L}) & y \text{ (peak area, AU)} & \ln(x) & \ln(y) \\ \hline \text{S1} & 1 & 550 & 0 & 6.309918 \\ \hline \text{S2} & 1 & 500 & 0 & 6.214608 \\ \hline \text{S3} & 1 & 450 & 0 & 6.109248 \\ \hline \text{S4} & 2 & 1100 & 0.693147 & 7.003065 \\ \hline \text{S5} & 2 & 1200 & 0.693147 & 7.090077 \\ \hline \text{S6} & 2 & 1300 & 0.693147 & 7.17012 \\ \hline \text{S7} & 10 & 6500 & 2.302585 & 8.779557 \\ \hline \text{S8} & 10 & 7000 & 2.302585 & 8.853665 \\ \hline \text{S9} & 10 & 7500 & 2.302585 & 8.922658 \\ \hline \text{S10} & 50 & 40000 & 3.912023 & 10.59663 \\ \hline \text{S11} & 50 & 45000 & 3.912023 & 10.71442 \\ \hline \text{S12} & 50 & 50000 & 3.912023 & 10.81978 \\ \hline \text{S13} & 125 & 140000 & 4.828314 & 11.8494 \\ \hline \text{S14} & 125 & 150000 & 4.828314 & 11.91839 \\ \hline \text{S15} & 125 & 160000 & 4.828314 & 11.98293 \\ \hline \text{S16} & 250 & 400000 & 5.521461 & 12.89922 \\ \hline \text{S17} & 250 & 420000 & 5.521461 & 12.94801 \\ \hline \text{S18} & 250 & 440000 & 5.521461 & 12.99453 \\ \hline \text{S19} & 500 & 1000000 & 6.214608 & 13.81551 \\ \hline \text{S20} & 500 & 1200000 & 6.214608 & 13.99783 \\ \hline \text{S21} & 500 & 1400000 & 6.214608 & 14.15198 \\ \hline \end{array}$$

The calibration curve for compound A

Applying equation (1) and exponentiating the result, the $LOD$ would come out as:

$$ LOD = \exp {\left(\dfrac{3.3 \times 0.177}{1.2265} \right) } = \pu{1.61 mg/L} $$

The strange thing for me is that if I multiply the original concentrations ($x$) by 1000, say to convert to from $\pu{mg/L}$ to $\pu{\mu g/L}$,and then generate the calibration curve of $\ln(y)$ on $\ln(x)$, the $LOD$ comes out at exactly the same value, i.e. 1.61. This is because the slope and the standard deviation of the regression fit are unchanged by multiplying the concentration by 1000 - only the intercept value changes. Note, given that concentration axis was in $\pu{\mu g/L}$ before $\ln$-transformation, this would presumably mean an $LOD$ of $\pu{1.61 \mu g/L}$?

My suspicion is that I've missed a crucial step in calculating $LOD$ from the $\ln$-transformed calibration curve model.

Thanks for any pointers you can provide!

Reference: https://www.ema.europa.eu/en/documents/scientific-guideline/ich-q-2-r1-validation-analytical-procedures-text-methodology-step-5_en.pdf

  • 1
    $\begingroup$ Very interesting question. I do not think the LOD equation is valid anymore after log-log transformation. The other issue by multiplication by 1000 seems like a coincidence. What instrument are you using? $\endgroup$ – M. Farooq Oct 11 '20 at 21:20
  • $\begingroup$ Hi M. Farooq, I'm using an LC-MS instrument that has a quadratic-type response and shows heteroscedasticity in the residuals of the regression when plotted in 'raw' form. By multiplying my concentrations by 1000, before ln-transformation, I see no change in the gradient of the slope. If I did, this approach would mostly likely work okay. $\endgroup$ – MRJ Oct 12 '20 at 7:04
  • $\begingroup$ Your data at low values of x is fairly linear with y , so you could find a (probably quite good) approximation to your limit from just this data, or better still, if it is possible, would be to add more data points in this concentration region and fit to a normal linear plot. $\endgroup$ – porphyrin Oct 12 '20 at 9:32
  • $\begingroup$ @MRJ, What type of LCMS is this? ESI-MS, Why do you think it is fundamentally quadratic? $\endgroup$ – M. Farooq Oct 12 '20 at 11:51
  • 1
    $\begingroup$ The ln() transformation functionally changes the weighting of the error at each point. In the linear regression, you assume that the magnitudes of the errors (ie the residuals) are independent of the true value of y. That is, a deviation of 0.1 from a true value of 0.1 is the same as a deviation of 0.1 from a true value of 1. With the ln() function, this is no longer true, since an 0.1 deviation from 0.1 results in a larger deviation of the ln function than an 0.1 deviation from 1. In some cases, this might in fact be a more accurate representation of the error of your machine. $\endgroup$ – Andrew Oct 12 '20 at 13:50

I can add some points for the OP which are too long to serve as comments. I have talked to a couple of leading persons who work with a lot a of LODs. The first lesson, which is important, that there is no "the limit of a detection" of an analytical procedure. Second, some analytical chemists also state that it should be abolished because of the same reason- there is no the limit of detection. Limit of detection varies from method to method of calculation. You will find a couple of common things in legally accepted methods for LODs as the German DIN-32645, and ISO-BS 11843-2:2000 (Incorporating corrigendum October 2007). They all deal with linear calibration curves and no one want to mess around with quadratic or log transformed calibration curves. In the literature, in my opinion, LOD is the most incorrectly reported number and the experts agree with this opinion in personal conversations.

Now quadratic calibration curves are completely kosher. They are being used in atomic absorption spectrophotometers for ages (~ 40 years). Similarly, another HPLC detector based on light scattering does a small manipulation that makes the calibration curve non-linear. These detectors are used in pharmaceutical industry all the time. So, if your sole purpose is to do quantification, within 1 to 500 mg/L, this calibration curve should serve you well but do not calculate the LODs using the formula 3.3xsigma/slope. The problem is the determination of sigma, and the multiplying factor. When Kaiser developed these formulae (mostly in German papers), he never bothered to say anything about nonlinearity because nobody wanted to work with non-linear calibration curves before the computers.

The log transformation as suggested in the comments changes the noise behavior. If you really wish to determine a number called LOD for this instrument you can do the following (for the satisfaction of a novice reviewer of your paper or another higher up who wants a LOD number from you).

P.S. Do not round off concentrations and areas, the instrument readings look simulated to me because they are only changing in units of 50 and 100. Round off is a bad idea, it should only be done at the end.

a) Check the linearity range of your method. I plotted the data up to 50 mg/L. It is linear.

b) Obtain the readings in the linear range in triplicate or quadruplicate. You already have this data.

c) Look at the standard deviation of peak area vs. concentration curve. As you can see it is not a flat line (slope of 0). Hence the Kaiser's formula fails right here. You cannot apply 3.3*sigma/slope formula. The noise is heteroscedastic. Kaiser's formula is valid for homoscedastic noise.

Suggestions to obtain an LOD of your method.

d) Make about 7-8 standards in the range 0.1 mg/L to 20 mg/L. Repeat the process a to c. What we need is a standard near the estimated LOD.

e) If you still see a non-zero slope for the peak area standard deviation vs. concentration, apply a weighted least squares method. ISO-BS 11843-2:2000 has all the solved examples.

Calibration curve

  • $\begingroup$ Your answer is very interesting. But it makes me wonder if you understand the meaning of a LOD, or if you just question its utility? To me the meaning is simple. Too small a signal (here <3.3 sigmas) can't be distinguished from random noise. Assuming a linear fit is appropriate and noise is uniform (no heterosc), do you see an inherent problem with application of the equation in the OP? To me it seems simple and intuitive. It seems to me the biggest problem the OP is facing is the nonlinear nature of his problem, but for application under ideal conditions the LOD computation seems just fine. $\endgroup$ – Buck Thorn Oct 12 '20 at 18:48
  • $\begingroup$ So determining LOD is simple: determine a function for the noise, establish if it depends on the concentration of sample, and determine a function for the signal. $\endgroup$ – Buck Thorn Oct 12 '20 at 18:49
  • 1
    $\begingroup$ Buck Thorn, Well, I wish it were that easy. You will have to talk to experts to fully understand the intricacies of correct LOD and the philosophy behind it. I have had detailed discussions with people who have written an entire book on the limit of detection or written at least 20-30 articles on this topic. The deeper you are exposed...the lesser you feel to know. I feel I now know very little about LODs. However, I now refuse the accept the typical undergraduate spam taught in standard analytical textbooks. $\endgroup$ – M. Farooq Oct 12 '20 at 18:54
  • 1
    $\begingroup$ The problem is the behavior of noise at very low concentrations. Second question is how do you define noise? Is it the noise in the baseline (the way chromatographers accept it) or is it the noise in the signal (the way spectroscopists define it). $\endgroup$ – M. Farooq Oct 12 '20 at 18:56
  • 2
    $\begingroup$ LOD is one of the most hotly debated topic in analytical chemistry. The debate will never end. I will quote a sentence from a speaker in a conference, that analytical chemists do not know enough statistics and statisticians do not know enough analytical chemistry. My contacts who are cited couple of times in your linked pdf agree with that the way undergraduate LOD concept is taught is garbage. $\endgroup$ – M. Farooq Oct 12 '20 at 19:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.