Experimental error in physics

Experimental Error in Physics,
A Few Brief Remarks…
[What Every Physicist SHOULD Know]
L. Pinsky

July 2004

© 2004 L. Pinsky

1


Outline of This Talk…
Overview
Systematic and Statistical Errors
Kinds of Statistics
The Interval Distribution
Drawing Conclusions

July 2004

© 2004 L. Pinsky

2


What your should take away
In Science, it is NOT the value you
measure, BUT how well you know that
value that really counts…
Appreciation of the accuracy of the

information is what distinguishes REAL
Science from the rest of human
speculation about nature…
…And, remember, NOT all measurements
are statistical, BUT all observations have a
some sort of associated confidence level…
July 2004

© 2004 L. Pinsky

3


Almighty Chance…
Repeatability is the cornerstone of Science!
…BUT, No observation or measurement is truly
repeatable!
The challenge is to understand the differences
between successive measurements…
Some observations differ because they are
genuinely unique!


(e.g. Supernovae, Individual Human Behavior, etc.)

Some are different because of RANDOM CHANCE
Most real measurements are a combination of
BOTH…


(Even the most careful preparation cannot guarantee
identical initial conditions…)

July 2004

© 2004 L. Pinsky

4


The Experimentalist’s Goal
The Experimental Scientist seeks to
observe nature and deduce from those
observations, generalizations about the
Universe.
The generalizations are typically
compared with representations of nature
(theoretical models) to gain insight as to
how well those representations do in
mimicking nature’s behavior…
July 2004

© 2004 L. Pinsky

5


Tools of the Trade
The techniques associated with STATISTICS are
employed to focus the analysis in cases where

RANDOM CHANCE is present in the measurement.


(e.g. Measuring the individual energy levels in an atom)

Statistical analysis is generally combined with a
more global attempt to place the significance of
the observation within the broader context of
similar or related phenomena.


(e.g. Fitting the measured energy levels into a Quantum
Mechanical Theory of atomic structure…)

What we typically want to know is whether, and
to what extent the measurements support or
contradict the Theory…
July 2004

© 2004 L. Pinsky

6


Blunders
These can be either Explicit or Implicit
 Explicit—Making an overt mistake ( i.e. intending to do the
right thing, but accidentally doing something else, and not
realizing it…)
 (e.g. Using a mislabeled reagent bottle…)



Implicit—Thinking some principle is true, which is not, and
proceeding on that assumption.
 (e.g. Believing that no pathogens can survive 100 C)

Blunders can only be guarded against by vigilance, and are
NOT reflected in error bars when the data are presented…
Confidence against Explicit blunders can be enhanced by
independent repetition.
Protection against Implicit blunders can be enhanced by
carefully considering (and disclosing) the details regarding
ALL procedures and assumptions…
July 2004

© 2004 L. Pinsky

7


Systematic Error
Generally, this includes all of the KNOWN uncertainties that are
related to the nature of the observations being made.
 Instrumental Limitations (e.g. resolution or calibration)
 Human Limitations (e.g. gauge reading ability)
 Knowledge limitations (e.g. the accuracy with which needed
fundamental constants are known)
Usually, Systematic Error is quoted independently from Statistical
Error. However, like all combinations of errors, effects that are

independent of one an other can be added in “Quadrature”:
 (i.e. Etotal = [ E12 + E22]1/2 )
Increased statistics can NEVER reduce Systematic Error !
Even Non-Statistical measurements are subject to Blunders and
Systematic Error…

July 2004

© 2004 L. Pinsky

8


Quantitative v. Categorical
Statistics
Quantitative—When the measured variable takes
NUMERICAL values, so that differences and
averages between the values make sense…




Continuous—The variable is a continuous real
number… (e.g. kinematic elastic scattering angles)
Discrete—The variable can take on only discrete
“counting” values… (e.g. demand as a function of
price in Economics)

Categorical—When the variable can only have an
exclusive value (e.g. your country of residence),

and arithmetic operations have no meaning with
respect to the categories…

July 2004

© 2004 L. Pinsky

9


Getting the Right Parent
Distribution
Generally, the issue is to find the proper
PARENT DISTRIBUTION—(i.e. the
probability distribution that is actually
responsible for the data…)
In most cases the PARENT DISTRIBUTION
is complex and unknown…
…BUT, in most cases it may be reasonably
approximated by one of the well known
distribution functions…
July 2004

© 2004 L. Pinsky

10


Deviation, Variance
and Standard Deviation

The Mean Square Deviation refers to the actual data:
 s2 = Σ (x – m)2/(N-1), and is an experimental statement of fact!
i
Standard Deviation (σ) and the Variance (σ2) refer to the PARENT
DISTRIBUTION:
σ2 = Lim[Σ (χi – µ)2/N], and is a mathematically useful concept!
One can calculate “Confidence Limits” (the fraction of the time the
truth is within) from Standard Deviations, not Deviations!
This is because you can integrate the PARENT DISTRIBUTION
to determine the fraction inside ±nσ!
The Mean Square Deviation is sometimes used as an estimate of
the Variance when one assumes a particular PARENT
DISTRIBUTION!
 …BUT, the resulting Confidence Limit is only as good as the
assumption about the PARENT DISTRIBUTION!
 The wrong DISTRIBUTION gives you a false Confidence Limit!
July 2004

© 2004 L. Pinsky

11


Categorical Distributions
The CATEGORIES must be EXCLUSIVE!
 (i.e. Being a member of one CATEGORY precludes being a
member of any other within the distribution…)
Sometimes there are “Explanatory” and “Response” variables,
where the “Response” variable is Quantative, and the
“Explanatory” Variable is Categorical.

 (e.g. Annual Income is the [Quantative] “Response” variable
and Educational Degree [i.e. high school, B.S., M.S., Ph.D.] is
the [Categorical] “Explanatory” variable).
 The “Response” variable is called the Dependent variable, with
the “Explanatory” variable being the Independent variable…
 We can use statistical methods on the individual categorical
“Response” variables. Note that in some Categorical
Distributions the “Explanatory” variable can be Quantative.
(e.g. In the example above, the Education Degree could be
replaced by Number of Years of Education).
July 2004

© 2004 L. Pinsky

12


The Binomial Distribution
Where the SAMPLE SIZE is FIXED, and one has a
“Bernoulli” (Yes or No) Variable, the PARENT
DISTRIBUTION is a Binomial…


N independent trials, each with a probability of “success”
of π. (e.g. The number of fatal accidents per every 100
highway accidents)

P(y) = [N! πy (1 – π)N-y] / [y! (N – y)!]
With: y = 0, 1, 2…

µ(y) = N π, and σ(y) = [N π (1 – π)]1/2



The Binomial Variance, σ(y)2 is always smaller than µ(y).
It is impractical to evaluate P(y) exactly for large N…

July 2004

© 2004 L. Pinsky

13


The Poisson Distribution
Where many identical measurements are made, and during
which, some variable number, y, of sought after events
occur…
 (e.g. The number of radioactive decays/sec)
P(y) = ( e-µ µy ) / y! ( y = 0, 1, 2, …)
µ = Distribution mean & σ = µ1/2
σ increases with the value of µ
When the Experimental Variance exceeds σ, it is called
“Overdispersion” and is usually due to differences in the
conditions from one measurement to the next…
The distribution of counts within an INDIVIDUAL category
over multiple experiments is Poisson!
When N is Large and π is small (such that µ = Nπ << N) a
Binomial Distribution tends towards a Poisson Distribution.

July 2004

© 2004 L. Pinsky

14


The Pervasive Gaussian:
The NORMAL Distribution
P(y) = e-(1/2)[(y-µ)/σ] /(σ [2π]1/2)
Characterized solely by µ and σ, and the average is the best
estimate of the mean.
In the limit of large N, with a non-vanishing Nπ, (i.e. Nπ>>1)
the Normal Distribution approximates a Binomial Distribution…
Also, in the limit where Nπ>>1 a Poisson Distribution tends
towards a Normal Distribution.
…Because P(y) is symmetric, σ2 = 1/(N-1)
dP(y)/dy = 0 at y = µ, and d2P(y)/dy2 = 0 at y = µ ± σ.
±σ ~ 68%, ±2σ ~ 95%, and ±3σ ~ 99.7%. FWHM = 2.354σ
Although it is by far the most common and likely PARENT
DISTRIBUTION encountered in Experimental Science, it is
NOT the only one!
2

July 2004

© 2004 L. Pinsky

15

The Central Limit Theorem
Any Distribution that is the sum of many
SMALL effects, which are each due to
some RANDOM DISTRIBUTION, will tend
towards a Normal Distribution in the limit
of large statistics, REGARDLESS of the
nature of the individual random
distributions!

July 2004

© 2004 L. Pinsky

16


Other Distributions to Know
Lorentzian (Cauchy) Distribution—Used to
describe Resonant behavior:




P(y) = (Γ/2)/{π[(y-µ)2 + (Γ/2)2]}, Γ =FWHM
Here, π means 3.14159… & σ has no meaning!
…Instead, the FWHM is the relevant parameter!

Landau Distribution—in Particle Physics…
Boltzmann Distribution—in Thermo…

Bose-Einstein Distribution—in QM…
Fermi-Dirac Distribution—in QM…
…and others…
July 2004

© 2004 L. Pinsky

17


Maximum Likelihood
The “Likelihood” is simply the product of the
probabilities for each individual outcome in a
measurement, or an estimate for the total actual
probability of the observed measurement being
made.
If one has a candidate distribution that is a
function of some parameter, then the value of
that parameter that maximizes the likelihood of
the observation is the best estimate of that
parameter’s value.
The catch is, one has to know the correct
candidate distribution for this to have any
meaning…
July 2004

© 2004 L. Pinsky

18

Drawing Conclusions
Rejecting Hypotheses:




Relatively Easy if the form of the PARENT
Distribution is known: just show a low
probability of fit. The χ2 technique is perhaps
the best known method.
A more general technique is the F-Test, which
allows one to separate the deviation of the
data from the Estimated Distribution AND the
discrepancy between the Estimated
Distribution and the PARENT DISTRIBUTION.

July 2004

© 2004 L. Pinsky

19


Comparing Alternatives
This is much tougher…
 Where χ2 tests favor one hypothesis over another, but not
decisively, one must take great care. It is very east to be
fooled into rejecting the correct alternative…




Generally, a test is based on some statistic (e.g. χ2) that
estimates some parameter in a hypothesis. Values of the
estimate of the parameter far from that specified by the
hypothesis gives evidence against it…
One can ask, given a hypothesis, for the probability of
getting a set of measurements farther from the one
obtained assuming the hypothesis is correct. The lower the
probability, the less the confidence in the hypothesis being
correct…

July 2004

© 2004 L. Pinsky

20


Fitting Data
Fitting to WHAT???


Phenomenological (Generic)
 Linear
 LogLinear
 Polynomial



Hypothesis Driven

 Functional Form From Hypothesis

Least Squares Paradigm…


Minimizing the Mean Square Error is the Best
Estimate of Fit…

July 2004

© 2004 L. Pinsky

21


Errors in Comparing Hypotheses:
Choice of Tests
Type I Error—Rejecting a TRUE Hypothesis


The Significance Level of any fixed level confidence
test is the probability of a Type I Error. More serious,
so choose a strict test.

Type II Error—Accepting a FALSE Hypothesis


The Power of a fixed level test against a particular
alternative is 1 – the probability of a Type II Error.
Choose a test that makes the probability of a Type II
Error as small as possible.

July 2004

© 2004 L. Pinsky

22


The INTERVAL DISTRIBUTION
This is just an aside that needs mentioning:
For RANDOMLY OCCURING EVENTS, the
Distribution of TIME INTERVALS between
successive events is given by:
I(t) = (1/τ) e-t/τ
The mean value is τ.
I(0) = τ, or in words: the most likely value is 0.
Thus, there are far more short intervals than long
ones! BEWARE: As such, truly RANDOM EVENTS
TO THE NAÏVE EYE APPEAR TO “CLUSTER”!!!
July 2004

© 2004 L. Pinsky

23

Time Series Analysis
Plotting Data taken at fixed time intervals is
called a Time Series.


(e.g. The closing Dow Jones Average each day)

If nothing changes in the underlying PARENT
DISTRIBUTION, then Poisson Statistics apply…
BUT, in the real world one normally sees
changes from period to period.
Without specific hints as to causes, one can look
for TRENDS and CYCLES or“SEASONS.”
Usually, the problem is filtering these out from
large variation background fluctuations…
July 2004

© 2004 L. Pinsky

24


Bayesian Statistics
A Field of Statistics that takes into account
the degree of “Belief” in a Hypothesis:



P(H|d) = P(d|H) P(H)/P(d)
P(d) = Σi P(d|Hi) P(Hi), for multiple hypotheses

Can be useful for non-repeatable events
Can be applied to multiple sets of prior
knowledge taken under differing conditions
Bayes Theorem: P(B|A) P(A) = P(A|B) P(B)


Where P(A) and P(B) are unconditional or a
priori probabilities…

July 2004

© 2004 L. Pinsky

25