Numerical Errors

Engineering
Computation
Lecture 2
Errors
v20160922
E. T. S. I. Caminos, Canales y Puertos. Santander
1
APPROXIMATION AND ERRORS
Numerical Methods
Instead of solving for the exact solution we solve
math problems with a series of arithmetic
operations.
b
Example:
1
∫ x dx
a
analytical solution: ln(b) – ln(a)
numerical solution e. g., Trapezoidal Rule
Error Analysis
(a) identify the possible sources of error
(b) estimate the magnitude of the error
(c) determine how to minimize and control errors
E. T. S. I. Caminos, Canales y Puertos. Santander
2
Approximations and Rounding Errors
• Unfortunately, computers introduce errors in the calculations.
However, since many engineering problems have not analytical
solution, we are forced to use numerical methods
(approximations). The only option we have is to accept the error
and try to reduce it up to a tolerable level.
• The only way of minimizing the errors is by knowing and
understanding why they occur and how we can diminish them.
• The most frequent errors are:
– Rounding errors, due to the fact that computers can work
only with a finite representation of numbers.
– Truncation errors, due to differences between the exact
and the approximate (numeric) formulations of the
mathematical problem being dealt with.
• Before analyzing each one of them, we will see some important
concepts on the computer representation of numbers.
E. T. S. I. Caminos, Canales y Puertos. Santander
3
Approximations and Rounding Errors
• Significant figures of a number:
– The significant figures of a number are those which can be
used with confidence.
. A taylor’s ruler: 87.5 cms: 3 significant figures
. Car’s odometer: 87324.6 kms: 6 significant figures
. 0.00001845, 18.45, 0.001845 : 4 significant figures
. 4.54 104, 4.530 104, 4.5300 104: 3, 4, 5 significant figures
Implications
1. An approximation is acceptable when it is correct for a given number of
significant figures.
2. There are magnitudes or constants that cannot be represented exactly:
π=3.14159265… , 17 =4.1231056… : 9, 8 significant figures
- Omitted significant figures: round-off error
E. T. S. I. Caminos, Canales y Puertos. Santander
4
Approximations and Rounding Errors
• Accuracy and precision:
– The errors associated with
calculations and
measurements can be
characterized observing their
accuracy and precision.
. Accuracy refers to how close are
values to the true one.
(inaccuracy or bias, systematic
deviation)
. Precision refers to how close
different measured/computed
values agree each other
(imprecision, uncertainty)
Numerical methods must be sufficiently exact (without bias) and
precise to satisfy the requirements of engineering problems. From
now on we will refer to error as the inaccuracy and lack of precision
of our predictions.
E. T. S. I. Caminos, Canales y Puertos. Santander
5
Approximations and Rounding Errors
• Error definitions:
– True value = approximation + true error.
– Absolute error = abs (true value - approximation).
– Relative error = absolute error / |true value|
εt =
absolute error
| true value |
100%
– In real-world cases, not always the true value can be known, thus:
εa =
aproximate abs error
100%
| approximate value |
– In many occasions a tolerance (error approximation) is calculated as
the difference between the previous and the current approximations.
εa =
| current approximation - previous approximation |
100%
| current approximation |
E. T. S. I. Caminos, Canales y Puertos. Santander
6
Approximations and Rounding Errors
– Thus, the stopping criterium of a numerical method can be:
εa < εs
ε s = prefixed percent tolerance
– It is convenient to relate the relative error when representing
a numerical value with the number of significant figures.
If the εa < εs
with
ε s = (0.5 * 102 − n )%
where εa approximates the relative error, one can say (C&C, Ex
3.2) that the result is correct to at least n significant figures.
E. T. S. I. Caminos, Canales y Puertos. Santander
7
Approximations and Rounding Errors
ERROR BOUNDS (more used in practice)
. Absolute error bound (EaB):
Any number greater than the absolute error.
. Relative error bound (ErB):
Any number greater than the relative error.
Example: a taylor uses a ruler marked each 0.5 cm:
Absolute error bound of his measurements: 0.25 cm (why?, he measures
to the closest mark)
Some properties (ErB(M)=EaB(M)/|M|)
* EaB(X1+X2+…+Xn)=EaB(X1)+EaB(X2)+…+EaB(Xn)
* ErB(X1+X2+…+Xn, {xi >0, i=1,…,n} )≈ ErB (term with greatest relative
error bound)
* ErB(X1·X2·…·Xn) ≈ ErB(X1)+ErB(X2)+…+ErB(Xn)
* ErB(N/D)≈ ErB(N)+ErB(D)
Examples:
Area of a triangle: ErB(b·h/2) ≈ ErB(b)+ErB(h)
Pressure on a wall: ErB(Force/Area) ≈ ErB(F)+ErB(A)
E. T. S. I. Caminos, Canales y Puertos. Santander
8
Representation of numbers
• Numerical systems:
– A numerical system is a
convention to represent
quantities. Since we have 10
fingers in our hands, the most
popular is the base-10 system
(arabic). It uses 10 different
digits,0 to 9.
104 103 102 101 100
8 6 4 0 9
a)
9x
1=
9
0x
10 =
0
4 x 100 = 400
6 x 1000 = 6000
8 x 10000 = 80000
86409
27 26 25 24 23 22 21 20
1 0 1 0 1 1 0 1
– However, computers, due to
the memory structure, can only
store two digits:
0 and 1.
Thus, they use the binary system
of numeric representation.
E. T. S. I. Caminos, Canales y Puertos. Santander
b)
1x 1= 1
0x 2= 0
1x 4= 4
1x 8= 8
0 x 16 = 16
1 x 32 = 32
0 x 64 = 64
1 x 128 = 128
173
9
Representation of numbers
Background:
•
•
•
How numbers are stored in a computer?
The fundamental unit, a "word," consists of a string of "bits" (binary digits).
Because computers are made up of gates or switches which are either
closed or open, we work in binary or base 2 system.
A number in base q will be denoted by
(anan-1...a1a0.b1b2..bk..)q
The conversion to base 10 is, by definition
(anan-1...a1a0.b1b2..bk..)q =anqn+an-1qn-1+...+a1q+a0q0+b1q-1+b2q-2+...
Example:
(1011.01)2=1x23+0x22+1x2+1x20+0x2-1+1x2-2=11.2510
E. T. S. I. Caminos, Canales y Puertos. Santander
10
Representation of numbers
Conversion from base 10 to base 2.
This is the recipe for conversion:
Integer part: we have to divide the integer part by 2 (successively) and
to retain the fractional part in each step.
Fractional part: we have to multiply by 2 and to retain the integer part
in each step.
Example:
(26.1)10=(11010 . 00011)2
E. T. S. I. Caminos, Canales y Puertos. Santander
11
Representation of numbers
Fraction
Integer
Representing 0.110 in base 2:
2610=110102=
=0*20+1*21+
+0*22+1*23+1*24=
=0 +2+0+8+16=2610
•
0.0 00110011…2
•
•
•
•
•
•
•
0.1*2 = 0.2---> 0
0.2*2 = 0.4---> 0
0.4*2 = 0.8---> 0
0.8*2 = 1.6---> 1
1.6-1=0.6, 0.6*2 = 1.2---> 1
1.2-1=0.2, 0.2*2 = 0.4---> 0
0.4*2 = 0.8---> 0
E. T. S. I. Caminos, Canales y Puertos. Santander
26:2 ->13, remainder=0;
13:2 ->6, rem=1,
6:2 ->3, rem=0,
3:2 ->1, rem=1
12
Round-off Errors
Example:
An 8 bit word representation of the integer "3510" is 001000112 or
0
0
1
0
0
0
1
1
25
24
23
22
21
20
±
26
+
0x26 1x25 0x24 0x23 0x22 1x21 1x20 = 35
32
2
1
Note: We can only represent a finite # of numbers; for our case:
–127
to +127
(127 = 27 – 1)
or a total of 255 numbers (including 0)
Biggest number:± 1 1 1 1 1 1 12= ± (1·20 + …+ 1· 26)= ±12710
E. T. S. I. Caminos, Canales y Puertos. Santander
13
Approximations and Rounding Errors
• Representation of integer numbers:
– To represent base 10 numbers in binary form the signed
magnitude method is used. The first digit stores the sign (0,
positive and 1, negative). The remaining bits are used to
store the number.
– A computer working with words of 16 bits can store integer
numbers in the range -32768 to 32767.
E. T. S. I. Caminos, Canales y Puertos. Santander
14
Approximations and Rounding Errors
• Floating point representation:
– This representation is used for fractional quantities. It has
the fraction part, called mantissa, and an integer part, called
exponent or characteristic.
m * be
– The mantissa is usually normalized, so that the value of m is
limited (b=2 in binary):
1
≤m<1
b
E. T. S. I. Caminos, Canales y Puertos. Santander
15
Approximations and Rounding Errors
• IEEE-floating point formats: there are two types of “precision”
(simple and double). They differ in the number of digits available
for storing the numbers:
• Simple precision (32 bits): 1 bit for the sign, 8 bits for the
exponent, 23 bits for the mantissa.
(largest:~1038 , smallest: ~10-38; ~7 decimal digits)
• Double precision (64 bits, two words of 32 bits): 1 bit for the
sign, 11 bits for the exponent, 52 bits for the mantissa.
(largest:~10308 , smallest: ~10-308; ~16 decimal digits)
The number of bits for the exponent and the mantissa
determine “underflow” and “overflow” numbers.
E. T. S. I. Caminos, Canales y Puertos. Santander
16
CLASSIFICATION OF ERRORS
1. Errors in Input/Initial Data
Concept: ILL CONDITIONED PROBLEM (Bad conditionning):
SMALL CHANGES IN THE DATA, BIG ERROR IN RESULTS
y
All operations in exact arithmetic!:
x+2y=10
x=(2·10-2·10.4)/(1·2-2·1.1) =4
1.1x+2y=10.4
y=(1·10.4-1.1·10)/(1·2-2·1.1) =3
Small change in data:
x+2y=10
1.05x+2y=10.4
‘The problem of the
unsharpened pencil’
x
x=(2·10-2·10.4)/(1·2-2·1.05) =8
y=(1·10.4-1.05·10)/(1·2-2·1.05) =1
2. Propagated errors: Errors which appear because we are basing
current calculations on previous calculations which also incurred
some form of error
E. T. S. I. Caminos, Canales y Puertos. Santander
17
CLASSIFICATION OF ERRORS
3. Errors generated by calculations:
- Model
Hooke’s law: σ = E · ε
- Truncation
ex ≈ 1+x+x2/2!+x3/3!+x4/4!
- Rounding ---> Concept: UNSTABILITY,
Sensitivity of a process to rounding errors
Example of Unstability
Evaluate for n=0, 1, 2, …the value of:
yn =
1 xn
∫0 x + 5dx
using the recurrence (exact for this example):
yn + 5·yn-1=1/n
It is a decreasing sequence of positive numbers (positive areas in [0,1])
E. T. S. I. Caminos, Canales y Puertos. Santander
18
CLASSIFICATION OF ERRORS
Scheme 1 (unstable): yn =1/n - 5·yn-1, n=0: ∫01dx/(x+5)=ln(x+5)]01≈0.183 (3 decim)
EaB=0.5·10-3
y1 =1 - 5·y0 ≈ 1-5·0.182=1-0.910=0.090
y2 =1/2 - 5·y1 ≈ 0.5-5·0.0.090=1/2-0.45=0.050
y3 =1/3 - 5·y2 ≈ 1/3-5·0.050 ≈ 0.333-0.250=0.083
y3>y2 !!
y4 =1/4 - 5·y3 ≈ 0.25-0.415 = -0.165 < 0 !!, WRONG!!
Rounding errors: y0 : 0.5·10-3 ; y1: 5·ε ; y2: 25·ε ; y3= 125·ε
y4: 625·ε=625·0.5·10-3=0.3125 >y0 !!
Scheme 2 (stable): yn-1 =1/(5·n) - yn/5, we divide by 5 the error at each step
Let us take y10 ≈ 0:
y9 ≈ 0.020; y8 ≈0.018; y7 ≈0.021 ; y6 ≈0.025 ; y5 ≈ 0.028
y4 ≈ 0.034; y3 ≈0.043; y2 ≈0.058 ; y1 ≈0.088 ; y5 ≈ 0.182
(From y8 until y0, error<0.001)
E. T. S. I. Caminos, Canales y Puertos. Santander
19
CLASSIFICATION OF ERRORS
Example of truncation error:
Taylor expansion for ex
ex =
1+x+x2/2!+x3/3!+x4/4!+…
If we use a truncated version 1+x+x2/2!
the Truncation Error is x3/3!+x4/4!+…
ex , Taylor approx, No of Terms: (2,*),(4,.),(6,+),(8,o)
8
7
6
ex y aprox
5
4
3
2
1
0
-1
-2
-1.5
-1
-0.5
E. T. S. I. Caminos, Canales y Puertos. Santander
0
x
0.5
1
1.5
2
20
CLASSIFICATION OF ERRORS
Example of Truncation Error: Taylor expansion for ex
% script comparing approximations of exp(x) by Taylor expansion
x=linspace(-2,2,40);
% x=linspace(-1,1,40);x=linspace(-5,5,40);
ex=exp(x); % ‘exact’ value
ex2=expTaylor(x,2);
ex4=expTaylor(x,4);
ex6=expTaylor(x,6);
ex8=expTaylor(x,8);
figure
plot(x,ex,'k-',x,ex2,'k*',x,ex4,'k.',x,ex6,'k+',x,ex8,'ko')
title('e^x, Taylor approx, No of Terms: (2,*),(4,.),(6,+),(8,o)')
xlabel('x'),ylabel('e^x y aprox')
function [y]=expTaylor(x,n)
% exp(x) , Taylor expansion around a x=0:
% exp(x)=1+x+x^2/2+x^3/3!+...
% input: vector x evaluation at x's , n number of terms
% output: vector con the approximation of exp(x)
y=ones(1,length(x)); %y=ones(size(x));
for i=1:n-1
y=y+x.^i/prod(1:i);% ‘.’(point) operator acts element by element
end
E. T. S. I. Caminos, Canales y Puertos. Santander
21
Round-off Error due to Arithmetic Operations
Subtractive Cancellation (subtracting numbers of almost
equal size) – too few significant figures left.
Examples:
x1 = 0.5764 ± 0.5·10-4, x2 = 0.5763 ± 0.5·10-4
D = x1-x2 = 0.0001 ± 0.0001, bound of εr =1=100%
1.
2.
3.
Use of the “standard” formulas for solving P(x)=0, being P(x)
a polynomial of degree 2.
Computation of f(x)=(x+1)1/2-x1/2 for large x.
Computation of g(x)=(1-cos(x))/x2 for small x.
Tricks:
. (x+1)1/2-x1/2 = (x+1-x)/((x+1)1/2+x1/2) =1/((x+1)1/2+x1/2)
. 1-cos(x)=(sin(x))2/(1+cos(x)) ,
. cos(x+a)-cos(x)= - 2sin(a/2)·sin(x+a/2), |a|<<|x|
>> Use of higher precision can be an alternative
E. T. S. I. Caminos, Canales y Puertos. Santander
22
Round-off Error due to Arithmetic Operations
Smearing
Occurs when individual terms are larger than summation itself.
Consider the exponential exp(x) with x = -15, with a formula such as
(Taylor Mac-Laurin):
F(x)=ex = 1+x+x2/2!+x3/3!+x4/4!+…
With 15-decimal-digit accuracy (Matlab):
correct answer
= 3.059…e-07 (Matlab exp(-15))
computed answer = 5.289…e-04, (45 terms, alternate signs )
Largest intermediate terms are:
15th = 3.348…e+05 & 16th = = -3.348…e+05
Alternative calculation: proceed with F(+15) and then 1/F(15)=F(-15)
This gives, with 45 terms: F(+15)= 3.269…e+06, 1/F(15)= 3.059…e-07
E. T. S. I. Caminos, Canales y Puertos. Santander
23
Error when evaluating a function
Let us consider a function of one variable: y=f(x)
We suppose that f(x) is differentiable.
|∆x| is the absolute error of x. For small |∆x|,
|∆f|=|f(x+ ∆x)-f(x)| ≈ |f’(x)·∆x| , (remember Taylor series)
(EaB : absolute Error Bound, ErB : relative Error Bound)
EaB(f) ≈ |f’(x)|·EaB(x), greater slope, greater EaB(f)
uncertainty in x is amplified by ƒ’(x)
ErB(f(x))=EaB(f(x)) / |f(x)| ≈ |f’(x)|·EaB(x)/|f(x)|=
= |f’(x)·x|/|f(x)|·EaB(x)/|x|=|f’(x)·x|/|f(x)|·ErB(x)
Cond(f(x)) = |f’(x)·x|/|f(x)| : CONDITION NUMBER of f(x)
"well conditioned" vs "ill conditioned"
Better if Cond(f(x)) is small. Bad if Cond(f(x)) is big
E. T. S. I. Caminos, Canales y Puertos. Santander
24
Examples of errors
Shot an Arrow in the Air,
The Sad Story of the Ariane 5
On June 4 1996 Ariane 5 was launched at Kourou.
All went well for 36 sec. At second 37, the rocket veered off
course and self-destructed.
The Cause of: Data error
The problem was in the Inertial Reference System
The IRS tried to convert a 64-bit floating point number to a 16-bit
fixed point number.
The number was too large, which triggered an Operand Error and
sent an diagnostic word to the On-Board Computer.
The OBC interpreted the diagnostic word as flight
data.
The Irony of It All: The program that caused the crash was legacy
software from the Ariane 4. It was not needed on the Ariane 5 !!
E. T. S. I. Caminos, Canales y Puertos. Santander
25
Examples of errors
• Cancellation because of fixed mantissa (big and small numbers)
• Consider the sum
• 472635.0000
+
27.5013
- 472630.0000
32.5013 (Exact)
• If we compute it to six figures we get
• 472635.0000
+
27.5013
472663.0000 we have thrown away the last four digits of 27.5013!
When searching where a calculation has gone wrong,
And
observe where you have lost information
• 472663.0000
- 472630.0000
33.0000 (no error in substraction)
• There are only two accurate figures.
E. T. S. I. Caminos, Canales y Puertos. Santander
26
Examples of errors
Solving a quadratic equation:
The quadratic equation x2 -56x + 1 = 0
. x1=28- 783 ≈ 28-27.982=0.018±0.5·10-3 , Abs Error Bound<= 0.5·10-3
. x2=28+ 783 ≈ 28+27.982=55.9828±0.5·10-3
Er (x1)≈ 0.5·10-3/0.018=2.78%
Er (x2) ≈ 0.5·10-3/55.982=0.893·10-5=0.000893%
Non equilibrated treatment of error for x1 and x2
Cardano formulas: x1·x2=c/a, x1+x2=-b/a, in ax2+bx+c=0
Compute first the largest root in absolute value (avoid cancellation)!!
Then, in this case, x1=(c/a)/x2=1/55.982 ≈ 0.017863, similar error level than x2
Relative error bound of a quotient ≈ Sum of relative error bounds of numerator and
denominator
E. T. S. I. Caminos, Canales y Puertos. Santander
27
Approximations and Rounding Errors
• Precautions:
– Sums of large and small numbers: loose of digits. They
are common in sums of infinite series where the
individual terms are very small when compared with the
accumulated sum. This error can be reduced by
summing first the small terms and using double
precision.
– Cancellation of the subtraction: The subtraction of very
similar numbers.
– Smearing: The individual terms are larger than the total
sum.
– Inner products: They are prone to rounding errors. Thus,
it is convenient to use double precision in this type of
calculations.
n
∑ xiyi = x1y1 + x 2y 2 +  + xnyn
i =1
E. T. S. I. Caminos, Canales y Puertos. Santander
28