Perhaps the most fundamental statistical summary beyond simple counting or totaling is the mean. The mean reduces a collection of numbers to a single value, and is one of a number of measures of location. The mean is by far the most commonly used and widely understood way of averaging data, but it is not the only one, nor is it always the "best" one. In terms of popularity, the median is a distant second, but it offers a mixture of behaviors which make it an appealing alternative in many circumstances.
One important property of the median is that it is not affected- at all- by extreme values. No matter how we change any observation, its value will have zero effect on the sample median unless it changes from being above the median to below it, or vice versa. If the maximum value in a data set is multiplied by a million, the median will not change. This behavior is usually characterized as a benefit of the median, as it means that noisy or extreme values will not tug at the median. This is in stark contrast to the sample mean, which blows around in the wind of any change to the data set. Indeed, this is one of the nice things about working with the median: It is highly resistant to noise. Also typically cast as a benefit is the median's resistance to being tugged to extreme values by long-tailed distributions. Whether this is truly a strength or a weakness, though, I will leave to the analyst to decide, since capturing the nature of a long-tail may be important for some problems.
Another important quality of the sample median is that, for odd numbers of observations, it is always a value from the original data set. This is also true for many data sets with even numbers of observations. In some situations, this is desirable. Data summaries being shown to non-technical people often provide more aesthetic appeal if there are no "odd" values. Consider a retail store in which all products are prices with .99 at the end, yet the reported mean ends in .73- this is normally not an issue with the median. Replacing missing values with means very likely introduces new values to the data set, and this is especially awkward when dealing with variables which present limited distributions, such as counts. As an example, consider a database with missing values for "number of children", which (one would expect!) would always be an integer. Substituting the mean for missings may result in observations sporting "1.2" children. This is not a problem with the median. Note that this behavior has a darker side for signal- and image-processing problems: data treated with a windowed median tend to form stepped plateaus, rather than smooth curves. Such artifacts can be distracting or worse.
On the downside, the median is not always the most statistically efficient summary. This is a technical issue which means simply that the median may "wander" from the "true" value (the population parameter) more than other summaries when finite data are available. For instance, when the data are known to be drawn from a true normal distribution, the sample mean is known to "wander" the least from the true value. I'm sparing the reader the technical details here, but suffice it to say that, though the mean or median might be closer to the "true" value in any particular situation, the mean is likely to be closer most of the time. Statistical efficiency may be measured, and it is worth noting that different summaries achieve different relative efficiencies, depending on the size of the data sample, and the shape of the population distribution.
Hopefully, this short note has whet your appetite for the median and other alternatives to the mean. I encourage the reader to explore yet other alternatives, such as trimmed means, of which the mean and median are special cases.
Showing posts with label summaries. Show all posts
Showing posts with label summaries. Show all posts
Tuesday, July 31, 2012
Friday, April 13, 2007
Basic Summary Statistics in MATLAB
This posting covers basic summary statistics in MATLAB.
First, note that MATLAB has a strong array-orientation, so data sets to be analyzed are most often stored as a matrix of values. Note that the convention in MATLAB is for variables to be stored in columns, and observations to be stored in rows. This is not a hard-and-fast rule, but it is much more common than the alternative (variables in rows, observations in columns). Besides, most MATLAB routines (whether from the MathWorks or elsewhere) assume this convention.
Basic summaries are easy to obtain from MATLAB. For the examples below, the following matrix of data, A, will be used (No, it's not very exciting, but it will do for our purposes):
>> A = [1 2 3 4; -1 10 8 5; 9 8 7 0; 0 0 0 1]
A =
1 2 3 4
-1 10 8 5
9 8 7 0
0 0 0 1
MATLAB matrices are indexed as: MatrixName(row,column):
>> A(2,1)
ans =
-1
Common statistical summaries are available in MATLAB, such as: mean (arithmetic mean), median (median), min (minimum value), max (maximum value) and std (standard deviation). Their use is illustrated below:
>> mean(A)
ans =
2.2500 5.0000 4.5000 2.5000
>> median(A)
ans =
0.5000 5.0000 5.0000 2.5000
>> min(A)
ans =
-1 0 0 0
>> max(A)
ans =
9 10 8 5
>> std(A)
ans =
4.5735 4.7610 3.6968 2.3805
Note that each of these functions operate along the columns, yielding one summary for each, stored in a row vector. Sometimes it is desired to calculate along the rows instead. Some routines can be redirected by another parameter, like this:
>> mean(A,2)
ans =
2.5000
5.5000
6.0000
0.2500
The above calculates the arithmetic means of each row, storing them in a column vector. The second mean parameter, if it is specified, indicates the dimension along which mean is to operate.
For routines without this capability, the data matrix may be transposed (rows become columns and columns become rows) using the apostrophe operator while feeding it to the function:
>> mean(A')
ans =
2.5000 5.5000 6.0000 0.2500
Note that, this time, the result is stored in a row vector.
The colon operator, :, can be used to dump all of the contents of an array into one giant column vector. The result of this operation can then be fed to any of our summary routines:
>> A(:)
ans =
1
-1
9
0
2
10
8
0
3
8
7
0
4
5
0
1
>> mean(A(:))
ans =
3.5625
The reader will find more information on summary routines in base MATLAB through:
help datafun
The MATLAB Statistics Toolbox
MATLAB users lucky enough to own the Statistics Toolbox will have available still more summaries, such as iqr (inter-quartile range), trimmean (trimmed mean) and geomean (geometric mean). Also, there are extended versions of several summary functions, such as nanmean and nanmax, which will ignore NaN (IEEE floating point "not-a-number") values, which are commonly used to represent missing values in MATLAB.
To learn more, see the "Descriptive Statistics" section when using:
help stats
First, note that MATLAB has a strong array-orientation, so data sets to be analyzed are most often stored as a matrix of values. Note that the convention in MATLAB is for variables to be stored in columns, and observations to be stored in rows. This is not a hard-and-fast rule, but it is much more common than the alternative (variables in rows, observations in columns). Besides, most MATLAB routines (whether from the MathWorks or elsewhere) assume this convention.
Basic summaries are easy to obtain from MATLAB. For the examples below, the following matrix of data, A, will be used (No, it's not very exciting, but it will do for our purposes):
>> A = [1 2 3 4; -1 10 8 5; 9 8 7 0; 0 0 0 1]
A =
1 2 3 4
-1 10 8 5
9 8 7 0
0 0 0 1
MATLAB matrices are indexed as: MatrixName(row,column):
>> A(2,1)
ans =
-1
Common statistical summaries are available in MATLAB, such as: mean (arithmetic mean), median (median), min (minimum value), max (maximum value) and std (standard deviation). Their use is illustrated below:
>> mean(A)
ans =
2.2500 5.0000 4.5000 2.5000
>> median(A)
ans =
0.5000 5.0000 5.0000 2.5000
>> min(A)
ans =
-1 0 0 0
>> max(A)
ans =
9 10 8 5
>> std(A)
ans =
4.5735 4.7610 3.6968 2.3805
Note that each of these functions operate along the columns, yielding one summary for each, stored in a row vector. Sometimes it is desired to calculate along the rows instead. Some routines can be redirected by another parameter, like this:
>> mean(A,2)
ans =
2.5000
5.5000
6.0000
0.2500
The above calculates the arithmetic means of each row, storing them in a column vector. The second mean parameter, if it is specified, indicates the dimension along which mean is to operate.
For routines without this capability, the data matrix may be transposed (rows become columns and columns become rows) using the apostrophe operator while feeding it to the function:
>> mean(A')
ans =
2.5000 5.5000 6.0000 0.2500
Note that, this time, the result is stored in a row vector.
The colon operator, :, can be used to dump all of the contents of an array into one giant column vector. The result of this operation can then be fed to any of our summary routines:
>> A(:)
ans =
1
-1
9
0
2
10
8
0
3
8
7
0
4
5
0
1
>> mean(A(:))
ans =
3.5625
The reader will find more information on summary routines in base MATLAB through:
help datafun
The MATLAB Statistics Toolbox
MATLAB users lucky enough to own the Statistics Toolbox will have available still more summaries, such as iqr (inter-quartile range), trimmean (trimmed mean) and geomean (geometric mean). Also, there are extended versions of several summary functions, such as nanmean and nanmax, which will ignore NaN (IEEE floating point "not-a-number") values, which are commonly used to represent missing values in MATLAB.
To learn more, see the "Descriptive Statistics" section when using:
help stats
Subscribe to:
Posts (Atom)