Bxdty

Equivalently, about variance?

I realize it measures the spread of a distribution, but many other metrics could do the same (e.g., the average absolute deviation). What is its deeper significance? Does it have

• a particular geometric interpretation (in the sense, e.g., that the mean is the balancing point of a distribution)?


• any other intuitive interpretation that differentiates it from other possible measures of spread?

What's so special about it that makes it act as a normalizing factor in all sorts of situations (for example, convert covariance to correlation)?

There's a very nice geometric interpretation.

Random variables of finite mean form a vector space. Covariance is a useful inner product on that space. Oh, wait, that's not quite right: constant variables are orthogonal to themselves in this product, so it's only positive semi-definite. So, let me be more precise - on the quotient space formed by the equivalence relation "differs from by a constant", covariance is a true inner product. (If quotient spaces are an unfamiliar concept, just focus on the vector space of zero-mean variables; it gets you the same outcome in this context.)

Right, let's carry on. In the norm this inner product induces, standard deviation is a variable's length, while the correlation coefficient between two variables (their covariance divided by the product of their standard deviations) is the cosine of the "angle" between them. That the correlation coefficient is in $[-1,,1]$ is then a restatement of the vector space's Cauchy-Schwarz inequality.

I take it as unproblematic that the standard deviation is important in the normal distribution since the standard deviation (or variance) is one of its parameters (though it could doubtless be reparameterized in various ways). By the Central Limit Theorem, the normal distribution is in turn relevant for understanding just about any distribution: If $X$ is a normal variable with mean $mu$ and standard deviation $sigma$, then for large $n$

$$frac{overline{X} - mu}{frac{sigma}{sqrt{n}}}$$

is approximately standard normal. No other measure of dispersion can so relate $X$ with the normal distribution. Said simply, the Central Limit Theorem in and of itself guarantees that the standard deviation plays a prominent role in statistics.

An interesting feature of the standard deviation is its connection to the (root) mean square error. This measures how well a predictor does in predicting the values. The root mean square error of using the mean as a predictor is the standard deviation, and this is the least root mean square error that you can get with a constant predictor.

(This, of course, shifts the question to why the root mean squared error is interesting. I find it a bit more intuitive than the standard deviation, though: you can see it as the $L_2$ norm of the error vector, corrected for the number of points.)

When defining "standard deviation", we want some way to take a bunch of deviations from a mean and quantify how big they typically are using a single number in the same units as the deviations themselves. But any definition of "standard deviation" induces a corresponding definition of "mean" because we want our choice of "mean" to always minimize the value of our "standard deviation" (intuitively, we want to define "mean" to be the "middlemost" point as measured by "standard deviation"). Only by defining "standard deviation" in the usual way do we recover the arithmetic mean while still having a measure in the right units. (Without getting into details, the key point is that the quadratic becomes linear when we take the derivative to find its critical point.)

If we want to use some other mean, we can of course find a different "standard deviation" that will match that mean (the progress is somewhat analogous to integration), but in practice it's just easier to transform the data so that the arithmetic mean is appropriate.

The normal distribution has maximum entropy among real distributions supported on $(-infty, infty)$ with specified standard deviation (equivalently, variance). (Reference.) Consequently, if the only thing you know about a real distribution supported on $mathbb{R}$ is its mean and variance, the distribution that presumes the least prior information is the normal distribution.

I don't tend to think of the statement above as the important fact. It's more: normal distributions appear frequently and knowing the location parameter (mean) is reasonable. So what else do I have to know to make the least presumptive model be the normal distribution? The dispersion (variance).