Covariance

Covariance is a measure of how much two random variables change together. For random variables X and Y with finite expected values, the population covariance is defined as Cov(X,Y) = E[(X - E[X])(Y - E[Y])] = E[XY] - E[X]E[Y].

Properties include symmetry (Cov(X,Y) = Cov(Y,X)) and linearity in each argument: Cov(aX + b, cY + d) = ac Cov(X,Y) for constants a, c, b, d. Cov(X, cY) = c Cov(X,Y), and Cov(X, c) = 0 for a constant c. Scaling yields Cov(aX, bY) = ab Cov(X,Y). If X and Y are independent, Cov(X,Y) = 0; however, zero covariance does not in general imply independence unless certain conditions hold (e.g., joint normality). The covariance matrix is positive semidefinite: for any vector a, a^T Cov(X) a ≥ 0.

The sample covariance between observed pairs (xi, yi), i = 1,...,n, is Sxy = (1/(n-1)) Σ (xi - x̄)(yi - ȳ). This is an unbiased estimator of the population covariance when sampling without bias. Diagonal elements correspond to variances.

Relation to correlation: Corr(X,Y) = Cov(X,Y) / (σ_X σ_Y), where σ_X and σ_Y are the standard deviations of X and Y. Correlation is dimensionless and ranges between -1 and 1, providing a standardized measure of linear association.

Applications and interpretation: covariance is fundamental in multivariate statistics and finance, underpinning the covariance matrix used in PCA and in portfolio variance calculations. Limitations include its dependence on the units of X and Y and its restriction to capturing linear relationships; non-linear dependencies may not be reflected by covariance alone.