COvariances

Covariance is a measure of how two random variables vary together. For random variables X and Y with finite means μx and μy, the covariance is Cov(X,Y) = E[(X−μx)(Y−μy)]. A positive covariance indicates that X and Y tend to increase together, while a negative covariance indicates that one tends to increase when the other decreases.

Covariance is symmetric and bilinear: Cov(X,Y) = Cov(Y,X) and Cov(aX+bZ, Y) = a Cov(X,Y) + b Cov(Z,Y). If X and Y are independent, Cov(X,Y) = 0, but zero covariance does not necessarily imply independence except in special cases such as joint normal distributions. The correlation coefficient, defined as ρ = Cov(X,Y)/(σx σy), standardizes covariance to a dimensionless measure between −1 and 1.

The concept extends to vectors through the covariance matrix. For a random vector X = (X1,…,Xk), the Covariance matrix has entries Cov(Xi,Xj). It is symmetric and positive semi-definite, and its diagonal entries equal the variances of the components. The matrix describes the linear relationships among all components and forms the basis for many multivariate techniques, including principal component analysis and multivariate regression.

Estimation from data uses the sample covariance. Given n paired observations (xi, yi), the sample covariance is sXY = (1/(n−1)) ∑ (xi − x̄)(yi − ȳ). The corresponding sample covariance matrix is formed similarly for vectors. Interpretation relies on the scale of the variables; correlations are often preferred for comparing relationships across different units.