PCA Dimension Reduction

What Is Principal Component Analysis (PCA)?

PCA is a dimensionality reduction technique that transforms correlated variables into a smaller set of uncorrelated variables called principal components. Each component captures the maximum remaining variance in the data, ordered by importance. The first few components often capture 80-95% of total variance.

Mathematically, PCA computes the eigenvectors and eigenvalues of the data's covariance (or correlation) matrix. The eigenvectors define the directions (loadings) of the new axes, and the eigenvalues quantify the variance captured along each axis. Data is typically standardized (zero mean, unit variance) before PCA to prevent variables with larger scales from dominating.

The scree plot shows eigenvalues vs. component number — the 'elbow' suggests how many components to retain. The biplot visualizes both scores (data points in PC space) and loadings (original variable contributions), enabling interpretation of what each component represents.

Formula: Covariance Matrix: C = (1/(n−1)) XᵀX (X centered) Eigendecomposition: C·v = λ·v PC scores: Z = X·V Variance explained = λₖ / Σλᵢ Standardized (default): divide each centered column by its standard deviation first — C becomes the correlation matrix R, and Σλᵢ = number of variables

Example Calculation

Five parts measured on three dimensions in mm: length 24/26/25/28/27, width 35/35/37/38/40, height 13/14/14/15/14. The covariance matrix has eigenvalues λ₁ = 6.21, λ₂ = 1.22, λ₃ = 0.07, a total variance of 7.50. PC1 explains 6.21/7.50 = 82.7%, PC2 16.3%, cumulative 99.0% — keeping 2 components reduces three dimensions to two and preserves 99% of the variance. All three variables share one unit, so the covariance matrix is the right one to decompose; variables in different units would need standardizing first, or the one with the largest numbers dominates PC1. To reproduce these numbers in the calculator, clear Standardize variables — it is on by default, which decomposes the correlation matrix instead.

When to Use This Calculator

• A data scientist reducing a high-dimensional feature set (50+ variables) to a manageable number of components before feeding into a machine learning model
• A researcher visualizing high-dimensional data in 2D or 3D using the first 2-3 principal components to detect patterns or clusters
• A quality engineer identifying the main sources of variation in a multi-variable manufacturing process using factor analysis
• An analyst removing multicollinearity from predictor variables by replacing correlated features with uncorrelated principal components

Common Mistakes to Avoid

• Not standardizing data before PCA — variables with larger scales dominate the variance and the first component merely reflects the highest-variance variable; always standardize to zero mean and unit variance
• Keeping too few components and losing important information — if cumulative variance explained is below 70%, important signal is being discarded; aim for 80-95%
• Interpreting principal components as individual variables — PCs are linear combinations of all original variables; they represent directions of maximum variance, not individual features
• Applying PCA to data with very few observations relative to variables — PCA needs at least as many observations as variables (ideally 5-10x) to produce stable results

How to Interpret Results

• If the first 2-3 components explain 80%+ of variance, your data is effectively low-dimensional and the biplot provides a meaningful summary
• Loadings with high absolute values indicate which original variables contribute most to each component — this helps interpret what the component represents
• Large gaps between consecutive eigenvalues in the scree plot indicate natural dimensionality boundaries — components after the gap contribute mostly noise

Related Standards & References

• Pearson, K. (1901), 'On Lines and Planes of Closest Fit to Systems of Points in Space', Philosophical Magazine — geometric origin of principal component analysis
• Hotelling, H. (1933), 'Analysis of a Complex of Statistical Variables into Principal Components', Journal of Educational Psychology — variance-maximizing formulation of PCA
• Jolliffe, I. T., Principal Component Analysis (Springer) — comprehensive modern reference including component-retention rules

Frequently Asked Questions

Should I standardize my data before PCA?

Yes, almost always. If variables have different units or scales (e.g., height in cm and weight in kg), PCA on raw data will be dominated by the variable with the largest variance. Standardizing to zero mean and unit variance ensures all variables contribute equally. Use the correlation matrix instead of the covariance matrix for the same effect.

How many principal components should I keep?

Common rules: (1) Keep components until cumulative variance explained reaches 80-95%. (2) Use the scree plot 'elbow' — retain components before the curve flattens. (3) Kaiser's rule — keep components with eigenvalue > 1 (for correlation-based PCA). The best choice depends on the trade-off between information loss and dimensionality reduction needed.