Outlier Detection

How does Isolation Forest detect outliers?

Isolation Forest is an unsupervised machine learning algorithm for anomaly detection, based on the principle that outliers are 'few and different' — they are easier to isolate than normal points. Unlike distance-based methods, it does not compute distances or densities, making it highly efficient for large datasets.

The algorithm builds an ensemble of random isolation trees. Each tree recursively partitions data by randomly selecting a feature and a split value. Anomalies, being rare and distinct, require fewer splits to isolate (shorter path length from root to leaf). Normal points, being clustered, require more splits.

The anomaly score is derived from the average path length across all trees, normalized by the expected path length for the dataset size. Scores close to 1 indicate anomalies; scores close to 0.5 indicate normal points. The contamination parameter sets the expected proportion of anomalies, determining the score threshold for classification.

Formula: Anomaly Score s(x, n) = 2^(-E(h(x)) / c(n)) c(n) = 2H(n-1) - 2(n-1)/n H(i) = ln(i) + Euler's constant s ≈ 1: anomaly s ≈ 0.5: normal s ≈ 0: very dense cluster

Example Calculation

100 data points with contamination = 0.05 (expect ~5 outliers). 100 isolation trees built. Point A has average path length 3.2 (quickly isolated → anomaly score 0.82 → flagged). Point B has average path length 8.7 (hard to isolate → anomaly score 0.45 → normal). 4 points exceed the threshold — flagged as anomalies.

When to Use This Calculator

• During incoming inspection data review to flag anomalous measurements that may indicate supplier quality issues or measurement errors
• When cleaning datasets before statistical analysis — undetected outliers can severely distort means, standard deviations, and capability indices
• In high-volume manufacturing SPC to detect unusual production patterns that standard control charts may miss in multivariate data
• When analyzing sensor or IoT data from equipment monitoring to identify early signs of machine degradation or process drift
• During quality audits to find data anomalies that may indicate recording errors, test equipment malfunction, or process upsets

Common Mistakes to Avoid

• Automatically removing all flagged anomalies without investigation — outliers may represent real and important process events that need corrective action, not just data cleanup
• Setting contamination too high, flagging normal variation as anomalous — start conservative (0.01-0.05) and increase only if domain expertise suggests a higher anomaly rate
• Using Isolation Forest on very small datasets (under 50 points) — the algorithm needs sufficient data to build meaningful isolation trees; use simpler methods (Z-score, IQR) for small samples
• Ignoring the anomaly score distribution — rather than relying solely on the binary classification, examine the score distribution to understand the severity gradient of detected anomalies

How to Interpret Results

• Anomaly score close to 1.0 — the point is strongly isolated and very likely a true anomaly; investigate immediately
• Anomaly score 0.6-0.8 — the point is moderately anomalous; review in context of the domain to determine if it warrants action
• Anomaly score around 0.5 — the point is normal; it is embedded within the main data distribution
• Anomaly fraction significantly higher than expected — either the contamination parameter is set too high, or the process has more issues than assumed
• Clusters of anomalies at similar feature values — these may represent a distinct sub-population rather than random outliers; investigate for a systematic cause

Related Standards & References

• Liu, Ting & Zhou (2008), "Isolation Forest", IEEE ICDM — The original paper defining the algorithm and anomaly score implemented here
• ISO 16269-4 — Statistical interpretation of data: detection and treatment of outliers, the international standard framing outlier handling decisions
• ASTM E178 — Standard practice for dealing with outlying observations, covering when removal of flagged points is justified
• NIST/SEMATECH e-Handbook §1.3.5.17 — Classical univariate outlier tests (Grubbs) useful as a cross-check on single-feature data

Frequently Asked Questions

How do I set the contamination parameter?

Contamination is the expected proportion of anomalies in your data. If you know the historical anomaly rate (e.g., 2% defect rate), use that value. If unknown, start with 0.05-0.10 and adjust based on domain expertise. Too low → miss real anomalies; too high → flag normal points as anomalies. Visual inspection of flagged points helps calibrate.

When should I use Isolation Forest vs other outlier detection methods?

Isolation Forest excels with high-dimensional data, large datasets, and when outliers are globally distinct. Use LOF when local density matters (outliers relative to their neighborhood). Use statistical methods (Z-score, IQR) for simple univariate data. Use DBSCAN when you also need clustering. Isolation Forest's O(n log n) complexity makes it the most scalable option.