Data Profiler

What Is Data Profiling?

Data profiling is the process of examining a dataset to collect summary statistics, assess data quality, and understand its structure. Key metrics include central tendency (mean, median), dispersion (standard deviation, range), distribution shape (skewness, kurtosis), and completeness (missing values).

Skewness measures asymmetry: positive skew means a right tail (e.g., income data), negative skew means a left tail. Kurtosis measures tail heaviness relative to a normal distribution: excess kurtosis > 0 (leptokurtic) indicates heavier tails and more outliers; < 0 (platykurtic) indicates lighter tails.

Data profiling is the essential first step in any analysis pipeline. It reveals data quality issues (missing values, outliers, unexpected distributions) before modeling, preventing garbage-in-garbage-out problems.

Formula: Skewness = [n/((n−1)(n−2))] × Σ((xᵢ − x̄)/s)³ Kurtosis = [n(n+1)/((n−1)(n−2)(n−3))] × Σ((xᵢ − x̄)/s)⁴ − 3(n−1)²/((n−2)(n−3)) Q1 = 25th percentile, Q3 = 75th percentile IQR = Q3 − Q1

Example Calculation

Dataset: [2, 4, 4, 4, 5, 5, 7, 9]. n=8, Mean=5.0, Median=4.5, Std=2.14 (sample), Min=2, Q1=4, Q3=5.5 (linear interpolation), Max=9. Skewness ≈ 0.82 (sample-adjusted; right-skewed). Excess kurtosis ≈ 0.94 (sample-adjusted; somewhat heavier tails than normal). Missing = 0.

When to Use This Calculator

• A data analyst performing the first step of any data analysis project — understanding data quality, distributions, and anomalies before modeling
• A database administrator auditing data quality by checking for missing values, outliers, and unexpected distributions across columns
• A researcher preparing data for publication by documenting summary statistics including skewness and kurtosis for each variable
• A machine learning engineer assessing feature distributions to decide on preprocessing steps like normalization, outlier removal, or imputation

Common Mistakes to Avoid

• Skipping data profiling and jumping straight to modeling — this leads to garbage-in-garbage-out; always profile first to catch data quality issues, unexpected distributions, and outliers
• Ignoring skewness and kurtosis — these shape metrics reveal whether parametric methods (which assume normality) are appropriate; high skewness or kurtosis suggests the need for data transformation or non-parametric methods
• Treating all missing values the same way — MCAR (missing completely at random) can be safely deleted, but MAR and MNAR require imputation or specialized methods to avoid bias
• Looking only at individual column statistics without considering relationships between columns — two columns with normal individual profiles can still have a non-linear or unexpected joint relationship

How to Interpret Results

• Skewness near 0 and kurtosis near 0 (excess) indicate a distribution close to normal — parametric statistical methods are appropriate
• Large differences between mean and median indicate significant skewness — the median is more representative of the typical value in this case
• High missing percentage (above 20%) in a column suggests the variable may be unreliable or systematically absent — consider excluding it from analysis or investigating the cause

Related Standards & References

• Tukey, J. W. (1977), Exploratory Data Analysis — foundational text for quartile-based data summaries and exploratory profiling
• Joanes & Gill (1998), 'Comparing Measures of Sample Skewness and Kurtosis', The Statistician — reference comparison of sample skewness and kurtosis estimators
• NIST/SEMATECH e-Handbook of Statistical Methods (Section 1, Exploratory Data Analysis) — reference definitions for quantiles and distribution-shape measures

Frequently Asked Questions

What does negative skewness mean?

Negative skewness (left-skewed) means the left tail is longer and the bulk of values are concentrated on the right. Examples include exam scores on an easy test (most students score high) or failure times for well-designed components. The mean is typically less than the median in left-skewed distributions.

How should I handle missing values detected by profiling?

Options include: deletion (listwise or pairwise — acceptable if missing is < 5% and random), imputation (mean/median for numeric, mode for categorical), or model-based imputation (KNN, regression, MICE). The choice depends on the missing data mechanism: MCAR, MAR, or MNAR. Always report the proportion of missing data.