Data Analysis

Overview

Turn raw data into actionable insight statements. This skill covers the full EDA pipeline: profiling the dataset, exploring distributions, finding correlations, detecting anomalies, and -- critically -- translating statistical findings into business-language narratives tagged with story intents for downstream visualization.

The cardinal rule: statistics are not insights. "Mean revenue is $4.2M" is a statistic. "Revenue grew 34% YoY but growth is decelerating -- Q3 peak was 8% vs. 22% last year" is an insight.

Module 1: Data Profiling

First pass on any dataset. Compute before exploring.

Profiling Output Template

Dataset: {name} ({rowCount} rows x {colCount} columns)
Source:  {source} | Format: {format} | Encoding: {encoding}

Column Summary:
  {name}: {type} | {nullPct}% null | {uniqueCount} unique | min={min} max={max} mean={mean}
  ...

Quality Score: {score}/100
  - Completeness: {completeness}% (columns with <5% nulls)
  - Consistency: {consistency}% (columns with single type)
  - Uniqueness: {uniqueness}% (no unexpected duplicates)

Warnings:
  - Column "X" has 23% nulls -- consider imputation or exclusion
  - Column "Y" has mixed types (78% number, 22% string) -- needs cleaning

Module 2: Descriptive Statistics

What to compute first for every numeric column.

Rule of Thumb: Mean vs. Median

• If |mean - median| / median > 0.1 (10%), report median as the "typical" value
• Always report both -- the gap itself is an insight

Module 3: Distribution Analysis

Normality Quick Check

1. Compare mean to median (>10% gap = non-normal)
2. Check skewness (|skew| > 1 = non-normal)
3. If important: Shapiro-Wilk test (n < 5000) or Anderson-Darling

Module 4: Correlation & Relationship

Simpson's Paradox Awareness

Always check: does the correlation reverse when you split by a categorical variable?

Overall: Ad spend positively correlates with sales (+0.6)
By region: In 3 of 4 regions, correlation is NEGATIVE
Cause: High-spend region has higher baseline sales (confound)

Rule: If a strong correlation exists, segment by the top 2-3 categorical variables and re-check.

Module 5: Segmentation

Group-by patterns for discovering sub-populations.

"So What?" for Segments

For each segment found, answer: "If I could only act on ONE segment, which one and why?"

Module 6: Time-Series Decomposition

Rolling Average Windows

Module 7: Anomaly Detection

Anomaly Protocol

1. Detect and flag -- never auto-remove
2. Investigate: is it a data error, a real outlier, or a different population?
3. Document the decision: kept (real), removed (error), or separated (sub-population)

Module 8: "So What?" Translation (DIKW)

The most important module. Convert statistics into business language.

Insight Statement Template

[WHAT]: {metric} is {value/behavior}
[SO WHAT]: This means {business implication}
[NOW WHAT]: Consider {action or follow-up question}
[STORY INTENT]: {compare|trend|deviation|distribution|relationship|part-to-whole|flow|hierarchy|spatial}
[CHART]: {recommended chart type} because {rationale}

Example Insight Statements

[WHAT]: California's population (39M) is 5x the median state (7.5M)
[SO WHAT]: Resource allocation models using state averages will dramatically under-serve CA
[NOW WHAT]: Segment by population tier, not just state count
[STORY INTENT]: Compare
[CHART]: Horizontal bar (sorted descending) because ranking 50 items with long labels

[WHAT]: Support ticket resolution time has a bimodal distribution (peaks at 2h and 48h)
[SO WHAT]: Two distinct processes exist -- quick fixes and escalated investigations
[NOW WHAT]: Separate the two populations before setting SLA targets
[STORY INTENT]: Distribution
[CHART]: Histogram with two highlighted peaks, or violin plot for visual impact

Module 9: Hypothesis Framework

Structured approach to moving from observation to testable claim.

Anti-Patterns