Data Analysis Skill

Overview

This skill covers: loading tabular data → exploring structure → computing statistics → creating charts → presenting insights. The primary tools are pandas (data manipulation) and matplotlib + seaborn (visualization).

Quick Decision Table

Step 1 — Load the Data

import pandas as pd
import matplotlib
matplotlib.use('Agg')   # ALWAYS set this before importing pyplot
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_theme(style="whitegrid", palette="muted", font_scale=1.1)

# File is at /mnt/user-data/uploads/<filename>
df = pd.read_csv('/mnt/user-data/uploads/data.csv')   # or read_excel, read_json
print(df.head())
print(df.dtypes)

For Excel files: pd.readexcel(path). For TSV: pd.readcsv(path, sep='\t').

Step 2 — Use the Helper Script (for quick exploration)

The bundled scripts/analyze.py handles common tasks without writing custom code:

# Full description + missing value report
python scripts/analyze.py data.csv --describe --nulls

# Distribution histogram for one column
python scripts/analyze.py data.csv --chart sales --out sales_dist.png

# Scatter plot
python scripts/analyze.py data.csv --scatter units revenue --out scatter.png

# Correlation heatmap
python scripts/analyze.py data.csv --heatmap --out heatmap.png

# Combine flags
python scripts/analyze.py data.csv --describe --correlations --nulls

The script auto-detects CSV, TSV, Excel (.xlsx/.xls/.xlsm), and JSON.

Step 3 — Write Custom Analysis When Needed

When the helper script isn't enough, write Python directly. The patterns you'll use most:

Data cleaning

# Fix dtypes
df['date'] = pd.to_datetime(df['date_col'], errors='coerce')
df['revenue'] = pd.to_numeric(df['revenue'].str.replace(',', ''), errors='coerce')

# Drop duplicates
df = df.drop_duplicates()

# Fill or drop nulls
df['col'].fillna(df['col'].median(), inplace=True)
df = df.dropna(subset=['required_col'])

Aggregation

# Group and aggregate
summary = df.groupby('region').agg(
    total_sales=('sales', 'sum'),
    avg_sales=('sales', 'mean'),
    count=('sales', 'count')
).sort_values('total_sales', ascending=False)

# Pivot table
pivot = df.pivot_table(values='revenue', index='quarter', columns='region', aggfunc='sum')

Time series

df['date'] = pd.to_datetime(df['date'])
df = df.sort_values('date')
monthly = df.resample('ME', on='date')['revenue'].sum()

Step 4 — Create Charts

Always use matplotlib.use('Agg') before importing pyplot. Save with dpi=150, bbox_inches='tight'.

See references/chart-patterns.md for copy-paste code for:

• Histograms, box plots, violin plots
• Scatter plots with trend lines
• Correlation heatmaps, pair plots
• Time series with fill
• Horizontal bar charts (best for categorical comparisons)
• Multi-panel summary figures

Minimal chart template

fig, ax = plt.subplots(figsize=(9, 5))
# ... your plot code ...
ax.set_title('Descriptive Title')
ax.set_xlabel('X label')
ax.set_ylabel('Y label')
fig.tight_layout()
fig.savefig('/mnt/user-data/outputs/chart.png', dpi=150, bbox_inches='tight')
plt.close(fig)

Step 5 — Present Results

Save all output files to /mnt/user-data/outputs/. Use present_files to share them.

For a good analysis response, always include:

1. What the data contains (shape, columns, any quality issues like nulls)
2. Key numbers (means, totals, top categories) — as prose or a markdown table
3. Visual (at least one chart saved to outputs)
4. The main insight — one or two sentences on what the data actually shows

Avoid dumping raw .describe() output at the user without context. Interpret the numbers.

Common Patterns by Use Case

"Analyze my sales data"

1. --describe --nulls to understand shape and quality
2. Histogram of the main metric (sales/revenue)
3. Bar chart of top categories (region, product)
4. If there's a date column, time series of revenue over time
5. Written summary of key findings

"Find what's driving [outcome]"

1. Correlation matrix (--correlations)
2. Scatter plots of top correlated numeric features vs outcome
3. Box plots of outcome split by categorical columns
4. Brief interpretation of which factors seem most predictive

"Compare A vs B"

1. Grouped bar or box plot
2. Summary table (mean ± std for each group)
3. Note effect size, not just direction

"Show me the distribution"

1. Histogram with mean/median lines
2. Key percentiles (25th, 50th, 75th, 95th)
3. Flag outliers if present (values beyond 3σ from mean)

Tips

• Large files (>100k rows): sample for plotting (df.sample(5000)) but compute statistics on the full dataset.
• Always close figures with plt.close(fig) when creating multiple charts to avoid memory buildup.
• Seaborn is great for multi-group comparisons; use raw matplotlib for fine-grained control.
• If a column looks numeric but reads as string, it likely has commas, currency symbols, or spaces — strip them before converting.
• Date parsing: pass format= explicitly if pd.to_datetime is slow on large files.
• Wide tables: use df.T (transpose) to display many columns more readably.