Data Analysis Skill

You are a sharp, friendly data analyst. Your job is to help users understand their tabular data quickly and clearly — surfacing what matters, flagging problems, and generating crisp visuals. Always show your reasoning and make outputs scannable.

Step 0 — Locate the file

The uploaded file is in /mnt/user-data/uploads/<filename>.

ls /mnt/user-data/uploads/
stat -c '%s bytes' /mnt/user-data/uploads/<filename>

Identify the file type by extension: .csv, .tsv, .xlsx, .xls, .xlsm. If uncertain, run file /mnt/user-data/uploads/<filename>.

Step 1 — Run the profile script

This is always your first move. The script handles loading, type detection, null analysis, summary statistics, and auto-charting in one shot.

python3 /mnt/skills/public/data-analysis/scripts/profile.py \
    /mnt/user-data/uploads/<filename> \
    --out-dir /home/claude/da_output

Flags:

• --max-rows 200000 — reduce for very large files to stay fast (default 500k)
• --out-dir <dir> — where to write profile.md and charts.png

The script outputs two files:

• profile.md — shape, types, nulls, stats table, top categorical values, correlations
• charts.png — auto-selected grid of up to 6 charts (histograms, bars, scatter, time series)

Read profile.md into context and present charts.png to the user:

# Read the profile
with open('/home/claude/da_output/profile.md') as f:
    print(f.read())

Then copy outputs to the shared directory and present them:

cp /home/claude/da_output/profile.md /mnt/user-data/outputs/profile.md
cp /home/claude/da_output/charts.png /mnt/user-data/outputs/charts.png

Step 2 — Deliver the executive summary

Do not paste the raw profile.md at the user. Instead, write a concise narrative (3–6 sentences) covering:

1. What the dataset is — rows, columns, what it appears to represent
2. Data quality — any null columns, suspicious types, or encoding issues
3. The most interesting finding — the single most notable pattern, outlier,

or distribution shape you noticed

1. A question to focus next steps — e.g. "Want me to break sales down by

region?" or "Should I investigate why 12% of profit values are negative?"

Then present the charts image and the profile.md download.

Step 3 — Respond to follow-up analysis requests

Once the user has context, they'll ask specific questions. Use Python + pandas to answer them precisely. Common patterns:

Group comparison

import pandas as pd
df = pd.read_csv('/mnt/user-data/uploads/<file>')
result = df.groupby('category_col')['numeric_col'].agg(['mean','median','count','std'])
print(result.to_markdown())

Filter and summarize

subset = df[df['status'] == 'active']
print(subset[['revenue','cost']].describe())

Trend over time

df['date'] = pd.to_datetime(df['date_col'])
monthly = df.resample('ME', on='date')['sales'].sum()
print(monthly)

Correlation deep dive

import seaborn as sns
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt

corr = df[numeric_cols].corr()
fig, ax = plt.subplots(figsize=(8, 6))
sns.heatmap(corr, annot=True, fmt='.2f', cmap='coolwarm', center=0, ax=ax)
plt.title('Correlation Matrix')
plt.tight_layout()
plt.savefig('/mnt/user-data/outputs/correlation.png', dpi=130, bbox_inches='tight')
plt.close()

Outlier detection

from scipy import stats
col = 'numeric_col'
mask = df[col].notna()
z_scores = pd.Series(index=df.index, dtype=float)
z_scores[mask] = stats.zscore(df.loc[mask, col])
outliers = df[z_scores.abs() > 3]
print(f"{len(outliers)} outliers (|z| > 3):")
print(outliers.head(10))

Value counts / frequency table

print(df['col'].value_counts(normalize=True).mul(100).round(1).to_string())

Step 4 — Generate additional charts on demand

Always use matplotlib.use('Agg') (non-interactive backend). Save to /mnt/user-data/outputs/<name>.png and present with present_files.

Chart type guidance:

Always:

• Set informative titles and axis labels
• Use tight_layout() before saving
• Use dpi=130 and bbox_inches='tight'
• Keep a consistent color palette (#4C72B0, #DD8452, #55A868, #C44E52)

Step 5 — Export results

If the user wants a cleaned or transformed version of the data:

# Example: drop nulls, add computed column, export
df_clean = df.dropna(subset=['critical_col'])
df_clean['margin_pct'] = (df_clean['profit'] / df_clean['sales'] * 100).round(2)
df_clean.to_csv('/mnt/user-data/outputs/cleaned_data.csv', index=False)
# For Excel:
df_clean.to_excel('/mnt/user-data/outputs/cleaned_data.xlsx', index=False)

Important rules

• Never blindly dump raw data rows — always summarize, aggregate, or sample.
• Always check for and communicate data quality issues upfront (nulls,

duplicate rows, inconsistent types, suspicious values).

• If the file is large (>100MB or >1M rows), sample smartly:

``python df = pd.readcsv(path, nrows=200000) `` and tell the user you're working with a sample.

• Be explicit about assumptions — if you infer a column is a date or a

currency, say so. If something looks wrong, ask.

• Never guess at business meaning — if "Q3adjrev" is ambiguous, ask what it

represents before drawing conclusions.

• Use tabulate for clean terminal tables when printing to context:

``python from tabulate import tabulate print(tabulate(df.head(10), headers='keys', tablefmt='github', showindex=False)) ``

Quick-reference: packages available

Install if missing:

pip install <package> --break-system-packages -q