SKILL.md
Experiment Tracking
Track ML experiments, metrics, and models.
Comparison
| Platform | Best For | Self-hosted | Visualization |
|---|---|---|---|
| MLflow | Open-source, model registry | Yes | Basic |
| W&B | Collaboration, sweeps | Limited | Excellent |
| Neptune | Team collaboration | No | Good |
| ClearML | Full MLOps | Yes | Good |
MLflow
Open-source platform from Databricks.
Core components:
- Tracking: Log parameters, metrics, artifacts
- Projects: Reproducible runs (MLproject file)
- Models: Package and deploy models
- Registry: Model versioning and staging
Strengths: Self-hosted, open-source, model registry, framework integrations Limitations: Basic visualization, less collaborative features
Key concept: Autologging for major frameworks - automatic metric capture with one line.
Weights & Biases (W&B)
Cloud-first experiment tracking with excellent visualization.
Core features:
- Experiment tracking: Metrics, hyperparameters, system stats
- Sweeps: Hyperparameter search (grid, random, Bayesian)
- Artifacts: Dataset and model versioning
- Reports: Shareable documentation
Strengths: Beautiful visualizations, team collaboration, hyperparameter sweeps Limitations: Cloud-dependent, limited self-hosting
Key concept: wandb.init() + wandb.log() - simple API, powerful features.
What to Track
| Category | Examples |
|---|---|
| Hyperparameters | Learning rate, batch size, architecture |
| Metrics | Loss, accuracy, F1, per-epoch values |
| Artifacts | Model checkpoints, configs, datasets |
| System | GPU usage, memory, runtime |
| Code | Git commit, diff, requirements |
Model Registry Concepts
| Stage | Purpose |
|---|---|
| None | Just logged, not registered |
| Staging | Testing, validation |
| Production | Serving live traffic |
| Archived | Deprecated, kept for reference |
Decision Guide
| Scenario | Recommendation |
|---|---|
| Self-hosted requirement | MLflow |
| Team collaboration | W&B |
| Model registry focus | MLflow |
| Hyperparameter sweeps | W&B |
| Beautiful dashboards | W&B |
| Full MLOps pipeline | MLflow + deployment tools |
Resources
- MLflow: <https://mlflow.org/docs/latest/>;
- W&B: <https://docs.wandb.ai/>;