EvalScope Documentation

EvalScope is ModelScope's LLM/VLM evaluation framework providing benchmarking, performance testing, arena mode, and visualization.

• GitHub | Docs

CRITICAL: grep references/ for detailed docs. references/benchmarks/ has 153 individual benchmark files with full details, metrics, and usage examples.

Quick Start

pip install evalscope

# Evaluate a model on datasets (CLI)
evalscope eval --model Qwen/Qwen2.5-0.5B-Instruct --datasets gsm8k arc --limit 5

# Evaluate OpenAI-compatible API
evalscope eval \
  --model qwen2.5 \
  --eval-type openai_api \
  --api-url http://127.0.0.1:8801/v1 \
  --datasets gsm8k mmlu

# Performance stress test
evalscope perf --model Qwen/Qwen2.5-0.5B-Instruct --url http://127.0.0.1:8801/v1

from evalscope import run_task, TaskConfig

task_cfg = TaskConfig(
    model='Qwen/Qwen2.5-0.5B-Instruct',
    datasets=['gsm8k', 'arc'],
    limit=5,
    generation_config={'max_tokens': 1024, 'temperature': 0.0}
)
run_task(task_cfg)

Key Topics

Get Started

• references/get_started/installation.md — Installation
• references/getstarted/basicusage.md — Quick start guide
• references/get_started/parameters.md — All CLI/TaskConfig parameters
• references/get_started/faq.md — FAQ

Supported Datasets

• references/getstarted/supporteddataset/index.md — Dataset overview
• references/getstarted/supporteddataset/llm.md — LLM benchmarks (MMLU, GSM8K, HumanEval, etc.)
• references/getstarted/supporteddataset/vlm.md — VLM benchmarks
• references/getstarted/supporteddataset/agent.md — Agent benchmarks
• references/getstarted/supporteddataset/aigc.md — AIGC (T2I) benchmarks
• references/benchmarks/ — 153 individual benchmark docs with metrics, examples, prompt templates
• references/datasets.csv — Full registry (190 datasets: name, backends, tags, metrics, subsets)

Backends

• references/user_guides/backend/index.md — Backend selection guide
• references/userguides/backend/opencompassbackend.md — OpenCompass backend
• references/userguides/backend/vlmevalkitbackend.md — VLMEvalKit backend
• references/userguides/backend/ragevalbackend/ — RAGEval (RAGAS, MTEB, CLIP)

Performance Testing

• references/userguides/stresstest/index.md — Stress test overview
• references/userguides/stresstest/quick_start.md — Quick start
• references/userguides/stresstest/parameters.md — All perf parameters
• references/userguides/stresstest/examples.md — Examples

Advanced

• references/advancedguides/customdataset/ — Custom datasets (LLM, VLM, CLIP, Embedding)
• references/advanced_guides/collection/ — Dataset collection management
• references/advancedguides/addbenchmark.md — Add custom benchmark
• references/advancedguides/custommodel.md — Custom model integration
• references/user_guides/arena.md — Arena (model comparison) mode
• references/user_guides/sandbox.md — Sandbox evaluation
• references/user_guides/service.md — Service mode (online API eval)
• references/get_started/visualization.md — WebUI visualization

Best Practices

• references/best_practice/ — Qwen3, DeepSeek-R1, QwQ, GPT, VLM, T2I evaluation guides
• references/third_party/ — SWE-Bench, BFCL, ToolBench, LongWriter, NeedleHaystack

References

• references/ — 228 doc files from docs/en/
• references/benchmarks/ — 153 benchmark docs (name, dataset_id, metrics, examples, usage)
• references/datasets.csv — Full dataset registry (190 datasets)