Prometheus API Skill

Query Prometheus monitoring systems via HTTP API at /api/v1.

Quick Reference

Instant Query

curl 'http://<prometheus>:9090/api/v1/query?query=<promql>&time=<timestamp>'

Range Query

curl 'http://<prometheus>:9090/api/v1/query_range?query=<promql>&start=<ts>&end=<ts>&step=<duration>'

Response Format

All responses return JSON:

{
  "status": "success" | "error",
  "data": <result>,
  "errorType": "<string>",
  "error": "<string>",
  "warnings": ["<string>"]
}

HTTP codes: 400 (bad params), 422 (expression error), 503 (timeout).

Query Endpoints

Metadata & Status Endpoints

Admin Endpoints (require --web.enable-admin-api)

Common PromQL Patterns

# Rate of counter over 5m
rate(http_requests_total[5m])

# Sum by label
sum by (job) (rate(http_requests_total[5m]))

# Percentile from histogram
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Filter by label
up{job="prometheus", instance=~".*:9090"}

# Increase over time
increase(http_requests_total[1h])

# Average over time range
avg_over_time(process_cpu_seconds_total[5m])

Result Types

• vector: [{"metric": {...}, "value": [timestamp, "value"]}]
• matrix: [{"metric": {...}, "values": [[ts, "val"], ...]}]
• scalar: [timestamp, "value"]
• string: [timestamp, "string"]

Scripts

Query script: scripts/prom_query.py

# Instant query
python scripts/prom_query.py http://localhost:9090 'up'

# Range query
python scripts/prom_query.py http://localhost:9090 'rate(http_requests_total[5m])' \
  --start '2024-01-01T00:00:00Z' --end '2024-01-01T01:00:00Z' --step '1m'

# Output: table, json, csv
python scripts/prom_query.py http://localhost:9090 'up' --format table

Health check: scripts/prom_health.py

python scripts/prom_health.py http://localhost:9090

Detailed Reference

For complete API documentation: [references/apireference.md](references/apireference.md)

For PromQL functions: [references/promqlfunctions.md](references/promqlfunctions.md)

Gotchas

• rate() over a counter that resets too often: math is correct but meaningless — use increase() and divide by interval explicitly when counters don't survive scrapes.
• up{} per-target gauge: a flaky target shows up=0 but doesn't trigger alerts unless for is met. Set short for for liveness, long for noise.
• Recording rules evaluate at fixed interval; missed evaluations don't backfill — gaps in the recording series during incidents.
• Federation match[] parameter requires ALL matchers to match — an empty matcher returns no series, which looks like a working query with no data.
• Stale-marker semantics: a series stops being scraped → stale marker after 5 min by default → queries see "no data" not "0". Affects alerts on absent().
• Service Discovery + relabel_config: a bad regex in keep action silently drops all targets — verify with /api/v1/targets after each config change.