Juicebox Observability

Overview

Juicebox provides AI-powered people search and analysis where query performance, dataset ingestion rates, and quota consumption are the primary observability concerns. Monitor analysis completion times to ensure interactive UX, track ingestion pipeline health for data freshness, and watch quota usage to prevent mid-workflow cutoffs. Slow queries or failed ingestions degrade recruiter productivity and data accuracy.

Key Metrics

Instrumentation

async function trackJuiceboxCall(operation: string, fn: () => Promise<any>) {
  const start = Date.now();
  try {
    const result = await fn();
    metrics.histogram('juicebox.api.latency', Date.now() - start, { operation });
    metrics.increment('juicebox.api.calls', { operation, status: 'ok' });
    return result;
  } catch (err) {
    metrics.increment('juicebox.api.errors', { operation, error: err.code });
    throw err;
  }
}

Health Check Dashboard

async function juiceboxHealth(): Promise<Record<string, string>> {
  const searchP95 = await metrics.query('juicebox.api.latency', 'p95', '5m');
  const errorRate = await metrics.query('juicebox.api.error_rate', 'avg', '5m');
  const quota = await juiceboxAdmin.getQuotaUsage();
  return {
    search_latency: searchP95 < 2000 ? 'healthy' : 'slow',
    error_rate: errorRate < 0.01 ? 'healthy' : 'degraded',
    quota: quota.pct < 0.7 ? 'healthy' : 'at_risk',
  };
}

Alerting Rules

const alerts = [
  { metric: 'juicebox.search.latency_p95', condition: '> 5s', window: '10m', severity: 'warning' },
  { metric: 'juicebox.api.error_rate', condition: '> 0.05', window: '5m', severity: 'critical' },
  { metric: 'juicebox.quota.daily_pct', condition: '> 0.85', window: '1h', severity: 'warning' },
  { metric: 'juicebox.ingestion.rate', condition: '< 50/s', window: '15m', severity: 'critical' },
];

Structured Logging

function logJuiceboxEvent(event: string, data: Record<string, any>) {
  console.log(JSON.stringify({
    service: 'juicebox', event,
    operation: data.operation, duration_ms: data.latency,
    result_count: data.resultCount, query_length: data.queryLen,
    // Redact candidate PII — log only aggregate counts
    timestamp: new Date().toISOString(),
  }));
}

Error Handling

Prerequisites

• An approved telemetry schema, sandbox synthetic fixture, redaction policy, source/destination allowlists, suppression controls, retention window, and an owner for alert response.

Instructions

1. Instrument aggregate operational signals only; reject raw queries, contact fields, enrichment values, credentials, and unapproved exports.
2. Validate a sandbox canary, confirm suppression, retention, redaction, and contacts_exported=0, then compare it with the approved baseline.
3. Pause ingestion or downstream delivery on scope, policy, quota, or retention drift and return to the last approved configuration.
4. Keep only redacted aggregate evidence and delete test telemetry after its approved window.

Output

Produce an observability receipt with environment, event classes, aggregate volume/error/latency signals, redaction/suppression/no-export outcomes, alert owner, retention/deletion proof, and rollback reference.

Examples

env=ci-synthetic; events=aggregate-only; latencyp95=within-baseline; suppression=pass; contactsexported=0; retention=24h; cleanup=verified is a valid canary record.

Resources

• Juicebox Dashboard

Next Steps

See juicebox-incident-runbook.