AQUA Model Evaluation

Use this skill when the user wants to evaluate LLM models on OCI Data Science using AI Quick Actions.

Supported Metrics

Dataset Format

JSONL format with required prompt and completion keys, optional category:

{"prompt": "Summarize this dialog:\nAmanda: I baked cookies...", "completion": "Amanda baked cookies and will bring some for Jerry tomorrow."}
{"prompt": "Translate to French: Hello world", "completion": "Bonjour le monde", "category": "translation"}
{"prompt": "What is 2+2?", "completion": "4", "category": "math"}

The category field dimensions evaluation metrics in the report (e.g., see accuracy per category). When omitted, defaults to "_" (unknown).

Official sample datasets (10 prompts each, math + logic categories):

Python SDK Usage

Import

from ads.aqua.evaluation import AquaEvaluationApp
eval_app = AquaEvaluationApp()

Create Evaluation

from ads.aqua.evaluation.entities import CreateAquaEvaluationDetails

details = CreateAquaEvaluationDetails(
    evaluation_source_id="ocid1.datasciencemodeldeployment.oc1.iad.xxx",  # Deployment OCID
    evaluation_name="llama-3.1-8b-eval-bertscore",
    dataset_path="oci://my-bucket@my-namespace/datasets/eval_data.jsonl",
    report_path="oci://my-bucket@my-namespace/eval-reports/",
    model_parameters={
        "max_tokens": 500,
        "temperature": 0.7,
        "top_p": 0.9,
    },
    shape_name="VM.Standard.E4.Flex",
    block_storage_size=50,
    compartment_id="ocid1.compartment.oc1..xxx",
    project_id="ocid1.datascienceproject.oc1.iad.xxx",
    log_group_id="ocid1.loggroup.oc1.iad.xxx",
    log_id="ocid1.log.oc1.iad.xxx",
    metrics=[
        {"name": "bertscore"},
        {"name": "rouge"},
    ],
)
evaluation = eval_app.create(create_evaluation_details=details)
print(f"Evaluation: {evaluation.id} | State: {evaluation.lifecycle_state}")

Evaluate a Model (not deployment) Directly

You can pass a model OCID instead of a deployment OCID:

details = CreateAquaEvaluationDetails(
    evaluation_source_id="ocid1.datasciencemodel.oc1.iad.xxx",  # Model OCID
    # ... rest of params same as above
)

Evaluate Stacked/Multi-Model Deployment

For stacked or multi-model deployments, specify which model to evaluate:

details = CreateAquaEvaluationDetails(
    evaluation_source_id="ocid1.datasciencemodeldeployment.oc1.iad.xxx",
    model_parameters={
        "max_tokens": 500,
        "temperature": 0.7,
        "model": "llama-3.1-8b-customer-support",  # Target specific model in deployment
    },
    # ... rest of params
)

With Experiment Tracking

details = CreateAquaEvaluationDetails(
    evaluation_source_id="ocid1.datasciencemodeldeployment.oc1.iad.xxx",
    evaluation_name="llama-eval-v2",
    dataset_path="oci://my-bucket@my-namespace/datasets/eval_data.jsonl",
    report_path="oci://my-bucket@my-namespace/eval-reports/",
    model_parameters={"max_tokens": 500, "temperature": 0.7},
    shape_name="VM.Standard.E4.Flex",
    block_storage_size=50,
    experiment_name="llama-evaluations",  # Groups evaluations together
    experiment_description="Llama 3.1 evaluation experiments",
    metrics=[{"name": "bertscore"}, {"name": "rouge"}],
)

List Evaluations

evaluations = eval_app.list(compartment_id="ocid1.compartment.oc1..xxx")
for e in evaluations:
    print(f"{e.display_name} | {e.lifecycle_state}")

Get Evaluation Details

evaluation = eval_app.get(eval_id="ocid1.datasciencemodel.oc1.iad.xxx")

CLI Usage

Create Evaluation

ads aqua evaluation create \
  --evaluation_source_id "ocid1.datasciencemodeldeployment.oc1.iad.xxx" \
  --evaluation_name "llama-eval-bertscore" \
  --dataset_path "oci://my-bucket@my-namespace/datasets/eval_data.jsonl" \
  --report_path "oci://my-bucket@my-namespace/eval-reports/" \
  --model_parameters '{"max_tokens": 500, "temperature": 0.7}' \
  --shape_name "VM.Standard.E4.Flex" \
  --block_storage_size 50 \
  --compartment_id "ocid1.compartment.oc1..xxx" \
  --project_id "ocid1.datascienceproject.oc1.iad.xxx" \
  --metrics '[{"name": "bertscore"}, {"name": "rouge"}]'

List / Get Evaluations

ads aqua evaluation list --compartment_id "ocid1.compartment.oc1..xxx"
ads aqua evaluation get --eval_id "ocid1.datasciencemodel.oc1.iad.xxx"

Interpreting Results

BERTScore

The evaluation produces a Model Catalog entry with:

• Precision: How much of the generated text is semantically represented in the reference
• Recall: How much of the reference text is captured by the generated text
• F1: Harmonic mean of precision and recall

Higher scores = better quality. Scores clustered around the mean indicate consistent performance.

ROUGE

• ROUGE-1: Unigram overlap
• ROUGE-2: Bigram overlap
• ROUGE-L: Longest common subsequence

BERTScore Limitations

• May favor models mirroring its own architecture
• Lacks consideration for sentence-level syntax
• Diminished effectiveness for context beyond word-level (idioms, cultural nuances)
• Not suitable for evaluating coding models on programming tasks

Key Source Files

• ads/aqua/evaluation/evaluation.py — AquaEvaluationApp (create, list, get, load_metrics)
• ads/aqua/evaluation/entities.py — CreateAquaEvaluationDetails, AquaEvalMetrics
• ads/aqua/config/evaluation/evaluationserviceconfig.py — Metric configuration