Agent Evaluation Methodology

Overview

Evaluating AI agents is fundamentally different from evaluating traditional software. Agents are non-deterministic, may take multiple valid paths to a solution, and produce outputs that span multiple quality dimensions simultaneously. This guide provides a structured methodology for measuring, comparing, and improving agent performance.

Core principle: Judge results, not exact paths. An agent that takes a different route to the correct answer is not wrong -- it may even be better.

When to Use

• Testing a new agent or command before deployment
• Comparing two prompt variants to determine which performs better
• Regression testing after modifying an agent's system prompt or tools
• Establishing baseline quality metrics for continuous monitoring
• Debugging inconsistent agent behavior across different inputs

Core Evaluation Challenges

Agent evaluation must account for three fundamental difficulties:

1. Non-Determinism

The same agent with the same input may produce different outputs across runs. Temperature, sampling, and internal state contribute to variance.

Implication: Single-run evaluation is unreliable. Measure across multiple runs (minimum 3-5) and report distributions, not single scores.

2. Multiple Valid Paths

For most real-world tasks, there is more than one correct approach. A code review agent might organize findings by file or by severity -- both are valid.

Implication: Rubrics must evaluate outcome quality, not path similarity. Avoid comparing against a single "gold standard" output.

3. Composite Quality

Agent output quality is not a single number. An output can follow instructions perfectly but be poorly organized, or be well-written but miss half the requirements.

Implication: Use multi-dimensional rubrics that score each quality aspect independently.

Multi-Dimensional Evaluation Rubric

Score agent outputs across five weighted dimensions:

Weighted Score Calculation

Final Score = (Instruction × 0.30) + (Completeness × 0.25) +
              (Efficiency × 0.20) + (Reasoning × 0.15) +
              (Coherence × 0.10)

Score interpretation:

Dimension-Level Rubric Example

Instruction Following (Weight: 0.30):

Evaluation Methodologies

LLM-as-Judge: Direct Scoring

Use a separate LLM instance to score agent outputs against the rubric. This is the most scalable approach for automated evaluation.

Setup:

You are an expert evaluator. Score the following agent output against
each rubric dimension on a 1-5 scale.

## Task Description
[Original task given to the agent]

## Agent Output
[Output to evaluate]

## Rubric
[Full rubric with dimension descriptions and scale definitions]

## Instructions
For each dimension:
1. Quote specific parts of the output that inform your score
2. Explain your reasoning in 1-2 sentences
3. Assign a score (1-5)
4. Calculate the weighted final score

Advantages: Scalable, consistent, can evaluate hundreds of outputs per hour. Limitations: Judge LLM has its own biases; requires calibration.

LLM-as-Judge: Pairwise Comparison

Instead of absolute scoring, present two agent outputs and ask which is better. This is more reliable for detecting subtle quality differences.

Setup:

You are an expert evaluator. Compare these two agent outputs for the
same task and determine which is better.

## Task Description
[Original task]

## Output A
[First agent's output]

## Output B
[Second agent's output]

## Instructions
For each rubric dimension:
1. Compare both outputs
2. State which is better and why
3. Assign a preference: A >> B, A > B, A = B, B > A, B >> A

Finally, give an overall winner with reasoning.

Advantages: More reliable for close comparisons; eliminates absolute scale calibration issues. Limitations: Only provides relative ranking; does not tell you if both outputs are bad.

Combined Approach

For thorough evaluation, use both methods:

1. Direct scoring to establish absolute quality baselines
2. Pairwise comparison to validate that improvements are real
3. Cross-validation by running the same evaluation with different judge prompts

Bias Mitigation

LLM judges carry systematic biases that can distort evaluation results. Identify and mitigate each one.

Position Bias

Problem: The judge favors whichever output appears first (or last) in the comparison prompt.

Mitigation:

• Run each pairwise comparison twice with swapped positions (A|B and B|A)
• Only count a preference if the judge is consistent across both orderings
• Discard results where the judge flips preference based on position

Length Bias (Verbosity Bias)

Problem: The judge favors longer, more verbose outputs regardless of information density.

Mitigation:

• Add explicit instruction: "A concise output that covers all requirements is preferable to a verbose output with padding"
• Include examples where the shorter output is scored higher
• Normalize scores by information density, not raw token count

Self-Enhancement Bias

Problem: When the judge model is the same as (or similar to) the agent model, it may favor outputs that match its own style.

Mitigation:

• Use a different model as judge when possible
• Include diverse example outputs in the judge prompt to calibrate expectations
• Cross-validate with human evaluation on a sample

Authority Bias

Problem: The judge defers to outputs that cite sources, use confident language, or invoke authority, even when the content is incorrect.

Mitigation:

• Instruct the judge to verify factual claims, not just accept them
• Include test cases where a confident but wrong output should score lower than a hedged but correct one
• Separate "confidence" from "correctness" in the rubric

Test Set Design

The quality of your evaluation is bounded by the quality of your test set.

Complexity Stratification

Design test cases across three complexity levels:

Representative Sampling

Ensure your test set covers:

• Input variety: Different phrasing, different levels of detail in user requests
• Domain spread: If the agent works across multiple domains, test all of them
• Edge cases: Empty input, malformed input, conflicting requirements, very large inputs
• Failure modes: Inputs designed to trigger known weaknesses

Test Case Template

## Test Case: [ID]-[Brief Description]

**Complexity:** Simple | Moderate | Complex
**Category:** [Domain or feature being tested]

**Input:**
[Exact input to the agent]

**Context:**
[Any additional context the agent receives: files, history, etc.]

**Expected Behavior:**
- [Requirement 1]
- [Requirement 2]
- [Requirement 3]

**Evaluation Notes:**
[What to look for in the output; common failure modes for this case]

Rubric Generation

For a new agent, generate a tailored rubric before running any evaluations.

Step-by-Step Rubric Creation

1. Define the task domain. What does this agent do? What does a good output look like?

2. Identify quality dimensions. Start with the five standard dimensions, then add domain-specific ones if needed (e.g., "Security Correctness" for a security review agent).

3. Write level descriptions. For each dimension, define what a score of 1, 3, and 5 looks like. Fill in 2 and 4 by interpolation.

4. Add characteristics. For each level, list 2-3 observable characteristics that a judge can check for.

5. Include examples. Provide at least one example output for a score of 5 and one for a score of 2. This anchors the judge's calibration.

6. Document edge cases. Describe scenarios where scoring is ambiguous and provide guidance.

Example Domain-Specific Dimension

Security Correctness (for a security review agent):

Evaluation Workflows

Workflow 1: Testing a New Agent

1. Create test set (10-20 cases across complexity levels)
2. Run agent on all test cases (3 runs each)
3. Score all outputs using direct scoring
4. Compute per-dimension and overall averages
5. Identify dimensions scoring below 3.5
6. Refine agent prompt targeting weak dimensions
7. Re-run and compare using pairwise evaluation
8. Iterate until all dimensions reach target threshold

Workflow 2: Comparing Prompt Variants

1. Select 10 representative test cases
2. Run variant A on all cases (3 runs each)
3. Run variant B on all cases (3 runs each)
4. Run pairwise comparison (with position swapping) on all pairs
5. Calculate win rate for each variant
6. Check for statistical significance (min 60% consistent preference)
7. Analyze dimension-level differences for insights

Workflow 3: Regression Testing

1. Maintain a fixed test set of 15-20 cases (do not modify between runs)
2. Run baseline agent and store scores
3. After any agent modification, run the same test set
4. Compare per-dimension scores against baseline
5. Flag any dimension that drops more than 0.3 points
6. Investigate and fix regressions before deployment

Workflow 4: Continuous Quality Monitoring

1. Sample 5% of production agent invocations
2. Score sampled outputs using automated direct scoring
3. Track weekly averages per dimension
4. Alert if any dimension drops below threshold
5. Monthly: review trend data and identify systematic patterns

Metrics Reference

Anti-Patterns