Concept

• ACI Component Ablation - Measure resolve rate with each ACI piece swapped or removed so human-UI instincts do not silently regress the agent harness.
• Agent Eval Outcome vs Transcript - A trial's transcript is what the agent claims happened; its outcome is the actual resulting environment state — grade the outcome, since a transcript can narrate success that never occurred.
• Agent Grader Types - Code-based, model-based, and human graders trade objectivity and cost against nuance and scale — choose the grader type the check actually needs, not whichever is easiest to wire up.
• Agent Runtime Fault Taxonomy - Runtime faults in agent systems fall into availability, performance, control/routing and data/policy classes, and how well a model can diagnose them from a trace varies sharply by class.
• Application Evaluation Criteria - Score any foundation-model application against four buckets — domain capability, generation quality, instruction-following, and cost/latency — rather than a single vague "quality" number.
• Balanced Eval Problem Sets - Test both cases where a behavior should occur and cases where it shouldn't — a one-sided eval only rewards one-sided optimization.
• Budget-Matched Agent Ablation - Hold total queries or model calls constant when ablating an orchestration component so quality gains are not confused with extra compute.
• Canned Conversations and Model-Mocked User - Two ways to define a test "example" for a multi-call conversational loop — replay a scripted transcript, or have a model play the user's side too.
• Capability vs Regression Evals - Capability evals should start hard and pass rarely, to give a target to climb toward; regression evals should pass almost always, so any drop is a signal something broke.
• Coding Agent Eval Design - Coding agents grade naturally against deterministic test pass/fail signals, with transcript-level heuristics and rubrics layered on top for quality beyond correctness.
• Compare (Model, Program, Strategy) Triples, Not Models Alone - "How do models compare on this benchmark" is underspecified for LM pipelines — report accuracy against a fixed program and compiling/ optimization strategy so the comparison is reproducible.
• Compare Against a Known-Good Baseline - Diagnose an agent system by diffing it against a known-good reference, such as a fault-free trace of the same workflow or a recorded healthy fingerprint calibrated for normal noise.
• Component-Level Harness Ablation - Isolate one evolved harness component at a time to see where an evolution loop's gain actually lives — components own different failure surfaces and interact non-additively, capping the aggregate improvement.
• Computer-Use Agent Eval State Checks - Verify a computer-use agent by inspecting real backend and application state after the fact, not the on-screen confirmation alone, and account for its DOM-versus-screenshot interaction trade-off.
• Conversational Agent Eval Simulation - Grading a conversational agent needs a second LLM playing the user, because the quality of the interaction itself — not just the end state — is part of what's being measured.
• Count Infrastructure Failures As Task Failures - When computing an eval's pass rate, score a trial that dies on an infrastructure exception as a failure rather than discarding it, so the metric can't be inflated by silently dropping hard trials.
• Cross-Validating an LLM Judge Against an Independent Metric - Earn trust in a model-based grader by measuring its case-by-case agreement with a separately computed, non-LLM metric for the same quality, and report that agreement rate and the judge's non-convergence rate alongside its scores.
• Deep Memory Retrieval - Probe long-conversation consistency with questions answerable only from prior sessions — favoring full recall plus paginated search over lossy summary baselines.
• Deterministic Session Replay in CI - Curated slices of recorded agent sessions become golden fixtures that CI replays offline, without models, tools or network, to check that a harness change has not silently altered the context the model is shown.
• Domain-Capability Evaluation - Measure whether a model understands the subject matter at all — usually via domain benchmarks — without mistaking multiple-choice discrimination for open-ended generation skill.
• Episode-Scale Evaluation vs Request-Scale APM - Judge harness changes on episode outcomes, trajectory phases, and budgets — not on sub-second per-tool-call latency habits borrowed from request APM.
• Error Analysis First - Read and code real traces into a failure taxonomy before automating evals, and build evaluators only for failures actually observed, cheapest first.
• Eval Environment Isolation - Start every eval trial from a clean environment — shared state between trials creates correlated failures that look like agent performance and can let an agent cheat on leftovers from a previous run.
• Eval Observation Anti-Leakage - Censor environment observations that overlap the question or reference answer so agents cannot solve the eval by copying leaked gold text.
• Eval Sample Sourcing - Three ways to find enough example problems for evaluation at scale — mine existing records, harvest app usage, or generate samples with an LLM.
• Eval Saturation - A capability eval near 100% pass rate has stopped giving improvement signal, and near-saturation score deltas understate real capability gains because only the hardest remaining tasks can still move the number.
• Eval Suite As Product-Research Interface - A shared eval suite resolves spec ambiguity between people up front and becomes the highest-bandwidth channel through which a product team hands research or engineering a concrete target to optimize against.
• Eval Test Granularity - Test a single model pass or the whole application loop depending on what changed — unit-test isolated prompts, regression-test architecture changes.
• Eval-Driven Development - Define evaluation criteria before building (or before a capability exists), so shipped features stay measurable and future-model bets become trackable instead of wishful.
• Eval/Monitoring Feedback Loop - Couple an agent's offline eval suite and its production telemetry in both directions: eval scores should predict what production shows, and every production failure should come back as an eval case.
• Example Suite - A minimal offline-eval starting point — a handful of example inputs run through the app, with results eyeballed by hand rather than auto-scored.
• Factual Consistency Evaluation - Check whether generated claims are supported by a scoped context (local) or by world knowledge (global), using judges, self-consistency, search augmentation, or textual entailment — not fluency alone.
• Functional Completion Testing - When gold answers are absent, score whether the completion works — parseable, valid tool calls, or passing existing unit tests.
• Generation-Capability Evaluation - For strong models, de-emphasize fluency and coherence checks and invest evaluation budget in factual consistency and safety — the generation failures that still ship as user-visible harm.
• Gold-Standard Matching - Score a completion by comparing it against a trusted reference solution, matching on one carefully chosen aspect rather than the whole output.
• Grader Correctness as Eval Hygiene - A low agent score can mean the grader is broken, not the agent — check the grader before concluding capability is lacking, and design it so passing requires actually solving the task rather than exploiting a loophole.
• Grader Score Combination - Decide up front whether a task passes when every grader agrees or when a weighted blend clears a threshold — the combination rule changes what "passing" actually means as much as the graders themselves do.
• Grounding LLM Assessment in Human Evaluation - Validate an LLM-as-judge pipeline by checking it doesn't widen disagreement beyond what a pool of human assessors already disagrees on among themselves.
• Grounding Quality Dimensions - Citation presence, entailment, inference legitimacy, relevance, and viewpoint coverage must be evaluated separately in grounded agent outputs.
• Harness Cards - Agent results should ship with a structured disclosure of the harness they ran in, plus measures of harness and model variance, so that score differences can be attributed to the right cause.
• Harness Drift Awareness - Prompt templates, user behaviour, and silent provider model swaps all change agent behaviour without code diffs — version and detect them as first-class maintenance risks.
• Inferential-Link Grounding Failure - A synthesis can connect individually supported facts in a relationship that no cited source actually establishes.
• Instruction-Following Evaluation - Measure whether the model obeys your stated constraints — format, length, vocabulary, style, content limits — separately from domain skill or prose quality, and build custom suites for the instructions your product actually uses.
• Intermediate Artifact Evaluation - Evaluating planning artifacts separately reveals whether a workflow fails in research and organization before final generation obscures the cause.
• Judge Reliable Lift Alongside Mean Lift - Report a lower-percentile held-out "reliable lift" and repeatability alongside mean gain when judging harness changes, because search procedures can select brittle edits with large average improvements.
• LLM Self-Assessment Grading Bias - An LLM judging its own model family's output grades less accurately unless it's framed as reviewing a third party's work, not its own.
• LLM SRE Agent Success Depends on Benchmark and Fault Class - Published success rates for SRE and AIOps agents vary widely by benchmark, date and fault class, so an operations agent should propose evidenced diagnoses and act alone only on fault classes it has been measured to diagnose reliably.
• Localise the Root-Cause Step - Errors in tool-using agents cascade from an early root cause, so diagnosis should find the step where the cascade began and feed back a correction from that point.
• MAST Multi-Agent Failure Taxonomy - Annotated traces sort multi-agent failures into system design, inter-agent misalignment and task verification, and targeted prompt or topology fixes give only modest gains.
• Memory Challenge Question Design - Build long-memory eval questions that require the old conversation log — never answerable from persona summaries alone — so retrieval, not priors, is tested.
• Model-Family Independence Rule - Enforce that release verification agents belong to a different model family than the synthesis agents that authored the implementation or tests.
• Offline Evaluation Proxies - Test new prompt or harness ideas against a simulated success signal before exposing them to real users, since there is no live feedback signal offline.
• Offline Example Suites - Run 5–20 representative inputs through prompt assembly, save prompts and completions, and eyeball diffs — the first offline harness before automated scoring.
• Offline Prompt Evaluation - Prototype and score prompt or task changes against proxies or LLM judges before shipping, exercising as much of the real feedforward path as possible.
• Outcome-Based Partial-Credit Grading - Grade what the agent produced or achieved, not the specific tool-call sequence it followed to get there — and give multi-step tasks a continuum of credit instead of one binary pass/fail.
• Owner Validation Decides Success - Agents claim checks and saves they never performed, so whether an operational action succeeded must be decided by an independent owner check recorded as a machine-readable verdict, never by the agent's own report.
• Pass@k and Pass^k - Two opposite ways to summarize an agent's per-trial success rate across k attempts — pass@k asks whether at least one attempt succeeds, pass^k asks whether all of them do — and they tell opposite stories as k grows.
• Per-Task Offline Harness Tests - Collect task I/O examples and run offline prompt harnesses so each workflow step can change safely without regressing expected completions.
• Practise with Fault Injection - Exercise and evaluate operations agents in environments that inject faults under synthetic load, because response paths decay unless practised and each injected fault yields a labelled failure to measure against.
• Preference-Comparison Baseline Selection - Choose what a generated answer gets compared against as carefully as the comparison procedure itself — a mismatched baseline makes the result look objective while actually measuring something else.
• Proxy-Validation Failure Pattern - An agent closes a task on a self-invented or shallow check that resembles success — row counts, file existence, a help flag — instead of reproducing the actual evaluator's exact contract.
• Research Agent Eval Groundedness - Score a research agent's output on groundedness, coverage, and source quality together, since no single check like unit tests exists for open-ended synthesis.
• Self-Generated Test Evaluator - Let a code agent write and run its own unit-test suite so sparse pass/fail signals ground reflection without an external oracle — enabling pass@1 loops.
• Separate the Generator from the Evaluator - A separately tuned, sceptical evaluator that does not share the generator's context is more tractable than self-critique, but it pays only when the task is beyond what the model does reliably on its own.
• SOMA LLM Assessment - Grade free-form completions with Specific questions, Ordinal scales, and Multi-Aspect criteria — as a third-party judge, validated against humans.
• Source Defect Transfer - A grounded generator can faithfully reproduce promotional tone, dominant viewpoints, staleness, and other defects present in retrieved sources.
• Successes Per Million Tokens - A cost-normalized accuracy metric — expected task successes per million tokens spent — for comparing harness configurations that trade accuracy against token cost.
• Test Suite Fidelity - Self-written evaluator tests bound reflection quality: false positives cause premature wrong submits; false negatives are safer because bad tests can be spotted.
• Tool Selection Ablation - Empirically size and prune an agent's tool inventory by measuring performance with and without each tool and replacing chronically misused ones.