Concept

• Agent Debugger Trajectory Distillation - Turn millions of raw trajectory tokens into a navigable, per-task root-cause report plus a single benchmark-level overview, so a downstream agent can act on evidence without reading every trace.
• Automated ACI Iteration - Turn recurring failed trajectory patterns into ACI changes automatically — tips, command docs, guardrails — instead of only hand-editing the interface.
• Automated Agent Optimization Taxonomy - Classify automated agent-improvement methods by what evidence the optimizer observes and what surface it is allowed to edit, to see where full-harness evolution sits relative to single-surface methods.
• Brevity Bias - Prompt and context optimizers tend to converge on short, generic instructions that drop domain heuristics, tool-use guidance, and known failure modes, which hurts detail-hungry agent and domain tasks.
• Cluster Failures by Signature Before Fixing - Grouping failed traces by a verifier-grounded failure signature before proposing minimal fixes per group targets recurring mechanisms rather than one-off errors.
• Component Observability - Expose every editable harness piece as its own file at a fixed mount point so each failure pattern localizes to one component and every edit is a reviewable, revertible diff.
• Compound AI System Optimization - Treat multi-component LLM systems as optimizable computation graphs so prompts, solutions, and tools improve via automated feedback—not only craft.
• Context Adaptation - Improve an LLM system by iteratively revising its inputs — system prompt, memory, strategies, evidence — from natural-language feedback on execution, instead of updating model weights.
• Context Adaptation Cost Profile - Token cost of self-adapting contexts splits between an adaptation stage, where prompt evolution is dominated by candidate validation while delta playbooks stay cheap, and a serving stage, where longer playbooks cost more per request unless prefix caching absorbs them.
• Context Adaptation Feedback Dependence - A self-adapting context's gain depends on what feedback its reflection step sees (ground-truth labels, execution signals, or neither), and weak or absent signals can make adaptation worse than none.
• Context Adaptation Noise Robustness - An itemized, delta-updated context degrades gradually under weak or occasionally harmful reflections, falling below baseline only when every step is corrupted, which a harmful-reflector stress test can verify.
• Context Collapse - When an LLM monolithically rewrites an accumulated context at each adaptation step, it can abruptly compress thousands of tokens of learned detail into a short summary, which in an observed case dropped accuracy below the no-adaptation baseline.
• Coverage-Guided Prompt Mutation - Direct test-generation prompts toward unexercised execution paths by dynamically injecting coverage feedback into subsequent iteration rounds.
• Critique–Optimizer Separation - Keep the feedback engine to criticism only; let a separate optimizer step propose the next variable value — do not conflate diagnose and rewrite.
• Curator Delta Operation Contract - Prompt a Curator to emit only new, non-redundant playbook additions as typed JSON operations (type, section, content), never a regenerated playbook, leaving IDs and counters to deterministic system code.
• DSPy Compiler Three Stages - A teleprompter compiles a DSPy program through candidate generation, parameter selection, and optional program-structure search — discrete stages, not a single optimization pass.
• DSPy Module Parameterization - Wrap a signature in a callable module whose LM choice, instructions, and demonstrations are explicit parameters a compiler can search over, so published prompting techniques become swappable, signature-agnostic components.
• DSPy Signature Abstraction - Declare what a text transformation needs to do — typed input/output field names, not a hand-written prompt string — so a compiler can generate and refine the actual prompting strategy.
• DSPy Teacher-Program Distillation - Compile an expensive teacher program first, then bootstrap a cheap student program's demonstrations or finetuning data from the teacher's own traces — even when no labels exist for the student's intermediate steps.
• Evidence-Driven Change Manifest - Attach every harness edit to a self-declared, falsifiable prediction of what it will fix and what it might break, then verify that prediction against the next round's real outcomes.
• Failure-Derived Prompt Tips - Encode recurring failed strategies seen in development trajectories as short imperative tips in the instance prompt to steer later episodes.
• Generator–Reflector–Curator Loop - ACE's three-role architecture for self-improving contexts — a Generator solves tasks, a Reflector distills lessons from its trajectories, and a Curator turns them into delta updates that deterministic code merges into an evolving playbook.
• Grow-and-Refine Context Maintenance - Keep an incrementally growing bullet context compact by appending new bullets, updating existing ones in place, and periodically de-duplicating via semantic embeddings, either proactively or lazily.
• Harness Component-Level Selection - Match a harness fix to the component level whose enforcement strength and scope fits the failure — and if a level doesn't stick after repeated attempts, pivot to a different level rather than iterating harder there.
• Harness Evolution Outer Loop - The rollout-attribute-distill-edit-commit cycle that drives automatic harness evolution one iteration at a time, with multiple rollouts per task so partial-pass signal survives into diagnosis.
• Harness Evolution Transfer and Generalization - Freeze an evolved harness and move it to an unseen benchmark or a different base model without further evolution — the result separates general engineering experience from benchmark-specific tuning.
• Harness Self-Attribution's Regression Blindness - An evolve loop's self-declared fix predictions land far above chance, but its regression predictions barely beat chance — it can justify why an edit helps but not reliably foresee what it will break.
• Incremental Delta Context Updates - Evolve an agent's context by emitting small sets of candidate bullets to merge into the existing store instead of regenerating the whole context each step, keeping update output small and avoiding knowledge loss.
• Instance vs Prompt Optimization - Optimize one test-time solution for a single query, or optimize a prompt that must generalize across many queries — different variables, same feedback loop.
• Instruction vs Demonstration Optimization - Instruction-only optimizers rewrite system prompts; demonstration bootstrappers add few-shot traces — complementary levers that can be combined.
• Itemized Bullet Context - Represent an evolving agent context as a set of small, individually addressable bullets with IDs and helpful/harmful counters, not one monolithic prompt, so updates and retrieval can be localized.
• Minimal Seed Harness - Start an automatic harness-evolution loop from a deliberately minimal seed so every component the loop adds must earn its place against measured rollouts, not inherit unattributed credit from hand-tuning.
• Natural-Language Optimizer Constraints - Append hard format or domain rules as natural-language constraint blocks to the textual optimizer so updates must satisfy them.
• Optimization Variable Role Description - A short role string on each optimizable variable steers what textual-gradient descent is allowed to rewrite — answer-only vs reasoning-plus-answer, and more.
• Optimizer In-Context Examples - Show the textual optimizer worked examples of desired variable properties so rewrites match target style, format, or quality traits.
• Prompt Evolution vs Playbook Accumulation - Choose between evolving whole candidate prompts (GEPA-style) and accumulating itemized insights by delta updates (ACE-style) based on how many granular, long-lived rules the task needs to retain.
• Prompt Optimization Tooling - Automated search and AI-assisted rewriting can explore the prompt space, but multiply API calls and hide template mistakes unless you inspect outputs.
• Raw Traces over Summaries - Optimisers and debuggers that read raw execution traces outperform those given only scores or LLM summaries, because summaries compress away the diagnostic detail needed to find causes.
• Reflector Diagnostic Prompt Contract - Prompt a dedicated Reflector to diagnose one attempt against ground truth and environment feedback (plus unit-test reports in the AppWorld variant), returning JSON diagnostic fields and, in the FINER variant, per-bullet helpful/harmful/neutral tags.
• Regression Control Inside the Loop - Harness optimisers compound only when edits that lose previously solved tasks are rejected inside the search loop; without that gate, gains transfer negatively or stall on re-optimisation.
• Self-Adapting Context Evaluation Protocol - Evaluate a system that rewrites its own context by predicting before updating on each online sample, using the same model for every role, and reporting with and without labels against a no-adaptation baseline.
• Self-Modifying Harness Controllability - Scope a harness-editing agent's write access to the harness workspace alone so every measured gain is attributable to a harness edit, not a disabled check or a swapped model.
• Strong-to-Weak Prompt Optimization - Optimize prompts for a cheaper inference model with feedback from a stronger model so you pay a fixed optimization cost and keep inference cheap.
• Test-Time Code Refinement - Treat candidate code as an instance-optimization variable scored by local tests plus an LLM evaluator that critiques failures without rewriting code.
• Test-Time Solution Refinement - At inference, treat the current answer as an optimizable variable and iterate self-supervised LLM critiques via textual gradient descent before committing.
• Textual Gradient Descent - Update a graph variable by asking an LLM to incorporate textual-gradient criticisms — the natural-language analogue of an SGD step.
• Textual Gradients - Natural-language criticisms of a variable’s role in a computation graph, produced by an LLM as the analogue of a numerical gradient.
• Textual Optimizer Momentum - Keep a window of past variable values in the optimizer context so updates see trajectory history, not only the latest textual gradients.