Concept

• Agent = Model + Harness - An agent's behaviour comes from the model weights plus all the non-weight runtime code around them (loop, tools, context, safety, orchestration, extensions), and that harness can be engineered independently of the model.
• Autoregressive Generation - A model produces one committed token at a time with no ability to pause, backtrack, or edit — any correction has to be engineered on top, not expected from the raw generation process.
• Chat vs. Completion Tradeoffs - Chat models buy unambiguous turn-taking and instruction compliance at the cost of alignment tax, excess chattiness, and loss of expressive range.
• ChatML Format - A markup language tagging conversation turns with roles — system, user, assistant — that chat models are fine-tuned to complete unambiguously.
• Completion Confidence Scoring - Average a completion's token logprobs into a confidence signal that can drive application behavior, without treating it as an absolute quality score.
• Context Engineering - Deliberate structuring of instructions, examples, retrieved material, and working memory inside the context window for reliable model behaviour.
• Critical Point Detection via Logprobs - Requesting logprobs on the prompt itself, not just the completion, surfaces surprising, anomalous, or high-information-density passages in the input.
• Fine-Tuning Approach Comparison - Full fine-tuning, LoRA, and soft prompting trade training-document volume and duration against how much genuinely new behavior they can teach.
• Fine-Tuning as Prompt Continuation - Treat fine-tuning as baking static instructions and few-shots into weights so prompts shrink — LoRA for format and priors, full FT for new domains.
• Fine-Tuning Decision - Decide whether to fine-tune based on whether you can gather enough correct, well-formatted, on-domain training examples — not on model quality alone.
• GenAI Radical Fragility - Small prompt or model changes produce unforeseeable behavioural shifts, so agent harnesses need continuous offline regression—not one-time validation.
• Generality–Strength Tradeoff - Today's LLMs trade generality (working across any domain) against strength (reliably solving complex tasks); narrowing scope is how you buy the latter.
• Generation Repetition Trap - Once a repetitive pattern starts by chance, continuing it is statistically more likely at every subsequent step than breaking it, since the model has no mechanism for getting bored.
• Hallucination - A model cannot internally distinguish a recalled fact from a plausible invention, so confident-sounding fabrication is a structural property of generation, not an occasional bug.
• Harness as a Performance Lever - Holding the base model fixed, harness design alone materially shifts task completion — and the best harness is model-specific, so it must be re-adapted every time the base model changes.
• Harness Components Encode Stale Assumptions - Each harness component encodes an assumption about something the model cannot do, so components should be re-tested and removed when the model changes, as with dropped context resets and an 80% system-prompt cut.
• Inner and Outer Harness - "Harness" names both the agent runtime (loop, tools, context) and the repository and environment the agent works in (AGENTS.md, linters, tests, observability), and a team running agents usually has to engineer both.
• LLM Agent - An LLM agent is a model that perceives an environment and acts on it through a tool inventory, with the model itself serving as the planner.
• LLM Application Loop - An LLM application is a transformation layer converting a user's problem into a document the model completes, then converting the completion back.
• LLM Inference Latency Metrics (TTFT, TPOT) - LLM latency splits into time to first token (driven by prompt length) and time per output token (paid on every generated token), so prompt and harness choices move the two halves independently.
• LLM Model Selection Criteria - Rank candidate models by intelligence, speed, cost, ease of use, functionality, and special constraints — and keep the choice swappable in code.
• LLM Token and Cost Metrics - Input tokens, output tokens, request rate, and cost per request are the units in which an agent design's cost and rate-limit exposure are paid, so they belong in harness and prompt decisions, not just on the invoice.
• Logprob Classification Calibration - Shift a classifier's per-token logprobs by a constant to match the confidence threshold the application actually needs, not the model's default.
• Logprob Completion Quality Signal - Use average token logprobs (or early-token probabilities) as a relative confidence signal to gate show/warn/retry/escalate decisions in the harness.
• Logprobs Fundamentals - A model computes a full probability distribution over the next token, not just the one it picks, and can return those log-probabilities at no extra cost.
• Model Gateway - A unified interface to self-hosted and commercial models that centralizes access control, fallbacks, logging hooks, and API-change maintenance.
• Model Router - Classify intent and send each query to the right model, human, FAQ, or tool path so simple work stays cheap and out-of-scope work is declined early.
• Model Selection Tradeoffs - Choosing a model for a task balances output quality against cost, latency tolerance, and whether the base model's knowledge or behavior needs changing.
• Output Consistency and Robustness - Because generation is probabilistic, the same or a near-identical prompt can yield meaningfully different completions across calls — a property to manage, not a bug to eliminate.
• Prompt Engineering - Crafting instructions that elicit a desired model outcome without changing weights — the first adaptation lever before heavier techniques like finetuning.
• Prompt Engineering Sophistication Levels - Rank LLM apps from thin wrappers through context injection and tools up to goal-directed agency — each level adds harness obligations, not just wording.
• Prompt Logprob Critical Points - Echo prompt-token logprobs to flag anomalies (typos, odd spans) and high-information passages — thresholds must tolerate model and position drift.
• Provider Selection Guidance - Narrow model choice by picking a provider first, then capability tier, then model size within that provider's lineup — hosted API or self-hosted.
• Response Caching - Exact or semantic reuse of prior model or retrieval results to cut latency and cost — with eviction, cacheability rules, and cross-user leak vigilance.
• Shared Token Prefix Trap - Classification labels that share a starting token have their probabilities compound at that token, which can make a worse option outrank a better one.
• Stop Sequences - Halt generation at known end markers server-side (or cancel a stream) so postscripts never burn tokens after the extractable answer is complete.
• Subtoken Task Avoidance - A model can't examine or manipulate individual letters within its own tokens, so any task requiring that should be handled outside the LLM.
• Temperature and Sampling - A model computes a full probability distribution over the next token, and sampling — controlled by temperature — decides how faithfully generation follows versus deviates from the single most likely choice.
• Thin vs Thick Harness - Model-lab voices argue capability makes scaffolding redundant while harness builders show large harness-only gains, and the resolution is that harness thickness should track model capability and how verifiable the task is.
• Token Boundary Inertness - Tokenizing two concatenated strings together can yield a different token count than tokenizing them separately, so token budgets aren't additive.
• Tokenization Fundamentals - LLMs process text as deterministic multicharacter tokens, not letters, which makes token count — not character count — the real unit of prompt length.
• Truth Bias - A model presented with a false premise tends to continue assuming it's true rather than self-correcting, because self-correcting documents are rare in training data.
• User Problem Domain Complexity - Scope an LLM application by four independent axes — medium, abstraction level, context required, and statefulness — before designing the harness.

Subdirectories

• Context - What goes into the context window, in what order and at what budget? Prompt assembly, element importance and position, compression, snippet formatting and long-context failure modes.
• Agentic Development - How do humans build software with agents? Spec fidelity and ambiguity, spec-driven synthesis, coding-agent patch discipline and human-agent collaboration patterns.
• Evaluation - How do you know it works? Graders, eval suites, pass@k, offline proxies, ablation design and eval hygiene.
• Harness & Tools - What does the agent see and touch? Agent-computer interfaces, tool definitions and inventories, function calling, observation design, episode prompt stacks, approval gates and MCP.
• Instructions & Skills - How do you write standing agent instructions (AGENTS.md and steering files, skills, glossaries) that load cheaply and steer well? Per-call prompt text belongs in prompting/.
• Knowledge & Memory - What lives outside the window, and how does it get back in? Retrieval-augmented generation, retrieval and query construction, agent memory tiers, and knowledge-base architecture and maintenance.
• Optimization - How does the system get improved automatically or improve itself? Textual gradients, DSPy-style compilation, harness evolution and self-adapting contexts.
• Orchestration - How are model calls and agents composed over time? ReAct and Reflexion loops, planning, workflows and routing, multi-agent architectures and delegation.
• Prompting - What text do you write? Instruction wording, prompt anatomy, few-shot examples as format or task signal, document framing, completion boundaries and structured output. Notes about the reasoning procedure a prompt induces belong in reasoning/.
• Reasoning - What reasoning procedure do you induce or run within a call? Chain-of-thought and its variants, decomposition, search, sampling and voting, self-critique and test-time compute, including when the procedure is triggered purely by prompt text. Multi-step pipelines belong in orchestration/.
• Security - How are agents attacked, and how are they defended? Prompt injection, jailbreaks, instruction hierarchy, guardrails and blast-radius containment.