Hypothesis:​

• AI agents are not good at all aspects of engineering work.
• How can we optimize their strengths while keeping humans in the loop for weaknesses?

Questions:​

• What are agents good at?
• What are agents bad at?
• What does engineering SDLC currently look like?
• Where do agents fit into that loop?
• Where can agents bridge gaps

Notes:​

• Where should agentic work start and stop?
• I think using agents for a "first pass" is a mistake. It will introduce unexamined assumptions
  • PRs are hard to review, particularly from agents
  • Context gathering is the most critical piece that needs a human -- someone needs a theory of software!
• HITL work needs humans at the start and end of any process.
  • Agents should accelerate single-human tasks
• That doesn't mean we should maintain current role boundaries
  • Expanding/adjusting roles may be necessary

References:​

Notes:​

• AI Agent skills should not be thought of as workflows
• Instead they should be considered context injectors
• Procedures are acceptable in a skill
• Workflows (defined as deterministic event sequences) are agent harnesses
  • Harnesses are code that manages agentic conversations

Specifically, skills should not define strict requirements. They're too flakey to be relied on for critical rules.

References:​

Necessary research:​

• I need to figure out how agentic hooks work
• I suspect workflows need more than just subagents, but I don't know yet.

Notes:​

• Why does the generator/judge split work?
  • Naive approaches: multiple models
    • Can reduce per-token costs, but not necessarily task cost
  • Separate out context
    • Task executions poison context. It's much harder to ask an agent "is this correct" when it just went through the reasoning process to justify it.
  • Prompt differences.
    • Unique prompts can help isolate the actual intent from incidental requirements present in each prompt
• How to implement
  • Orchestration layer
  • Context isolation

References:​

• Addy Osmani Long Running Agents

Having smarter models means that a lot of the weird tricks that were shared ("You are a brilliant software engineer. Have fun while completing this task") are no longer applicable. At the same time, the more serious research has moved on to solving the general problem of context hydration. (LongRoPE context within LLMs, and RAG hydration in agentic systems).

While models and agents may be better, that doesn't mean prompting strategy is irrelevant. As AI use is becoming more central to professional work, it's more important than ever to know how the get the most out of them.

This post is intended to provide practical guidance for engineers and other AI agent users who already understand software systems but haven't yet built a strong mental model for how agentic LLM workflows actually function. The goal isn't to memorize vocabulary or give research-level insights, it's to help you reason about context tradeoffs in your own workflow so you can get better results more consistently.

If there's anything to take away from this post, it's that the model is only one part of the system. The agent harness shapes your interacting with the model, and the model context is the working memory that makes LLM inference both useful and expensive.

Before talking about cost or quality, though, lets briefly define the harness around the model.

Harnesses: the runtime around the model​

When people talk about using an AI agent, they usually focus on the model — Claude, GPT, Gemini, or whichever brand-name is in play. The application that actually enables users to do useful work with the model is the harness: the control layer that assembles context, exposes functional capabilities, and defines the mode of interaction between the user and model.

All of the concepts related to model context that we'll discuss later (tools, skills, pruning, compaction) live inside this layer.

The standard loop​

The interaction between you and the model follows an incredibly simple cycle:

1. Your request enters the harness.
2. The harness assembles the context window with system instructions, available tool definitions, and the conversation history,.
3. The harness sends this assembled context to the model via a provider API call.
4. The model processes the context and either answers directly, spends time thinking through a reasoning chain, or requests a tool call.
5. If a tool call is requested, the harness executes the tool and feeds the result into the conversation history for another turn of the loop.
6. The loop ends when the model produces a final answer or the harness stops it (due to turn limits, token budgets, or other constraints).

Tools​

Tools are advertised extensions that the model can invoke through the harness. Models are simple text prediction systems; things like searching a codebase, reading and writing files, executing code, querying a database, or making calls to an MCP server require the model to produce a formatted tool call block to request the harness to perform an action on it's behalf.

For the purpose of this discussion, the most important piece is that tool definitions and outputs become part of the working context. Every tool the model is given access to, and the resulting search, file read, or API response the model requests gets appended to the conversation, consuming tokens and bringing more information to compete for the model's attention with everything else in the context window.

Context management​

As the loop runs, the conversation grows and tool results accumulate. The context continues to fill with each call and response. Harnesses manage this growth through two primary mechanisms:

• Pruning removes stale or low-value context. The harness may discard old tool outputs, trim earlier turns of conversation, or drop information that it decides is no longer relevant to the current task.
• Compaction replaces a long history with a shorter retained state. Rather than keeping every turn of a long-running conversation, the harness can summarize the conversation so far into a condensed block that preserves the most important information while discarding the raw back-and-forth.

We'll return to these mechanisms in more detail later; harnesses have a few options for managing context when it grows beyond the model limits, but there are quality implications to both strategies.

Why it matters​

The harness is the layer that controls what the model sees and what it can do; how the harness manages the context impacts the cost and quality of the result almost as much as the underlying model being used. Context is the entire working set of information that is processed on every turn of the loop, and its size, shape, and freshness determine both how much you pay for each request and the quality of the returned response.

Every LLM provider has more than one harness for use with their models, and there are hundreds of third-party options. They all have slightly different strategies for managing context, tooling, and interaction. These nuances can lead to significantly different cost and quality differences even when the workflow and underlying model remains the same.

Costs: context discipline matters even when subscriptions hide it​

Model selection is the most apparent cost when using an AI model. Small or old models are cheaper, while large frontier models are expensive. Rates change depending on the model, but the read driver of billing is token usage. Most providers make token usage feel invisible, but understanding what drives cost helps you reason about efficiency, which directly informs how far your workflow can stretch.

Tokens: the billing unit​

Every interaction with an LLM is measured in tokens; the atomic units the model reads and writes.

Long-running agent loops don't just send a prompt once. Each turn of the harness loop resends the accumulated context — system prompt, conversation history, tool definitions, and all prior tool outputs. It then appends whatever new information was added on the last turn. The model processes the full context on every turn, and you (mostly) pay for it each time.

Token pricing​

Most provider APIs charge less for input tokens (what you send to the model) and more for output tokens (what the model generates back). Pricing also varies by capability tier. A frontier reasoning model costs more per token (both input and output) than a smaller, faster model.

The practical consequence is that a "cheap" model can still become expensive if you feed it bloated, undisciplined context on every turn that leads to low-quality, verbose output. Conversely, a more capable and expensive model used with lean, well-shaped context can end up costing less per task than a cheaper model drowning in noise.

Prompt caching​

Provider APIs offer prompt caching to reduce cost and latency when the leading portion of the prompt (which includes your system prompt and conversation history) stays stable across requests. When you send a request, their system checks if a prefix of your prompt matches something already cached from a recent call. If it does, the cached portion is reused at a significantly reduced cost — often 90% cheaper than processing it fresh.

Cache reuse rewards stable prefixes. Harnesses can take advantage of this by keeping your system prompt, tool definitions, and other often-repeated context at the front of the prompt where they can be matched across requests. Some will even allow you to fork a conversation so you can return to an earlier version with an already-cached prompt. Caching can significantly reduce the cost of long-running conversations, where early parts of the context remain unchanged every round.

When building custom agentic systems, this means prompt construction isn't just about what information the model needs. It's also about structuring that information so the expensive parts stay cached. For harness users, understanding prompt caching can help optimize cost if you're able to re-use the same conversation context for multiple tasks.

While prompt caching significantly reduces the cost of maintaining large contexts, that can hide another issue: large or irrelevant information in the context can degrade a model's ability to reason about the problem at hand, which can result in more output tokens being used to reach a quality output.

That brings us to the second half of the story. The same context decisions that drive token cost also affect whether the model can reliably use the information you give it, which can further drive up costs if more prompting is required to produce a satisfactory output.

Output quality: how context impacts performance​

It's tempting to assume that more context means better answers, but research and practice both show that isn't the case. The larger available context windows in the latest frontier models haven't automatically solve retrieval, salience, or reasoning problems even when all of the needed information is technically present.

Fact retrieval degrades with context length​

Models perform best when relevant facts are easy to locate in the prompt. In a short, focused prompt, the signal-to-noise ratio is high and the model can quickly find and use the information it needs.

As context grows, more competing noise is introduced. Long contexts create competition among instructions, examples, tool outputs, and retrieved documents. The model's attention (the mechanism it uses to determine which tokens are relevant to which) gets stretched across more and more material. Research indicates that even on intentionally simple retrieval tasks, models are less able to identify relevant answers within the context as length increases, often in non-uniform and surprising ways. Information can be present but not effectively usable.

The missing middle​

One of the most well-documented patterns in long-context behavior is the "lost in the middle" effect. It was found that language models tend to pay the most attention to information near the beginning and end of their input context, with significantly degraded performance for information placed in the middle, even for models explicitly designed for long contexts.

Long agent workflows naturally accumulate material in the middle of the context. Early system instructions sit at the top. The most recent user message sits at the bottom. All of the prior turns, tool results, and intermediate reasoning pile up in exactly the region where the model is least likely to attend to it carefully.

Context rot​

Beyond positional effects, there is a broader phenomenon called context rot: the gradual decline in context quality over a long interaction. Context rot isn't just about raw length, it's the accumulation of low-value information (old assumptions, verbose tool outputs, contradictory instructions left over from previous turns).

Anthropic has observed that context must be treated as a finite resource with diminishing marginal returns. Every new token introduced depletes the model's "attention budget" by some amount.

The upshot​

These are all context-construction problems, and there are solutions to all of them available to users. So if long context can weaken retrieval and reasoning, the question becomes: how do you construct context so the right information stays salient? More broadly, how do you get the most useful output from an agent with the lowest cost?

Context building: keeping the working set useful​

Context building is how you respond to the problems described above. The goal is not to fill the agent's context, it's to add only useful context: a working set small enough to keep the model focused and rich enough provide the information necessary for the task at hand.

Think of context building as an engineering activity. It's the process of designing the information environment the model will operate in, and the quality of that environment directly determines the quality of the output.

System prompt​

The system prompt is a durable instruction layer provided by whatever agent is being run by the harness. It defines the model's behavior, identity, and constraints across every turn of the conversation. The system prompt remains stable at the front of the context window, where it benefits from both positional salience (due to the "missing middle") and prompt caching.

A well-crafted system prompt preserves key rules even as the rest of the context drifts. It's where you define the persona, safety constraints, output format expectations, and the behavioral boundaries that define the role of the agent as a tool for the user.

The worst quality issues happen when the system prompt is misaligned with the task at hand, e.g. asking a marketing agent for a bash script, or asking a planner to write tests. At best the agent will fail. At worst, it may struggle against the restrictions in the system prompt and complete the task with a significant token spend and extremely low quality.

Tools​

Tools expand what the model can do — search, read, write, compute, query. Less obviously, tools pull targeted information into context. Tool calls allow the model to pull more data into the context on-the-fly, which enables agents to perform long-running tasks independantly.

Tool design matters for output quality. A well-designed tool returns scoped, well-shaped output that's easy for the model to use — a focused search result, a specific file section, a clean API response. A poorly designed tool dumps verbose, unstructured output into context, recreating exactly the retrieval and attention problems described in the previous section.

Since the model needs to be made aware of the available tools, each one has to be included in the model context. Having too many tools loaded or having tools that aren't applicable to the task at hand will increase the context length and reduce output quality. Limiting the available tool selection can increase the agent's reasoning quality, but can also handicap the agent if it doesn't have access to the tools it needs.

Conversely, tools that are overly specific can also reduce quality. Agents are trained (and therefore more capable with) general tools like file readers and web search. Highly specific tools like documentation search may not be as useful to the agent as a general web search. Always be considerate when giving an agent access to a tool.

Skills​

Skills are modular instruction or workflow packages that can be loaded only when necessary. Instead of stuffing every possible instruction into the system prompt, skills let the agent load specialized guidance only when it determines the guidance is relevant.

Anthropic describes skills as working through progressive disclosure: the harness will only load the name and description of each available skill into the context at first. When the model encounters a task that matches a skill, it instructs the harness to load the full instructions into context. If a skill contains even more detailed reference material, the agent can discover and load those additional files only as needed.

This design directly addresses the context efficiency issues that normal tools introduce. Instead of loading everything into the context at once, specific instructions and guidance stay out of the context until it's actually relevant, reducing background clutter and preserving the model's attention budget for the information that matters right now. In the same way that tools allow the model to load new working data into the context on the fly, skills essentially allow the model to load new tools.

Keep in mind that, while skills are progressively disclosed, the skill descriptions are always given to the agent to determine if they're necessary. Making hundreds of skills available can still add noise to your context, and having multiple similar skills can make it more difficult for an agent to decide that any single one is relevant to the task at hand.

A good skill adds reusable procedure, structure, or domain judgment that makes the agent measurably more effective at a class of tasks. Be skeptical of anything that adds tokens without adding much capability. A skill that injects 2,000 tokens of generic advice on every prompt is a net negative; the harm from increasing context length will almost certainly outweigh any marginal benefit the advice could provide. The best skills are focused, modular, and loaded only when they're relevant.

Conversely, a small skill that is only useful in highly-specific circumstances can also be harmful. The skill description will be included in many contexts where it doesn't provide value, and when the skill is relevant, the model may have been able to accomplish the task without the skill anyway (albiet with an increased token count for those few instances).

The most useful skills are ones that provide the agent context for a task within your personal working domain. Skills that point out which folders contain shared marketing material or that provide example commands for connecting to a database instance are irreplacable — the agent may have no other way of finding out that information. Design best practices, or project layout information is much less valuable. The first is likely already part of the underlying model's training data, and the second can be easily discovered via another tool call.

Agents and prompts​

This is the user-configurable request framing layer where you describe what you want, how you want it, and what constraints apply. In many harnesses, this takes the form of custom agent configurations, persona definitions, or specialized prompt templates.

Better request framing reduces ambiguity and keeps the model focused. Vague instructions force the model to guess at intent or lead it down irrelevant paths that waste tokens and pollute context with noise.

This is also where agent specialization becomes valuable. Rather than using one general-purpose agent with access to every tool and every domain, consider defining focused agents for different kinds of work:

• A code review agent that only has access to file reading and diff tools, keeping its context focused on the code under review.
• A research agent that has access to web search and document retrieval but doesn't need file-editing capabilities (and the associated context bloat)
• A writing agent that works with style guides and reference documents but doesn't need a code execution environment.

Each specialized agent can carry a leaner context with fewer tool definitions, more focused instructions, and less noise.

Subagents​

Subagents are a way to allow an agent to isolate side work in an independent context instead of letting it flood the main conversation thread. When a task requires an investigation, a calculation, or a subsidiary piece of work, the harness can spin up a separate agent with its own context window to handle it. When the task is complete, the results come back without details of the intermediary steps.

Without subagents, every subsidiary task expands the main context — adding tool outputs, intermediate reasoning, and exploratory dead ends to a window that's already trying to track the primary task. With subagents, the main thread stays clean and focused.

Because each subagent starts with a fresh context window, multiple sub-agents can share the same cached context. Because subagents handle isolated workstreams, it's easy to switch to smaller, cheaper models for simpler tasks while reserving more capable models for complex tasks.

Practical approaches: habits that improve outcomes without overengineering​

The mechanisms above are how the harness manages context, but many of the most impactful improvements come from how you interact with the agent on a day-to-day basis (after all, agents are meant to imitate what a person can do).

Be specific​

When you give the model a task, include the goal, the constraints, any prior attempts, the expected output format, and anything you already know about the problem. Don't make the model guess at information you have.

In a long context window, the model has to identify which information is relevant to the current task. Specific, well-structured instructions make the important facts easier to find and harder to bury under noise. Even if the model can technically access the information through it's own tools, providing it upfront (or just telling it to retrieve that data specifically) can dramatically reduce the number of tokens the model will use just to figure out that the data is necessary. In the worst case, allowing a model to learn for itself that a particular approach to the problem won't work will cost more tokens and leave a large chunk of information in the context window that is directly misleading the agent in future rounds of conversation.

Tell the model to clarify ambiguities rather than letting it silently guess. A clarifying question costs a few tokens, but will help prevent the model exploring an incorrect path.

Treat the interaction like mentorship, not magic​

A useful mental model: you're giving a scoped task to a capable but not omniscient collaborator. This collaborator is very fast, has broad general knowledge, and can use tools, but it doesn't know your project history, your preferences, or your unstated constraints unless you tell it.

This means:

• Give concrete, checkable tasks rather than vague delegation. "Refactor the authentication module to use JWT tokens, keeping the existing session-based auth as a fallback" is better than "improve the auth system."
• Break large tasks into smaller steps rather than dumping an entire project specification into one prompt. Smaller tasks mean smaller contexts, which means better reasoning.
• Review and redirect rather than hoping the model gets everything right on the first try. Iteration is normal and expected.

Treat the interaction like magic, not mentorship​

On the other hand, LLMs can be very useful for analyzing from huge amounts of unstructured data. When you have no idea where to even start with an issue, it can be surprisingly helpful to simply dump a ton of possibly-relevant data into the context window and asking what's going on. With this kind of interaction, the goal should be to output some insight rather than to perform a task.

This means:

• Load whatever data that might be relevant, including text files, documentation, or logs. Make sure to inform the agent where the data comes from and what it (probably) represents.
• Be mindful about your own assumptions. If you aren't absolutely certain of something, be careful about telling the agent. Providing speculation may lead the agent down the wrong reasoning chain.
• Check and recheck the resulting insight. Starting a new session and providing a slightly different set of inputs can confirm results. For example, if the agent claims the logs show a bug in one of the source code files you provided, asking the same question with the source code excluded should confirm whether the logs actually point to that source code file, or if your inclusion of the file led the agent to make an assumption.

These kinds of sessions are generally more expensive (particularly if you're regenerating similar large context prompts repeatedly), but they can provide insights that would otherwise take significantly more time. Once an insight is generated, it's generally better to extract those details into a new session rather than continuing, since the large raw data in the context probably can't provide further value for more task-oriented conversations.

Compact or reset when context gets stale​

When a conversation becomes long, repetitive, or stale (or when you've extracted the specific answers you needed from an earlier exchange) consider compacting or starting a new session. Most agent harnesses support this automatically or through an explicit action.

The goal of compaction is to preserve only what still matters: goals, decisions made, constraints, open questions, and source-of-truth facts. Everything else (the exploratory back-and-forth, verbose tool outputs, intermediate reasoning steps, etc.) gets summarized or discarded. Harnesses generally ask the underlying model to perform compaction directly, and differences in the prompt used by the harness can change the quality or focus of the compacted information.

If the context has become stale, (whether because the task has drifted or the underlying data has changed since the last interaction) it may be better to start fresh than to rely on the model to compact the context. Use what you learned over the course of that interaction to provide a more concise and up-to-date context to the new session.

Don't rely on the model to assemble context for you​

The harness does its best to build a useful context window, but you often know important facts that the harness does not. You know which files are relevant. You know what you tried yesterday. You know the business constraint the model has never been told about.

Participate actively in context selection and shaping. Paste in relevant code snippets, link to the right documentation and state your constraints explicitly. The model is good at working with information, but it can only work with what it has.

Even in the agent can discover the information on it's own, directly providing details you know are relevant will save time and tokens and increase quality.

Conclusion: context as a first-class engineering constraint​

The model matters. But much of the behavior users observe — the quality of the output, the relevance of the reasoning, the efficiency of the interaction — comes from the harness and the context it assembles. A frontier model with poorly managed context will consistently underperform against a smaller model with well-curated context.

Better context management improves both cost and output quality. You send fewer tokens, which costs less. The tokens you do send are higher-signal, which means the model reasons more effectively.

Being specific in your prompts, breaking large tasks into smaller ones, compacting stale conversations, choosing the right tools, shaping what the model sees — these are practical skills that are rapidly becoming requirements across the professional world. As agents become more capable and the quality ceiling shifts, what you can do matters a lot less than what you can get a model to do.

Sources and further reading​

• Anthropic, Effective context engineering for AI agents
• Anthropic, Effective harnesses for long-running agents
• Anthropic, Equipping agents for the real world with Agent Skills
• Anthropic, Contextual Retrieval in AI Systems
• Anthropic, Prompt caching
• Anthropic, Compaction
• OpenAI, Prompt guidance
• OpenAI, Function calling
• OpenAI, What are tokens and how to count them?
• Model Context Protocol, What is MCP?
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts (TACL 2024)
• Ling et al., LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion (arXiv 2025)
• Chroma, Context Rot: How Increasing Input Tokens Impacts LLM Performance
• Claude Code docs, Create custom subagents
• Addy Osmani, Agent Skills
• Anthropic, How Claude Code works in large codebases