🧠 AI Context Engineering (Part 5): Context Optimization - Give AI What It Needs, Not Everything You Have

We've come a long way in this series. In Part 1, we introduced Context Engineering and why it goes beyond writing better prompts. In Part 2, we explored tokens, context windows and memory. In Part 3, we looked at how AI applications get information through RAG, Tool Calling and MCP. And in Part 4, we explored how AI agents use context and tools to execute multi-step tasks. Now there's a problem. A problem that becomes increasingly important as your AI application becomes more capable. Imagine an agent working on a task for 30 minutes. During that time, it: Reads 20 files. Calls 15 tools. Retrieves dozens of documents. Receives thousands of lines of tool output. Makes several decisions. Encounters and resolves errors. Maintains conversation history. Eventually, you could end up with something like: System instructions + Conversation history + Tool definitions + Retrieved documents + Tool results + Agent state + Memory + Current request ↓ LLM You might think: "Great. The model has everything it needs." But there's a catch. Having everything doesn't necessarily mean having the right context. And that's what Part 5 is about. 🎯 The Real Goal of Context Engineering The goal isn't: ❌ Put as much information as possible into the context window. It's: ✅ Build the smallest useful set of information required for the model to make the current decision correctly. Think of context as a limited working budget. You have a task. You have a certain amount of useful information. Your job is to decide: What should stay? What should go? What should be summarized? What should be retrieved later? What should be cached? What should never enter the context? That's Context Engineering. 🚨 More Context Can Become a Problem Consider this request: "Why is the payment service returning 500?" Suppose your system retrieves: ✓ Payment service logs ✓ Recent deployment ✓ Database errors ✓ Relevant code ✓ Configuration ✓ Recent incidents ✓ Unrelated authentication logs ✓ Old deployment logs ✓ 500 pages of documentation ✓ 20 previous conversations Technically, the model has more information. But did we improve the context? Not necessarily. We introduced: More tokens More latency More cost More irrelevant information More opportunities for conflicting information The useful information can become harder to distinguish from everything else. This is why: Context quality matters more than context quantity. Recent production guidance and research on context management similarly emphasize that long agent trajectories can accumulate irrelevant history and tool output, making selection and compression important engineering problems. 🧩 Five Ways to Optimize Context A practical context pipeline usually has several levers: Information │ ▼ ┌───────────┐ │ Select │ └─────┬─────┘ │ ▼ ┌───────────┐ │ Retrieve │ └─────┬─────┘ │ ▼ ┌───────────┐ │ Compress │ └─────┬─────┘ │ ▼ ┌───────────┐ │ Cache │ └─────┬─────┘ │ ▼ ┌───────────┐ │ LLM │ └───────────┘ Let's understand each one. 1️⃣ Selection — Don't Send What You Don't Need The cheapest token is often the token you never send. Suppose an AI coding assistant is debugging: payment-service Does it need the entire repository? Probably not. It might only need: payment/ ├── handler.go ├── service.go ├── payment.go └── payment_test.go Instead of: Entire Repository ↓ LLM Use: User Question ↓ Identify Relevant Files ↓ Only Relevant Files ↓ LLM This is context selection. And it can happen before compression. Why compress information that shouldn't have been included in the first place? 💡 A Simple Rule Before adding something to context, ask: "Does the model need this information to make the current decision?" If the answer is no, don't send it. This sounds obvious. But it's one of the easiest things to get wrong in real systems. 2️⃣ Retrieval — Fetch Information Just in Time Part 3 introduced RAG. Now we can look at RAG from a different perspective. Instead of loading everything upfront: User Request ↓ Load 1,000 Documents ↓ LLM retrieve information when it is actually relevant: User Request ↓ Understand Need ↓ Retrieve Relevant Information ↓ LLM This is especially useful for large knowledge bases. For example: "What's our refund policy for enterprise customers?" There is no reason to load: HR policies Engineering documentation Marketing documentation Old product specifications The retrieval layer should narrow the information down. 🔍 Retrieval Is a Trade-Off There is no universal rule that says: "Always use RAG." If you have a short document that fits comfortably into the context window and is directly relevant, retrieving chunks may add unnecessary complexity. For example: Document: 2 pages User: "Summarize this document." Just provide the document. But if you have: 10 million documents you obviously cannot send them all. Retrieval becomes valuable. The engineering question is: What information should be available now and what can be fetched later? 3️⃣ Compression — Keep the Meaning, Remove the Bulk Now let's say we really do need the information. But the information is huge. Consider a tool response: GET /orders/12345 { "id": 12345, "customer": {...}, "shipping": {...}, "billing": {...}, "items": [...], "auditLogs": [...], "metadata": {...}, "internalDebugInformation": {...} } Maybe the agent only needs: Order ID: 12345 Status: FAILED Payment Status: DECLINED Failure Reason: Insufficient funds Why send the entire response? We can transform: Large Tool Output ↓ Relevant Information ↓ Compact Context This is context compression. 🗜️ Compression Is Not the Same as Deletion This distinction matters. Suppose we have: 50 messages and replace them with: "The user wants to migrate PostgreSQL from version 14 to 16 and cannot tolerate more than 5 minutes of downtime." We've reduced the number of tokens. But we also potentially lost information. Compression is therefore lossy unless the representation preserves everything required for the task. Recent research on long-horizon agents specifically studies this trade-off: compressing accumulated interaction history can reduce memory requirements while trying to preserve task performance. So don't blindly summarize everything. Instead, identify what must survive. 🧠 What Should Survive Compression? For an agent working on a task, a useful compact state might preserve: Goal Current progress Important decisions Constraints Known failures Important outputs Pending actions Relevant identifiers For example: Before User: We need to migrate the database. Assistant: Let's inspect the schema. Tool: 500 lines... Assistant: There are 3 large tables. Tool: 800 lines... Assistant: We should migrate table A first. Tool: ... Assistant: Migration failed because... ... After compaction Goal: Migrate PostgreSQL database. Progress: Tables A and B analyzed. Decision: Migrate table A first. Constraint: Maximum downtime: 5 minutes. Failure: Previous migration failed due to missing index. Next step: Create the required index and retry. Much smaller. But much more useful. ⚠️ Don't Summarize Blindly This is one of the most important practical lessons. A generic instruction like: "Summarize everything above." doesn't guarantee that the summary will preserve the information your application needs later. Imagine the original context contains: Maximum allowed downtime: 5 minutes If the summary accidentally removes that constraint, the agent might make a technically valid but operationally dangerous decision. So compression should have explicit preservation rules. For example: Always preserve: - User requirements - Hard constraints - IDs - Decisions - Pending actions - Important errors - Security restrictions This is much safer than generic summarization. 4️⃣ Caching — Don't Recompute What Hasn't Changed Now let's talk about something different. Suppose your application repeatedly sends a large, mostly stable piece of context: System Instructions + Tool Definitions + Large Reference Information + User Request If the first three parts rarely change, repeatedly processing them can be wasteful. That's where prompt caching / provider-side caching mechanisms can help, depending on the model provider. Conceptually: ┌─────────────────────────────┐ │ Stable Context │ │ │ │ System Instructions │ │ Tool Definitions │ │ Reference Information │ └──────────────┬──────────────┘ │ CACHE │ ▼ Current User Request │ ▼ LLM The exact caching behavior, eligibility rules, pricing and cache lifetime depend on the provider. So don't assume: "Caching means the model remembers everything." It doesn't. Caching is an infrastructure optimization, not the same thing as memory. 🧠 Memory ≠ Cache ≠ Context This is worth clarifying because these concepts are frequently mixed together. Context Information available to the model for the current inference. Current request + Relevant history + Retrieved information Memory Information intentionally retained for future interactions. User preference + Long-term facts + Previous task state Cache Previously processed/reusable information used to reduce repeated work, depending on the implementation. Stable prompt prefix + Reusable computation They solve different problems. 5️⃣ Structure — Context Isn't Just About What You Send There's another subtle aspect: How you organize the information matters. Imagine giving an engineer this: Database: PostgreSQL Issue: Connection timeout Logs: ... User: ... Instructions: ... Relevant configuration: ... Versus: SYSTEM INSTRUCTIONS TASK CONSTRAINTS RELEVANT EVIDENCE TOOL RESULTS CURRENT STATE USER REQUEST Structured context makes it easier for both developers and models to distinguish different types of information. A useful pattern is to separate: Stable information + Dynamic information + Retrieved information + Current task This also makes your system easier to debug. When something goes wrong, you can inspect which context component caused the problem. 💰 Token Optimization Is More Than "Use Fewer Tokens" Suppose an agent executes 20 steps. And each step sends: 10,000 input tokens That's potentially: 20 × 10,000 = 200,000 input tokens across the run. Now imagine that 6,000 of those tokens were: Old tool outputs Repeated instructions Irrelevant history Duplicate documents You may have spent tokens carrying information the model didn't need. This is why optimizing an agent requires looking at the whole execution, not just one prompt. Recent work on context optimization explicitly treats long-running agents as a lifecycle problem involving accumulated histories, tool outputs, retrieval and repeated model calls. 📊 Think in Terms of a Context Budget Instead of asking: "How much context can my model handle?" ask: "How much context should I spend on this task?" For example: Context Budget: 20,000 tokens System Instructions 2,000 Tool Definitions 3,000 Conversation 2,000 Retrieved Documents 5,000 Agent State 2,000 Current Request 1,000 Safety / Constraints 1,000 -------------------------------- Total 16,000 You still have some room. But if retrieval suddenly returns: 15,000 tokens you have a problem. A context budget makes that visible. 🧹 What Should We Remove First? When context becomes too large, don't immediately summarize everything. Use a hierarchy. First: Remove unnecessary information Irrelevant tool output Old duplicate results Unrelated documents Then: Reduce verbose outputs 10,000-line log ↓ Relevant error lines Then: Retrieve only what matters 100 documents ↓ Top relevant documents Then: Compress history 50 conversation turns ↓ Task state + important decisions Finally: Consider architectural changes For example: Separate agent contexts Store large artifacts externally Retrieve information on demand Split complex tasks This approach avoids using expensive compression when simple filtering would have solved the problem. 🧰 Keep Large Artifacts Outside the Context This is particularly important for coding agents and document-heavy systems. Suppose a tool generates: 50,000 lines of logs Don't necessarily place all 50,000 lines into the next model request. Instead: Tool ↓ Large Artifact ↓ External Storage ↓ Relevant Extract ↓ LLM The model can receive: "Application crashed at 14:32:08. Relevant stack trace: ..." while the complete artifact remains available for retrieval if needed. This pattern is especially useful when dealing with: Logs Large files Database exports Build artifacts Test reports Large API responses 🧠 Context Engineering Is a Selection Problem At this point, we can define the problem more precisely. Every model call has a potential information pool: ALL AVAILABLE INFORMATION │ ┌────────────────┼────────────────┐ ▼ ▼ ▼ Memory RAG Tools │ │ │ └────────────────┼────────────────┘ ▼ SELECT / RANK │ ▼ COMPRESS │ ▼ CONTEXT BUDGET │ ▼ LLM The LLM doesn't need everything your system knows. It needs the right working set for the current decision. That's perhaps the most important idea in this entire series. 🧪 But How Do You Know Your Context Strategy Is Working? This is where many AI projects stop too early. They measure: Did the model produce a good answer? But that's not enough. You should also ask: Retrieval Did we retrieve the right documents? How often are irrelevant documents included? Context How large is the context? Which components consume most tokens? How often are contexts compressed? Agent How many steps does a task require? How often does the agent repeat a tool call? How often does it need to recover from an error? Cost Input tokens per request Output tokens Tokens per successful task Cache utilization where applicable Quality Task success rate Incorrect answers Missing information Failures after compression A context optimization that saves 50% of tokens but causes 20% more task failures isn't necessarily an optimization. ⚖️ Optimize for Cost and Quality This is an important mindset. Don't optimize: Tokens ↓↓↓ at any cost. Optimize: Quality ▲ │ │ ● │ ● │ ● └────────────────► Cost The goal is to find a useful balance between: Accuracy Reliability Latency Cost Context size Sometimes sending more context is absolutely worth it. Sometimes it isn't. Context Engineering is about making that decision intentionally. 🚨 Common Context Engineering Mistakes ❌ 1. "Let's send everything" More context isn't automatically better. ❌ 2. "Let's summarize everything" Compression can remove information that becomes important later. ❌ 3. "The model has a huge context window, so we're fine" A large context window is a capability—not a reason to fill it unnecessarily. ❌ 4. "RAG will solve it" Poor retrieval produces poor context. RAG doesn't magically make irrelevant documents relevant. ❌ 5. "Just truncate the oldest messages" The oldest information isn't necessarily the least important. A previous decision or constraint may still matter. ❌ 6. "We'll optimize tokens later" Context architecture can affect: Cost Latency Retrieval Memory Agent behavior It becomes increasingly difficult to retrofit once the application grows. 🏗️ A Practical Context Pipeline Putting everything together: USER REQUEST │ ▼ ┌──────────────────┐ │ Understand Task │ └────────┬─────────┘ │ ┌────────────────┼────────────────┐ ▼ ▼ ▼ Memory RAG Tools │ │ │ └────────────────┼────────────────┘ ▼ SELECT RELEVANT │ ▼ REMOVE NOISE │ ▼ COMPRESS IF NEEDED │ ▼ CONTEXT BUDGET │ ▼ LLM │ ▼ Evaluate Result │ ┌────────┴────────┐ │ │ Continue Done │ ▼ Next Agent Step This is the mental model I'd keep. 🚀 A Simple Production Checklist Before shipping an AI feature, ask: Context [ ] What information does the model actually need? [ ] Which information is unnecessary? [ ] What is retrieved dynamically? [ ] What is persistent memory? Tokens [ ] Do we have a context budget? [ ] Are large tool outputs filtered? [ ] Are duplicate documents removed? [ ] Are long histories compressed when appropriate? Agents [ ] What is the maximum number of steps? [ ] What happens when a tool fails? [ ] What happens when the model chooses the wrong tool? [ ] Can the agent loop indefinitely? Cost & Performance [ ] Are repeated stable inputs cacheable? [ ] Are large artifacts kept outside the context? [ ] Are we measuring tokens per successful task? [ ] Are we measuring latency? Quality [ ] Does compression preserve important constraints? [ ] Are retrieved documents relevant? [ ] Are tool results trustworthy and validated? [ ] Do we have evaluations for context-related failures? 🧠 The Bigger Picture Let's step back. When we started this series, Context Engineering sounded like: "Writing better context for an LLM." After five parts, we can see that it's much broader. A production AI system may need to decide: What should the model see? ↓ What should it retrieve? ↓ What should it remember? ↓ What should it forget? ↓ What should be compressed? ↓ What should be cached? ↓ What should be fetched only when needed? ↓ What should the model act on? ↓ What should never enter the context? That is a very different problem from simply writing a better prompt. 🎯 The Five-Part Mental Model Let's bring the entire series together. Part 1 — Context Engineering Give the model the right information. ↓ Part 2 — Tokens, Context & Memory Understand the constraints around that information. ↓ Part 3 — RAG, Tools & MCP Retrieve information and connect the model to external capabilities. ↓ Part 4 — AI Agents Let the system use that context and tools across multiple steps. ↓ Part 5 — Context Optimization Continuously decide what information should enter, stay, change or leave the context. And that gives us the bigger picture: USER │ ▼ ┌──────────┐ │ GOAL │ └────┬─────┘ │ ▼ ┌───────────────┐ │ Context Layer │ │ │ │ Memory │ │ Retrieval │ │ Tools │ │ History │ └───────┬───────┘ │ ▼ Select / Rank │ ▼ Compress │ ▼ Context Budget │ ▼ LLM │ ▼ Agent │ ┌─────┴─────┐ ▼ ▼ Tool Done │ ▼ New Context │ └──────────► And that loop can continue for as long as the task requires. 🏁 Final Thoughts One of the biggest shifts happening in AI engineering is that we're moving away from thinking: "How do I write the perfect prompt?" and toward: "How do I build the right information environment for the model at every step?" That's Context Engineering. The model matters. The prompt matters. But so do: retrieval, memory, tools, context selection, compression, caching, state, evaluation and orchestration. And as AI systems become more agentic, this problem becomes even more important. Because the question isn't simply: How much can the model see? It's: What should the model see right now? That is the engineering problem. 📌 Key Takeaways More context isn't automatically better context. Treat the context window as a resource that needs to be managed. Remove irrelevant information before trying to compress it. Retrieve information when it's needed instead of loading everything upfront. Compress long histories carefully and preserve important decisions, constraints and state. Keep large artifacts outside the context when possible and retrieve only the relevant portions. Don't confuse context, memory and caching—they solve different problems. Set practical token budgets for agent runs. Measure task success, not just token reduction. Context optimization is a trade-off between quality, cost, latency and reliability. The goal isn't to give the model everything. The goal is to give the model what it needs for the decision it has to make right now.