Maintaining context in a chatbot requires actively managing the finite token budget of the language model across multi-turn conversations. Developers achieve this by combining token budgeting, strategic history handling, and conversation summarisation. Platforms like VideoSDK provide built-in context management and memory features through their AI Voice Agent SDK and Conversational Graph to preserve state reliably. Start by auditing your token allocation and implementing a sliding window strategy to keep your bot remembering.
A forgetting chatbot is more than a frustrating user experience. It directly impacts your bottom line through lost support tickets, abandoned carts, and higher escalation costs to human agents. When a conversational agent asks a user to repeat information they provided just three turns ago, trust evaporates instantly.
Context is a finite resource. Every language model has a maximum token limit per call, and every piece of information you inject consumes a slice of that budget. If you exceed the limit, the model truncates the input, leading to context rot and lost state. Managing this budget is the difference between a chatbot that feels intelligent and one that feels broken. This guide breaks down how to maintain context in a chatbot across three core pillars: token budgeting, history handling, and summarisation.
What is a Chatbot Context Window?
A chatbot context window is the total token budget available for a single model call. It represents the maximum amount of text the model can process at one time, encompassing everything from the initial system prompt to the final generated response.
When you send a request to an LLM, the context window is typically consumed by five contributors. First, the system prompt sets the bot persona and rules. Second, the current user message provides the immediate query. Third, retrieved documents from your knowledge base inject external context. Fourth, the conversation history replays previous turns. Finally, you must reserve response space for the model to generate a meaningful answer.
Understanding these contributors is the first step in learning how to maintain context in a chatbot. If you do not account for all five, you will eventually exceed the window and silently lose critical conversation state.
Token Budgeting: Allocating Space Wisely
Effective context management starts with treating your token limit like a financial budget. You have a fixed amount to spend, and every component of your prompt costs tokens. If you do not track these costs, your chatbot will overspend and crash.
To calculate token usage, you need to estimate the size of each contributor before sending the request to the model. The system prompt is usually a fixed cost. The current user message is variable but typically small. The real variability comes from retrieved documents and conversation history. You must measure the token count of your retrieved passages and your chat history dynamically on every turn.
After calculating the input cost, set a hard ceiling for the response. If your model supports a 4,000 token output, but you only need a short sentence, cap the response tokens at 100. This leaves substantial headroom for your input context. A common mistake is allocating too much space to the response, forcing the model to truncate the conversation history.
Here is a checklist to audit your token allocation:
- Calculate the exact token cost of your system prompt.
- Dynamically measure the token count of retrieved documents before injection.
- Set a strict maximum token limit for the model response.
- Subtract the system prompt, user message, and response ceiling from the total context window to find your available history budget.
- Implement a hard stop if the total input exceeds 90 percent of the window.
Managing Conversation History
Replaying the full chat history on every turn is the default approach for many developers, but it is a flawed strategy. As conversations grow longer, the history consumes the entire token budget. This leads to a phenomenon known as "lost in the middle," where the model pays attention to the beginning and end of the prompt but ignores the middle context. To prevent this, developers must implement structured history handling patterns.
Sliding Window (Retain Most Recent N Turns)
The sliding window strategy involves keeping only the most recent N turns of conversation and discarding everything else. This is highly effective for task-oriented bots where the immediate context is the only relevant context. For example, a food ordering chatbot only needs to know the current item being customized, not the greeting from ten turns ago. The sliding window is simple to implement but risks losing critical early context like user identification details.
Selective Truncation (Drop Low-Signal Turns)
Selective truncation is a smarter approach where you filter out low-signal turns. Low-signal turns include greetings, acknowledgements, and filler conversation. You retain turns where the user provided specific data or made decisions. This pattern requires a lightweight classifier or heuristic to score the importance of each turn before assembling the prompt. It preserves valuable context while freeing up token space.
Priority Tagging (Protect Critical Turns)
Priority tagging involves marking certain turns as protected. For instance, if a user states their account number or a specific constraint, that turn is tagged as high priority. When the history grows too large, the system drops unprotected turns first. VideoSDK's Conversational Graph handles this elegantly by allowing developers to define specific state variables and extractors that persist throughout the conversation, ensuring critical data survives any history truncation.
Summarising and Compressing Older Turns
When you cannot afford to keep raw history but still need the information contained within it, conversation summarisation is the solution. Summarisation compresses older turns into a dense, high-signal paragraph that consumes far fewer tokens than the original text.
There are two primary approaches to summarisation: extractive and abstractive. Extractive summarisation pulls the most important sentences directly from the original text. It is fast and avoids hallucination, but it often produces choppy, disjointed summaries. Abstractive summarisation uses the language model to generate a new, coherent paragraph that captures the meaning of the previous turns. This approach costs more compute but produces a much cleaner context for the model to read.
To implement summarisation, follow this step-by-step workflow:
- Identify the trigger: Set a threshold, such as when history exceeds 50 percent of the available token budget.
- Generate the summary: Send the oldest chunk of conversation history to the model with a prompt instructing it to write a concise summary of the key facts and user intent.
- Replace the original segment: Swap the raw conversation turns with the generated summary in your context window.
- Maintain a rolling summary: Update the summary as the conversation progresses, ensuring it always reflects the latest state.
Retrieval-Augmented Context
Retrieval-augmented generation (RAG) is a powerful way to give your chatbot access to a vast knowledge base. However, external documents compete directly with your conversation history for token space. If you retrieve too much information, you will crowd out the chat history and lose the conversational thread.
The key to managing RAG context is aggressive filtering. Instead of injecting entire documents, narrow your retrieval to the top-k most relevant passages. Use relevance scores from your vector database to set a strict cutoff. If a passage scores below a certain threshold, do not include it.
Furthermore, ensure your retrieval pipeline accounts for the current conversation state. If the user is discussing a specific error code, your retrieval query should be augmented with that context to ensure the returned passages are highly relevant. This minimizes the number of passages needed and maximizes the value of every token spent on external context.
Practical Workflow Diagram
Visualising the per-turn decision process helps clarify how these components fit together. The following diagram illustrates the flow from receiving a user message to assembling the final prompt, ensuring the context window is never exceeded.
Monitoring and Alerting
You cannot manage what you do not measure. Monitoring your chatbot's context usage is critical for preventing silent failures. You need visibility into how your token budget is being spent across real conversations.
Track these key metrics on every turn: total token usage per turn, percentage of the context window filled, and the frequency of history truncation events. If your truncation frequency is high, your users are likely experiencing context rot. If your token usage is consistently low, you might be under-utilizing the model's capacity and could provide richer responses.
Set alerts when usage exceeds 80 percent of the total budget. This gives you a buffer to handle edge cases where a user sends an unexpectedly long message. If you are using a platform like VideoSDK Agent Cloud, leverage the built-in pipeline observability to monitor these metrics without building custom logging infrastructure.
When to Hand Off to a Human or Reset the Session
Sometimes, context cannot be maintained. The conversation might become too complex, or the user might introduce conflicting information that breaks the bot's logic. In these scenarios, you need graceful degradation strategies.
If the context window is completely full and summarisation is losing critical details, the bot should warn the user. A simple message like, "I am losing track of our earlier conversation, could we start fresh?" sets expectations honestly. Offer a fresh start by clearing the history and resetting the state.
For high-stakes applications like customer support, the ultimate graceful degradation is escalation. If the bot detects it has lost context on a critical issue, it should trigger a warm transfer to a live agent. VideoSDK's AI Voice Agent SDK supports seamless call transfers, allowing the bot to pass the current conversation summary to the human agent so the user does not have to repeat themselves.
Definitions Glossary
Context Window: The maximum number of tokens a language model can process in a single call, encompassing the system prompt, user message, history, and response.
Sliding Window: A context management strategy that retains only the most recent N turns of conversation, discarding older history to save tokens.
Extractive Summarisation: A compression method that pulls the most important sentences directly from original text without rewriting the content.
Conversational Graph: A deterministic, graph-based orchestration layer from VideoSDK that manages conversation state and context flow independently of the LLM.
Agent Worker: The Python process in VideoSDK that runs an AI agent, manages the session lifecycle, and handles the STT, LLM, and TTS pipeline.
Key Takeaways
- Context is a finite resource that requires active budgeting across system prompts, history, and retrieved documents.
- Avoid replaying full chat history by using sliding windows, selective truncation, or priority tagging to preserve critical state.
- Implement abstractive summarisation to compress older turns into dense, high-signal context blocks.
- Monitor token usage per turn and set alerts at 80 percent capacity to prevent silent context rot.
- Use graceful degradation strategies like session resets or warm transfers to human agents when context limits are reached.
Conclusion
Disciplined context management is the foundation of a reliable chatbot experience. By mastering token budgeting, strategic history handling, and conversation summarisation, you can build conversational agents that remember user intent across long sessions. Stop letting your chatbot forget critical details. Apply the token audit checklist, monitor your context metrics, and iterate on your history handling patterns. To explore how VideoSDK handles context management and memory natively, check out the VideoSDK AI Agents documentation or the Conversational Graph guide. What are you building with VideoSDK? Drop a comment below, I would love to hear what kind of chatbot or voice agent use case you are working on.
Free $20 Balance for AI Voice Agents & Video Calls
FAQ