Maintain Context in Chatbots with AI

Step-by-step guide to building AI Voice Agents that maintain context in chatbots using VideoSDK.

Introduction to AI Voice Agents in how to maintain context in a chatbot

What is an AI

Voice Agent

?

An AI

Voice Agent

 is a software entity designed to interact with users through voice commands and responses. It employs technologies like Speech-to-Text (STT), Language Models (LLM), and Text-to-Speech (TTS) to process and generate human-like conversations. These agents are capable of understanding spoken language, processing it to derive meaning, and responding appropriately, making them ideal for applications in customer service, virtual assistants, and more.

Why are they important for the how to maintain context in a chatbot industry?

In the chatbot industry, maintaining context is crucial for delivering coherent and meaningful interactions. AI Voice Agents play a significant role by ensuring that conversations are fluid and contextually aware. They can remember previous interactions, allowing them to respond in a way that feels natural and relevant to the user. This capability is essential for applications like customer support, where understanding the user's history can lead to better service.

Core Components of a

Voice Agent

  • Speech-to-Text (STT): Converts spoken language into text.
  • Language Models (LLM): Processes the text to understand and generate responses.
  • Text-to-Speech (TTS): Converts the generated text back into spoken language.

What You'll Build in This Tutorial

In this tutorial, you will learn how to build an AI

Voice Agent

 using the VideoSDK framework. The agent will be capable of maintaining context during chatbot interactions, ensuring seamless and coherent conversations.

Architecture and Core Concepts

High-Level Architecture Overview

The architecture of an AI

Voice Agent

 involves several key components working together to process user input and generate responses. The process begins with the user speaking into a microphone. The audio is captured and sent to the Speech-to-Text (STT) module, which converts it into text. This text is then processed by a Language Model (LLM) to generate a response, which is finally converted back into speech by the Text-to-Speech (TTS) module.
Diagram

Understanding Key Concepts in the VideoSDK Framework

  • Agent: The core class that represents your bot, responsible for handling interactions.
  • CascadingPipeline: Manages the flow of audio processing through STT, LLM, and TTS. Learn more about the

    Cascading pipeline in AI voice Agents

    .
  • VAD & TurnDetector: These components help the agent determine when to listen and when to speak, ensuring smooth interaction. Explore the

    Turn detector for AI voice Agents

     for more details.

Setting Up the Development Environment

Prerequisites

Before starting, ensure you have Python 3.11+ installed and a VideoSDK account. Sign up at app.videosdk.live to access the necessary API keys.

Step 1: Create a Virtual Environment

Create a virtual environment to manage your project dependencies:
1python -m venv venv
2source venv/bin/activate  # On Windows use `venv\\Scripts\\activate`
3

Step 2: Install Required Packages

Install the required packages using pip:
1pip install videosdk python-dotenv
2

Step 3: Configure API Keys in a .env file

Create a .env file in your project directory and add your VideoSDK API keys:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

To build your AI Voice Agent, we'll start by presenting the complete code block and then break it down into manageable parts for a detailed explanation.
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "You are a Context-Aware Conversational AI Agent designed to assist users in maintaining context during chatbot interactions. Your primary role is to ensure seamless and coherent conversations by remembering past interactions and using them to inform current responses. You should be able to:\n\n1. Recall previous user inputs and responses to maintain context.\n2. Provide contextually relevant answers based on the conversation history.\n3. Handle multi-turn conversations without losing track of the topic.\n4. Offer suggestions or clarifications when the context is unclear.\n\nConstraints and Limitations:\n- You are not capable of making decisions or providing advice outside the scope of maintaining conversation context.\n- You must inform users that you are a conversational AI and not a human.\n- You should not store personal data beyond the session duration and must comply with privacy regulations.\n- You cannot provide context for conversations that occurred outside the current session."
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=DeepgramSTT(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=SileroVAD(threshold=0.35),
32        turn_detector=TurnDetector(threshold=0.8)
33    )
34
35    session = AgentSession(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63

Step 4.1: Generating a VideoSDK Meeting ID

To interact with your agent, you'll need a meeting ID. Use the following curl command to generate one:
1curl -X POST "https://api.videosdk.live/v1/meetings" \
2-H "Authorization: YOUR_API_KEY" \
3-H "Content-Type: application/json"
4

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class extends the Agent class, providing custom behavior for entering and exiting sessions. It uses predefined instructions to maintain conversation context.
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6

Step 4.3: Defining the Core Pipeline

The CascadingPipeline orchestrates the flow of audio data through various processing stages:
  • STT (DeepgramSTT): Converts audio to text.
  • LLM (OpenAILLM): Processes the text to generate responses.
  • TTS (ElevenLabsTTS): Converts responses back to audio.
  • VAD (SileroVAD) & TurnDetector: Manage when the agent listens and speaks. The

    Silero Voice Activity Detection

     is crucial for detecting voice activity accurately.
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8

Step 4.4: Managing the Session and Startup Logic

The start_session function initializes the agent session and connects it to the VideoSDK framework. The make_context function creates a job context with room options. Learn more about

AI voice Agent Sessions

 to manage these interactions effectively.
1def make_context() -> JobContext:
2    room_options = RoomOptions(
3    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
4        name="VideoSDK Cascaded Agent",
5        playground=True
6    )
7
8    return JobContext(room_options=room_options)
9
10async def start_session(context: JobContext):
11    agent = MyVoiceAgent()
12    conversation_flow = ConversationFlow(agent)
13    pipeline = CascadingPipeline(
14        stt=DeepgramSTT(model="nova-2", language="en"),
15        llm=OpenAILLM(model="gpt-4o"),
16        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
17        vad=SileroVAD(threshold=0.35),
18        turn_detector=TurnDetector(threshold=0.8)
19    )
20    session = AgentSession(
21        agent=agent,
22        pipeline=pipeline,
23        conversation_flow=conversation_flow
24    )
25    try:
26        await context.connect()
27        await session.start()
28        await asyncio.Event().wait()
29    finally:
30        await session.close()
31        await context.shutdown()
32
33if __name__ == "__main__":
34    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
35    job.start()
36

Running and Testing the Agent

Step 5.1: Running the Python Script

To run your AI Voice Agent, execute the following command in your terminal:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Once the agent is running, you will see a playground link in the console. Use this link to join the session and interact with your AI Voice Agent. You can test its ability to maintain context by engaging in multi-turn conversations.

Advanced Features and Customizations

Extending Functionality with Custom Tools

The VideoSDK framework allows you to extend your agent's functionality by integrating custom tools. This can include additional plugins or custom logic tailored to specific use cases.

Exploring Other Plugins

While this tutorial uses specific plugins for STT, LLM, and TTS, the VideoSDK framework supports various other options. You can experiment with different plugins to find the best fit for your application.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure your API key is correctly configured in the .env file. Double-check the key for typos or missing characters.

Audio Input/Output Problems

Verify that your microphone and speakers are properly connected and configured. Check your system settings to ensure the correct audio devices are selected.

Dependency and Version Conflicts

If you encounter issues with package dependencies, ensure all required packages are installed and up to date. Use a virtual environment to manage dependencies effectively.

Conclusion

Summary of What You've Built

In this tutorial, you've built an AI Voice Agent capable of maintaining context in chatbot interactions using the VideoSDK framework. This agent can handle multi-turn conversations and provide contextually relevant responses.

Next Steps and Further Learning

To further enhance your AI Voice Agent, consider exploring additional plugins and customizations. The VideoSDK documentation provides extensive resources for building more sophisticated and capable agents.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ