Master Multi-Turn Conversations with AI Voice Agents

Build AI Voice Agents for multi-turn conversations with VideoSDK. Step-by-step guide with code examples.

Introduction to AI Voice Agents in Multi-Turn Conversation Handling

What is an AI

Voice Agent

?

An AI

Voice Agent

 is a sophisticated software application designed to interact with users through voice commands and responses. These agents are capable of understanding natural language, processing speech, and responding in a human-like manner. They utilize technologies like Speech-to-Text (STT), Text-to-Speech (TTS), and Language Models (LLM) to facilitate seamless interactions.

Why are they important for the multi-turn conversation handling industry?

In industries where customer interaction is key, such as healthcare, customer service, and personal assistance, AI Voice Agents play a crucial role. They handle complex, multi-turn conversations, maintaining context across interactions to provide coherent and relevant responses. This capability is essential for applications like virtual health assistants, where understanding and maintaining the context of previous interactions can significantly enhance user experience.

Core Components of a

Voice Agent

  • Speech-to-Text (STT): Converts spoken language into text.
  • Language Models (LLM): Processes text to understand and generate responses.
  • Text-to-Speech (TTS): Converts text responses back into spoken language.

What You'll Build in This Tutorial

In this tutorial, we will build an AI

Voice Agent

 using the VideoSDK framework. This agent will be capable of handling multi-turn conversations, particularly in the healthcare domain, providing users with information and assistance while maintaining context throughout the interaction.

Architecture and Core Concepts

High-Level Architecture Overview

The architecture of our AI

Voice Agent

 involves several key components working in harmony. The agent listens to user input, processes it through a pipeline of speech and language models, and responds appropriately. Here’s a high-level sequence of operations:
Diagram

Understanding Key Concepts in the VideoSDK Framework

  • Agent: This is the core class that represents the bot. It handles the interaction logic and manages the conversation flow.

  • Cascading Pipeline in AI voice Agents

    :     This pipeline handles the flow of audio processing, converting speech to text, processing it, and then converting the response back to speech.
  • VAD &

    Turn Detector for AI voice Agents

    :     These components help the agent determine when to listen and when to speak, ensuring smooth conversation flow.

Setting Up the Development Environment

Prerequisites

To get started, ensure you have Python 3.11+ installed and a VideoSDK account. You can sign up at app.videosdk.live.

Step 1: Create a Virtual Environment

Create a virtual environment to manage dependencies for your project:
1python -m venv venv
2source venv/bin/activate  # On Windows use `venv\Scripts\activate`
3

Step 2: Install Required Packages

Install the necessary packages using pip:
1pip install videosdk
2pip install python-dotenv
3

Step 3: Configure API Keys in a .env file

Create a .env file in your project directory and add your VideoSDK API key:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

Below is the complete code for our AI Voice Agent. We will break it down step-by-step in the following sections.
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "You are a helpful healthcare assistant specializing in multi-turn conversation handling. Your primary role is to assist users by answering questions about symptoms, providing general health advice, and helping schedule appointments. You must maintain context across multiple interactions to ensure a coherent and helpful conversation flow. However, you are not a medical professional, and you must always include a disclaimer advising users to consult a doctor for medical advice. You should prioritize user privacy and data security, ensuring that no personal health information is stored or shared. Your responses should be concise, informative, and empathetic, aiming to provide a supportive experience for users seeking health-related information."
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=DeepgramSTT(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=[Silero Voice Activity Detection](https://docs.videosdk.live/ai_agents/plugins/silero-vad)(threshold=0.35),
32        turn_detector=TurnDetector(threshold=0.8)
33    )
34
35    session = [AI voice Agent Sessions](https://docs.videosdk.live/ai_agents/core-components/agent-session)(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63

Step 4.1: Generating a VideoSDK Meeting ID

To interact with the agent, you need a meeting ID. You can generate one using the following curl command:
1curl -X POST "https://api.videosdk.live/v1/meetings" \
2-H "Authorization: YOUR_API_KEY" \
3-H "Content-Type: application/json"
4

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class is where we define the behavior of our agent. It inherits from the Agent class and uses the provided instructions to guide its interactions. The on_enter and on_exit methods define what the agent says when a session starts and ends.

Step 4.3: Defining the Core Pipeline

The CascadingPipeline is central to processing the audio data. It consists of:
  • STT (DeepgramSTT): Converts user speech into text.
  • LLM (OpenAILLM): Processes the text and generates a response.
  • TTS (ElevenLabsTTS): Converts the response text back into speech.
  • VAD (SileroVAD): Detects when the user is speaking.
  • TurnDetector: Manages conversation turns by detecting pauses.

Step 4.4: Managing the Session and Startup Logic

The start_session function initializes the agent and starts the session. The make_context function sets up the room options, and the if __name__ == "__main__": block runs the job.

Running and Testing the Agent

Step 5.1: Running the Python Script

To start the agent, run the following command in your terminal:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Once the script is running, check the console for a playground link. Use this link to join the meeting and interact with your agent. You can speak to the agent and receive responses in real-time.

Advanced Features and Customizations

Extending Functionality with Custom Tools

You can enhance your agent by adding custom tools to perform specific tasks. This can be done by integrating additional plugins or writing custom logic.

Exploring Other Plugins

While we used specific plugins for STT, LLM, and TTS, the VideoSDK framework supports various other options. Experiment with different models to find what best suits your needs.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure your API keys are correctly set in the .env file and that they have the necessary permissions.

Audio Input/Output Problems

Check your microphone and speaker settings to ensure they are correctly configured and accessible by the application.

Dependency and Version Conflicts

Use a virtual environment to manage dependencies and avoid conflicts with other Python packages.

Conclusion

Summary of What You’ve Built

In this tutorial, we built a sophisticated AI Voice Agent capable of handling multi-turn conversations using the VideoSDK framework. We explored the architecture, set up the environment, and implemented a working agent.

Next Steps and Further Learning

Explore additional features of the VideoSDK framework, experiment with different plugins, and consider building agents for other domains to expand your understanding and capabilities.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ