Building an AI Voice Agent with VideoSDK

Step-by-step guide to building a conversational AI voice agent using VideoSDK, complete with code examples and testing instructions.

Introduction to AI Voice Agents in what is conversational ai

In the rapidly evolving field of artificial intelligence, AI Voice Agents have emerged as a transformative technology. These agents are essentially software programs that can interpret human speech, process it, and respond in a natural and engaging manner. They are a subset of conversational AI, which focuses on enabling machines to understand and interact with human language.

What is an AI

Voice Agent

?

An AI

Voice Agent

 is a system designed to interact with users through speech. It uses various technologies such as Speech-to-Text (STT), Text-to-Speech (TTS), and Language Models (LLM) to convert spoken language into text, process the text to understand the intent, and then generate a spoken response.

Why are they important for the what is conversational ai industry?

AI Voice Agents are crucial in the conversational AI industry because they provide a more natural and intuitive way for users to interact with machines. They are used in various applications, including customer service, virtual assistants, and smart home devices, making interactions more efficient and user-friendly.

Core Components of a

Voice Agent

  • Speech-to-Text (STT): Converts spoken language into written text.
  • Language Models (LLM): Processes the text to understand user intent and generate appropriate responses.
  • Text-to-Speech (TTS): Converts the text response back into spoken language.
For a comprehensive understanding, refer to the

AI voice Agent core components overview

.

What You'll Build in This Tutorial

In this tutorial, you will learn how to build a simple yet effective AI

Voice Agent

 using the VideoSDK framework. We will guide you through setting up the environment, writing the code, and testing your agent.

Architecture and Core Concepts

High-Level Architecture Overview

The architecture of an AI

Voice Agent

 involves several components working together seamlessly. When a user speaks, the agent captures the audio and processes it through the following steps:
  1. Voice

    Activity Detection

     (VAD)    : Identifies when the user is speaking.
  2. Speech-to-Text (STT): Transcribes the spoken words into text.
  3. Language Model (LLM): Analyzes the text to determine the appropriate response.
  4. Text-to-Speech (TTS): Converts the response text into spoken words.
Diagram

Understanding Key Concepts in the VideoSDK Framework

  • Agent: The core class representing your bot, responsible for managing interactions.
  • CascadingPipeline: Manages the flow of audio processing, linking STT, LLM, and TTS components.
  • VAD & TurnDetector: Tools to detect when the user starts and stops speaking, ensuring smooth interactions. For more details, check out the

    Turn detector for AI voice Agents

    .

Setting Up the Development Environment

Prerequisites

Before you begin, ensure you have Python 3.11+ installed and a VideoSDK account. You can sign up at app.videosdk.live.

Step 1: Create a Virtual Environment

Creating a virtual environment helps manage dependencies and isolate your project:
1python -m venv myenv
2source myenv/bin/activate  # On Windows use `myenv\Scripts\activate`
3

Step 2: Install Required Packages

Install the necessary packages using pip:
1pip install videosdk
2pip install python-dotenv
3

Step 3: Configure API Keys in a .env file

Create a .env file in your project directory and add your VideoSDK API key:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

Let's start by presenting the complete code block that you will be working with:
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "You are an AI Voice Agent specialized in explaining the concept of 'conversational AI'. Your persona is that of a knowledgeable and friendly tech expert who can simplify complex topics for a general audience. Your primary capability is to provide clear and concise explanations about what conversational AI is, including its applications, benefits, and limitations. You can also answer related questions and provide examples of conversational AI in use today. However, you must avoid technical jargon unless specifically asked for by the user, and you should always aim to make the information accessible to non-experts. You are not a developer or a business consultant, so you should not provide coding advice or business strategies. Always remind users to consult professional sources for in-depth technical or business guidance."
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=DeepgramSTT(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=SileroVAD(threshold=0.35),
32        turn_detector=TurnDetector(threshold=0.8)
33    )
34
35    session = AgentSession(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63

Step 4.1: Generating a VideoSDK Meeting ID

To interact with your AI Voice Agent, you'll need a meeting ID. You can generate one using the VideoSDK API. Here's how you can do it using curl:
1curl -X POST \
2  https://api.videosdk.live/v1/rooms \
3  -H "Authorization: Bearer YOUR_VIDEOSDK_API_KEY" \
4  -H "Content-Type: application/json" \
5  -d '{"name": "My Meeting Room"}'
6

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class is a custom implementation of the Agent class. It defines the agent's behavior when entering and exiting a session:
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6

Step 4.3: Defining the Core Pipeline

The CascadingPipeline is where the magic happens. It connects the STT, LLM, and TTS components:
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8

Step 4.4: Managing the Session and Startup Logic

The start_session function manages the session lifecycle, and the if __name__ == "__main__": block starts the agent:
1async def start_session(context: JobContext):
2    agent = MyVoiceAgent()
3    conversation_flow = ConversationFlow(agent)
4    pipeline = CascadingPipeline(
5        stt=DeepgramSTT(model="nova-2", language="en"),
6        llm=OpenAILLM(model="gpt-4o"),
7        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
8        vad=SileroVAD(threshold=0.35),
9        turn_detector=TurnDetector(threshold=0.8)
10    )
11    session = AgentSession(
12        agent=agent,
13        pipeline=pipeline,
14        conversation_flow=conversation_flow
15    )
16    try:
17        await context.connect()
18        await session.start()
19        await asyncio.Event().wait()
20    finally:
21        await session.close()
22        await context.shutdown()
23
24def make_context() -> JobContext:
25    room_options = RoomOptions(
26        name="VideoSDK Cascaded Agent",
27        playground=True
28    )
29    return JobContext(room_options=room_options)
30
31if __name__ == "__main__":
32    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
33    job.start()
34

Running and Testing the Agent

Step 5.1: Running the Python Script

To start your agent, run the following command in your terminal:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Once the script is running, you will see a link to the

AI Agent playground

 in the console. Open it in your browser to interact with your agent. You can speak to the agent, and it will respond based on the instructions provided.

Advanced Features and Customizations

Extending Functionality with Custom Tools

The VideoSDK framework allows you to extend your agent's capabilities by integrating custom tools. This feature enables you to add specific functionalities tailored to your needs.

Exploring Other Plugins

While we've used specific plugins for STT, LLM, and TTS, the VideoSDK framework supports various other options. Consider exploring different plugins to find the best fit for your application.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure that your API key is correctly set in the .env file and that you have the necessary permissions to access the VideoSDK API.

Audio Input/Output Problems

Check your microphone and speaker settings to ensure they are properly configured and not muted.

Dependency and Version Conflicts

If you encounter issues with package versions, ensure that all dependencies are up to date and compatible with your Python version.

Conclusion

Summary of What You've Built

In this tutorial, you've built a functional AI Voice Agent using the VideoSDK framework. You've learned about the core components, set up the environment, and implemented a complete solution.

Next Steps and Further Learning

To further enhance your skills, explore additional features of the VideoSDK framework, experiment with different plugins, and consider building more complex agents for various applications. For more advanced implementations, delve into

AI voice Agent Sessions

 to understand session management in depth.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ