Build a Python Real-Time Voice AI Agent

Step-by-step guide to building a Python real-time voice AI agent using VideoSDK.

Introduction to AI Voice Agents in Python Real-Time Voice AI

What is an AI

Voice Agent

?

An AI

Voice Agent

 is a software system designed to interact with users through voice commands. These agents can understand spoken language, process the information, and respond in a conversational manner. They are often used in applications like virtual assistants, customer service bots, and smart home devices.

Why are they important for the Python Real-Time Voice AI industry?

AI Voice Agents are crucial in the real-time voice AI industry as they enable hands-free interaction with technology. This is particularly valuable in environments where manual operation is impractical, such as driving or operating machinery. They also enhance accessibility for users with disabilities and provide a more natural interface for interacting with digital systems.

Core Components of a

Voice Agent

  • Speech-to-Text (STT): Converts spoken language into text.
  • Large Language Model (LLM): Processes the text to understand and generate responses.
  • Text-to-Speech (TTS): Converts text responses back into spoken language.

What You'll Build in This Tutorial

In this tutorial, you will build a Python-based real-time voice AI agent using the VideoSDK framework. The agent will listen to user commands, process them using AI models, and respond in real-time.

Architecture and Core Concepts

High-Level Architecture Overview

The architecture of a real-time voice AI agent involves several components working in tandem. The user's speech is first captured and processed by a

Deepgram STT Plugin for voice agent

. The transcribed text is then analyzed by an

OpenAI LLM Plugin for voice agent

 to generate a suitable response. Finally, the response is converted back to speech using

ElevenLabs TTS Plugin for voice agent

 technology.
Diagram

Understanding Key Concepts in the VideoSDK Framework

  • Agent: The core class representing your bot, responsible for managing interactions.
  • CascadingPipeline: Manages the flow of audio processing, linking STT, LLM, and TTS.
  • VAD & TurnDetector: These components, including

    Silero Voice Activity Detection

    , help the agent determine when to listen and when to respond, ensuring smooth interaction.

Setting Up the Development Environment

Prerequisites

To follow along with this tutorial, you will need Python 3.11+ and a VideoSDK account, which you can create at app.videosdk.live.

Step 1: Create a Virtual Environment

Create a virtual environment to manage your project's dependencies. Run the following command in your terminal:
1python -m venv venv
2source venv/bin/activate  # On Windows use `venv\Scripts\activate`
3

Step 2: Install Required Packages

Install the necessary packages using pip:
1pip install videosdk
2pip install asyncio
3pip install python-dotenv
4

Step 3: Configure API Keys in a .env file

Create a .env file in your project directory to store your VideoSDK API keys:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

Here is the complete Python code for building your AI Voice Agent:
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "{\n  \"persona\": \"Real-time Python Voice AI Assistant\",\n  \"capabilities\": [\n    \"Process and respond to voice commands in real-time using Python.\",\n    \"Provide information and assistance on various topics as requested by the user.\",\n    \"Integrate with Python libraries to perform tasks such as data retrieval, calculations, and more.\",\n    \"Support continuous conversation flow with context awareness.\"\n  ],\n  \"constraints\": [\n    \"The agent is not capable of making decisions or providing professional advice.\",\n    \"Responses should be generated within a few seconds to maintain real-time interaction.\",\n    \"The agent must clearly state when it cannot fulfill a request or when a task is beyond its capabilities.\",\n    \"Ensure user privacy and data security by not storing any personal information.\"\n  ]\n}"
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=DeepgramSTT(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=SileroVAD(threshold=0.35),
32        turn_detector=TurnDetector(threshold=0.8)
33    )
34
35    session = AgentSession(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63

Step 4.1: Generating a VideoSDK Meeting ID

To generate a meeting ID, you can use the following curl command. This ID is essential for setting up the communication session with your agent.
1curl -X POST \
2  https://api.videosdk.live/v1/meetings \
3  -H "Authorization: YOUR_API_KEY" \
4  -H "Content-Type: application/json"
5

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class is where you define the behavior of your voice agent. It inherits from the Agent class and uses the agent_instructions to initialize its capabilities and constraints. The on_enter and on_exit methods define what the agent says when a session starts and ends.
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6

Step 4.3: Defining the Core Pipeline

The

Cascading pipeline in AI voice Agents

 is crucial for processing audio data. It connects the STT, LLM, and TTS components, allowing the agent to understand and respond to user commands.
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8

Step 4.4: Managing the Session and Startup Logic

The start_session function initializes the agent session, connects to the VideoSDK platform, and starts the conversation flow. The make_context function sets up the job context, including room options for the session.
1def make_context() -> JobContext:
2    room_options = RoomOptions(
3    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
4        name="VideoSDK Cascaded Agent",
5        playground=True
6    )
7
8    return JobContext(room_options=room_options)
9

Running and Testing the Agent

Step 5.1: Running the Python Script

To run your agent, execute the Python script using the following command:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Once the script is running, you will see a playground link in the console. Open this link in your browser to interact with your agent. You can speak commands and receive real-time responses.

Advanced Features and Customizations

Extending Functionality with Custom Tools

You can extend your agent's capabilities by integrating custom tools. This involves creating new functions that the agent can call to perform specific tasks, such as querying a database or interacting with an API.

Exploring Other Plugins

The VideoSDK framework supports various plugins for STT, LLM, and TTS. You can experiment with different models to optimize performance and accuracy for your specific use case.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure that your API keys are correctly configured in the .env file and that they have the necessary permissions.

Audio Input/Output Problems

Check your microphone and speaker settings to ensure they are correctly connected and configured.

Dependency and Version Conflicts

Make sure all dependencies are installed and compatible with your Python version. Use a virtual environment to manage these dependencies effectively.

Conclusion

Summary of What You've Built

In this tutorial, you built a real-time voice AI agent using Python and the VideoSDK framework. The agent can process voice commands, generate responses, and interact with users in real-time.

Next Steps and Further Learning

Consider exploring additional features of the VideoSDK framework, such as integrating with other AI models or expanding the agent's capabilities with custom tools. Continue learning by experimenting with different plugins and configurations to enhance your agent's performance. For a comprehensive understanding, refer to the

AI voice Agent core components overview

 and explore

AI voice Agent Sessions

 for more insights.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ