Build a Real-Time Voice AI Agent

Step-by-step guide to building a real-time voice AI agent using VideoSDK's framework. Includes complete code examples and testing instructions.

Introduction to AI Voice Agents in Real-Time Voice AI Framework

AI Voice Agents are revolutionizing the way we interact with technology by providing seamless voice-based interfaces. These agents are crucial in real-time voice AI frameworks, enabling applications to process and respond to user queries instantly. In this tutorial, we will explore how to build a real-time voice AI agent using the VideoSDK framework.

What is an AI

Voice Agent

?

An AI

Voice Agent

 is a software application designed to understand and respond to human speech. It leverages technologies like Speech-to-Text (STT), Language Learning Models (LLM), and Text-to-Speech (TTS) to process speech input, generate a response, and convert it back to speech.

Why Are They Important for the Real-Time Voice AI Framework Industry?

AI Voice Agents are pivotal in industries like customer service, healthcare, and smart home automation. They provide real-time assistance, improve user engagement, and enhance accessibility.

Core Components of a

Voice Agent

  • STT (Speech-to-Text): Converts spoken language into text.
  • LLM (Language Learning Model): Processes the text to generate a meaningful response.
  • TTS (Text-to-Speech): Converts the response text back into speech.
For a comprehensive understanding, refer to the

AI voice Agent core components overview

.

What You'll Build in This Tutorial

In this guide, you'll build a real-time voice AI agent using VideoSDK's framework, integrating key components like STT, LLM, and TTS.

Architecture and Core Concepts

High-Level Architecture Overview

The architecture of a real-time voice AI agent involves several stages: capturing audio input, processing it through a

cascading pipeline in AI voice Agents

 of STT, LLM, and TTS, and delivering a spoken response.
Diagram

Understanding Key Concepts in the VideoSDK Framework

  • Agent: Represents the core logic of your voice assistant.
  • CascadingPipeline: Manages the flow of data from STT to LLM to TTS.
  • VAD & TurnDetector: Determine when the agent should listen and respond.
For more details on managing sessions, see

AI voice Agent Sessions

.

Setting Up the Development Environment

Prerequisites

To get started, ensure you have Python 3.11+ installed and a VideoSDK account at app.videosdk.live.

Step 1: Create a Virtual Environment

Create a virtual environment to manage dependencies:
1python -m venv venv
2source venv/bin/activate # On Windows use `venv\\Scripts\\activate`
3

Step 2: Install Required Packages

Install the necessary Python packages using pip:
1pip install videosdk
2

Step 3: Configure API Keys in a .env file

Create a .env file in your project directory and add your VideoSDK API keys:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

Here is the complete, runnable code for the AI Voice Agent:
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "{\n  \"persona\": \"real-time voice AI assistant\",\n  \"capabilities\": [\n    \"Provide real-time responses to user queries using the VideoSDK framework\",\n    \"Assist users in navigating the framework's features\",\n    \"Offer guidance on integrating the framework into existing systems\",\n    \"Answer technical questions related to the framework's capabilities\"\n  ],\n  \"constraints\": [\n    \"You are not a certified technical support agent and should advise users to consult official documentation for complex issues\",\n    \"You must not provide any personal opinions or unverified information\",\n    \"Ensure user privacy and data protection by not storing any personal information\"\n  ]\n}"
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=DeepgramSTT(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=SileroVAD(threshold=0.35),
32        turn_detector=TurnDetector(threshold=0.8)
33    )
34
35    session = AgentSession(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63

Step 4.1: Generating a VideoSDK Meeting ID

To generate a meeting ID, use the following curl command:
1curl -X POST https://api.videosdk.live/v1/meetings -H "Authorization: Bearer YOUR_API_KEY"
2

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class extends the Agent class, defining custom behavior for entering and exiting a session:
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6

Step 4.3: Defining the Core Pipeline

The

CascadingPipeline

 manages the flow of audio data through various processing stages:
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8

Step 4.4: Managing the Session and Startup Logic

The start_session function initializes the agent session, connecting it to the VideoSDK framework:
1async def start_session(context: JobContext):
2    # Create agent and conversation flow
3    agent = MyVoiceAgent()
4    conversation_flow = ConversationFlow(agent)
5
6    # Create pipeline
7    pipeline = CascadingPipeline(
8        stt=DeepgramSTT(model="nova-2", language="en"),
9        llm=OpenAILLM(model="gpt-4o"),
10        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
11        vad=SileroVAD(threshold=0.35),
12        turn_detector=TurnDetector(threshold=0.8)
13    )
14
15    session = AgentSession(
16        agent=agent,
17        pipeline=pipeline,
18        conversation_flow=conversation_flow
19    )
20
21    try:
22        await context.connect()
23        await session.start()
24        # Keep the session running until manually terminated
25        await asyncio.Event().wait()
26    finally:
27        # Clean up resources when done
28        await session.close()
29        await context.shutdown()
30
The make_context function sets up the room options:
1def make_context() -> JobContext:
2    room_options = RoomOptions(
3    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
4        name="VideoSDK Cascaded Agent",
5        playground=True
6    )
7
8    return JobContext(room_options=room_options)
9
Finally, the script is started with:
1if __name__ == "__main__":
2    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
3    job.start()
4

Running and Testing the Agent

Step 5.1: Running the Python Script

Run the script using:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

After running the script, find the playground link in the console to interact with your agent. You can test different queries and observe how the agent responds in real-time.

Advanced Features and Customizations

Extending Functionality with Custom Tools

You can extend the agent's functionality by integrating custom tools using the function_tool concept, allowing for more complex interactions and capabilities.

Exploring Other Plugins

Explore other STT, LLM, and TTS options available in the VideoSDK framework to tailor the agent to specific needs. For instance, consider using the

Deepgram STT Plugin for voice agent

 for enhanced speech-to-text capabilities or the

OpenAI LLM Plugin for voice agent

 for advanced language processing.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure your API keys are correctly configured in the .env file and that they have the necessary permissions.

Audio Input/Output Problems

Check your microphone and speaker settings to ensure they are correctly configured and accessible by the application. Utilizing

Silero Voice Activity Detection

 can help manage audio input more effectively.

Dependency and Version Conflicts

Ensure all dependencies are installed and compatible with your Python version. Use a virtual environment to manage them effectively.

Conclusion

Summary of What You've Built

In this tutorial, you built a real-time voice AI agent using the VideoSDK framework, integrating STT, LLM, and TTS components.

Next Steps and Further Learning

Explore additional plugins and customization options to enhance your agent's capabilities further. Consider contributing to open-source projects or developing your own plugins. For example, you might explore the

Turn detector for AI voice Agents

 to improve interaction flow.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ