Build an Open Source AI Voice Agent

Step-by-step guide to building an open source AI voice agent with VideoSDK. Includes code examples and testing.

Introduction to AI Voice Agents in Open Source AI Voice Agent

In recent years, AI voice agents have become integral to many applications, from virtual assistants like Siri and Alexa to customer service bots. These agents can process human speech, understand the intent, and respond in a natural language, making them invaluable in enhancing user interaction.

What is an AI Voice Agent?

An AI voice agent is a software application that uses artificial intelligence to interpret and respond to human speech. These agents typically leverage technologies like Speech-to-Text (STT), Language Models (LLM), and Text-to-Speech (TTS) to perform their tasks.

Why are they Important for the Open Source AI Voice Agent Industry?

In the open source sector, AI voice agents can democratize access to technology, enabling developers to build customizable and cost-effective solutions. They can be used in various scenarios, such as automating customer support, providing hands-free control of applications, and more.

Core Components of a Voice Agent

  • STT (Speech-to-Text): Converts spoken language into text.
  • LLM (Language Model): Processes the text to understand and generate responses.
  • TTS (Text-to-Speech): Converts text responses back into speech.

What You'll Build in This Tutorial

In this tutorial, you will learn how to build an open source AI voice agent using the VideoSDK framework. We will guide you through setting up the environment, writing the code, and testing your agent. For a detailed walkthrough, refer to the

Voice Agent Quick Start Guide

.

Architecture and Core Concepts

High-Level Architecture Overview

The architecture of an AI voice agent involves several components working together to process user input and generate responses. Here’s a high-level overview:
Diagram

Understanding Key Concepts in the VideoSDK Framework

Setting Up the Development Environment

Prerequisites

Before starting, ensure you have Python 3.11+ installed and a VideoSDK account. You can sign up at the VideoSDK website.

Step 1: Create a Virtual Environment

Create a virtual environment to manage dependencies:
1python -m venv venv
2source venv/bin/activate  # On Windows use `venv\\Scripts\\activate`
3

Step 2: Install Required Packages

Install the necessary packages using pip:
1pip install videosdk
2pip install python-dotenv
3

Step 3: Configure API Keys in a .env File

Create a .env file in your project directory and add your VideoSDK API key:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

Now, let’s dive into building the AI voice agent. Below is the complete code that we will break down step-by-step:
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "You are an open source AI voice agent designed to assist users with general inquiries and provide information about open source projects. Your persona is that of a knowledgeable and friendly guide who is passionate about open source technology. Your capabilities include answering questions about various open source projects, providing guidance on how to contribute to open source, and offering insights into the benefits of using open source software. However, you must adhere to certain constraints: you are not a legal advisor and cannot provide legal advice on open source licenses; you should always encourage users to consult official documentation or legal professionals for such matters. Additionally, you must respect user privacy and not store any personal data. Your goal is to promote the use and understanding of open source technology while ensuring a positive and informative user experience."
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=DeepgramSTT(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=SileroVAD(threshold=0.35),
32        turn_detector=TurnDetector(threshold=0.8)
33    )
34
35    session = AgentSession(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63

Step 4.1: Generating a VideoSDK Meeting ID

To interact with your AI voice agent, you need a meeting ID. You can generate one using the VideoSDK API:
1curl -X POST \
2  https://api.videosdk.live/v1/rooms \
3  -H "Authorization: Bearer YOUR_API_KEY" \
4  -H "Content-Type: application/json" \
5  -d '{"name": "My Meeting"}'
6

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class extends the Agent class from the VideoSDK framework. It defines the agent's behavior when entering and exiting a session:
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6
This class uses the agent_instructions to define the agent's persona and capabilities.

Step 4.3: Defining the Core Pipeline

The CascadingPipeline is crucial for processing audio data. It connects various plugins to handle STT, LLM, TTS, VAD, and turn detection:
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8
Each component plays a specific role:

Step 4.4: Managing the Session and Startup Logic

The start_session function initializes the agent, pipeline, and session:
1async def start_session(context: JobContext):
2    agent = MyVoiceAgent()
3    conversation_flow = ConversationFlow(agent)
4
5    pipeline = CascadingPipeline(
6        stt=DeepgramSTT(model="nova-2", language="en"),
7        llm=OpenAILLM(model="gpt-4o"),
8        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
9        vad=SileroVAD(threshold=0.35),
10        turn_detector=TurnDetector(threshold=0.8)
11    )
12
13    session = AgentSession(
14        agent=agent,
15        pipeline=pipeline,
16        conversation_flow=conversation_flow
17    )
18
19    try:
20        await context.connect()
21        await session.start()
22        await asyncio.Event().wait()
23    finally:
24        await session.close()
25        await context.shutdown()
26
The make_context function sets up the room options, and the if __name__ == "__main__": block starts the job:
1def make_context() -> JobContext:
2    room_options = RoomOptions(
3        name="VideoSDK Cascaded Agent",
4        playground=True
5    )
6
7    return JobContext(room_options=room_options)
8
9if __name__ == "__main__":
10    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
11    job.start()
12

Running and Testing the Agent

Step 5.1: Running the Python Script

To run your AI voice agent, execute the script:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Once the script is running, you will receive a link to the VideoSDK playground in the console. Open this link in your browser to interact with your agent.

Advanced Features and Customizations

Extending Functionality with Custom Tools

You can enhance your agent by integrating custom tools and functionalities. The VideoSDK framework allows you to add plugins and tools to extend the agent's capabilities.

Exploring Other Plugins

While this tutorial uses specific plugins, VideoSDK supports various STT, LLM, and TTS options. Explore these to find the best fit for your needs.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure your API key is correctly set in the .env file and that it has the necessary permissions.

Audio Input/Output Problems

Check your audio settings and ensure your microphone and speakers are working correctly.

Dependency and Version Conflicts

Ensure all dependencies are installed with compatible versions. Use a virtual environment to manage these effectively.

Conclusion

Summary of What You've Built

In this tutorial, you built an open source AI voice agent using the VideoSDK framework. You learned how to set up the environment, write the code, and test your agent. For more information on managing sessions, refer to

AI voice Agent Sessions

.

Next Steps and Further Learning

Explore additional features and plugins offered by VideoSDK to enhance your agent. Consider contributing to open source projects to further your learning and impact.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ