Why use an open source AI voice agent SDK?

Open source SDKs allow developers to integrate voice capabilities into applications without proprietary solutions, enabling innovation and customization.

What are the core components of a voice agent?

The core components are Speech-to-Text (STT), Large Language Model (LLM), and Text-to-Speech (TTS).

How do I generate a VideoSDK meeting ID?

Use the VideoSDK API with a `curl` command to generate a meeting ID.

What should I do if I encounter API key errors?

Ensure your API keys are correct and placed in the `.env` file, checking for typos or missing information.

Build an AI Voice Agent with Open Source SDK

Q: What is an AI Voice Agent?

An AI Voice Agent is a software system that interacts with users through voice commands, using natural language processing to understand and respond to spoken language.

Step-by-step guide to create an AI Voice Agent using an open source SDK with full code and testing instructions.

Introduction to AI Voice Agents in open source ai voice agent sdk

What is an AI Voice Agent?

AI Voice Agents are sophisticated software systems designed to interact with users through voice commands. They utilize natural language processing to understand spoken language, process the information, and respond appropriately. These agents are increasingly used in various applications, from virtual assistants to customer service bots, enhancing user experience by providing hands-free interaction.

Why are they important for the open source ai voice agent sdk industry?

In the realm of open source AI voice agent SDKs, these agents are crucial as they enable developers to integrate voice capabilities into applications without the need for proprietary solutions. This democratizes access to advanced voice technology, allowing for innovation and customization in diverse fields such as home automation, telecommunication, and accessibility tools.

Core Components of a Voice Agent

The core components of a voice agent include:

Speech-to-Text (STT): Converts spoken language into text.
Large Language Model (LLM): Processes the text to understand and generate responses.
Text-to-Speech (TTS): Converts text responses back into spoken language.

For a comprehensive understanding of these components, refer to the

AI voice Agent core components overview

What You'll Build in This Tutorial

In this tutorial, you will build a fully functional AI Voice Agent using an open source SDK. We will guide you through setting up the environment, writing the code, and testing the agent in a playground environment. For a quick setup, you can follow the

Voice Agent Quick Start Guide

Architecture and Core Concepts

High-Level Architecture Overview

The AI Voice Agent processes user speech through a series of stages: capturing audio input, converting it to text, processing the text to generate a response, and finally converting the response back to speech. This flow ensures a seamless interaction between the user and the agent.

Understanding Key Concepts in the VideoSDK Framework

Agent: The core class representing your bot, responsible for handling interactions.
CascadingPipeline: Manages the audio processing flow, coordinating between STT, LLM, and TTS. Learn more about the
Cascading pipeline in AI voice Agents
.
VAD & TurnDetector: These components help the agent determine when to listen and when to respond, ensuring smooth interaction. For more details, see the
Turn detector for AI voice Agents
.

Setting Up the Development Environment

Prerequisites

To begin, ensure you have Python 3.11+ installed and a VideoSDK account. You can sign up at app.videosdk.live.

Step 1: Create a Virtual Environment

Create a virtual environment to manage dependencies:

1python -m venv venv
2source venv/bin/activate  # On Windows use `venv\Scripts\activate`
3

Step 2: Install Required Packages

Install the necessary packages using pip:

1pip install videosdk
2pip install python-dotenv
3

Step 3: Configure API Keys in a `.env` file

Create a .env file in your project directory and add your VideoSDK API keys:

1VIDEOSDK_API_KEY=your_api_key
2VIDEOSDK_SECRET_KEY=your_secret_key
3

Building the AI Voice Agent: A Step-by-Step Guide

Here is the complete, runnable code for the AI Voice Agent:

1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "You are an AI Voice Agent built using an open source AI voice agent SDK. Your persona is that of a friendly and knowledgeable tech assistant. Your primary capabilities include providing information about various open source AI voice agent SDKs, guiding users on how to implement these SDKs in their projects, and offering troubleshooting tips for common issues. You can also suggest best practices for integrating voice capabilities into applications. However, you must not provide any proprietary or confidential information, and you should always encourage users to refer to official documentation for detailed technical guidance. Additionally, you are not a substitute for professional technical support and should remind users to seek expert advice for complex issues."
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=DeepgramSTT(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=SileroVAD(threshold=0.35),
32        turn_detector=TurnDetector(threshold=0.8)
33    )
34
35    session = AgentSession(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63

Step 4.1: Generating a VideoSDK Meeting ID

To generate a meeting ID, you can use the VideoSDK API. Here is an example using curl:

1curl -X POST "https://api.videosdk.live/v1/rooms" \
2-H "Authorization: Bearer YOUR_API_KEY" \
3-H "Content-Type: application/json"
4

This command will return a JSON response with the meeting ID you can use.

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class is a custom implementation of the Agent class. It defines how the agent interacts with users:

1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6

This class uses the agent_instructions to define the agent's persona and capabilities.

Step 4.3: Defining the Core Pipeline

The CascadingPipeline is central to processing audio and generating responses:

1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8

Each component in the pipeline has a specific role: STT converts speech to text, LLM processes the text, and TTS converts the response back to speech. For more information on the plugins used, check out the

Deepgram STT Plugin for voice agent

OpenAI LLM Plugin for voice agent

, and

ElevenLabs TTS Plugin for voice agent

Step 4.4: Managing the Session and Startup Logic

The start_session function manages the lifecycle of the agent session:

1async def start_session(context: JobContext):
2    agent = MyVoiceAgent()
3    conversation_flow = ConversationFlow(agent)
4    pipeline = CascadingPipeline(
5        stt=DeepgramSTT(model="nova-2", language="en"),
6        llm=OpenAILLM(model="gpt-4o"),
7        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
8        vad=SileroVAD(threshold=0.35),
9        turn_detector=TurnDetector(threshold=0.8)
10    )
11    session = AgentSession(
12        agent=agent,
13        pipeline=pipeline,
14        conversation_flow=conversation_flow
15    )
16    try:
17        await context.connect()
18        await session.start()
19        await asyncio.Event().wait()
20    finally:
21        await session.close()
22        await context.shutdown()
23

The make_context function sets up the environment for the agent:

1def make_context() -> JobContext:
2    room_options = RoomOptions(
3        name="VideoSDK Cascaded Agent",
4        playground=True
5    )
6    return JobContext(room_options=room_options)
7

Finally, the script is executed with the if __name__ == "__main__": block:

1if __name__ == "__main__":
2    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
3    job.start()
4

Running and Testing the Agent

Step 5.1: Running the Python Script

To run the agent, execute the Python script:

1python main.py
2

This will start the agent and display a playground link in the console.

Step 5.2: Interacting with the Agent in the Playground

Open the playground link in your browser to interact with the agent. Speak into your microphone, and the agent will respond based on the instructions provided.

Advanced Features and Customizations

Extending Functionality with Custom Tools

The function_tool concept allows you to extend the agent's capabilities by integrating custom tools that can perform specific tasks or access external APIs.

Exploring Other Plugins

The VideoSDK framework supports various plugins for STT, LLM, and TTS. Explore options like Cartesia for STT or Google Gemini for LLM to customize your agent further.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure your API keys are correct and placed in the .env file. Check for any typos or missing information.

Audio Input/Output Problems

Verify your microphone and speaker settings. Ensure the correct devices are selected in your system preferences.

Dependency and Version Conflicts

Use a virtual environment to manage dependencies. Ensure all packages are compatible with your Python version.

Conclusion

Summary of What You've Built

You have successfully built an AI Voice Agent using an open source SDK, complete with speech recognition, language processing, and voice synthesis capabilities.

Next Steps and Further Learning

Explore additional features and plugins in the VideoSDK framework to enhance your agent. Consider integrating with other APIs to expand functionality. For a quick start, revisit the