Build an Open Source AI Voice Bot

Step-by-step guide to building an AI voice bot using open-source tools. Perfect for developers.

Introduction to AI Voice Agents in ai voice bot open source

What is an AI

Voice Agent

?

An AI

Voice Agent

 is a sophisticated software application designed to interact with users through voice commands. It leverages technologies like Speech-to-Text (STT), Natural Language Processing (NLP), and Text-to-Speech (TTS) to understand spoken language, process it, and respond in a human-like manner. These agents can perform tasks ranging from answering questions to controlling smart devices.

Why are they important for the ai voice bot open source industry?

AI Voice Agents are crucial in the open-source industry as they democratize access to advanced voice interaction capabilities. By utilizing open-source frameworks, developers can customize and deploy voice agents in various applications, from customer support to home automation, without the high costs associated with proprietary solutions.

Core Components of a

Voice Agent

  • Speech-to-Text (STT): Converts spoken language into text.
  • Large Language Model (LLM): Processes the text to understand intent and generate responses.
  • Text-to-Speech (TTS): Converts text responses back into spoken language.

What You'll Build in This Tutorial

In this tutorial, you will build a fully functional AI Voice Bot using open-source tools and the VideoSDK framework. This bot will be capable of understanding and responding to user queries about open-source software.

Architecture and Core Concepts

High-Level Architecture Overview

The architecture of an AI

Voice Agent

 involves several key components working together to process user input and generate responses. The data flow begins with capturing user speech, converting it to text, processing it through an LLM, and finally converting the response back to speech.
Diagram

Understanding Key Concepts in the VideoSDK Framework

  • Agent: The core class representing your bot, responsible for managing interactions.

  • Cascading Pipeline in AI voice Agents

    :     Manages the flow of audio processing from STT to LLM to TTS.
  • VAD & TurnDetector: These components help the agent determine when to listen and speak, ensuring smooth interactions.

Setting Up the Development Environment

Prerequisites

To get started, ensure you have Python 3.11+ installed and a VideoSDK account. Sign up at app.videosdk.live to access the necessary API keys.

Step 1: Create a Virtual Environment

Begin by creating a virtual environment to manage dependencies:
1python -m venv venv
2source venv/bin/activate  # On Windows use `venv\Scripts\activate`
3

Step 2: Install Required Packages

Install the necessary packages using pip:
1pip install videosdk
2pip install python-dotenv
3

Step 3: Configure API Keys in a .env file

Create a .env file in your project directory and add your VideoSDK API keys:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

Let's start by presenting the complete, runnable code for the AI Voice Agent:
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "You are an AI Voice Bot designed to assist users with open-source software inquiries. Your persona is that of a knowledgeable and friendly tech assistant. Your primary capabilities include answering questions about various open-source software projects, providing guidance on how to contribute to open-source projects, and offering insights into the benefits and challenges of using open-source software. You can also direct users to relevant online resources and communities for further assistance. However, you are not a licensed software developer, and you must include a disclaimer advising users to consult professional developers for complex technical issues. Additionally, you should refrain from providing legal advice regarding software licenses and instead direct users to consult legal professionals for such matters."
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=[Deepgram STT Plugin for voice agent](https://docs.videosdk.live/ai_agents/plugins/stt/deepgram)(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=[Silero Voice Activity Detection](https://docs.videosdk.live/ai_agents/plugins/silero-vad)(threshold=0.35),
32        turn_detector=[Turn detector for AI voice Agents](https://docs.videosdk.live/ai_agents/plugins/turn-detector)(threshold=0.8)
33    )
34
35    session = [AI voice Agent Sessions](https://docs.videosdk.live/ai_agents/core-components/agent-session)(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63

Step 4.1: Generating a VideoSDK Meeting ID

To interact with the AI Voice Agent, you need a meeting ID. You can generate one using the following curl command:
1curl -X POST "https://api.videosdk.live/v1/meetings" \
2-H "Authorization: Bearer YOUR_API_KEY" \
3-H "Content-Type: application/json"
4

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class is where you define the behavior of your AI Voice Bot. It inherits from the Agent class and implements the on_enter and on_exit methods to handle user interactions.
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6

Step 4.3: Defining the Core Pipeline

The CascadingPipeline is crucial as it defines the flow of audio processing through various stages:
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8
  • STT (DeepgramSTT): Converts spoken words into text.
  • LLM (OpenAILLM): Processes the text to generate a response.
  • TTS (ElevenLabsTTS): Converts the response text back into speech.
  • VAD (SileroVAD): Detects when the user is speaking.
  • TurnDetector: Determines when the agent should respond.

Step 4.4: Managing the Session and Startup Logic

The start_session function sets up the agent session and manages its lifecycle:
1async def start_session(context: JobContext):
2    # Create agent and conversation flow
3    agent = MyVoiceAgent()
4    conversation_flow = ConversationFlow(agent)
5
6    # Create pipeline
7    pipeline = CascadingPipeline(
8        stt=DeepgramSTT(model="nova-2", language="en"),
9        llm=OpenAILLM(model="gpt-4o"),
10        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
11        vad=SileroVAD(threshold=0.35),
12        turn_detector=TurnDetector(threshold=0.8)
13    )
14
15    session = AgentSession(
16        agent=agent,
17        pipeline=pipeline,
18        conversation_flow=conversation_flow
19    )
20
21    try:
22        await context.connect()
23        await session.start()
24        # Keep the session running until manually terminated
25        await asyncio.Event().wait()
26    finally:
27        # Clean up resources when done
28        await session.close()
29        await context.shutdown()
30
The make_context function prepares the job context, including room options:
1def make_context() -> JobContext:
2    room_options = RoomOptions(
3    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
4        name="VideoSDK Cascaded Agent",
5        playground=True
6    )
7
8    return JobContext(room_options=room_options)
9
Finally, the script's entry point ensures the agent starts when the script is run:
1if __name__ == "__main__":
2    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
3    job.start()
4

Running and Testing the Agent

Step 5.1: Running the Python Script

To run your AI Voice Agent, execute the following command in your terminal:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Once the script is running, you will receive a playground link in the console. Open this link in your browser to interact with your AI Voice Bot. You can speak to the bot and receive responses in real-time.

Advanced Features and Customizations

Extending Functionality with Custom Tools

The VideoSDK framework allows you to extend your agent's functionality using custom tools. These tools can be integrated into the pipeline to add new capabilities.

Exploring Other Plugins

While this tutorial uses specific plugins for STT, LLM, and TTS, you can explore other options like Cartesia for STT or Google Gemini for LLM to suit your needs.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure your API keys are correctly set in the .env file. Double-check the VideoSDK dashboard for accurate credentials.

Audio Input/Output Problems

Verify your microphone and speaker settings. Ensure your browser has permission to access these devices.

Dependency and Version Conflicts

Use a virtual environment to manage dependencies. Ensure all packages are compatible with Python 3.11+.

Conclusion

Summary of What You've Built

You've successfully built an AI Voice Bot using open-source tools and the VideoSDK framework. This bot can understand and respond to user queries about open-source software.

Next Steps and Further Learning

Consider exploring additional plugins and customizations to enhance your bot's capabilities. Engage with the community and contribute to open-source projects to further your learning.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ