Build an AI Voice Agent with Vosk

Step-by-step guide to building a Vosk-based AI Voice Agent using VideoSDK.

Introduction to AI Voice Agents in vosk

What is an AI

Voice Agent

?

AI Voice Agents are software programs designed to interact with users through voice commands. These agents leverage technologies like Speech-to-Text (STT), Text-to-Speech (TTS), and Natural Language Processing (NLP) to understand and respond to user queries. They are increasingly used in various industries to automate customer service, provide hands-free assistance, and more.

Why are they important for the vosk industry?

In the vosk industry, AI Voice Agents play a crucial role in enhancing user experiences by providing real-time transcription services, supporting multiple languages, and handling various audio formats. They are essential for applications that require accurate and efficient voice recognition capabilities.

Core Components of a

Voice Agent

  • STT (Speech-to-Text): Converts spoken language into text.
  • LLM (Large Language Model): Processes and understands the text to generate appropriate responses.
  • TTS (Text-to-Speech): Converts text responses back into spoken language.

What You'll Build in This Tutorial

In this tutorial, you will build a Vosk-based AI

Voice Agent

 using the VideoSDK framework. This agent will transcribe audio to text, process the text using an LLM, and respond with synthesized speech.

Architecture and Core Concepts

High-Level Architecture Overview

The architecture of an AI

Voice Agent

 involves several components working together to process user input and generate responses. The data flow begins with capturing user speech, which is then converted to text using STT. The text is processed by an LLM to understand the intent and generate a response. Finally, TTS converts the response text back into speech.
Diagram

Understanding Key Concepts in the VideoSDK Framework

  • Agent: The core class representing your bot, responsible for managing interactions.

  • Cascading Pipeline in AI voice Agents

    :     Manages the flow of audio processing, connecting STT, LLM, and TTS components.
  • VAD & TurnDetector: Used to determine when the agent should listen and respond, ensuring smooth communication.

Setting Up the Development Environment

Prerequisites

Before you begin, ensure you have Python 3.11+ installed and a VideoSDK account, which you can create at app.videosdk.live.

Step 1: Create a Virtual Environment

Create a virtual environment to manage dependencies:
1python -m venv venv  
2source venv/bin/activate  
3

Step 2: Install Required Packages

Install the necessary packages using pip:
1pip install videosdk-agent videosdk-plugins  
2

Step 3: Configure API Keys in a .env file

Create a .env file in your project directory and add your VideoSDK API keys:
1VIDEOSDK_API_KEY=your_api_key_here  
2

Building the AI Voice Agent: A Step-by-Step Guide

Below is the complete, runnable code for the AI Voice Agent. We will break it down into smaller sections for detailed explanations.
1import asyncio, os  
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow  
3from videosdk.plugins.silero import SileroVAD  
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model  
5from videosdk.plugins.deepgram import DeepgramSTT  
6from videosdk.plugins.openai import OpenAILLM  
7from videosdk.plugins.elevenlabs import ElevenLabsTTS  
8from typing import AsyncIterator  
9
10# Pre-downloading the Turn Detector model  
11pre_download_model()  
12
13agent_instructions = "{  
14  \"persona\": \"helpful transcription assistant\",  
15  \"capabilities\": [  
16    \"transcribe audio to text using the Vosk speech recognition toolkit\",  
17    \"support multiple languages for transcription\",  
18    \"provide real-time transcription updates\",  
19    \"handle audio files of various formats and qualities\"  
20  ],  
21  \"constraints\": [  
22    \"you are not capable of translating text\",  
23    \"you must inform users that transcription accuracy may vary based on audio quality\",  
24    \"you cannot process audio longer than 2 hours in a single session\",  
25    \"you must include a disclaimer that the transcriptions are for informational purposes only and should be verified by a human\"  
26  ]  
27}"  
28
29class MyVoiceAgent(Agent):  
30    def __init__(self):  
31        super().__init__(instructions=agent_instructions)  
32    async def on_enter(self): await self.session.say("Hello! How can I help?")  
33    async def on_exit(self): await self.session.say("Goodbye!")  
34
35async def start_session(context: JobContext):  
36    # Create agent and conversation flow  
37    agent = MyVoiceAgent()  
38    conversation_flow = ConversationFlow(agent)  
39
40    # Create pipeline  
41    pipeline = CascadingPipeline(  
42        stt=[Deepgram STT Plugin for voice agent](https://docs.videosdk.live/ai_agents/plugins/stt/deepgram)(model="nova-2", language="en"),  
43        llm=[OpenAI LLM Plugin for voice agent](https://docs.videosdk.live/ai_agents/plugins/llm/openai)(model="gpt-4o"),  
44        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),  
45        vad=[Silero Voice Activity Detection](https://docs.videosdk.live/ai_agents/plugins/silero-vad)(threshold=0.35),  
46        turn_detector=[Turn detector for AI voice Agents](https://docs.videosdk.live/ai_agents/plugins/turn-detector)(threshold=0.8)  
47    )  
48
49    session = AgentSession(  
50        agent=agent,  
51        pipeline=pipeline,  
52        conversation_flow=conversation_flow  
53    )  
54
55    try:  
56        await context.connect()  
57        await session.start()  
58        # Keep the session running until manually terminated  
59        await asyncio.Event().wait()  
60    finally:  
61        # Clean up resources when done  
62        await session.close()  
63        await context.shutdown()  
64
65def make_context() -> JobContext:  
66    room_options = RoomOptions(  
67    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create  
68        name="VideoSDK Cascaded Agent",  
69        playground=True  
70    )  
71
72    return JobContext(room_options=room_options)  
73
74if __name__ == "__main__":  
75    job = WorkerJob(entrypoint=start_session, jobctx=make_context)  
76    job.start()  
77

Step 4.1: Generating a VideoSDK Meeting ID

To interact with the AI Voice Agent, you need a meeting ID. Use the following curl command to generate one:
1curl -X POST https://api.videosdk.live/v1/meetings -H "Authorization: Bearer YOUR_API_KEY"  
2

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class defines the behavior of the AI Voice Agent. It inherits from the Agent class and uses predefined instructions to guide its interactions.

Step 4.3: Defining the Core Pipeline

The CascadingPipeline is central to the agent's functionality. It connects various plugins:
  • DeepgramSTT: Handles speech-to-text conversion.
  • OpenAILLM: Processes text to understand user intent.
  • ElevenLabsTTS: Converts text responses to speech.
  • SileroVAD & TurnDetector: Manage when the agent listens and speaks.

Step 4.4: Managing the Session and Startup Logic

The start_session function initializes the agent and manages the session lifecycle. The make_context function creates a job context with room options for the agent to operate in. The main block starts the worker job.

Running and Testing the Agent

Step 5.1: Running the Python Script

Execute the script using Python to start the agent:
1python main.py  
2

Step 5.2: Interacting with the Agent in the Playground

Once running, the console will display a playground link. Use this link to join the session and interact with the agent. The agent will greet you and respond to your queries.

Advanced Features and Customizations

Extending Functionality with Custom Tools

VideoSDK allows you to extend your agent's functionality by integrating custom tools using the function_tool feature. This enables more tailored interactions and capabilities.

Exploring Other Plugins

Consider experimenting with different STT, LLM, and TTS plugins to suit your specific needs. VideoSDK supports a variety of options, allowing for flexible customization.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure your API keys are correctly set in the .env file. Double-check for typos or missing entries.

Audio Input/Output Problems

Verify that your microphone and speakers are functioning correctly. Check system settings and permissions.

Dependency and Version Conflicts

Ensure all dependencies are up-to-date and compatible with your Python version. Use a virtual environment to manage packages.

Conclusion

Summary of What You've Built

In this tutorial, you've built a fully functional AI Voice Agent using Vosk and VideoSDK. This agent can transcribe, process, and respond to user queries in real-time.

Next Steps and Further Learning

To enhance your agent, explore additional plugins and custom tools. Continue learning about AI and voice technologies to expand your skill set.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ