Build an AI Voice Agent with Streaming API

Step-by-step guide to building an AI Voice Agent with VideoSDK's streaming API, complete with code examples and testing instructions.

Introduction to AI Voice Agents in ai voice agent streaming api

AI Voice Agents are sophisticated systems designed to interact with users through voice commands. They leverage technologies such as Speech-to-Text (STT), Language Learning Models (LLM), and Text-to-Speech (TTS) to process user input, generate responses, and communicate back to the user. In the context of streaming APIs, these agents can provide real-time assistance, automate customer service, and enhance user engagement.

What is an AI Voice Agent?

An AI Voice Agent is a software program that uses voice recognition and natural language processing to understand and respond to human speech. These agents can perform tasks, provide information, and interact with users in a conversational manner.

Why are they important for the ai voice agent streaming api industry?

In the streaming API industry, AI Voice Agents can streamline user interactions, provide instant support, and automate routine tasks, making them invaluable for improving efficiency and user satisfaction.

Core Components of a Voice Agent

  • STT (Speech-to-Text): Converts spoken language into text.
  • LLM (Language Learning Model): Processes the text to understand and generate responses.
  • TTS (Text-to-Speech): Converts text responses back into spoken language.

What You'll Build in This Tutorial

In this tutorial, you will build a fully functional AI Voice Agent using VideoSDK's streaming API. You will learn how to set up the development environment, create a custom agent, and test it in a real-time environment. For a comprehensive guide, refer to the

Voice Agent Quick Start Guide

.

Architecture and Core Concepts

High-Level Architecture Overview

The AI Voice Agent's architecture involves several key components working together to process user input and generate responses. The data flow begins with user speech, which is captured and converted to text using STT. This text is then processed by an LLM to generate a response, which is converted back to speech using TTS.
Diagram

Understanding Key Concepts in the VideoSDK Framework

  • Agent: The core class representing your bot, responsible for managing interactions and responses.
  • CascadingPipeline: A sequence of processes that handle audio input, language processing, and audio output. Learn more about the

    Cascading pipeline in AI voice Agents

    .
  • VAD & TurnDetector: Tools that help the agent determine when to listen and when to speak, such as the

    Turn detector for AI voice Agents

    .

Setting Up the Development Environment

Prerequisites

Before starting, ensure you have Python 3.11+ installed and a VideoSDK account. You can sign up at app.videosdk.live.

Step 1: Create a Virtual Environment

Create a virtual environment to manage dependencies:
1python -m venv venv
2source venv/bin/activate  # On Windows use `venv\Scripts\activate`
3

Step 2: Install Required Packages

Install the necessary Python packages using pip:
1pip install videosdk-agents videosdk-plugins
2

Step 3: Configure API Keys in a .env file

Create a .env file in your project directory and add your VideoSDK API key:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

Here is the complete code for your AI Voice Agent. We will break it down in the following sections:
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "You are an AI Voice Agent specialized in providing information and assistance related to streaming APIs. Your persona is that of a knowledgeable and friendly tech support assistant. Your primary capabilities include answering questions about streaming API functionalities, guiding users through API integration processes, and troubleshooting common issues related to streaming APIs. You can also provide best practices for optimizing API usage and suggest resources for further learning. However, you are not a developer and cannot write or debug code. Always remind users to consult official documentation or a professional developer for complex technical issues. You must include a disclaimer that your advice is based on general knowledge and may not apply to specific cases."
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=DeepgramSTT(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=SileroVAD(threshold=0.35),
32        turn_detector=TurnDetector(threshold=0.8)
33    )
34
35    session = AgentSession(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63

Step 4.1: Generating a VideoSDK Meeting ID

To create a meeting ID, use the following curl command:
1curl -X POST \
2  https://api.videosdk.live/v1/meetings \
3  -H "Authorization: YOUR_API_KEY" \
4  -H "Content-Type: application/json"
5

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class extends the base Agent class. It defines the agent's behavior when entering and exiting a session:
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6

Step 4.3: Defining the Core Pipeline

The CascadingPipeline is crucial as it defines how the agent processes input and generates output. It involves various plugins, including the

Deepgram STT Plugin for voice agent

 and

ElevenLabs TTS Plugin for voice agent

.
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8

Step 4.4: Managing the Session and Startup Logic

This section sets up the session and handles the startup logic, ensuring the agent is ready to interact. For more details, see the

AI voice Agent Sessions

.
1async def start_session(context: JobContext):
2    agent = MyVoiceAgent()
3    conversation_flow = ConversationFlow(agent)
4    pipeline = CascadingPipeline(
5        stt=DeepgramSTT(model="nova-2", language="en"),
6        llm=OpenAILLM(model="gpt-4o"),
7        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
8        vad=SileroVAD(threshold=0.35),
9        turn_detector=TurnDetector(threshold=0.8)
10    )
11    session = AgentSession(
12        agent=agent,
13        pipeline=pipeline,
14        conversation_flow=conversation_flow
15    )
16    try:
17        await context.connect()
18        await session.start()
19        await asyncio.Event().wait()
20    finally:
21        await session.close()
22        await context.shutdown()
23
24def make_context() -> JobContext:
25    room_options = RoomOptions(
26        name="VideoSDK Cascaded Agent",
27        playground=True
28    )
29    return JobContext(room_options=room_options)
30
31if __name__ == "__main__":
32    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
33    job.start()
34

Running and Testing the Agent

Step 5.1: Running the Python Script

Execute the script using the command:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Once the script is running, you will see a link in the console to join the agent in the playground. Use this link to interact with your AI Voice Agent.

Advanced Features and Customizations

Extending Functionality with Custom Tools

You can enhance your agent by integrating custom tools using the function_tool concept, allowing for more specialized interactions.

Exploring Other Plugins

Explore other plugins for STT, LLM, and TTS to customize your agent's capabilities further. Options include Cartesia for STT and Google Gemini for LLM. For a detailed overview, refer to the

AI voice Agent core components overview

.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure your API key is correctly set in the .env file and that you have the necessary permissions.

Audio Input/Output Problems

Verify your microphone and speaker settings. Check if the correct devices are selected in your system settings.

Dependency and Version Conflicts

Ensure all dependencies are installed with compatible versions. Use a virtual environment to manage dependencies effectively.

Conclusion

Summary of What You've Built

You have successfully built an AI Voice Agent capable of interacting with users in real-time using VideoSDK's streaming API.

Next Steps and Further Learning

Explore additional features and plugins to enhance your agent's capabilities. Consider integrating with other APIs for more complex interactions, such as the

OpenAI LLM Plugin for voice agent

 or

Silero Voice Activity Detection

.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ