Build Real-Time Audio Streaming AI Voice Agent

Step-by-step guide to building a real-time audio streaming AI Voice Agent using VideoSDK.

Introduction to AI Voice Agents in Real-Time Audio Streaming

AI Voice Agents are intelligent systems designed to interact with users through voice commands. In the context of real-time audio streaming, these agents can enhance user experience by providing seamless interaction, support, and automation.

What is an AI

Voice Agent

?

An AI

Voice Agent

 is a software entity that uses artificial intelligence to understand and respond to human speech. It leverages technologies like Speech-to-Text (STT), Text-to-Speech (TTS), and Large Language Models (LLM) to process and generate human-like responses.

Why are they important for the real-time audio streaming industry?

In real-time audio streaming, AI Voice Agents can facilitate tasks such as content navigation, user support, and interactive experiences. They are crucial in applications like live broadcasts, virtual events, and customer support systems.

Core Components of a

Voice Agent

  • STT (Speech-to-Text): Converts spoken language into text.
  • LLM (Large Language Model): Processes the text to understand context and generate responses.
  • TTS (Text-to-Speech): Converts text responses back into spoken language.
For a comprehensive understanding, refer to the

AI voice Agent core components overview

.

What You'll Build in This Tutorial

In this tutorial, you will build an AI

Voice Agent

 capable of interacting with users in real-time audio streaming scenarios using the VideoSDK framework.

Architecture and Core Concepts

High-Level Architecture Overview

The AI

Voice Agent

 processes user speech through a series of steps: capturing audio, converting it to text, generating a response, and converting the response back to speech.
Diagram

Understanding Key Concepts in the VideoSDK Framework

  • Agent: Represents your bot, handling user interactions.

  • Cascading Pipeline in AI voice Agents

    : Manages the flow of audio processing from STT to LLM to TTS.
  • VAD & TurnDetector: Detects when the user has finished speaking and when the agent should respond.

Setting Up the Development Environment

Prerequisites

Ensure you have Python 3.11+ installed and a VideoSDK account at app.videosdk.live.

Step 1: Create a Virtual Environment

1python -m venv venv
2source venv/bin/activate  # On Windows use `venv\\Scripts\\activate`
3

Step 2: Install Required Packages

1pip install videosdk
2

Step 3: Configure API Keys in a .env file

Create a .env file in your project root and add your VideoSDK API key:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

Here is the complete code to get started:
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "{\n  \"persona\": \"Real-time Audio Streaming Specialist\",\n  \"capabilities\": [\n    \"Provide detailed information about real-time audio streaming technologies and protocols.\",\n    \"Assist users in setting up and troubleshooting real-time audio streaming systems.\",\n    \"Offer guidance on optimizing audio quality and reducing latency in streaming.\",\n    \"Explain the benefits and applications of real-time audio streaming in various industries.\"\n  ],\n  \"constraints\": [\n    \"You are not a certified audio engineer and should advise users to consult professionals for complex technical issues.\",\n    \"Avoid providing legal advice regarding copyright and licensing of streamed content.\",\n    \"Ensure users are aware of privacy and data protection considerations when streaming audio.\"\n  ]\n}"
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=DeepgramSTT(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=SileroVAD(threshold=0.35),
32        turn_detector=TurnDetector(threshold=0.8)
33    )
34
35    session = AgentSession(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63

Step 4.1: Generating a VideoSDK Meeting ID

To generate a meeting ID, use the following curl command:
1curl -X POST "https://api.videosdk.live/v1/meetings" \
2-H "Authorization: Bearer YOUR_API_KEY" \
3-H "Content-Type: application/json"
4

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class defines the agent's behavior. It inherits from the Agent class and uses the agent_instructions to guide its interactions.
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6

Step 4.3: Defining the Core Pipeline

The CascadingPipeline manages the audio processing flow:
  • STT: Converts speech to text using DeepgramSTT.
  • LLM: Processes text with

    OpenAI LLM Plugin for voice agent

    .
  • TTS: Converts text back to speech using ElevenLabsTTS.
  • VAD & TurnDetector: Detects when to start and stop listening.
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=[Silero Voice Activity Detection](https://docs.videosdk.live/ai_agents/plugins/silero-vad)(threshold=0.35),
6    turn_detector=[Turn detector for AI voice Agents](https://docs.videosdk.live/ai_agents/plugins/turn-detector)(threshold=0.8)
7)
8

Step 4.4: Managing the Session and Startup Logic

The start_session function initializes and starts the agent session, while make_context sets up the room options.
1def make_context() -> JobContext:
2    room_options = RoomOptions(
3        name="VideoSDK Cascaded Agent",
4        playground=True
5    )
6    return JobContext(room_options=room_options)
7
1if __name__ == "__main__":
2    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
3    job.start()
4

Running and Testing the Agent

Step 5.1: Running the Python Script

Execute the script using:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Once the script is running, find the playground link in the console. Join the session and interact with your agent.

Advanced Features and Customizations

Extending Functionality with Custom Tools

You can add custom tools to your agent to extend its capabilities. This involves defining function_tool methods that the agent can call during interactions.

Exploring Other Plugins

Consider exploring other STT, LLM, and TTS plugins supported by VideoSDK to enhance your agent's performance.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure your API key is correctly set in the .env file and that it has the necessary permissions.

Audio Input/Output Problems

Check your microphone and speaker settings. Ensure they are properly configured and not muted.

Dependency and Version Conflicts

Ensure all dependencies are installed with compatible versions. Use a virtual environment to manage dependencies.

Conclusion

Summary of What You've Built

In this tutorial, you built a real-time audio streaming AI Voice Agent using the VideoSDK framework. You learned how to set up the environment, create an agent, and test it in a playground.

Next Steps and Further Learning

Explore more advanced features of VideoSDK, such as integrating with other APIs or customizing the agent's behavior further.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ