Running LLMs Locally with AI Voice Agents

Build an AI Voice Agent to run LLMs locally using VideoSDK. Follow this step-by-step guide with complete code examples.

Introduction to AI Voice Agents in Running LLMs Locally

AI Voice Agents are advanced systems designed to interact with users through spoken language. They leverage technologies such as Speech-to-Text (STT), Large Language Models (LLM), and Text-to-Speech (TTS) to process and respond to user queries. These agents are particularly valuable in environments where hands-free operation is essential, such as in industrial settings or for accessibility purposes.
In the context of running Large Language Models (LLMs) locally, AI Voice Agents can provide real-time assistance, guiding users through complex setups and offering insights into hardware and software requirements. This tutorial will guide you through building an AI

Voice Agent

 using the VideoSDK framework, enabling you to run LLMs locally with ease.

Core Components of a

Voice Agent

  • Speech-to-Text (STT): Converts spoken language into text.
  • Large Language Model (LLM): Processes the text to generate meaningful responses.
  • Text-to-Speech (TTS): Converts the generated text back into speech.

What You'll Build in This Tutorial

In this tutorial, you will create a fully functional AI

Voice Agent

 capable of assisting users in running LLMs locally. The agent will utilize VideoSDK's framework to integrate STT, LLM, and TTS components seamlessly.

Architecture and Core Concepts

High-Level Architecture Overview

The AI

Voice Agent

 architecture consists of several components that work together to process user input and generate responses. The flow begins with capturing user speech, converting it to text, processing it through an LLM, and finally converting the response back to speech.
Diagram

Understanding Key Concepts in the VideoSDK Framework

Setting Up the Development Environment

Prerequisites

To begin, ensure you have Python 3.11+ installed and a VideoSDK account. You can sign up at the VideoSDK website.

Step 1: Create a Virtual Environment

1python3 -m venv my_voice_agent_env
2source my_voice_agent_env/bin/activate
3

Step 2: Install Required Packages

Install the necessary packages using pip:
1pip install videosdk
2pip install python-dotenv
3

Step 3: Configure API Keys in a .env File

Create a .env file in your project directory and add your VideoSDK API key:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

Below is the complete, runnable code for the AI Voice Agent. We will break it down in the following sections.
1import asyncio, os
2from videosdk.agents import Agent, [AgentSession](https://docs.videosdk.live/ai_agents/core-components/agent-session), CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import [Silero Voice Activity Detection](https://docs.videosdk.live/ai_agents/plugins/silero-vad)
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import [OpenAI LLM Plugin for voice agent](https://docs.videosdk.live/ai_agents/plugins/llm/openai)
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "You are a knowledgeable AI Voice Agent specializing in providing guidance on running large language models (LLMs) locally. Your primary role is to assist users in understanding the process, requirements, and best practices for setting up and operating LLMs on their local machines. You can offer insights into hardware specifications, software dependencies, and troubleshooting common issues. However, you are not a technical support engineer, and users should be advised to consult official documentation or professional support for complex technical problems. Always remind users to ensure they have the necessary permissions and licenses to run LLMs locally."
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=DeepgramSTT(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=SileroVAD(threshold=0.35),
32        turn_detector=TurnDetector(threshold=0.8)
33    )
34
35    session = AgentSession(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63

Step 4.1: Generating a VideoSDK Meeting ID

To create a meeting ID, use the following curl command:
1curl -X POST "https://api.videosdk.live/v1/meetings" \
2-H "Content-Type: application/json" \
3-H "Authorization: Bearer YOUR_API_KEY" \
4-d '{}'
5

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class extends the Agent class, providing custom instructions and handling entry and exit messages.
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6

Step 4.3: Defining the Core Pipeline

The CascadingPipeline integrates various plugins to handle STT, LLM, TTS, VAD, and turn detection.
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8

Step 4.4: Managing the Session and Startup Logic

The start_session function initializes the agent session and manages the connection lifecycle.
1async def start_session(context: JobContext):
2    # Create agent and conversation flow
3    agent = MyVoiceAgent()
4    conversation_flow = ConversationFlow(agent)
5
6    # Create pipeline
7    pipeline = CascadingPipeline(
8        stt=DeepgramSTT(model="nova-2", language="en"),
9        llm=OpenAILLM(model="gpt-4o"),
10        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
11        vad=SileroVAD(threshold=0.35),
12        turn_detector=TurnDetector(threshold=0.8)
13    )
14
15    session = AgentSession(
16        agent=agent,
17        pipeline=pipeline,
18        conversation_flow=conversation_flow
19    )
20
21    try:
22        await context.connect()
23        await session.start()
24        # Keep the session running until manually terminated
25        await asyncio.Event().wait()
26    finally:
27        # Clean up resources when done
28        await session.close()
29        await context.shutdown()
30
31if __name__ == "__main__":
32    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
33    job.start()
34

Running and Testing the Agent

Step 5.1: Running the Python Script

Execute the Python script using:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Once the script is running, find the playground link in the console to interact with your agent. Use Ctrl+C to gracefully shut down the session.

Advanced Features and Customizations

Extending Functionality with Custom Tools

You can enhance your agent's capabilities by integrating custom tools and functions into the pipeline.

Exploring Other Plugins

Consider experimenting with different STT, LLM, and TTS plugins to suit your specific needs.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure your API key is correctly set in the .env file and that you have the necessary permissions.

Audio Input/Output Problems

Check your microphone and speaker settings to ensure they are properly configured.

Dependency and Version Conflicts

Make sure all dependencies are compatible with your Python version and other installed packages.

Conclusion

Summary of What You've Built

In this tutorial, you've built a comprehensive AI Voice Agent capable of assisting users in running LLMs locally. You've learned to set up the environment, create a custom agent, and manage sessions.

Next Steps and Further Learning

Explore additional features and plugins offered by VideoSDK to further enhance your AI Voice Agent's capabilities, and refer to the

AI voice Agent core components overview

 for more insights.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ