AI Telephony Agent vs AI Voice Agent Guide

Explore the differences between AI Telephony and Voice Agents with a practical implementation guide using VideoSDK.

Introduction to AI Voice Agents in ai telephony agent vs ai voice agent

In the rapidly evolving world of artificial intelligence, AI Voice Agents have become a cornerstone technology, particularly in the realm of telephony services. These agents are designed to interact with users through voice, providing a seamless and intuitive experience. But what exactly is an AI Voice Agent, and why are they so important in the context of telephony?

What is an AI Voice Agent?

An AI Voice Agent is a software application that uses artificial intelligence to process and respond to human speech. These agents can perform a variety of tasks, from answering customer inquiries to providing detailed information on specific topics. They leverage technologies such as Speech-to-Text (STT), Language Learning Models (LLM), and Text-to-Speech (TTS) to understand and generate human-like responses.

Why are they important for the ai telephony agent vs ai voice agent industry?

In the telephony industry, AI Voice Agents play a crucial role in enhancing customer service and operational efficiency. They can handle large volumes of calls, provide consistent and accurate information, and operate 24/7, reducing the need for human operators and improving customer satisfaction.

Core Components of a Voice Agent

The core components of an AI Voice Agent include:
  • Speech-to-Text (STT): Converts spoken language into text.
  • Language Learning Model (LLM): Processes the text to understand and generate responses.
  • Text-to-Speech (TTS): Converts text responses back into spoken language.
For a detailed understanding of these components, refer to the

AI voice Agent core components overview

.

What You'll Build in This Tutorial

In this tutorial, you will build a fully functional AI Voice Agent using the VideoSDK framework. This agent will be capable of engaging in conversations, providing insights into the differences between AI telephony agents and AI voice agents, and answering related questions. You can get started with the

Voice Agent Quick Start Guide

.

Architecture and Core Concepts

High-Level Architecture Overview

The architecture of an AI Voice Agent involves several key components working in harmony. The process begins with user speech, which is captured and converted into text using STT. This text is then processed by an LLM to generate a suitable response, which is finally converted back into speech using TTS.
Diagram

Understanding Key Concepts in the VideoSDK Framework

  • Agent: The core class representing your bot, responsible for managing interactions.
  • CascadingPipeline: Manages the flow of audio processing, ensuring smooth transitions from STT to LLM to TTS. Learn more about the

    Cascading pipeline in AI voice Agents

    .
  • VAD & TurnDetector: These components help the agent understand when to listen and when to respond, ensuring natural conversation flow. For more details, check out the

    Turn detector for AI voice Agents

    .

Setting Up the Development Environment

Prerequisites

Before you begin, ensure you have the following:
  • Python 3.11+
  • A VideoSDK account (sign up at app.videosdk.live)

Step 1: Create a Virtual Environment

Create a virtual environment to manage your project dependencies:
1python -m venv myenv
2source myenv/bin/activate  # On Windows use `myenv\Scripts\activate`
3

Step 2: Install Required Packages

Install the necessary packages using pip:
1pip install videosdk-agents videosdk-plugins
2

Step 3: Configure API Keys in a .env file

Create a .env file in your project directory and add your VideoSDK API key:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

To build your AI Voice Agent, we'll start by presenting the complete, runnable code. This will give you an overview of what you'll be working towards.
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "You are an AI Voice Agent specializing in telephony services. Your persona is that of a knowledgeable and friendly customer service representative. Your primary role is to assist users in understanding the differences between AI telephony agents and AI voice agents. 
14
15Capabilities:
161. Provide clear explanations of what AI telephony agents and AI voice agents are, including their primary functions and use cases.
172. Compare and contrast the features and benefits of AI telephony agents versus AI voice agents.
183. Answer frequently asked questions about AI telephony and voice agents, such as their integration capabilities, cost implications, and technological requirements.
194. Offer guidance on choosing the right type of agent based on specific business needs or scenarios.
20
21Constraints and Limitations:
221. You are not a technical support agent and cannot provide detailed troubleshooting or technical setup instructions.
232. You must include a disclaimer that users should consult with a technical expert or service provider for personalized advice and implementation.
243. Avoid making definitive statements about future developments or capabilities beyond current technology trends.
254. Ensure all information provided is up-to-date and based on the latest industry standards and practices."
26
27class MyVoiceAgent(Agent):
28    def __init__(self):
29        super().__init__(instructions=agent_instructions)
30    async def on_enter(self): await self.session.say("Hello! How can I help?")
31    async def on_exit(self): await self.session.say("Goodbye!")
32
33async def start_session(context: JobContext):
34    # Create agent and conversation flow
35    agent = MyVoiceAgent()
36    conversation_flow = ConversationFlow(agent)
37
38    # Create pipeline
39    pipeline = CascadingPipeline(
40        stt=DeepgramSTT(model="nova-2", language="en"),
41        llm=OpenAILLM(model="gpt-4o"),
42        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
43        vad=SileroVAD(threshold=0.35),
44        turn_detector=TurnDetector(threshold=0.8)
45    )
46
47    session = AgentSession(
48        agent=agent,
49        pipeline=pipeline,
50        conversation_flow=conversation_flow
51    )
52
53    try:
54        await context.connect()
55        await session.start()
56        # Keep the session running until manually terminated
57        await asyncio.Event().wait()
58    finally:
59        # Clean up resources when done
60        await session.close()
61        await context.shutdown()
62
63def make_context() -> JobContext:
64    room_options = RoomOptions(
65    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
66        name="VideoSDK Cascaded Agent",
67        playground=True
68    )
69
70    return JobContext(room_options=room_options)
71
72if __name__ == "__main__":
73    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
74    job.start()
75

Step 4.1: Generating a VideoSDK Meeting ID

Before running your agent, you'll need a meeting ID. You can generate one using the VideoSDK API. Here's an example using curl:
1curl -X POST \
2  https://api.videosdk.live/v1/meetings \
3  -H "Authorization: Bearer YOUR_API_KEY" \
4  -H "Content-Type: application/json"
5

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class is where you define the behavior and personality of your AI Voice Agent. This class extends the base Agent class from the VideoSDK framework and includes methods for handling session entry and exit.
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6

Step 4.3: Defining the Core Pipeline

The CascadingPipeline is crucial for managing the flow of audio data through the system. It integrates STT, LLM, TTS, VAD, and TurnDetector plugins to ensure smooth and accurate processing. For TTS, you can utilize the

ElevenLabs TTS Plugin for voice agent

, and for STT, consider the

Deepgram STT Plugin for voice agent

.
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8

Step 4.4: Managing the Session and Startup Logic

The start_session function is responsible for initializing and managing the agent's session. It connects the agent to the VideoSDK service and starts the conversation flow.
1async def start_session(context: JobContext):
2    agent = MyVoiceAgent()
3    conversation_flow = ConversationFlow(agent)
4
5    pipeline = CascadingPipeline(
6        stt=DeepgramSTT(model="nova-2", language="en"),
7        llm=OpenAILLM(model="gpt-4o"),
8        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
9        vad=SileroVAD(threshold=0.35),
10        turn_detector=TurnDetector(threshold=0.8)
11    )
12
13    session = AgentSession(
14        agent=agent,
15        pipeline=pipeline,
16        conversation_flow=conversation_flow
17    )
18
19    try:
20        await context.connect()
21        await session.start()
22        await asyncio.Event().wait()
23    finally:
24        await session.close()
25        await context.shutdown()
26
The make_context function creates a JobContext with room options, enabling the agent to join a meeting or create a new one. You can experiment with your agent in the

AI Agent playground

.
1def make_context() -> JobContext:
2    room_options = RoomOptions(
3    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
4        name="VideoSDK Cascaded Agent",
5        playground=True
6    )
7
8    return JobContext(room_options=room_options)
9
The main block starts the agent job, which runs the session logic.
1if __name__ == "__main__":
2    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
3    job.start()
4

Running and Testing the Agent

Step 5.1: Running the Python Script

To run your AI Voice Agent, execute the following command in your terminal:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Once the script is running, you'll receive a playground URL in the console. Open this URL in your browser to interact with your agent. You can speak to the agent and receive responses based on the instructions you provided.

Advanced Features and Customizations

Extending Functionality with Custom Tools

The VideoSDK framework allows you to extend the functionality of your agent using custom tools. These tools can be integrated into the pipeline to add new capabilities or enhance existing ones.

Exploring Other Plugins

While this tutorial uses specific plugins for STT, LLM, and TTS, the VideoSDK framework supports various other options. You can explore these plugins to customize your agent further.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure your API key is correctly configured in the .env file. If you encounter authentication errors, verify your key and account status.

Audio Input/Output Problems

Check your microphone and speaker settings if you experience issues with audio input or output. Ensure your hardware is functioning correctly and is properly configured.

Dependency and Version Conflicts

If you encounter dependency issues, ensure all packages are up-to-date and compatible with your Python version. Use a virtual environment to manage dependencies effectively.

Conclusion

Summary of What You've Built

In this tutorial, you've built a fully functional AI Voice Agent capable of engaging in conversations and providing insights into AI telephony and voice agents.

Next Steps and Further Learning

To further enhance your agent, consider exploring additional plugins and customizing the agent's behavior to suit specific use cases. Continue learning about AI and voice technologies to stay ahead in this rapidly evolving field.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ