Build an AI Call Bot with VideoSDK

Learn to build a powerful AI call bot using VideoSDK with our step-by-step guide.

Introduction to AI Voice Agents in AI Call Bot

AI Voice Agents are sophisticated software programs designed to interact with users through voice commands. They are a core component of modern customer service solutions, providing efficient and scalable ways to handle customer inquiries and tasks.

What is an AI

Voice Agent

?

An AI

Voice Agent

 is a virtual assistant that uses speech recognition, natural language processing, and speech synthesis to understand and respond to user queries. These agents can perform tasks, answer questions, and provide information in a conversational manner.

Why are they important for the AI Call Bot industry?

In the AI Call Bot industry, voice agents play a crucial role in automating customer interactions. They help reduce wait times, increase customer satisfaction, and provide 24/7 service availability. Use cases include customer support, appointment scheduling, and information retrieval.

Core Components of a

Voice Agent

  • Speech-to-Text (STT): Converts spoken language into text.
  • Large Language Model (LLM): Processes the text to understand intent and generate responses.
  • Text-to-Speech (TTS): Converts text responses back into spoken language.

What You'll Build in This Tutorial

In this tutorial, you will build an AI Call Bot using the VideoSDK framework. This bot will be capable of handling customer service inquiries, providing information, and guiding users through processes.

Architecture and Core Concepts

High-Level Architecture Overview

The AI

Voice Agent

 processes user input through a series of steps, starting from capturing voice input to generating a spoken response. The flow involves converting speech to text, processing the text to determine intent, and synthesizing a spoken response.
Diagram

Understanding Key Concepts in the VideoSDK Framework

  • Agent: The core class representing your bot, responsible for managing interactions.
  • CascadingPipeline: A sequence of audio processing steps, including STT, LLM, and TTS.

  • Cascading pipeline in AI voice Agents

    :     This is a structured approach that ensures seamless integration of different audio processing components.
  • VAD & TurnDetector: Tools to determine when the agent should listen and speak.

Setting Up the Development Environment

Prerequisites

To get started, ensure you have Python 3.11+ installed and a VideoSDK Account. Sign up at app.videosdk.live.

Step 1: Create a Virtual Environment

Create a virtual environment to manage dependencies:
1python3 -m venv venv
2source venv/bin/activate  # On Windows use `venv\Scripts\activate`
3

Step 2: Install Required Packages

Install the necessary Python packages using pip:
1pip install videosdk
2pip install python-dotenv
3

Step 3: Configure API Keys in a .env file

Create a .env file in your project directory and add your VideoSDK API keys:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

Here is the complete code for the AI Voice Agent:
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "You are an AI Call Bot designed to assist users with customer service inquiries. Your primary role is to provide accurate information, resolve common issues, and guide users through processes related to their accounts or services. You can handle tasks such as answering frequently asked questions, processing simple transactions, and escalating complex issues to human representatives when necessary. However, you must adhere to the following constraints: you cannot provide legal or financial advice, you must always verify user identity before accessing sensitive information, and you should remind users to contact a human representative for unresolved issues or emergencies. Your tone should be professional, courteous, and efficient, ensuring a positive user experience."
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=DeepgramSTT(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=SileroVAD(threshold=0.35),
32        turn_detector=TurnDetector(threshold=0.8)
33    )
34
35    session = AgentSession(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63

Step 4.1: Generating a VideoSDK Meeting ID

To interact with your AI Voice Agent, you need a meeting ID. You can generate one using the VideoSDK API:
1curl -X POST \
2  https://api.videosdk.live/v1/meetings \
3  -H "Authorization: YOUR_API_KEY" \
4  -H "Content-Type: application/json"
5

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class inherits from the Agent class and defines the behavior of your AI Call Bot. It includes methods for entering and exiting interactions with users.
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6

Step 4.3: Defining the Core Pipeline

The CascadingPipeline is the backbone of the AI Voice Agent, integrating various plugins for processing audio and generating responses. This includes the

OpenAI LLM Plugin for voice agent

 and

Silero Voice Activity Detection

 to enhance interaction capabilities.
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8

Step 4.4: Managing the Session and Startup Logic

The session management involves setting up the

AI voice Agent Sessions

 and handling the lifecycle of the agent.
1def make_context() -> JobContext:
2    room_options = RoomOptions(
3    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
4        name="VideoSDK Cascaded Agent",
5        playground=True
6    )
7
8    return JobContext(room_options=room_options)
9
10if __name__ == "__main__":
11    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
12    job.start()
13

Running and Testing the Agent

Step 5.1: Running the Python Script

To start your AI Voice Agent, run the script using Python:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Once the script is running, you will see a test URL in the console. Use this link to join the meeting and interact with your AI Voice Agent.

Advanced Features and Customizations

Extending Functionality with Custom Tools

You can extend your AI Voice Agent by adding custom tools to handle specific tasks or integrate additional services.

Exploring Other Plugins

Explore different STT, LLM, and TTS plugins to enhance the capabilities of your AI Voice Agent. Consider utilizing the

Turn detector for AI voice Agents

 to improve conversational flow.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure your API keys are correctly configured in the .env file and that you have the necessary permissions.

Audio Input/Output Problems

Check your microphone and speaker settings to ensure they are correctly configured.

Dependency and Version Conflicts

Verify that all dependencies are installed and compatible with your Python version.

Conclusion

Summary of What You've Built

In this tutorial, you built a fully functional AI Call Bot using the VideoSDK framework, capable of handling customer inquiries and providing information. For a comprehensive understanding of the system, refer to the

AI voice Agent core components overview

.

Next Steps and Further Learning

Explore additional features and plugins to enhance your AI Voice Agent, and consider integrating it into a broader customer service platform.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ