Build AI Voice Agents for API Calls

Step-by-step guide to build AI Voice Agents for chatbot API calls using VideoSDK.

Introduction to AI Voice Agents in API Calls from a Chatbot

In today's digital landscape, AI Voice Agents are transforming the way we interact with technology. These agents are not only capable of understanding and responding to human speech but also executing tasks such as making API calls from a chatbot. This tutorial will guide you through building an AI

Voice Agent

 using the VideoSDK framework, focusing on executing API calls within a chatbot environment.

What is an AI

Voice Agent

?

An AI

Voice Agent

 is a software program that uses artificial intelligence to understand and respond to voice commands. It integrates various technologies like Speech-to-Text (STT), Language Learning Models (LLM), and Text-to-Speech (TTS) to facilitate seamless human-computer interaction.

Why are they important for the API Calls from a Chatbot Industry?

AI Voice Agents are crucial in the chatbot industry as they enhance user experience by allowing voice interactions, which are more natural and efficient. They enable chatbots to perform tasks such as making API calls, providing information, and assisting users in real-time.

Core Components of a

Voice Agent

The core components of a

voice agent

 include:
  • STT (Speech-to-Text): Converts spoken language into text.
  • LLM (Language Learning Model): Processes the text to understand and generate responses.
  • TTS (Text-to-Speech): Converts text responses back into speech.

What You'll Build in This Tutorial

In this tutorial, you'll build an AI Voice Agent that can assist users with making API calls from a chatbot. You'll learn how to set up the development environment, build the agent using the VideoSDK framework, and test it in a

playground environment

.

Architecture and Core Concepts

High-Level Architecture Overview

The architecture of an AI Voice Agent involves a flow of data from user speech to agent response. The process begins with capturing user speech, converting it to text, processing the text to generate a response, and finally converting the response back to speech.
Diagram

Understanding Key Concepts in the VideoSDK Framework

  • Agent: The core class representing your bot, responsible for handling interactions.

  • Cascading Pipeline

    :     Manages the flow of audio processing from STT to LLM to TTS.
  • VAD & TurnDetector: These components help the agent determine when to listen and when to speak, ensuring smooth interaction.

Setting Up the Development Environment

Prerequisites

Before you begin, ensure you have Python 3.11+ installed and a VideoSDK account. You can sign up at app.videosdk.live.

Step 1: Create a Virtual Environment

To keep your project dependencies isolated, create a virtual environment:
1python -m venv venv
2source venv/bin/activate  # On Windows use `venv\Scripts\activate`
3

Step 2: Install Required Packages

Install the necessary packages using pip:
1pip install videosdk agents
2

Step 3: Configure API Keys in a .env File

Create a .env file in your project directory and add your VideoSDK API keys:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

Here is the complete, runnable code for the AI Voice Agent:
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "You are an AI Voice Agent specialized in assisting users with making API calls from a chatbot. Your persona is that of a knowledgeable and efficient technical assistant. Your primary capabilities include guiding users through the process of setting up and executing API calls within a chatbot environment, providing examples of common API call structures, and troubleshooting common issues related to API integrations. You can also suggest best practices for secure and efficient API usage. However, you are not a substitute for professional software development advice, and you must remind users to consult with a qualified developer for complex integrations or security concerns. Additionally, you should not store or process any sensitive user data, and you must ensure that all interactions comply with relevant data protection regulations."
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=DeepgramSTT(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=[Silero Voice Activity Detection](https://docs.videosdk.live/ai_agents/plugins/silero-vad)(threshold=0.35),
32        turn_detector=[Turn detector for AI voice Agents](https://docs.videosdk.live/ai_agents/plugins/turn-detector)(threshold=0.8)
33    )
34
35    session = [AI voice Agent Sessions](https://docs.videosdk.live/ai_agents/core-components/agent-session)(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63

Step 4.1: Generating a VideoSDK Meeting ID

To interact with your agent, you need a meeting ID. Use the following curl command to generate one:
1curl -X POST https://api.videosdk.live/v1/meetings \
2-H "Authorization: Bearer your_api_key_here"
3

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class is where you define the behavior of your voice agent. It inherits from the Agent class and uses the agent_instructions to guide interactions.
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6

Step 4.3: Defining the Core Pipeline

The CascadingPipeline is crucial for processing audio data. It defines the flow from STT to LLM to TTS, using plugins like DeepgramSTT, OpenAILLM, and ElevenLabsTTS.
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8

Step 4.4: Managing the Session and Startup Logic

The start_session function manages the agent's session lifecycle, while make_context sets up the environment for the agent to run.
1def make_context() -> JobContext:
2    room_options = RoomOptions(
3    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
4        name="VideoSDK Cascaded Agent",
5        playground=True
6    )
7
8    return JobContext(room_options=room_options)
9
10if __name__ == "__main__":
11    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
12    job.start()
13

Running and Testing the Agent

Step 5.1: Running the Python Script

To start the agent, run the following command in your terminal:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Once the agent is running, you'll see a playground link in the console. Open it in your browser to interact with your voice agent.

Advanced Features and Customizations

Extending Functionality with Custom Tools

You can extend the agent's functionality by adding custom tools. This involves defining new capabilities that the agent can use during interactions.

Exploring Other Plugins

The VideoSDK framework supports various plugins for STT, LLM, and TTS. Consider exploring options like Cartesia for STT or Google Gemini for LLM to enhance your agent.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure your API keys are correctly configured in the .env file and that you're using the correct authorization headers.

Audio Input/Output Problems

Check your microphone and speaker settings. Ensure that the correct devices are selected in your system settings.

Dependency and Version Conflicts

Verify that all dependencies are installed and compatible with your Python version. Use pip freeze to check installed packages.

Conclusion

Summary of What You've Built

In this tutorial, you've built a fully functional AI Voice Agent capable of making API calls from a chatbot. You've learned how to set up the development environment, build the agent, and test it using the VideoSDK framework.

Next Steps and Further Learning

Explore additional features and plugins to enhance your agent's capabilities. Consider integrating more complex functionalities and learning about advanced AI and machine learning techniques.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ