Build an AI Voice Agent for E-commerce

Step-by-step guide to building an AI Voice Agent for the e-commerce industry using VideoSDK.

Introduction to AI Voice Agents in E-commerce

What is an AI

Voice Agent

?

An AI

Voice Agent

 is an advanced software application designed to interact with users through voice commands. These agents leverage technologies such as speech-to-text (STT), natural language processing (NLP), and text-to-speech (TTS) to understand and respond to user queries. They are often integrated into systems to automate customer service, provide information, and perform tasks based on voice commands.

Why are they important for the e-commerce industry?

In the e-commerce industry, AI Voice Agents can significantly enhance the customer experience by providing instant support and guidance. They can assist with product inquiries, order tracking, and even personalized recommendations, thereby improving customer satisfaction and potentially increasing sales. The ability to interact with customers in a natural, conversational manner makes voice agents a valuable tool for online retailers.

Core Components of a

Voice Agent

  1. Speech-to-Text (STT): Converts spoken language into text.
  2. Language Learning Model (LLM): Processes the text to understand the intent and context.
  3. Text-to-Speech (TTS): Converts the processed response back into spoken language.
For a comprehensive understanding, refer to the

AI voice Agent core components overview

.

What You'll Build in This Tutorial

In this tutorial, you will build a fully functional AI

Voice Agent

 tailored for the e-commerce industry using the VideoSDK framework. We will guide you through setting up the environment, writing the code, and testing the agent.

Architecture and Core Concepts

High-Level Architecture Overview

The architecture of an AI

Voice Agent

 involves several key components working in harmony to process user input and generate responses. The process begins with capturing the user's voice input, which is then converted into text using STT. The text is processed by an LLM to determine the appropriate response, which is then converted back to speech using TTS.
Diagram

Understanding Key Concepts in the VideoSDK Framework

  • Agent: The core class representing your bot, responsible for managing interactions.
  • CascadingPipeline: Defines the flow of audio processing from STT to LLM to TTS. Learn more about the

    Cascading pipeline in AI voice Agents

    .
  • VAD & TurnDetector: These components help the agent determine when to listen and when to speak, enhancing the natural flow of conversation.

Setting Up the Development Environment

Prerequisites

To get started, ensure you have Python 3.11 or higher installed on your system. Additionally, you will need a VideoSDK account, which you can create at app.videosdk.live.

Step 1: Create a Virtual Environment

Create a virtual environment to manage your project dependencies:
1python -m venv venv
2source venv/bin/activate  # On Windows use `venv\\Scripts\\activate`
3

Step 2: Install Required Packages

Install the necessary packages using pip:
1pip install videosdk-agents
2

Step 3: Configure API Keys in a .env file

Create a .env file in your project directory and add your VideoSDK API keys:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

Here is the complete code for building your AI Voice Agent:
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "You are an AI Voice Agent specialized in the e-commerce industry, designed to assist customers with their online shopping experience. Your primary role is to provide a seamless and efficient interaction for users seeking information about products, order status, and general inquiries related to e-commerce platforms. \n\nCapabilities:\n1. Answer questions about product details, availability, and pricing.\n2. Assist users in tracking their orders and providing updates on delivery status.\n3. Guide users through the process of making a purchase, including adding items to the cart and checking out.\n4. Provide information on return policies and customer support contact details.\n5. Offer personalized product recommendations based on user preferences and browsing history.\n\nConstraints:\n1. You are not authorized to process payments or handle sensitive financial information.\n2. You must always include a disclaimer that users should verify critical information on the official e-commerce website.\n3. You cannot provide legal or financial advice.\n4. You should not store any personal user data beyond the current session.\n5. You must ensure user privacy and adhere to data protection regulations."
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=DeepgramSTT(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=SileroVAD(threshold=0.35),
32        turn_detector=TurnDetector(threshold=0.8)
33    )
34
35    session = AgentSession(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63

Step 4.1: Generating a VideoSDK Meeting ID

To generate a meeting ID, you can use the following curl command:
1curl -X POST https://api.videosdk.live/v1/meetings -H "Authorization: Bearer YOUR_API_KEY"
2
This command will return a meeting ID that you can use to connect your agent.

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class inherits from the Agent class, which is the core of your voice agent. It defines how the agent interacts with users:
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6

Step 4.3: Defining the Core Pipeline

The CascadingPipeline is crucial as it defines the sequence of processing steps:
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8
Each component in the pipeline is responsible for a specific task: converting speech to text, processing the text, converting text back to speech, and managing the conversation flow.

Step 4.4: Managing the Session and Startup Logic

The session management and startup logic ensure that your agent runs smoothly:
1async def start_session(context: JobContext):
2    # Create agent and conversation flow
3    agent = MyVoiceAgent()
4    conversation_flow = ConversationFlow(agent)
5
6    # Create pipeline
7    pipeline = CascadingPipeline(
8        stt=DeepgramSTT(model="nova-2", language="en"),
9        llm=OpenAILLM(model="gpt-4o"),
10        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
11        vad=SileroVAD(threshold=0.35),
12        turn_detector=TurnDetector(threshold=0.8)
13    )
14
15    session = AgentSession(
16        agent=agent,
17        pipeline=pipeline,
18        conversation_flow=conversation_flow
19    )
20
21    try:
22        await context.connect()
23        await session.start()
24        # Keep the session running until manually terminated
25        await asyncio.Event().wait()
26    finally:
27        # Clean up resources when done
28        await session.close()
29        await context.shutdown()
30
The make_context function sets up the environment for your agent:
1def make_context() -> JobContext:
2    room_options = RoomOptions(
3    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
4        name="VideoSDK Cascaded Agent",
5        playground=True
6    )
7
8    return JobContext(room_options=room_options)
9

Running and Testing the Agent

Step 5.1: Running the Python Script

To run your agent, execute the following command in your terminal:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Once the script is running, look for the playground link in your console. Use this link to join the session and interact with your agent. You can test various scenarios such as asking about product details or order status.

Advanced Features and Customizations

Extending Functionality with Custom Tools

You can extend your agent's functionality by integrating custom tools. This involves creating new classes that can be plugged into the existing pipeline to perform additional tasks.

Exploring Other Plugins

The VideoSDK framework supports a variety of plugins for STT, LLM, and TTS. You can explore alternatives such as Cartesia for STT or Google Gemini for LLM to enhance your agent's capabilities. Consider using the

OpenAI LLM Plugin for voice agent

 for advanced language processing.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure that your API keys are correctly configured in the .env file. Double-check for any typos or missing entries.

Audio Input/Output Problems

If you encounter issues with audio input or output, verify your microphone and speaker settings. Ensure that the correct devices are selected in your system's audio settings.

Dependency and Version Conflicts

Make sure all dependencies are installed with compatible versions. Use a virtual environment to manage these dependencies and avoid conflicts.

Conclusion

Summary of What You've Built

In this tutorial, you have built a robust AI Voice Agent tailored for the e-commerce industry. You learned about the core components, set up the development environment, and implemented the agent using the VideoSDK framework. For insights into performance, explore

AI voice Agent Session Analytics

.

Next Steps and Further Learning

To further enhance your agent, consider exploring more advanced features such as integrating with external databases or APIs. Continue experimenting with different plugins and configurations to optimize your agent's performance.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ