Build a Conversational AI for eCommerce

Step-by-step guide to building a conversational AI voice agent for eCommerce using VideoSDK.

Introduction to AI Voice Agents in Conversational AI in eCommerce

In today's fast-paced digital landscape, eCommerce businesses are constantly seeking innovative ways to enhance customer engagement and streamline operations. One such innovation is the use of AI Voice Agents. These virtual assistants are designed to interact with users through natural language, providing a seamless and interactive shopping experience.

What is an AI

Voice Agent

?

An AI

Voice Agent

 is a software application that uses artificial intelligence to understand and respond to human speech. It combines various technologies like Speech-to-Text (STT), Language Models (LLM), and Text-to-Speech (TTS) to process user inputs and generate appropriate responses.

Why are they important for the Conversational AI in eCommerce industry?

AI Voice Agents are transforming the eCommerce industry by offering personalized shopping experiences, assisting with customer inquiries, and providing product recommendations. They help businesses reduce operational costs and improve customer satisfaction by automating routine tasks and offering 24/7 support.

Core Components of a

Voice Agent

  • STT (Speech-to-Text): Converts spoken language into text.
  • LLM (Language Model): Processes text to understand context and intent.
  • TTS (Text-to-Speech): Converts text responses back into spoken language.

What You'll Build in This Tutorial

In this tutorial, you'll learn how to build a Conversational AI

Voice Agent

 for eCommerce using the VideoSDK framework. We'll guide you through setting up the development environment, creating a custom agent, and testing it in a real-world scenario.

Architecture and Core Concepts

High-Level Architecture Overview

The architecture of a

voice agent

 involves several components working together to process user input and generate responses. Here’s a high-level overview of the data flow:
  1. User Speech: The user speaks into their device.
  2. STT Conversion: The speech is converted to text using an STT plugin.
  3. Language Processing: The text is processed by an LLM to determine the appropriate response.
  4. TTS Conversion: The response text is converted back to speech using a TTS plugin.
  5. Agent Response: The agent delivers the spoken response to the user.
Diagram

Understanding Key Concepts in the VideoSDK Framework

Setting Up the Development Environment

Prerequisites

Before you begin, ensure you have Python 3.11+ installed on your machine and a VideoSDK account. You can sign up at app.videosdk.live.

Step 1: Create a Virtual Environment

Create a virtual environment to manage your project dependencies:
1python -m venv venv
2source venv/bin/activate  # On Windows use `venv\Scripts\activate`
3

Step 2: Install Required Packages

Install the necessary packages using pip:
1pip install videosdk-agents videosdk-plugins
2

Step 3: Configure API Keys in a .env file

Create a .env file in your project directory and add your VideoSDK API keys:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

To build your AI Voice Agent, we'll start by presenting the complete code block and then break it down for detailed explanations.
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "You are a Conversational AI Agent specialized in eCommerce. Your persona is that of a friendly and knowledgeable shopping assistant. Your primary capabilities include assisting customers with product inquiries, providing personalized product recommendations, helping with order tracking, and answering questions about store policies and promotions. You can also guide users through the checkout process and provide information on payment options. However, you are not authorized to handle sensitive payment information directly, and you must always redirect users to secure payment gateways for transactions. Additionally, you should remind users that product availability and prices are subject to change and encourage them to check the latest details on the official website. Your responses should be concise, informative, and engaging, ensuring a seamless shopping experience for the user."
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=DeepgramSTT(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=SileroVAD(threshold=0.35),
32        turn_detector=TurnDetector(threshold=0.8)
33    )
34
35    session = AgentSession(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63

Step 4.1: Generating a VideoSDK Meeting ID

To interact with your agent, you need a meeting ID. You can generate one using the VideoSDK API. Here’s an example using curl:
1curl -X POST "https://api.videosdk.live/v1/meetings" \
2-H "Authorization: Bearer YOUR_API_KEY" \
3-H "Content-Type: application/json"
4

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class is where you define the behavior of your voice agent. It inherits from the Agent class and sets the agent's instructions:
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6

Step 4.3: Defining the Core Pipeline

The CascadingPipeline is crucial as it defines the flow of audio processing through various plugins:
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8
  • STT (DeepgramSTT): Converts user speech to text.
  • LLM (OpenAILLM): Processes text to generate responses.
  • TTS (ElevenLabsTTS): Converts text responses back to speech.
  • VAD (SileroVAD): Detects voice activity.
  • TurnDetector: Determines when to listen and when to speak.

Step 4.4: Managing the Session and Startup Logic

The start_session function manages the agent's session lifecycle, while make_context sets up the environment:
1async def start_session(context: JobContext):
2    # Create agent and conversation flow
3    agent = MyVoiceAgent()
4    conversation_flow = ConversationFlow(agent)
5
6    # Create pipeline
7    pipeline = CascadingPipeline(
8        stt=DeepgramSTT(model="nova-2", language="en"),
9        llm=OpenAILLM(model="gpt-4o"),
10        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
11        vad=SileroVAD(threshold=0.35),
12        turn_detector=TurnDetector(threshold=0.8)
13    )
14
15    session = AgentSession(
16        agent=agent,
17        pipeline=pipeline,
18        conversation_flow=conversation_flow
19    )
20
21    try:
22        await context.connect()
23        await session.start()
24        # Keep the session running until manually terminated
25        await asyncio.Event().wait()
26    finally:
27        # Clean up resources when done
28        await session.close()
29        await context.shutdown()
30
31def make_context() -> JobContext:
32    room_options = RoomOptions(
33    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
34        name="VideoSDK Cascaded Agent",
35        playground=True
36    )
37
38    return JobContext(room_options=room_options)
39
40if __name__ == "__main__":
41    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
42    job.start()
43

Running and Testing the Agent

Step 5.1: Running the Python Script

To run your agent, execute the following command in your terminal:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Once the script is running, you'll see a playground link in the console. Use this link to join the session and interact with your AI Voice Agent.

Advanced Features and Customizations

Extending Functionality with Custom Tools

The VideoSDK framework allows you to extend your agent's functionality by integrating custom tools. These tools can perform specific tasks, such as fetching data from external APIs.

Exploring Other Plugins

While this tutorial uses specific plugins, VideoSDK supports a variety of STT, LLM, and TTS options. Explore these to find the best fit for your needs.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure your API keys are correctly set in the .env file. Double-check for typos or missing keys.

Audio Input/Output Problems

Verify your microphone and speaker settings. Ensure your device permissions allow audio access.

Dependency and Version Conflicts

Use a virtual environment to manage dependencies and avoid version conflicts. Ensure all packages are up to date.

Conclusion

Summary of What You've Built

In this tutorial, you've built a Conversational AI Voice Agent for eCommerce using the VideoSDK framework. You’ve learned how to set up the environment, create a custom agent, and test it in a real-world scenario.

Next Steps and Further Learning

Continue exploring the VideoSDK framework and its plugins to enhance your agent's capabilities. Consider integrating additional features like sentiment analysis or multi-language support to further improve user experience. For a comprehensive understanding, refer to the

AI voice Agent core components overview

 and explore

AI voice Agent Sessions

 for deeper insights.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ