Build an AI Voice Agent for Hotels

Step-by-step guide to building an AI Voice Agent for hotels using VideoSDK.

Introduction to AI Voice Agents in Hotels

In recent years, AI Voice Agents have become an integral part of various industries, including hospitality. These agents are designed to interact with users through natural language, providing a seamless and efficient way to access information and services. In this tutorial, we will explore how to build an AI

Voice Agent

 specifically for hotel environments using the VideoSDK framework.

What is an AI

Voice Agent

?

An AI

Voice Agent

 is a software application that uses artificial intelligence to understand and respond to human speech. These agents leverage technologies such as Speech-to-Text (STT), Text-to-Speech (TTS), and Large Language Models (LLM) to process and generate natural language conversations.

Why are they important for the hotel industry?

In the hotel industry, AI Voice Agents can enhance guest experiences by providing instant access to information and services. They can assist with room service orders, provide local recommendations, and handle check-in and check-out procedures, allowing hotel staff to focus on more personalized guest interactions.

Core Components of a

Voice Agent

  • Speech-to-Text (STT): Converts spoken language into text.
  • Large Language Model (LLM): Processes the text to understand and generate responses.
  • Text-to-Speech (TTS): Converts text responses back into spoken language.
For a comprehensive understanding, refer to the

AI voice Agent core components overview

.

What You'll Build in This Tutorial

In this tutorial, you will build a fully functional AI

Voice Agent

 for hotels using the VideoSDK framework. We will guide you through setting up the development environment, building the agent, and testing it in a simulated environment.

Architecture and Core Concepts

High-Level Architecture Overview

The architecture of an AI

Voice Agent

 involves several components working together to process user input and generate responses. The data flow typically follows this sequence:
  1. User speaks into the microphone.
  2. The audio is captured and processed by the Speech-to-Text (STT) service.
  3. The transcribed text is sent to the Large Language Model (LLM) for understanding and response generation.
  4. The generated text response is converted back to audio using Text-to-Speech (TTS).
  5. The audio response is played back to the user.
Diagram

Understanding Key Concepts in the VideoSDK Framework

  • Agent: The core class representing your bot, responsible for handling user interactions.
  • CascadingPipeline: Manages the flow of audio processing through STT, LLM, and TTS. Learn more about the

    Cascading pipeline in AI voice Agents

    .
  • VAD & TurnDetector: These components help the agent determine when to listen and when to speak.

Setting Up the Development Environment

Prerequisites

Before you begin, ensure you have Python 3.11+ installed and a VideoSDK account. You can sign up at app.videosdk.live.

Step 1: Create a Virtual Environment

To manage dependencies, create a virtual environment:
1python -m venv venv
2source venv/bin/activate  # On Windows use `venv\Scripts\activate`
3

Step 2: Install Required Packages

Install the necessary packages using pip:
1pip install videosdk
2pip install python-dotenv
3

Step 3: Configure API Keys in a .env File

Create a .env file in your project directory and add your VideoSDK API key:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

To build the AI Voice Agent, we will use the following complete code block, which we will break down and explain in subsequent sections.
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "You are a friendly and knowledgeable AI Voice Agent designed specifically for hotel environments. Your primary role is to assist hotel guests by providing information and services related to their stay. You can answer questions about hotel amenities, provide local area recommendations, assist with room service orders, and help guests with check-in and check-out procedures. However, you are not a human concierge and cannot provide personal opinions or handle emergency situations. Always remind guests to contact hotel staff for urgent matters or personalized assistance. Your responses should be concise, polite, and informative, ensuring a pleasant experience for all guests."
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=DeepgramSTT(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=SileroVAD(threshold=0.35),
32        turn_detector=TurnDetector(threshold=0.8)
33    )
34
35    session = AgentSession(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63

Step 4.1: Generating a VideoSDK Meeting ID

To interact with your AI Voice Agent, you need a meeting ID. You can generate one using the VideoSDK API:
1curl -X POST "https://api.videosdk.live/v1/meetings" \
2-H "Authorization: Bearer YOUR_API_KEY" \
3-H "Content-Type: application/json"
4

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class is a custom implementation of the Agent class. It defines the agent's behavior when entering and exiting a session:
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6

Step 4.3: Defining the Core Pipeline

The CascadingPipeline is crucial for processing audio input and generating responses. It integrates various plugins:
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8

Step 4.4: Managing the Session and Startup Logic

The start_session function manages the session lifecycle, while make_context sets up the job context:
1async def start_session(context: JobContext):
2    # Create agent and conversation flow
3    agent = MyVoiceAgent()
4    conversation_flow = ConversationFlow(agent)
5
6    # Create pipeline
7    pipeline = CascadingPipeline(
8        stt=DeepgramSTT(model="nova-2", language="en"),
9        llm=OpenAILLM(model="gpt-4o"),
10        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
11        vad=SileroVAD(threshold=0.35),
12        turn_detector=TurnDetector(threshold=0.8)
13    )
14
15    session = AgentSession(
16        agent=agent,
17        pipeline=pipeline,
18        conversation_flow=conversation_flow
19    )
20
21    try:
22        await context.connect()
23        await session.start()
24        # Keep the session running until manually terminated
25        await asyncio.Event().wait()
26    finally:
27        # Clean up resources when done
28        await session.close()
29        await context.shutdown()
30
31def make_context() -> JobContext:
32    room_options = RoomOptions(
33    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
34        name="VideoSDK Cascaded Agent",
35        playground=True
36    )
37
38    return JobContext(room_options=room_options)
39
40if __name__ == "__main__":
41    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
42    job.start()
43

Running and Testing the Agent

Step 5.1: Running the Python Script

To run your AI Voice Agent, execute the following command:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Once the agent is running, you will find a playground link in the console output. Use this link to join the session and interact with the agent. The agent will respond to your queries as per the instructions defined. For detailed insights into session management, refer to

AI voice Agent Sessions

.

Advanced Features and Customizations

Extending Functionality with Custom Tools

The VideoSDK framework allows you to extend your agent's functionality using function_tool, enabling custom interactions and operations.

Exploring Other Plugins

While this tutorial uses specific plugins, VideoSDK supports various STT, LLM, and TTS options, allowing you to customize your agent further.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure your API key is correctly configured in the .env file and that it is valid.

Audio Input/Output Problems

Check your microphone and speaker settings to ensure they are correctly configured and operational.

Dependency and Version Conflicts

Ensure all dependencies are installed with compatible versions, and consider using a virtual environment to manage them.
For monitoring and debugging, explore

AI voice Agent tracing and observability

.

Conclusion

Summary of What You've Built

In this tutorial, you have built a fully functional AI Voice Agent for hotels using the VideoSDK framework. This agent can handle various guest interactions, enhancing the hospitality experience.

Next Steps and Further Learning

To expand your knowledge, explore additional VideoSDK plugins and consider integrating more advanced features into your AI Voice Agent.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ