Build an AI Voice Assistant for Hotels

Step-by-step tutorial to build a hotel AI voice assistant using VideoSDK with complete code examples.

Introduction to AI Voice Agents in Hotels

What is an AI Voice Agent?

An AI Voice Agent is a software application that uses artificial intelligence to interact with users through voice commands. These agents can understand spoken language, process the information, and respond in a human-like manner. They are commonly used in various industries to automate customer service, provide information, and enhance user experiences.

Why are they important for the hotel industry?

In the hotel industry, AI Voice Agents can significantly improve guest experiences by providing instant assistance with room service, offering local recommendations, and answering frequently asked questions. They help reduce the workload on human staff, allowing them to focus on more complex tasks that require personal interaction.

Core Components of a Voice Agent

To build an effective AI Voice Agent, three core components are essential:
  • Speech-to-Text (STT): Converts spoken language into text.
  • Large Language Model (LLM): Processes the text and generates a response.
  • Text-to-Speech (TTS): Converts the generated text back into spoken language.

What You'll Build in This Tutorial

In this tutorial, you will learn how to build an AI Voice Assistant tailored for hotel environments using the VideoSDK framework. We will guide you through setting up the development environment, building the agent, and testing it in a simulated environment.

Architecture and Core Concepts

High-Level Architecture Overview

The architecture of an AI Voice Agent involves several key components working together to process user input and generate responses. The data flow typically follows these steps:
  1. User Speech: The guest speaks into the system.
  2. Voice Activity Detection (VAD): Identifies when the user is speaking.
  3. Speech-to-Text (STT): Converts the speech into text.
  4. Language Processing (LLM): Analyzes the text and formulates a response.
  5. Text-to-Speech (TTS): Converts the response text back into speech.
  6. Agent Response: The system speaks back to the user.
1sequenceDiagram
2    participant User
3    participant VAD
4    participant STT
5    participant LLM
6    participant TTS
7    participant Agent
8    User->>VAD: Speak
9    VAD->>STT: Detect Speech
10    STT->>LLM: Convert to Text
11    LLM->>TTS: Generate Response
12    TTS->>Agent: Convert to Speech
13    Agent->>User: Speak Response
14

Understanding Key Concepts in the VideoSDK Framework

  • Agent: The core class representing your bot. It manages the interaction between the user and the system.
  • CascadingPipeline: A sequence of processes that handle audio input, language processing, and audio output. Learn more about the

    cascading pipeline in AI voice Agents

    .
  • VAD & TurnDetector: Tools that help the agent determine when to listen and when to respond, ensuring smooth interaction. Explore the

    turn detector for AI voice Agents

     for more details.

Setting Up the Development Environment

Prerequisites

Before you start, ensure you have the following:
  • Python 3.11+ installed on your system.
  • A VideoSDK account, which you can create at app.videosdk.live.

Step 1: Create a Virtual Environment

Open your terminal and run the following commands to create and activate a virtual environment:
1python -m venv hotel-voice-agent
2source hotel-voice-agent/bin/activate  # On Windows use `hotel-voice-agent\\Scripts\\activate`
3

Step 2: Install Required Packages

Install the necessary Python packages using pip:
1pip install videosdk-agents videosdk-plugins-silero videosdk-plugins-turn-detector videosdk-plugins-deepgram videosdk-plugins-openai videosdk-plugins-elevenlabs
2

Step 3: Configure API Keys in a .env file

Create a .env file in your project directory and add your API keys for VideoSDK and other services:
1VIDEOSDK_API_KEY=your_videosdk_api_key
2DEEPGRAM_API_KEY=your_deepgram_api_key
3OPENAI_API_KEY=your_openai_api_key
4ELEVENLABS_API_KEY=your_elevenlabs_api_key
5

Building the AI Voice Agent: A Step-by-Step Guide

Here is the complete code for the AI Voice Agent that we will build and explain in this section:
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "You are a helpful hotel concierge AI Voice Assistant designed to enhance guest experiences. Your primary capabilities include providing information about hotel amenities, assisting with room service orders, offering local area recommendations, and answering frequently asked questions about the hotel. You can also help guests with booking services such as spa appointments or restaurant reservations. However, you are not a human concierge and must inform guests that for any complex or personalized requests, they should contact the hotel staff directly. Additionally, you must respect guest privacy and not store any personal information."
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=DeepgramSTT(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=SileroVAD(threshold=0.35),
32        turn_detector=TurnDetector(threshold=0.8)
33    )
34
35    session = AgentSession(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63

Step 4.1: Generating a VideoSDK Meeting ID

To generate a meeting ID, you can use the following curl command. This ID is necessary for the agent to join a session:
1curl -X POST "https://api.videosdk.live/v1/meetings" -H "Authorization: Bearer YOUR_VIDEOSDK_API_KEY"
2

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class is where we define the behavior of our AI Voice Assistant. It inherits from the Agent class provided by VideoSDK. The on_enter and on_exit methods define what the agent says when a session starts and ends.
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6

Step 4.3: Defining the Core Pipeline

The CascadingPipeline is a crucial part of the agent, defining how audio is processed. It includes STT, LLM, TTS, VAD, and Turn Detector components. For speech-to-text, we use the

Deepgram STT Plugin for voice agent

, and for text-to-speech, the

ElevenLabs TTS Plugin for voice agent

 is utilized.
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8

Step 4.4: Managing the Session and Startup Logic

The start_session function initializes the agent session, connects to the context, and starts the session. The make_context function sets up the room options, and the main block runs the job.
1async def start_session(context: JobContext):
2    # Create agent and conversation flow
3    agent = MyVoiceAgent()
4    conversation_flow = ConversationFlow(agent)
5
6    # Create pipeline
7    pipeline = CascadingPipeline(...)
8
9    session = AgentSession(
10        agent=agent,
11        pipeline=pipeline,
12        conversation_flow=conversation_flow
13    )
14
15    try:
16        await context.connect()
17        await session.start()
18        # Keep the session running until manually terminated
19        await asyncio.Event().wait()
20    finally:
21        # Clean up resources when done
22        await session.close()
23        await context.shutdown()
24
25if __name__ == "__main__":
26    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
27    job.start()
28

Running and Testing the Agent

Step 5.1: Running the Python Script

To start the AI Voice Agent, run the following command in your terminal:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Once the agent is running, a playground link will appear in the console. Open this link in a browser to interact with your AI Voice Assistant. You can speak to it and receive spoken responses, simulating a real hotel environment.

Advanced Features and Customizations

Extending Functionality with Custom Tools

The VideoSDK framework allows you to extend your agent's functionality using custom tools. You can integrate additional APIs or services to enhance the agent's capabilities. For a comprehensive understanding, refer to the

AI voice Agent core components overview

.

Exploring Other Plugins

While this tutorial uses specific plugins for STT, LLM, and TTS, VideoSDK supports various other options. You can experiment with different plugins to find the best fit for your needs.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure that your API keys are correctly configured in the .env file. Double-check the keys for any typos or incorrect values.

Audio Input/Output Problems

Verify that your microphone and speakers are properly connected and configured. Test them with other applications to ensure they are working correctly.

Dependency and Version Conflicts

If you encounter issues with package versions, ensure that your virtual environment is activated and that all packages are up-to-date. Use pip list to check installed versions.

Conclusion

Summary of What You've Built

In this tutorial, you have successfully built an AI Voice Assistant tailored for hotel environments using the VideoSDK framework. This assistant can handle common guest requests and enhance the overall guest experience.

Next Steps and Further Learning

To further enhance your AI Voice Assistant, consider exploring additional plugins and custom tools. You can also experiment with different models and configurations to optimize performance and capabilities. For a quick start, refer to the

Voice Agent Quick Start Guide

.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ