Build AI Voice Agent for Utilities

Step-by-step guide to building an AI Voice Agent for the utilities industry using VideoSDK.

Introduction to AI Voice Agents in the Utilities Industry

AI Voice Agents are transforming how industries interact with their customers. In this tutorial, we will explore how to build an AI

Voice Agent

 specifically tailored for the utilities industry, which includes services like electricity, water, and gas. These agents can handle customer inquiries, provide billing information, report service outages, and even offer energy-saving tips.

What is an AI

Voice Agent

?

An AI

Voice Agent

 is a software program that uses artificial intelligence to understand and respond to human speech. It combines several technologies, such as Speech-to-Text (STT), Language Models (LLM), and Text-to-Speech (TTS), to process and respond to user queries.

Why are they important for the Utilities Industry?

In the utilities industry, AI Voice Agents can significantly enhance customer service by providing quick and accurate responses to common queries. They can assist with troubleshooting issues, guide users through billing processes, and connect them with human representatives for more complex problems.

Core Components of a

Voice Agent

  • STT (Speech-to-Text): Converts spoken language into text.
  • LLM (Large Language Model): Processes the text and generates a response.
  • TTS (Text-to-Speech): Converts the generated text response back into speech.

  • Cascading pipeline in AI voice Agents

    : Manages the flow of audio processing, integrating STT, LLM, and TTS components.

What You'll Build in This Tutorial

In this guide, you will learn how to build a fully functional AI

Voice Agent

 using the VideoSDK AI Agents framework. We will cover everything from setting up your development environment to deploying and testing the agent.

Architecture and Core Concepts

High-Level Architecture Overview

The architecture of an AI Voice Agent involves several components working together to process user input and generate a response. Here is a high-level overview of the data flow:
Diagram

Understanding Key Concepts in the VideoSDK Framework

  • Agent: Represents the core logic of your AI Voice Agent.
  • CascadingPipeline: Manages the flow of audio processing, integrating STT, LLM, and TTS components.
  • VAD & TurnDetector: These components help the agent determine when to listen and when to speak. The

    Silero Voice Activity Detection

     plugin is used to detect voice activity, while the

    Turn detector for AI voice Agents

     ensures smooth conversation flow.

Setting Up the Development Environment

Prerequisites

Before you start, ensure you have Python 3.11+ installed and a VideoSDK account. You can sign up at app.videosdk.live.

Step 1: Create a Virtual Environment

Create a virtual environment to manage your dependencies:
1python -m venv venv
2source venv/bin/activate  # On Windows use `venv\Scripts\activate`
3

Step 2: Install Required Packages

Install the necessary Python packages using pip:
1pip install videosdk-agents videosdk-plugins
2

Step 3: Configure API Keys in a .env File

Create a .env file in your project directory and add your VideoSDK API keys:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

To build the AI Voice Agent, we will use the complete code provided and break it down into smaller parts to understand each component.
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "You are a knowledgeable AI Voice Agent specialized in the utilities industry. Your primary role is to assist users with inquiries related to utility services such as electricity, water, and gas. You can provide information on billing, service outages, and energy-saving tips. Additionally, you can guide users through troubleshooting common issues and connect them with customer service representatives for more complex problems. However, you are not a certified technician and should always recommend consulting a professional for technical repairs. Always ensure user data privacy and adhere to industry regulations."
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=DeepgramSTT(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=SileroVAD(threshold=0.35),
32        turn_detector=TurnDetector(threshold=0.8)
33    )
34
35    session = [AI voice Agent Sessions](https://docs.videosdk.live/ai_agents/core-components/agent-session)(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63

Step 4.1: Generating a VideoSDK Meeting ID

To interact with your AI Voice Agent, you need a meeting ID. You can generate one using the following curl command:
1curl -X POST "https://api.videosdk.live/v1/meetings" \
2-H "Authorization: Bearer YOUR_API_KEY" \
3-H "Content-Type: application/json"
4

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class extends the Agent class from the VideoSDK framework. It defines specific behaviors when the agent enters or exits a conversation. The agent is designed to assist with utility-related inquiries.
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6

Step 4.3: Defining the Core Pipeline

The CascadingPipeline is a crucial component that defines how audio is processed. It integrates various plugins for STT, LLM, TTS, VAD, and turn detection.
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8

Step 4.4: Managing the Session and Startup Logic

The start_session function manages the agent's session, establishing the connection and handling the conversation flow. The make_context function sets up the environment, and the main block starts the agent.
1def make_context() -> JobContext:
2    room_options = RoomOptions(
3        name="VideoSDK Cascaded Agent",
4        playground=True
5    )
6    return JobContext(room_options=room_options)
7
8if __name__ == "__main__":
9    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
10    job.start()
11

Running and Testing the Agent

Step 5.1: Running the Python Script

To run your AI Voice Agent, execute the following command in your terminal:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Once the script is running, you will receive a playground link in the console. Open this link in your browser to interact with your agent. You can speak to the agent and receive responses in real-time.

Advanced Features and Customizations

Extending Functionality with Custom Tools

You can extend the functionality of your AI Voice Agent by integrating custom tools. This allows you to add specific capabilities tailored to your needs.

Exploring Other Plugins

The VideoSDK framework supports various plugins for STT, LLM, and TTS. Explore these options to find the best fit for your use case.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure your API keys are correctly configured in the .env file. Double-check the authorization headers in your requests.

Audio Input/Output Problems

Verify your microphone and speaker settings. Ensure that your system permissions allow audio access.

Dependency and Version Conflicts

Use a virtual environment to manage dependencies and avoid conflicts. Ensure all packages are up-to-date.

Conclusion

Summary of What You've Built

Congratulations! You've built a fully functional AI Voice Agent for the utilities industry using the VideoSDK framework. This agent can handle various customer inquiries and provide valuable assistance.

Next Steps and Further Learning

Explore additional plugins and features offered by the VideoSDK framework to further enhance your AI Voice Agent. Consider diving deeper into custom tool integration and advanced conversation flows. For more advanced deployment options, refer to the

AI voice Agent deployment

 documentation.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ