Build AI Voice Assistants for Public Services

Step-by-step guide to building AI voice assistants for public services using VideoSDK.

Introduction to AI Voice Agents in Public Services

AI Voice Agents are sophisticated software systems designed to interact with humans through voice commands. They utilize technologies such as Speech-to-Text (STT), Language Learning Models (LLM), and Text-to-Speech (TTS) to understand and respond to user queries. In the context of public services, these agents can significantly enhance user experience by providing instant information and assistance related to healthcare, transportation, and government facilities.

What is an AI

Voice Agent

?

An AI

Voice Agent

 is a digital assistant that processes voice inputs from users, interprets them using natural language processing, and provides responses through synthesized speech. These agents are capable of handling a wide range of tasks, from answering FAQs to guiding users through complex procedures.

Why are they important for Public Services?

In public services, AI Voice Agents can streamline operations and improve accessibility. They can handle high volumes of inquiries, provide 24/7 support, and reduce the workload on human staff. Use cases include automated customer service in healthcare, real-time transit updates, and guidance on government services.

Core Components of a

Voice Agent

  • Speech-to-Text (STT): Converts spoken language into text.
  • Language Learning Model (LLM): Processes the text to understand and generate responses.
  • Text-to-Speech (TTS): Converts the response text back into spoken language.

What You'll Build in This Tutorial

In this tutorial, you will learn how to build an AI Voice Assistant tailored for public services using the VideoSDK AI Agents framework. You will implement a complete solution that can be tested and customized for various public service applications.

Architecture and Core Concepts

High-Level Architecture Overview

The AI

Voice Agent

 architecture involves several key components working in tandem to process user input and generate responses. The data flow begins with the user's speech, which is captured and converted into text by the STT component. The text is then processed by the LLM to generate a suitable response, which is converted back into speech by the TTS component.
Diagram

Understanding Key Concepts in the VideoSDK Framework

  • Agent: The core class representing your bot, responsible for handling interactions.

  • Cascading pipeline in AI voice Agents

    :     Manages the flow of audio processing through STT, LLM, and TTS.
  • VAD & TurnDetector: These components help the agent determine when to listen and when to respond.

Setting Up the Development Environment

Prerequisites

To get started, ensure you have Python 3.11+ installed and a VideoSDK account. You can sign up at app.videosdk.live.

Step 1: Create a Virtual Environment

Create a virtual environment to manage dependencies:
1python -m venv venv
2source venv/bin/activate  # On Windows, use `venv\Scripts\activate`
3

Step 2: Install Required Packages

Install the necessary packages using pip:
1pip install videosdk
2

Step 3: Configure API Keys in a .env file

Create a .env file to store your API keys securely:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

Below is the complete code to build your AI Voice Agent. We will break it down into parts for detailed explanation.
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10pre_download_model()
11
12agent_instructions = "You are a knowledgeable and efficient AI Voice Assistant designed to support public services. Your primary role is to assist users by providing information and guidance related to various public services such as healthcare, transportation, and government facilities. You can answer frequently asked questions, provide step-by-step guidance on accessing services, and offer general information about public service operations. However, you must always include a disclaimer that you are not a human representative and that users should verify information with official sources. You are not authorized to make decisions on behalf of users or access personal data beyond what is necessary for the interaction. Always prioritize user privacy and data security."
13
14class MyVoiceAgent(Agent):
15    def __init__(self):
16        super().__init__(instructions=agent_instructions)
17    async def on_enter(self): await self.session.say("Hello! How can I help?")
18    async def on_exit(self): await self.session.say("Goodbye!")
19
20async def start_session(context: JobContext):
21    agent = MyVoiceAgent()
22    conversation_flow = ConversationFlow(agent)
23
24    pipeline = CascadingPipeline(
25        stt=DeepgramSTT(model="nova-2", language="en"),
26        llm=[OpenAI LLM Plugin for voice agent](https://docs.videosdk.live/ai_agents/plugins/llm/openai)(model="gpt-4o"),
27        tts=[ElevenLabs TTS Plugin for voice agent](https://docs.videosdk.live/ai_agents/plugins/tts/eleven-labs)(model="eleven_flash_v2_5"),
28        vad=[Silero Voice Activity Detection](https://docs.videosdk.live/ai_agents/plugins/silero-vad)(threshold=0.35),
29        turn_detector=[Turn detector for AI voice Agents](https://docs.videosdk.live/ai_agents/plugins/turn-detector)(threshold=0.8)
30    )
31
32    session = AgentSession(
33        agent=agent,
34        pipeline=pipeline,
35        conversation_flow=conversation_flow
36    )
37
38    try:
39        await context.connect()
40        await session.start()
41        await asyncio.Event().wait()
42    finally:
43        await session.close()
44        await context.shutdown()
45
46def make_context() -> JobContext:
47    room_options = RoomOptions(
48        name="VideoSDK Cascaded Agent",
49        playground=True
50    )
51
52    return JobContext(room_options=room_options)
53
54if __name__ == "__main__":
55    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
56    job.start()
57

Step 4.1: Generating a VideoSDK Meeting ID

To generate a meeting ID, use the following curl command:
1curl -X POST \
2  https://api.videosdk.live/v1/meetings \
3  -H "Authorization: Bearer YOUR_API_KEY" \
4  -H "Content-Type: application/json"
5

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class extends the Agent class, providing custom behavior for entering and exiting interactions. It uses predefined instructions to guide its responses.

Step 4.3: Defining the Core Pipeline

The CascadingPipeline is the backbone of the agent, integrating various plugins:
  • DeepgramSTT: Transcribes user speech into text.
  • OpenAILLM: Processes text to generate a response.
  • ElevenLabsTTS: Converts the response text to speech.
  • SileroVAD: Detects voice activity to manage listening.
  • TurnDetector: Manages conversation flow by detecting when to listen and respond.

Step 4.4: Managing the Session and Startup Logic

The start_session function initializes the agent session and manages the lifecycle of the interaction. The make_context function sets up the room options, and the main block starts the agent.

Running and Testing the Agent

Step 5.1: Running the Python Script

Execute the script using:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Upon running, the console will provide a playground link. Use this link to join and interact with your agent.

Advanced Features and Customizations

Extending Functionality with Custom Tools

The function_tool concept allows you to extend the agent's capabilities by integrating custom logic and tools.

Exploring Other Plugins

Consider experimenting with different STT/LLM/TTS plugins to enhance the agent's functionality and performance.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure your API keys are correct and stored securely in the .env file.

Audio Input/Output Problems

Verify your audio devices are configured correctly and compatible with the agent.

Dependency and Version Conflicts

Ensure all dependencies are installed with compatible versions as specified in the requirements.

Conclusion

Summary of What You've Built

You have successfully built an AI Voice Assistant for public services using the VideoSDK framework. This agent can handle various public service inquiries and provide valuable assistance.

Next Steps and Further Learning

Explore additional features and plugins to enhance your agent's capabilities. Continue learning about AI and voice technologies to build more advanced solutions.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ