Conversational AI in Insurance: Build a Voice Agent

Step-by-step guide to building a conversational AI voice agent for insurance using VideoSDK.

Introduction to AI Voice Agents in Conversational AI in Insurance

AI Voice Agents are transforming the way businesses interact with their customers. In the insurance industry, these agents can handle routine inquiries, provide policy information, and guide users through complex processes like claims and policy selection. This tutorial will guide you through building a conversational AI voice agent specifically tailored for the insurance sector using the VideoSDK framework.

What is an AI Voice Agent?

An AI Voice Agent is a software application that uses artificial intelligence to interpret and respond to human speech. These agents leverage technologies like Speech-to-Text (STT), Language Models (LLM), and Text-to-Speech (TTS) to understand and generate human-like conversations. For a comprehensive setup, refer to the

Voice Agent Quick Start Guide

.

Why are they important for the Insurance Industry?

In the insurance sector, AI Voice Agents can streamline customer service operations by handling common queries, providing policy details, and assisting with claims. This not only improves efficiency but also enhances customer satisfaction by providing quick and accurate responses.

Core Components of a Voice Agent

What You'll Build in This Tutorial

In this guide, you'll create an AI Voice Agent capable of assisting with insurance-related inquiries. You'll learn to set up the development environment, build the agent using the VideoSDK framework, and test its capabilities in a simulated environment.

Architecture and Core Concepts

To build a robust AI Voice Agent, understanding its architecture is crucial. The agent processes user speech, interprets it, and responds appropriately. Here’s a high-level overview of the architecture:

High-Level Architecture Overview

The process begins with the user’s speech being captured and converted into text using STT. The text is then processed by an LLM to generate a response, which is finally converted back into speech using TTS. This seamless flow allows for real-time interaction with users.
1sequenceDiagram
2    participant User
3    participant Agent
4    participant STT
5    participant LLM
6    participant TTS
7    User->>Agent: Speak
8    Agent->>STT: Convert Speech to Text
9    STT-->>Agent: Text
10    Agent->>LLM: Process Text
11    LLM-->>Agent: Response Text
12    Agent->>TTS: Convert Text to Speech
13    TTS-->>Agent: Speech
14    Agent->>User: Respond
15

Understanding Key Concepts in the VideoSDK Framework

  • Agent: The core class representing your bot. It handles interactions and manages the conversation flow.
  • CascadingPipeline: This defines the sequence of processing steps (STT -> LLM -> TTS) that the agent uses to handle user interactions. Learn more about the

    Cascading pipeline in AI voice Agents

    .
  • VAD & TurnDetector: These components help the agent determine when a user has finished speaking and when to start listening again. The

    Silero Voice Activity Detection

     and

    Turn detector for AI voice Agents

     are crucial for this functionality.

Setting Up the Development Environment

Before diving into code, let’s set up your development environment.

Prerequisites

Ensure you have Python 3.11+ installed and a VideoSDK account, which you can create at app.videosdk.live.

Step 1: Create a Virtual Environment

1python -m venv venv
2source venv/bin/activate  # On Windows use `venv\\Scripts\\activate`
3

Step 2: Install Required Packages

Install the necessary packages using pip:
1pip install videosdk-agents videosdk-plugins-silero videosdk-plugins-turn-detector videosdk-plugins-deepgram videosdk-plugins-openai videosdk-plugins-elevenlabs
2

Step 3: Configure API Keys in a .env file

Create a .env file in your project root and add your API keys:
1VIDEOSDK_API_KEY=your_videosdk_api_key
2DEEPGRAM_API_KEY=your_deepgram_api_key
3OPENAI_API_KEY=your_openai_api_key
4ELEVENLABS_API_KEY=your_elevenlabs_api_key
5

Building the AI Voice Agent: A Step-by-Step Guide

Now, let’s build the AI Voice Agent. Below is the complete code that we will break down step-by-step:
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "You are an AI Voice Agent specialized in the insurance industry, designed to assist users with their insurance-related inquiries. Your persona is that of a knowledgeable and friendly insurance advisor. Your primary capabilities include answering questions about various insurance policies (such as health, auto, home, and life insurance), providing quotes, explaining policy terms and conditions, and guiding users through the claims process. You can also help users find contact information for specific insurance providers and offer general advice on choosing the right insurance plan based on their needs.
14
15Constraints and limitations:
161. You are not a licensed insurance agent and cannot provide personalized financial advice or make policy recommendations.
172. Always include a disclaimer advising users to consult with a licensed insurance professional for detailed and personalized advice.
183. You cannot access or process personal data, and you must inform users that they should not share sensitive information such as policy numbers or personal identification details.
194. You are limited to providing information based on the data available up to October 2023 and cannot offer real-time updates or changes in insurance regulations or policies.
205. You must maintain a neutral tone and avoid endorsing any specific insurance company or product."
21
22class MyVoiceAgent(Agent):
23    def __init__(self):
24        super().__init__(instructions=agent_instructions)
25    async def on_enter(self): await self.session.say("Hello! How can I help?")
26    async def on_exit(self): await self.session.say("Goodbye!")
27
28async def start_session(context: JobContext):
29    # Create agent and conversation flow
30    agent = MyVoiceAgent()
31    conversation_flow = ConversationFlow(agent)
32
33    # Create pipeline
34    pipeline = CascadingPipeline(
35        stt=DeepgramSTT(model="nova-2", language="en"),
36        llm=OpenAILLM(model="gpt-4o"),
37        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
38        vad=SileroVAD(threshold=0.35),
39        turn_detector=TurnDetector(threshold=0.8)
40    )
41
42    session = AgentSession(
43        agent=agent,
44        pipeline=pipeline,
45        conversation_flow=conversation_flow
46    )
47
48    try:
49        await context.connect()
50        await session.start()
51        # Keep the session running until manually terminated
52        await asyncio.Event().wait()
53    finally:
54        # Clean up resources when done
55        await session.close()
56        await context.shutdown()
57
58def make_context() -> JobContext:
59    room_options = RoomOptions(
60    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
61        name="VideoSDK Cascaded Agent",
62        playground=True
63    )
64
65    return JobContext(room_options=room_options)
66
67if __name__ == "__main__":
68    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
69    job.start()
70

Step 4.1: Generating a VideoSDK Meeting ID

To interact with your agent, you need a meeting ID. Use the following curl command to generate one:
1curl -X POST https://api.videosdk.live/v1/meetings \
2  -H "Authorization: Bearer YOUR_VIDEOSDK_API_KEY" \
3  -H "Content-Type: application/json" \
4  -d '{}'
5

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class is where you define the agent’s behavior. It inherits from the Agent class and implements two methods: on_enter and on_exit. These methods define what the agent says when a session starts and ends.
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6

Step 4.3: Defining the Core Pipeline

The CascadingPipeline defines the flow of data from STT to LLM to TTS. Each component in the pipeline plays a crucial role in processing the user’s input and generating a response.
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8

Step 4.4: Managing the Session and Startup Logic

The start_session function manages the lifecycle of the agent’s session. It creates an AgentSession with the agent and pipeline, connects to the context, and starts the session. For more details on managing sessions, refer to

AI voice Agent Sessions

.
1async def start_session(context: JobContext):
2    agent = MyVoiceAgent()
3    conversation_flow = ConversationFlow(agent)
4    pipeline = CascadingPipeline(
5        stt=DeepgramSTT(model="nova-2", language="en"),
6        llm=OpenAILLM(model="gpt-4o"),
7        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
8        vad=SileroVAD(threshold=0.35),
9        turn_detector=TurnDetector(threshold=0.8)
10    )
11    session = AgentSession(agent=agent, pipeline=pipeline, conversation_flow=conversation_flow)
12    try:
13        await context.connect()
14        await session.start()
15        await asyncio.Event().wait()
16    finally:
17        await session.close()
18        await context.shutdown()
19
The make_context function sets up the job context, including room options for the agent to operate in.
1def make_context() -> JobContext:
2    room_options = RoomOptions(
3        name="VideoSDK Cascaded Agent",
4        playground=True
5    )
6    return JobContext(room_options=room_options)
7
Finally, the script’s main block initiates the agent’s operation.
1if __name__ == "__main__":
2    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
3    job.start()
4

Running and Testing the Agent

With the setup complete, it’s time to run and test your AI Voice Agent.

Step 5.1: Running the Python Script

Execute the script using Python:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Once the script is running, you’ll see a link to the VideoSDK playground in the console. Open this link in a browser to interact with your agent. Speak into your microphone, and the agent will respond based on the insurance-related instructions you provided.

Advanced Features and Customizations

As your AI Voice Agent becomes more sophisticated, you may want to extend its capabilities.

Extending Functionality with Custom Tools

VideoSDK allows you to integrate custom tools into your agent, enhancing its functionality beyond the built-in plugins.

Exploring Other Plugins

While this tutorial uses specific plugins for STT, LLM, and TTS, VideoSDK supports various other options you can explore to optimize your agent’s performance.

Troubleshooting Common Issues

Building an AI Voice Agent can come with challenges. Here are some common issues and solutions:

API Key and Authentication Errors

Ensure your API keys are correctly set in the .env file and that they have the necessary permissions.

Audio Input/Output Problems

Check your microphone and speaker settings to ensure they’re configured correctly.

Dependency and Version Conflicts

Use a virtual environment to manage dependencies and avoid version conflicts.

Conclusion

Congratulations! You’ve built a functional AI Voice Agent for the insurance industry. This agent can handle inquiries, provide policy information, and assist with claims. As a next step, consider exploring additional plugins and custom tools to further enhance your agent’s capabilities. Happy coding!

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ