Build an AI Voice Agent for BFSI

Step-by-step guide to build an AI Voice Agent for the BFSI sector using VideoSDK, complete with code and testing.

Introduction to AI Voice Agents in BFSI

AI Voice Agents are intelligent systems designed to interact with users through voice commands. They process spoken language, understand the context, and respond appropriately, making them invaluable in various industries, including BFSI (Banking, Financial Services, and Insurance). In BFSI, AI Voice Agents can handle customer inquiries, provide information on financial products, and assist in navigating complex services, enhancing customer experience and operational efficiency.

Why are they important for the BFSI industry?

In the BFSI sector, AI Voice Agents streamline customer service by offering 24/7 support, reducing wait times, and handling routine inquiries. They can provide details about account balances, recent transactions, loan options, and insurance policies, freeing human agents to focus on more complex tasks. This automation not only improves customer satisfaction but also reduces operational costs.

Core Components of a

Voice Agent

  • Speech-to-Text (STT): Converts spoken language into text.
  • Large Language Model (LLM): Processes the text to understand and generate responses.
  • Text-to-Speech (TTS): Converts the generated text response back into speech.
For a comprehensive understanding of these elements, refer to the

AI voice Agent core components overview

.

What You'll Build in This Tutorial

In this tutorial, you will build a fully functional AI

Voice Agent

 tailored for the BFSI industry using the VideoSDK framework. This agent will handle basic customer inquiries and provide information on financial products.

Architecture and Core Concepts

High-Level Architecture Overview

The AI

Voice Agent

 architecture involves several components that work together to process user input and generate responses. The process begins with capturing the user's speech, converting it into text using STT, processing the text with an LLM, and finally converting the response back into speech using TTS.
Diagram

Understanding Key Concepts in the VideoSDK Framework

  • Agent: The core class representing your bot, responsible for handling interactions.
  • CascadingPipeline: Manages the flow of audio processing from STT to LLM to TTS. For more details, see the

    Cascading pipeline in AI voice Agents

    .
  • VAD & TurnDetector: Ensure the agent listens and responds at the right times.

Setting Up the Development Environment

Prerequisites

Before starting, ensure you have Python 3.11+ installed and a VideoSDK account. Sign up at app.videosdk.live to access the necessary API keys.

Step 1: Create a Virtual Environment

Create a virtual environment to manage dependencies:
1python -m venv venv
2source venv/bin/activate  # On Windows use `venv\\Scripts\\activate`
3

Step 2: Install Required Packages

Install the VideoSDK and other required packages:
1pip install videosdk
2pip install python-dotenv
3

Step 3: Configure API Keys in a .env file

Create a .env file in your project directory and add your VideoSDK API keys:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

First, let's present the complete, runnable code:
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "You are an AI Voice Agent specialized in the BFSI (Banking, Financial Services, and Insurance) sector. Your persona is that of a knowledgeable and efficient financial assistant. Your primary capabilities include answering customer inquiries related to banking services, financial products, and insurance policies. You can provide information on account balances, recent transactions, loan options, and insurance coverage details. You can also assist users in navigating through various financial services and products offered by the institution. However, you are not a licensed financial advisor, and you must include a disclaimer advising users to consult with a certified financial professional for personalized advice. You must ensure user data privacy and comply with all relevant financial regulations and standards. You are not authorized to perform transactions or access sensitive personal information beyond what is necessary to answer inquiries."
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=DeepgramSTT(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=SileroVAD(threshold=0.35),
32        turn_detector=TurnDetector(threshold=0.8)
33    )
34
35    session = AgentSession(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63

Step 4.1: Generating a VideoSDK Meeting ID

To interact with your agent, you'll need a meeting ID. You can generate one using the VideoSDK API:
1curl -X POST \
2  https://api.videosdk.live/v1/meetings \
3  -H "Authorization: Bearer YOUR_API_KEY" \
4  -H "Content-Type: application/json"
5

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class extends the Agent class. It defines how the agent greets users when they enter and exit the session:
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6

Step 4.3: Defining the Core Pipeline

The CascadingPipeline orchestrates the flow of data through the agent:
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8
Each component in the pipeline plays a crucial role:

Step 4.4: Managing the Session and Startup Logic

The start_session function initializes the agent session and starts the conversation flow:
1async def start_session(context: JobContext):
2    agent = MyVoiceAgent()
3    conversation_flow = ConversationFlow(agent)
4    pipeline = CascadingPipeline(...)
5    session = AgentSession(
6        agent=agent,
7        pipeline=pipeline,
8        conversation_flow=conversation_flow
9    )
10    try:
11        await context.connect()
12        await session.start()
13        await asyncio.Event().wait()
14    finally:
15        await session.close()
16        await context.shutdown()
17
The make_context function sets up the room options for the agent:
1def make_context() -> JobContext:
2    room_options = RoomOptions(
3        name="VideoSDK Cascaded Agent",
4        playground=True
5    )
6    return JobContext(room_options=room_options)
7
Finally, the script starts the agent:
1if __name__ == "__main__":
2    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
3    job.start()
4

Running and Testing the Agent

Step 5.1: Running the Python Script

Run the script using Python:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Once the script is running, you'll see a playground link in the console. Use this link to join the meeting and interact with your AI Voice Agent. Speak into your microphone, and the agent will respond based on the BFSI context.

Advanced Features and Customizations

Extending Functionality with Custom Tools

You can extend the agent's functionality by integrating custom tools, allowing it to perform specific tasks or access additional data sources.

Exploring Other Plugins

VideoSDK supports various plugins for STT, LLM, and TTS. Explore options like Cartesia for STT or Google Gemini for LLM to enhance your agent's capabilities.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure your API key is correctly set in the .env file and that your account is active.

Audio Input/Output Problems

Check your microphone and speaker settings if the agent does not respond correctly.

Dependency and Version Conflicts

Ensure all dependencies are installed correctly and compatible with Python 3.11+.

Conclusion

Summary of What You've Built

You've built a comprehensive AI Voice Agent for the BFSI sector, capable of handling customer inquiries and providing information on financial services. For more details on managing sessions, refer to

AI voice Agent Sessions

.

Next Steps and Further Learning

Explore additional features and plugins to enhance your agent's capabilities. Consider integrating with other APIs to provide more personalized services.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ