AI Voice Agents for Healthcare

Step-by-step guide to building AI Voice Agents for healthcare using VideoSDK.

Introduction to AI Voice Agents in Healthcare

AI Voice Agents are sophisticated systems that leverage artificial intelligence to interpret and respond to human speech. In the healthcare industry, these agents can assist with tasks such as scheduling appointments, providing general health information, and answering common queries about symptoms. This technology is particularly valuable in healthcare for its ability to improve accessibility and efficiency.

What is an AI

Voice Agent

?

An AI

Voice Agent

 is a software application that uses voice recognition and natural language processing (NLP) to interact with users. These agents can understand spoken language, process the information, and respond appropriately, making them ideal for hands-free operations.

Why are they important for the healthcare industry?

In healthcare, AI Voice Agents can streamline operations by handling routine inquiries, freeing up medical staff to focus on more complex tasks. They can provide 24/7 assistance, ensuring patients receive timely information and support.

Core Components of a

Voice Agent

  • Speech-to-Text (STT): Converts spoken language into text.
  • Large Language Model (LLM): Processes the text to understand and generate responses.
  • Text-to-Speech (TTS): Converts text responses back into speech.

  • AI voice Agent core components overview

    : Provides a comprehensive understanding of how these components interact.

What You'll Build in This Tutorial

In this tutorial, you'll build a healthcare-focused AI

Voice Agent

 using the VideoSDK framework. This agent will be capable of answering healthcare-related questions and assisting with appointment scheduling.

Architecture and Core Concepts

High-Level Architecture Overview

The AI

Voice Agent

 processes user speech through several stages: speech is first converted to text (STT), then the text is analyzed and a response is generated (LLM), and finally, the response is converted back to speech (TTS).
Diagram

Understanding Key Concepts in the VideoSDK Framework

Setting Up the Development Environment

Prerequisites

Ensure you have Python 3.11+ installed and a VideoSDK account. Sign up at app.videosdk.live.

Step 1: Create a Virtual Environment

Create a virtual environment to manage dependencies:
1python -m venv venv
2source venv/bin/activate  # On Windows use `venv\\Scripts\\activate`
3

Step 2: Install Required Packages

Install the necessary packages:
1pip install videosdk-agents videosdk-plugins
2

Step 3: Configure API Keys in a .env file

Create a .env file in your project directory and add your VideoSDK API keys:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

Here is the complete code for the AI Voice Agent:
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "You are a helpful healthcare assistant AI Voice Agent designed to assist users with healthcare-related inquiries. Your primary capabilities include answering questions about common symptoms, providing general health tips, and assisting with scheduling appointments with healthcare providers. However, you are not a medical professional, and you must always include a disclaimer advising users to consult a qualified healthcare provider for medical advice, diagnosis, or treatment. You should be empathetic, informative, and concise in your responses. You must respect user privacy and adhere to data protection regulations. You cannot provide emergency assistance or handle sensitive personal health information."
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=DeepgramSTT(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=SileroVAD(threshold=0.35),
32        turn_detector=TurnDetector(threshold=0.8)
33    )
34
35    session = [AI voice Agent Sessions](https://docs.videosdk.live/ai_agents/core-components/agent-session)(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63

Step 4.1: Generating a VideoSDK Meeting ID

To interact with your agent, you'll need a meeting ID. Use the following curl command to generate one:
1curl -X POST "https://api.videosdk.live/v1/meetings" \
2-H "Authorization: Bearer YOUR_API_KEY" \
3-H "Content-Type: application/json"
4

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class extends the Agent class from the VideoSDK framework. It defines the agent's behavior when entering and exiting a session:
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6

Step 4.3: Defining the Core Pipeline

The CascadingPipeline orchestrates the flow of audio data through various processing stages:
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8

Step 4.4: Managing the Session and Startup Logic

The start_session function initializes and starts the agent session, while make_context sets up the job context:
1def make_context() -> JobContext:
2    room_options = RoomOptions(
3        name="VideoSDK Cascaded Agent",
4        playground=True
5    )
6    return JobContext(room_options=room_options)
7
8async def start_session(context: JobContext):
9    agent = MyVoiceAgent()
10    conversation_flow = ConversationFlow(agent)
11    pipeline = CascadingPipeline(
12        stt=DeepgramSTT(model="nova-2", language="en"),
13        llm=OpenAILLM(model="gpt-4o"),
14        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
15        vad=SileroVAD(threshold=0.35),
16        turn_detector=TurnDetector(threshold=0.8)
17    )
18    session = AgentSession(
19        agent=agent,
20        pipeline=pipeline,
21        conversation_flow=conversation_flow
22    )
23    try:
24        await context.connect()
25        await session.start()
26        await asyncio.Event().wait()
27    finally:
28        await session.close()
29        await context.shutdown()
30

Running and Testing the Agent

Step 5.1: Running the Python Script

Run your script using the following command:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Once the script is running, you'll receive a playground link in the console. Use this link to join the session and interact with your AI Voice Agent.

Advanced Features and Customizations

Extending Functionality with Custom Tools

Enhance your agent's capabilities by integrating custom tools using the function_tool concept in the VideoSDK framework.

Exploring Other Plugins

Consider experimenting with different STT, LLM, and TTS plugins to optimize performance and cost.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure your API keys are correctly configured in the .env file.

Audio Input/Output Problems

Verify your audio devices are functioning correctly and configured in your system settings.

Dependency and Version Conflicts

Ensure all dependencies are compatible with Python 3.11+ and update them as necessary.

Conclusion

Summary of What You've Built

You've successfully built an AI Voice Agent for healthcare using the VideoSDK framework, capable of handling healthcare-related inquiries.

Next Steps and Further Learning

Explore additional features and plugins offered by VideoSDK to further enhance your agent's capabilities.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ