AI Voice Assistants for Healthcare

Step-by-step guide to building AI voice assistants for healthcare using VideoSDK, complete with code and testing instructions.

Introduction to AI Voice Agents in AI Voice Assistants for Healthcare

What is an AI

Voice Agent

?

AI Voice Agents are sophisticated software systems designed to interact with users through voice commands. They leverage advanced technologies such as speech-to-text (STT), natural language processing (NLP), and text-to-speech (TTS) to understand and respond to user queries. These agents are increasingly becoming integral in various industries, providing seamless and efficient user interactions.

Why are they important for the AI Voice Assistants for Healthcare Industry?

In the healthcare sector, AI Voice Agents can revolutionize patient interaction by providing immediate responses to common health inquiries, assisting with appointment scheduling, and offering general health tips. They can enhance patient engagement and streamline administrative processes, allowing healthcare professionals to focus more on patient care.

Core Components of a

Voice Agent

  • STT (Speech-to-Text): Converts spoken language into text.
  • LLM (Large Language Model): Processes and understands text to generate meaningful responses.
  • TTS (Text-to-Speech): Converts text back into spoken language for user interaction.
For a comprehensive understanding, refer to the

AI voice Agent core components overview

.

What You'll Build in This Tutorial

In this tutorial, you will learn to build a healthcare-focused AI

Voice Agent

 using the VideoSDK AI Agents framework. This agent will be capable of answering health-related questions, providing general health advice, and assisting with appointment scheduling.

Architecture and Core Concepts

High-Level Architecture Overview

The AI

Voice Agent

 processes user speech through a series of components that convert speech to text, understand the query, and respond appropriately. Here’s a high-level overview of the data flow:
Diagram

Understanding Key Concepts in the VideoSDK Framework

  • Agent: The core class representing your bot, responsible for handling interactions.
  • CascadingPipeline: Manages the flow of audio processing from STT to LLM to TTS. Learn more about the

    Cascading pipeline in AI voice Agents

    .
  • VAD & TurnDetector: These components help the agent determine when to listen and when to speak, utilizing

    Silero Voice Activity Detection

    .

Setting Up the Development Environment

Prerequisites

Before you begin, ensure you have Python 3.11+ installed and a VideoSDK account. You can sign up at app.videosdk.live.

Step 1: Create a Virtual Environment

Create a virtual environment to manage your project dependencies:
1python -m venv venv
2source venv/bin/activate  # On Windows use `venv\Scripts\activate`
3

Step 2: Install Required Packages

Install the necessary packages using pip:
1pip install videosdk-agents videosdk-plugins
2

Step 3: Configure API Keys in a .env File

Create a .env file in your project directory and add your VideoSDK API key:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

Let's start by presenting the complete, runnable code for the AI Voice Agent:
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "You are a helpful healthcare assistant AI Voice Agent designed to assist users with healthcare-related inquiries. Your primary capabilities include answering questions about common symptoms, providing general health tips, and assisting with scheduling appointments with healthcare providers. You can also offer information about healthcare services and facilities. However, you are not a medical professional and must always include a disclaimer advising users to consult a qualified healthcare provider for medical advice, diagnosis, or treatment. You should prioritize user privacy and data security, ensuring that any personal information shared by users is handled with the utmost confidentiality. Your responses should be clear, concise, and empathetic, reflecting a supportive and understanding tone."
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=DeepgramSTT(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=SileroVAD(threshold=0.35),
32        turn_detector=TurnDetector(threshold=0.8)
33    )
34
35    session = [AgentSession](https://docs.videosdk.live/ai_agents/core-components/agent-session)(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63
Now, let’s break down this code to understand each part.

Step 4.1: Generating a VideoSDK Meeting ID

To interact with the agent, you need a meeting ID. You can generate this using the VideoSDK API:
1curl -X POST "https://api.videosdk.live/v1/meetings" \
2-H "Authorization: Bearer YOUR_API_KEY" \
3-H "Content-Type: application/json"
4

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class is where you define the behavior of your voice agent. It inherits from the Agent class and uses the agent_instructions to guide its interactions.
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6

Step 4.3: Defining the Core Pipeline

The CascadingPipeline is central to processing audio data. It incorporates various plugins for STT, LLM, TTS, VAD, and turn detection.
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8

Step 4.4: Managing the Session and Startup Logic

The start_session function initializes the session, connects to the context, and manages the lifecycle of the agent.
1async def start_session(context: JobContext):
2    agent = MyVoiceAgent()
3    conversation_flow = ConversationFlow(agent)
4    pipeline = CascadingPipeline(
5        stt=DeepgramSTT(model="nova-2", language="en"),
6        llm=OpenAILLM(model="gpt-4o"),
7        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
8        vad=SileroVAD(threshold=0.35),
9        turn_detector=TurnDetector(threshold=0.8)
10    )
11    session = AgentSession(
12        agent=agent,
13        pipeline=pipeline,
14        conversation_flow=conversation_flow
15    )
16    try:
17        await context.connect()
18        await session.start()
19        await asyncio.Event().wait()
20    finally:
21        await session.close()
22        await context.shutdown()
23
The make_context function sets up the room options for the session:
1def make_context() -> JobContext:
2    room_options = RoomOptions(
3        name="VideoSDK Cascaded Agent",
4        playground=True
5    )
6    return JobContext(room_options=room_options)
7
Finally, the entry point of the script starts the job:
1if __name__ == "__main__":
2    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
3    job.start()
4

Running and Testing the Agent

Step 5.1: Running the Python Script

To run the voice agent, execute the script in your terminal:
1python main.py
2

Step 5.2: Interacting with the Agent in the Playground

Once the script is running, you’ll receive a

playground

 link in the console. Use this link to interact with your AI Voice Agent and test its capabilities.

Advanced Features and Customizations

Extending Functionality with Custom Tools

The VideoSDK framework allows you to extend your agent’s capabilities by integrating custom tools that can handle specific tasks or queries, such as the

AI voice Agent Wake-Up Call Feature

.

Exploring Other Plugins

While this tutorial uses specific plugins, VideoSDK supports various options for STT, LLM, and TTS. Explore these to tailor the agent to your needs.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure your API key is correctly configured in the .env file. Verify your account status on the VideoSDK platform.

Audio Input/Output Problems

Check your microphone and speaker settings. Ensure the correct audio devices are selected.

Dependency and Version Conflicts

Ensure all dependencies are up-to-date and compatible with Python 3.11+.

Conclusion

Summary of What You've Built

You’ve successfully built an AI Voice Agent tailored for healthcare applications using the VideoSDK framework.

Next Steps and Further Learning

Explore additional features and plugins offered by VideoSDK to enhance your agent’s capabilities. Continue learning about AI and voice technologies to keep your skills sharp.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ