Build AI Voice Assistant for Health Support

Step-by-step guide to building an AI voice assistant for the health support industry using VideoSDK.

Introduction to AI Voice Agents in Health Support Industry

What is an AI

Voice Agent

?

AI Voice Agents are software applications that can understand and respond to human speech. They are designed to interact with users through voice commands, providing information or performing tasks. These agents use technologies like Speech-to-Text (STT), Text-to-Speech (TTS), and Natural Language Processing (NLP) to process and respond to voice inputs.

Why are they important for the Health Support Industry?

In the health support industry, AI Voice Agents can play a crucial role by providing immediate assistance to users. They can answer common health-related questions, offer general health tips, and even assist in scheduling appointments with healthcare providers. This can significantly enhance user experience by providing quick and reliable support.

Core Components of a

Voice Agent

The core components of a

voice agent

 include:
  • Speech-to-Text (STT): Converts spoken language into text.
  • Large Language Model (LLM): Processes the text to understand and generate responses.
  • Text-to-Speech (TTS): Converts the generated text response back into spoken language.

What You'll Build in This Tutorial

In this tutorial, you will learn how to build an AI

Voice Agent

 tailored for the health support industry using the VideoSDK AI Agents framework. We will walk through setting up the environment, creating the agent, and testing it.

Architecture and Core Concepts

High-Level Architecture Overview

The AI

Voice Agent

 listens to user speech, processes it to understand the intent, and responds appropriately. The data flow involves capturing audio, converting it to text, processing the text with an

OpenAI LLM Plugin for voice agent

, and then converting the response back to speech.
Diagram

Understanding Key Concepts in the VideoSDK Framework

  • Agent: The core class representing your bot, responsible for handling interactions.

  • Cascading Pipeline in AI voice Agents

    :     Manages the flow of audio processing from STT to LLM to TTS.
  • VAD & TurnDetector: Help the agent determine when to listen and when to speak.

Setting Up the Development Environment

Prerequisites

To get started, ensure you have Python 3.11+ installed and a VideoSDK account, which you can create at app.videosdk.live.

Step 1: Create a Virtual Environment

Create a virtual environment to manage dependencies:
1python -m venv health-agent-env
2source health-agent-env/bin/activate  # On Windows use `health-agent-env\\Scripts\\activate`
3

Step 2: Install Required Packages

Install the necessary packages using pip:
1pip install videosdk-python
2pip install python-dotenv
3

Step 3: Configure API Keys in a .env file

Create a .env file in your project directory and add your VideoSDK API key:
1VIDEOSDK_API_KEY=your_api_key_here
2

Building the AI Voice Agent: A Step-by-Step Guide

Here is the complete code to build the AI Voice Agent:
1import asyncio, os
2from videosdk.agents import Agent, AgentSession, CascadingPipeline, JobContext, RoomOptions, WorkerJob, ConversationFlow
3from videosdk.plugins.silero import SileroVAD
4from videosdk.plugins.turn_detector import TurnDetector, pre_download_model
5from videosdk.plugins.deepgram import DeepgramSTT
6from videosdk.plugins.openai import OpenAILLM
7from videosdk.plugins.elevenlabs import ElevenLabsTTS
8from typing import AsyncIterator
9
10# Pre-downloading the Turn Detector model
11pre_download_model()
12
13agent_instructions = "You are a helpful healthcare assistant AI Voice Agent designed to support users in the health support industry. Your primary capabilities include answering questions about common symptoms, providing general health tips, and assisting users in scheduling appointments with healthcare providers. You must always include a disclaimer that you are not a medical professional and advise users to consult a doctor for any medical concerns. You should maintain a friendly and empathetic tone, ensuring users feel supported and understood. You are not authorized to provide medical diagnoses or treatment plans. Your responses should be concise, informative, and within the scope of general health support."
14
15class MyVoiceAgent(Agent):
16    def __init__(self):
17        super().__init__(instructions=agent_instructions)
18    async def on_enter(self): await self.session.say("Hello! How can I help?")
19    async def on_exit(self): await self.session.say("Goodbye!")
20
21async def start_session(context: JobContext):
22    # Create agent and conversation flow
23    agent = MyVoiceAgent()
24    conversation_flow = ConversationFlow(agent)
25
26    # Create pipeline
27    pipeline = CascadingPipeline(
28        stt=DeepgramSTT(model="nova-2", language="en"),
29        llm=OpenAILLM(model="gpt-4o"),
30        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
31        vad=[Silero Voice Activity Detection](https://docs.videosdk.live/ai_agents/plugins/silero-vad)(threshold=0.35),
32        turn_detector=[Turn detector for AI voice Agents](https://docs.videosdk.live/ai_agents/plugins/turn-detector)(threshold=0.8)
33    )
34
35    session = AgentSession(
36        agent=agent,
37        pipeline=pipeline,
38        conversation_flow=conversation_flow
39    )
40
41    try:
42        await context.connect()
43        await session.start()
44        # Keep the session running until manually terminated
45        await asyncio.Event().wait()
46    finally:
47        # Clean up resources when done
48        await session.close()
49        await context.shutdown()
50
51def make_context() -> JobContext:
52    room_options = RoomOptions(
53    #  room_id="YOUR_MEETING_ID",  # Set to join a pre-created room; omit to auto-create
54        name="VideoSDK Cascaded Agent",
55        playground=True
56    )
57
58    return JobContext(room_options=room_options)
59
60if __name__ == "__main__":
61    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
62    job.start()
63

Step 4.1: Generating a VideoSDK Meeting ID

To generate a meeting ID, use the following curl command:
1curl -X POST https://api.videosdk.live/v1/meetings -H "Authorization: YOUR_API_KEY" -H "Content-Type: application/json"
2

Step 4.2: Creating the Custom Agent Class

The MyVoiceAgent class extends the base Agent class, allowing us to define custom behavior for entering and exiting conversations:
1class MyVoiceAgent(Agent):
2    def __init__(self):
3        super().__init__(instructions=agent_instructions)
4    async def on_enter(self): await self.session.say("Hello! How can I help?")
5    async def on_exit(self): await self.session.say("Goodbye!")
6
This class initializes with specific instructions and defines actions when the agent session starts and ends.

Step 4.3: Defining the Core Pipeline

The

CascadingPipeline

 is responsible for orchestrating the flow of data through the agent's components:
1pipeline = CascadingPipeline(
2    stt=DeepgramSTT(model="nova-2", language="en"),
3    llm=OpenAILLM(model="gpt-4o"),
4    tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
5    vad=SileroVAD(threshold=0.35),
6    turn_detector=TurnDetector(threshold=0.8)
7)
8
Each component in the pipeline plays a critical role in processing the user's speech and generating a response.

Step 4.4: Managing the Session and Startup Logic

The start_session function manages the lifecycle of the agent's session:
1async def start_session(context: JobContext):
2    agent = MyVoiceAgent()
3    conversation_flow = ConversationFlow(agent)
4    pipeline = CascadingPipeline(
5        stt=DeepgramSTT(model="nova-2", language="en"),
6        llm=OpenAILLM(model="gpt-4o"),
7        tts=ElevenLabsTTS(model="eleven_flash_v2_5"),
8        vad=SileroVAD(threshold=0.35),
9        turn_detector=TurnDetector(threshold=0.8)
10    )
11    session = AgentSession(
12        agent=agent,
13        pipeline=pipeline,
14        conversation_flow=conversation_flow
15    )
16    try:
17        await context.connect()
18        await session.start()
19        await asyncio.Event().wait()
20    finally:
21        await session.close()
22        await context.shutdown()
23
The make_context function sets up the session environment:
1def make_context() -> JobContext:
2    room_options = RoomOptions(
3        name="VideoSDK Cascaded Agent",
4        playground=True
5    )
6    return JobContext(room_options=room_options)
7
The main entry point of the script starts the agent:
1if __name__ == "__main__":
2    job = WorkerJob(entrypoint=start_session, jobctx=make_context)
3    job.start()
4

Running and Testing the Agent

Step 5.1: Running the Python Script

Run the script using:
1python main.py
2
This will start the agent and provide a link to the playground in the console.

Step 5.2: Interacting with the Agent in the Playground

Use the provided playground link to join the session and interact with your AI Voice Agent. You can speak to the agent and receive responses based on the defined capabilities.

Advanced Features and Customizations

Extending Functionality with Custom Tools

You can extend the agent's functionality by integrating custom tools that enhance its capabilities, such as additional data processing or integration with other services.

Exploring Other Plugins

Explore other STT, LLM, and TTS plugins supported by VideoSDK to tailor the agent's performance to your specific needs.

Troubleshooting Common Issues

API Key and Authentication Errors

Ensure your API keys are correctly configured in the .env file and that you have the necessary permissions.

Audio Input/Output Problems

Check your microphone and speaker settings to ensure proper audio input and output.

Dependency and Version Conflicts

Ensure all dependencies are installed with compatible versions as specified in the requirements.

Conclusion

Summary of What You've Built

In this tutorial, you've built a functional AI Voice Agent for the health support industry using VideoSDK. The agent can interact with users, providing information and assistance based on predefined capabilities.

Next Steps and Further Learning

Explore additional features and plugins to expand the agent's capabilities. Consider integrating more advanced NLP techniques or connecting with external health databases for richer interactions.

Start Building With Free $20 Balance

No credit card required to start.

Want to level-up your learning? Subscribe now

Subscribe to our newsletter for more tech based insights

FAQ