AI Agents: Rigid Control & Large-Scale Automation

Conversational Graphs: Deterministic Workflows

Agents SDK v1.0.6 introduces directed graph support. This allows you to build multi-turn agents where you define the exact transitions between states (e.g., "Greeting" → "Booking" → "Payment"). The engine follows your routing exactly - the LLM handles the conversation, but never makes unauthorized routing decisions.

📄 Read the Conversational Graphs docs

▶️ Watch Introduction Video

Batch Calling: Voice Outreach at Scale

Our new Batch Calling infrastructure allows you to trigger hundreds of AI-powered voice calls simultaneously from the dashboard. This is designed for high-concurrency outbound use cases like automated reminders and large-scale customer surveys.

AI-Native Development: Agents MCP Server

We've launched the official Agents MCP Server. This allows you to provide the full, real-time context of our Agents documentation to AI coding assistants like Claude, Cursor, and VS Code. By connecting your AI to our docs, you can generate implementation code that follows our latest v1 standards and patterns automatically.

👉 View the Agents MCP Server Setup Guide

Agent Intelligence: Smarter Memory & Gated Speech

ContextWindow: Intelligent Memory

Agents SDK v1.0.3 introduces the ContextWindow. Instead of "blindly" truncating the oldest messages when token limits are reached, this feature uses history compression and summarization. It preserves the last N turns raw to maintain conversation quality while summarizing older history.

from videosdk.agents import ContextWindow

pipeline = Pipeline(
    ....
    context_window=ContextWindow(
        max_tokens=4000,
        max_context_items=20,
        keep_recent_turns=3,          # Keep last 3 turns raw
        max_tool_calls_per_turn=10,   # Prevent infinite loops
    ),
)

Startup Speech Gating

To prevent the user from interrupting an agent's initial greeting, we have implemented Early Speech Gating (v1.0.7). User audio is now gated until the agent has initiated its first say() or reply(), ensuring a professional and clear start to every interaction.

Read full release notes on GitHub

Observability & Metrics

Unified Observability Configuration

You can now pass all observability settings - recording, traces, metrics, and logs - directly at session startup (v1.0.9).

await session.start(
   wait_for_participant=True,
   observability=ObservabilityOptions(
       recording=RecordingOptions(video=True),        
       traces=TracesOptions(export_url="https://otlp-endpoint"),
       logs=LoggingOptions(level=["INFO", "DEBUG"]),  
   ),
)

Pipeline & Metrics Hooks

Agents SDK v1.0.7 adds deep visibility hooks. Use @pipeline.metrics.on("stt") or @pipeline.metrics.on("realtime") to capture latency and token usage. You can also now use session.get_context_history to access session data for debugging or post-processing.

Native SDK Agent Metrics

Real-time agent performance data is now exposed directly in the client-side SDKs. Monitor latency, tokens, and component health via:

• JS SDK v0.7.2 - agent-metrics event on AgentParticipant
• React SDK v0.9.1 - onAgentMetrics() callback
• Flutter SDK v3.9.0 - Events.agentMetrics event

Hardware-Silence Detection

Available in JS SDK v0.7.2 and React SDK v0.9.1, the audio-input-silence event triggers when a microphone is publishing but no audio signal is detected. This allows you to alert users to hardware mutes or Bluetooth issues immediately.

Interactive Transcripts & Ecosystem

Word-level Timestamps & Interim Transcripts

Agents SDK v1.0.9 adds support for streaming agent transcripts with word-level timestamps (for Cartesia and ElevenLabs). Additionally, you can now enable interim user results to show text on the UI as the user is still speaking.

# Opt-in to word-level sync
tts = CartesiaTTS(word_timestamps=True)
# Opt-in to interim user results (v1.0.10)
stt = DeepgramSTT(forward_interim_transcripts=True)

Core SDK Stability & Performance

• iOS SDK v2.7.1 - Resolved rare crashes in screenshare/PubSub and optimized stats collection for large meetings.
• Android SDK v1.2.0 - Rewritten audioManager logic to resolve complex Bluetooth handoff issues.
• React SDK v0.9.1 and React Native SDK v0.10.1 - Fixed unnecessary re-render issues and improved webcam toggle state handling.

Dashboard Power-ups

• Agent Logs: A dedicated tab in session details to view raw execution logs for debugging.
• Composite Recording: Download a unified view of agent-user interactions in a single MP4.
• Batch Calling & Alerts: Centralized UI to manage large-scale campaigns and configure proactive quality alerts.

📚 New Content & Resources

• Introducing VideoSDK AI Voice Agents v1: The official v1 breakdown.
• Video: Proactive monitoring with RTC Alerts
• Video: Deterministic Agent Workflows with Graphs

Featured Videos

• ▶️ Building with Agents MCP server

  Project Directory

  This simple structure is our final goal for the worker. By following along, you'll create this complete project from scratch.

  worker/
  ├── Dockerfile              # Instructions to build the Docker container
  ├── main.py                 # The core logic for your AI voice agent
  ├── requirements.txt        # Python package dependencies
  ├── .env                    # Environments variable
  └── videosdk.yaml           # VideoSDK configuration for deploying the agent

  Architecture - Inbound/Outbound Calls for AI Phone Agent

  This architecture shows how your AI Phone Agent connects to the global phone network for both inbound and outbound SIP calls.

  Our container-based deployment flow handles the complex telephony.

  

  Setting Up Agent Worker for AI Phone Agent

  Let's build our main.py. This file uses three core VideoSDK components to define our agent's logic and connect it to a call.

  1. Agent:

  This class defines your agent's personality and conversational flow. You can also integrate advanced protocols like MCP and Agent 2 Agent to provide external context to your agent.

  2. Pipeline:

  The Pipeline is the engine that processes the audio stream. It takes the user's voice as input and produces the agent's voice as output. VideoSDK offers two types of pipelines to fit your needs

  This is the audio processing engine. You can choose between two types:

  

  

  3. Agent Session

  This brings it all together, connecting your Agent and Pipeline to a live VideoSDK Room to start the call.

  Project Modules requirements.txt

  videosdk-agents
  videosdk-plugins-google
  videosdk
  python-dotenv

  Here’s how these components are implemented in the code main.py:

  import asyncio
  from videosdk.agents import Agent, AgentSession, RealTimePipeline, JobContext, RoomOptions, WorkerJob, MCPServerStdio
  from videosdk.plugins.google import GeminiRealtime, GeminiLiveConfig
  from dotenv import load_dotenv
  import os
  
  load_dotenv(override=True)
  
  # Agent Component
  class MyVoiceAgent(Agent):
      def __init__(self):
          super().__init__(
              instructions="You are VideoSDK's AI Avatar Voice Agent with real-time capabilities. You are a helpful virtual assistant with a visual avatar that can answer questions about weather help with other tasks in real-time.",
          )
  
      async def on_enter(self) -> None:
          await self.session.say("Hello! I'm your real-time AI avatar assistant. How can I help you today?")
      
      async def on_exit(self) -> None:
          await self.session.say("Goodbye! It was great talking with you!")
  
  async def start_session(context: JobContext):
  
      # Initialize Gemini Realtime model
      model = GeminiRealtime(
          model="gemini-2.0-flash-live-001",
          # When GOOGLE_API_KEY is set in .env - DON'T pass api_key parameter
          api_key=os.getenv("GOOGLE_API_KEY"), 
          config=GeminiLiveConfig(
              voice="Leda",  # Puck, Charon etc
              response_modalities=["AUDIO"]
          )
      )
  
      # Create pipeline with avatar
      pipeline = RealTimePipeline(
          model=model,
      )
      
      session = AgentSession(
          agent=MyVoiceAgent(),
          pipeline=pipeline
      )
  
      try:
          await context.connect()
          await session.start()
          await asyncio.Event().wait()
      finally:
          await session.close()
          await context.shutdown()
  
  def make_context() -> JobContext:
      room_options = RoomOptions(
          auth_token=os.getenv("VIDEOSDK_TOKEN"),
          room_id="ln4i-tuwm-yzkq",
          name="AI Agent",
          playground=True,
          recording=False
      )
      return JobContext(room_options=room_options)
  
  
  if __name__ == "__main__":
      job = WorkerJob(entrypoint=start_session, jobctx=make_context)
      job.start() 

  Running the AI Phone Agent Locally

  Before we containerize our agent, let's run the Python script directly in a local development environment. This is the fastest way to test your agent's core logic.

  Prerequisite: Ensure you have Python 3.12 or newer installed on your machine.

  1. Create and Activate a Virtual Environment

  First, open your terminal in the worker directory. It's a best practice to use a virtual environment to manage project dependencies.

  Create the environment:

  python3 -m venv .venv

  Next, Activate it! Command differ based on your environment

  • On MacOS/Linux

  source .venv/bin/activate

  • On Windows

  .venv\Scripts\activate

  You'll know the environment is active when you see (.venv) at the beginning of your terminal prompt.

  2. Install Dependencies

  With the virtual environment active, install the necessary Python packages listed in your requirements.txt file:

  pip install -r requirements.txt

  3. Run the Python Script

  Finally, run the agent:

  python main.py

  This will start your agent and connect it to a playground session, as defined by playground=True in the make_context() function in your code. You can now interact with it for initial testing before moving on to a full deployment.

  Build and Test Your AI Phone Agent

  Before deploying our agent to the cloud, it's crucial to ensure it runs correctly on our local machine. The VideoSDK CLI makes this incredibly simple.

  In your terminal, navigate to the worker directory and run the following command:

  videosdk run

  This command reads your videosdk.yaml and Dockerfile, builds a local container, and starts your agent. You should see an output confirming that your worker is running. This is the perfect time to test your agent's basic functionality in a playground environment to catch any bugs before going live.

  

  Deploy Your AI Phone Agent to the Cloud

  Once you've confirmed the agent works locally, it's time to deploy it to VideoSDK's global infrastructure.

  1. Configure the Deployment Manifest

  First, make sure your videosdk.yaml file is configured for a cloud deployment. The most important line is cloud: true, which tells the CLI to push your container to our servers instead of just running it locally.

  version: "1.0"
  deployment:
    id: 82fed3a5-5316-4273-b29f-4bb26e885842
    entry:
      path: main.py
  
  deploy:
    cloud: true
  
  env:
    path: "./.env"
  
  secrets:
    VIDEOSDK_AUTH_TOKEN: # your_videosdk_token

  2. Deploy with a Single Command

  Now for the magic. Run the deploy command:

  videosdk deploy

  The CLI will now package your worker, build the container, and upload it to the VideoSDK cloud. You'll see a live log of the progress.

  Upon completion, you will get a Success! message along with your unique Worker ID.

  

  Crucial Step: Copy this Worker ID! You will need this unique identifier in the next step to connect your deployed agent to a phone number using a Routing Rule in the VideoSDK dashboard.

  Connect Your AI Phone Agent to the Phone Network

  Now we'll use the VideoSDK dashboard to connect our deployed agent to a real phone number.

  1. Set up an Inbound Gateway

  This tells your SIP provider (e.g., Twilio) where to forward incoming calls.

  • In the VideoSDK Dashboard, go to Telephony , in Inbound Gateways and click on Add Inbound Gateway.
  • Give it a name and add your phone number.
  • The dashboard will generate a unique Inbound Gateway URL. Copy this URL.
  • In your SIP provider's dashboard (like Twilio), paste this URL into the Origination SIP URI field for your SIP trunk.

  

  2. Set up an Outbound Gateway

  This tells VideoSDK where to send outgoing calls from your agent.

  • Go to Telephony Outbound Gateways and click Add Outbound Gateway.
  • Give it a name and add your phone number.
  • In the Address field, enter the Termination SIP URI provided by your SIP provider.

  

  3. Create a Routing Rule

  This is the final step that connects everything. The routing rule links a phone number to your specific deployed agent.

  • Go to Telephony, in Routing Rules click on Add Routing Rules Gateway.
  • Select a direction (e.g., inbound).
  • Choose the gateway you just created.
  • Set Agent Type to Cloud.
  • In the Deployment ID field, paste the Worker ID from your videosdk.yaml file.
  • Click Create.

  

  Making an Outbound Call

  To trigger an outbound call from your agent, you can make a simple API request to the VideoSDK SIP endpoint.

  Use a POST request with your VIDEOSDK_TOKEN for authorization. In the body, specify the gatewayId (from your Outbound Gateway) and the phone number to call in sipCallTo.

  curl --request POST \\\\
    --url <https://api.videosdk.live/v2/sip/call> \\\\
    --header 'Authorization: YOUR_VIDEOSDK_TOKEN' \\\\
    --header 'Content-Type: application/json' \\\\
    --data '{
      "gatewayId": "gw_123456789",
      "sipCallTo": "+14155550123"
    }'
  
  

  Managing Agent Sessions Programmatically

  While routing rules automatically manage sessions, you can also control them via API for advanced use cases, like starting a session on demand or for cost management.

  Start a Deployment Session:

  curl --request POST \\\\
    --url <https://api.videosdk.live/ai/v1/ai-deployment-sessions/start> \\\\
    --header 'Authorization: YOUR_VIDEOSDK_TOKEN' \\\\
    --header 'Content-Type: application/json' \\\\
    --data '{
      "deploymentId": "<your-deployment-id>"
    }'
  
  

  End a Deployment Session:

  curl --request POST \\\\
    --url <https://api.videosdk.live/ai/v1/ai-deployment-sessions/end> \\\\
    --header 'Authorization: YOUR_VIDEOSDK_TOKEN' \\\\
    --header 'Content-Type: application/json' \\\\
    --data '{
      "sessionId": "<session-id-from-start-response>"
    }'
  
  

  Next Step

  That's it! You've successfully built a Python AI agent, deployed it to the cloud, and connected it to the global telephone network for both inbound and outbound calls.

  • AI Agent Documentation
  • Opensource AgentSDK
  • Join Us on Discord

  ]]>https://www.videosdk.live/blog/ai-voice-agent-use-cases/689af94a64df6f042b4d41f9Tue, 12 Aug 2025 08:29:28 GMTAI voice agent use cases span customer service, healthcare, banking, retail, restaurants, and insurance, where agents handle scheduling, support, payments, and claims at scale. Teams deploy them to cut call volume, extend 24/7 coverage, and route complex issues to humans with full context. Start with high-volume, repeatable conversations on sub-300ms audio infrastructure.

  Introduction

  

  Developers and founders evaluating AI voice agent use cases need industry-specific examples tied to measurable deployment outcomes, not generic hype about conversational AI.

  A missed restaurant call during Friday dinner rush costs revenue. A healthcare clinic that puts every patient on hold loses trust before a nurse answers. AI voice agents solve these problems by answering, routing, and completing routine conversations without waiting on staff.

  This article maps the most valuable AI voice agent use cases by industry, explains how they work in production, and shows how developers build them with real-time voice infrastructure. You will see where voice AI delivers ROI, where it fails, and how to implement it without rebuilding your telephony stack from scratch.

  What is AI Voice Agent?

  An AI voice agent is defined as a software system that listens to spoken language, interprets intent with natural language processing, and responds with synthesized speech in real time. Unlike legacy interactive voice response menus that force callers through rigid button trees, modern voice agents hold multi-turn conversations and adapt based on context.

  An AI voice agent works by streaming audio through a pipeline of speech-to-text transcription, large language model reasoning, and text-to-speech synthesis, all orchestrated over a low-latency real-time transport layer. The agent connects to phone networks through SIP/PSTN gateways or to in-app sessions through WebRTC, depending on where customers call from.

  According to PwC's Global Artificial Intelligence Study, AI could contribute up to $15.7 trillion to the global economy by 2030, with customer-facing automation representing a large share of near-term enterprise spending. For developers and product teams, that shift means voice is no longer a novelty channel. It is a production interface that must meet telephony-grade reliability, compliance, and latency standards from day one.

  AI voice agents differ from chatbots because they operate on spoken audio in real time rather than typed text in asynchronous threads.

  How Do AI Voice Agents Work?

  AI voice agents process live audio in a continuous loop where each spoken utterance becomes text, text becomes a model response, and the response becomes speech streamed back to the caller without perceptible delay.

  Real-Time Audio Transport

  The transport layer carries bidirectional audio between the user and the agent. Sub-300 millisecond round-trip latency keeps conversations natural. When latency exceeds roughly 500 milliseconds, callers perceive awkward pauses and repeat themselves, which increases handle time and abandonment rates.

  Speech-to-Text and Text-to-Speech

  Speech-to-text engines like Google Cloud Speech-to-Text and OpenAI Whisper convert incoming audio into text for the language model. Text-to-speech providers like ElevenLabs and OpenAI TTS render responses as human-sounding audio. Provider choice affects accent coverage, emotional tone, and per-minute cost at scale.

  Agent Orchestration and Escalation

  Orchestration coordinates STT, business logic or an LLM, TTS, and session state in a single pipeline. Production agents also need fallback paths for unrecognized intents, error handling, and warm handoff to human agents with conversation transcripts attached. Platforms like VideoSDK bundle this orchestration so teams focus on conversation design rather than wiring raw audio streams.

  Every production voice agent runs the same listen-think-speak loop, and platform choice determines how reliably that loop completes under load.

  

  The Most Impactful AI Voice Agent Use Cases by Industry

  High-ROI AI voice agent deployments target repetitive, high-volume conversations where automation saves measurable agent hours without sacrificing compliance or customer trust.

  Healthcare: Scheduling and Patient Follow-Up

  Healthcare organizations deploy voice agents for appointment booking, reminder calls, and post-discharge check-ins. An agent can collect symptoms during intake, confirm insurance details, and flag urgent responses for nurse review.

  HIPAA compliance is non-negotiable. Agents handling protected health information need encrypted transport, access controls, and business associate agreements with vendors. Teams building telehealth workflows often pair voice agents with HIPAA compliance.

  Customer Service: First-Line Support and Smart Routing

  Contact centers use voice agents as the first line for order status, account balance, password resets, and business hours inquiries. Intent detection routes complex or emotionally charged calls to specialists while passing full conversation context.

  According to IBM's Cost of a Data Breach Report, faster incident response reduces breach costs, and the same principle applies to customer service: resolving simple queries instantly prevents queue buildup that damages satisfaction scores. Voice agents that resolve tier-one tickets in under 90 seconds free human agents for retention-sensitive conversations.

  Banking and Financial Services: Payments and Fraud Alerts

  Banks automate EMI reminders, loan servicing outreach, and balance inquiries through outbound and inbound voice agents. Voice biometrics add a verification layer before discussing account details.

  Financial deployments require PCI-aware data handling, call recording policies, and clear disclosure when callers interact with AI. Regulated industries should treat every voice session as auditable, with transcripts stored under retention rules that match local banking regulations.

  E-Commerce and Retail: Order Tracking and Campaigns

  Retailers answer "where is my order?" at scale through CRM and order-management API integrations. Outbound agents run promotional campaigns and post-purchase feedback calls during peak seasons without hiring temporary call staff.

  Personalization raises conversion on these calls. When an agent greets a returning buyer by name and references their open shipment, completion rates on upsell offers rise compared with generic scripts.

  Restaurants: Orders, Reservations, and Delivery Coordination

  Restaurants lose revenue when staff cannot answer phones during service peaks. Voice agents take complex food orders, manage reservations, and relay modifications to point-of-sale and delivery systems in real time.

  Low-latency audio matters here more than in many other verticals. A two-second delay during order confirmation causes callers to repeat items, which introduces errors into kitchen tickets.

  Insurance: Claims Intake and Policy Renewals

  Insurers automate First Notice of Loss collection, policy renewal reminders, and identity verification through voice biometrics. Complex claims escalate to human adjusters, sometimes with a switch to video for virtual damage inspection.

  The National Insurance Crime Bureau estimates that insurance fraud costs the United States more than $308 billion annually [UPDATE: verify date], which makes secure voice authentication and suspicious-pattern flagging high-value agent capabilities.

  Across these six industries, the strongest AI voice agent use cases share structured dialog, clear escalation paths, and integrations with existing CRM, EHR, or order management systems.

  

  Why Latency and Architecture Decide Voice Agent Success

  Voice agents that feel robotic are usually slow agents, not poorly scripted ones, because human conversation tolerates only brief gaps before trust erodes.

  The Latency Budget

  A production voice agent should target under 300 milliseconds from end of user speech to start of agent audio for routine replies. That budget splits across voice activity detection (roughly 50 to 100 milliseconds), STT processing (80 to 150 milliseconds), LLM inference (50 to 200 milliseconds for cached or short responses), and TTS generation (50 to 100 milliseconds). Exceeding 500 milliseconds total round-trip produces the "are you still there?" effect that drives hang-ups.

  Cascading vs. End-to-End Pipelines

  Most production systems use a cascading pipeline (STT, then LLM, then TTS) because it allows provider swapping and fine-grained logging. End-to-end speech models reduce pipeline stages but offer less control over compliance logging and vendor selection. Teams in regulated industries typically choose cascading architectures for auditability.

  Caching and Edge Deployment

  Caching pre-rendered TTS audio for top 20 FAQ responses cuts latency and API cost simultaneously. Deploying agent workers in regions close to callers reduces transport delay for international brands. VideoSDK's cloud deployment model supports regional worker placement so agents join sessions near the user.

  Sub-300ms latency and a cascading STT-LLM-TTS architecture are the two technical decisions that separate trusted voice agents from frustrating ones.

  

  When AI Voice Agents Are the Wrong Choice

  Voice automation fails when organizations deploy agents before mapping conversation complexity, emotional stakes, and regulatory exposure.

  Choose human-first handling when the conversation involves legally binding consent, nuanced medical diagnosis, or high-emotion disputes where empathy outweighs speed. Debt collection in certain jurisdictions, mental health triage, and executive escalations are poor fits for unsupervised voice AI.

  Skip voice agents when your call volume is too low to justify integration cost. If a clinic receives 30 inbound calls daily and half require clinical judgment, a part-time receptionist plus online scheduling delivers better ROI than a full voice stack.

  Avoid voice-only design when visual confirmation improves outcomes. Insurance adjusters inspecting vehicle damage and bankers walking customers through complex product comparisons often need screen sharing or video, not audio alone. VideoSDK supports voice-to-video escalation precisely for these hybrid workflows.

  Teams that force full automation without a human escape hatch see higher churn. Always design a zero-friction path to a live agent within two conversational turns.

  Voice AI delivers the worst ROI when teams automate conversations that demand clinical judgment, legal consent, or high-empathy dispute resolution without a human backup.

  Compliance, Security, and Data Handling by Industry

  Regulated voice agent deployments require encryption, access logging, and vendor contracts that match the data each industry touches.

  Healthcare agents need HIPAA-aligned infrastructure, minimum necessary data collection, and BAAs with every subprocessors handling transcripts. Banking agents must respect call recording consent laws that vary by U.S. state and EU market. Retail agents interacting with European customers fall under GDPR purpose-limitation and deletion requirements.

  In practice, engineering teams that implement voice agents for regulated clients report that transcript retention policy is decided before model selection, not after launch. Store only what you need, redact payment card numbers at the STT layer, and route sensitive fields through tokenized APIs rather than raw LLM prompts.

  PCI DSS requirements apply when agents process or repeat payment card data aloud. SOC 2 Type II certification from your platform vendor provides third-party validation of security controls for enterprise procurement reviews. Call recording laws in two-party consent states like California and Florida require explicit disclosure at the start of each recorded session.

  For authoritative guidance on healthcare privacy obligations, see HHS HIPAA guidance on telehealth and remote communications. Pair that reference with your legal team's state-by-state recording checklist before launching outbound campaigns.

  Compliance architecture is a prerequisite for regulated AI voice agent use cases, not a post-launch patch.

  How to Measure ROI on Voice Agent Deployments

  Voice agent programs earn executive support when teams track operational metrics tied to revenue and cost, not vanity automation rates.

  Track containment rate (calls resolved without human transfer), average handle time, cost per contained call, and customer satisfaction score segmented by agent vs. human resolution. Compare baseline metrics from the 30 days before launch against the same period after reaching stable traffic.

  A mid-size e-commerce brand processing 12,000 monthly support calls illustrates the math. If tier-one inquiries represent 55% of volume and a voice agent contains 70% of those at $0.08 per minute versus $4.50 per human-handled minute, monthly savings exceed $15,000 while shaving average speed-to-answer from 4 minutes to under 10 seconds. Name your baseline, set a 90-day review, and kill workflows that transfer more than 40% of calls without improving CSAT.

  According to McKinsey's research on AI in customer service, generative AI applied to customer care functions can improve issue resolution speed and customer satisfaction when paired with clear escalation design. Track first-call resolution alongside containment rate so you do not optimize for cheap transfers that frustrate callers.

  Executives approve voice agent budgets when teams present containment rate, cost per call, and CSAT trends on a single dashboard updated monthly.

  Implementing AI Voice Agents with VideoSDK

  Understanding the potential of AI voice agents is the first step; building them is the next. A successful implementation requires orchestrating several complex technologies to create a fluid, human-like conversational experience. This is where a dedicated SDK designed for real-time communication becomes invaluable.

  Getting Started with an AI Voice Agent SDK

  To bring the use cases discussed above to life, developers need a streamlined way to integrate AI capabilities into a communication framework. An AI Voice Agent SDK, such as the one offered by VideoSDK, provides pre-built functionalities that handle the underlying complexities of real-time communication and AI integration. This allows developers to focus on crafting the agent's logic and personality rather than building the foundational infrastructure from scratch. The core of such an SDK revolves around four key components working in perfect harmony.

  Overview of Core Components

  1. Real-Time Streaming: This is the backbone of any live conversation. The SDK must manage the low-latency, bidirectional streaming of audio data between the user and the AI agent, ensuring the conversation flows naturally without awkward delays or interruptions.
  2. Speech-to-Text (STT): To understand the user, the AI agent needs to convert their spoken words into text. The SDK integrates with powerful STT engines that transcribe the user's audio in real-time, providing an accurate textual input for the AI model to process.
  3. Text-to-Speech (TTS): Once the AI has formulated a response, it needs to be converted back into natural-sounding speech. The SDK uses advanced TTS engines to generate high-quality, human-like audio, which is then streamed back to the user. The quality of the TTS is critical for user adoption and a positive experience.
  4. Agent Orchestration: This is the brain of the operation. The SDK orchestrates the entire workflow, managing the real-time flow of data between the STT service, your business logic or large language model (LLM), and the TTS service. This ensures that the agent can listen, think, and speak in a seamless, uninterrupted loop.

  Supported Integrations for Maximum Flexibility

  No single AI provider excels at everything. A flexible platform should allow developers to choose the best tools for their specific needs. VideoSDK's AI Voice Agent framework is designed to be plug-and-play, supporting integrations with leading AI services. Developers can mix and match providers for different components, including:

  • Speech-to-Text: Integrations with powerful engines like Google STT and OpenAI's Whisper ensure high-accuracy transcriptions across various languages and accents.
  • Text-to-Speech: To create lifelike and emotionally resonant voices, the platform supports leading TTS providers like ElevenLabs and services from OpenAI.

  This "bring your own AI" model gives developers the freedom to leverage the best-in-class technology and future-proof their applications against a rapidly evolving AI landscape.

  SIP/PSTN Integration for Telephony-Grade Quality

  While many AI interactions happen within apps, the ability to connect with traditional phone networks is crucial for countless business use cases, from customer service call centers to automated appointment reminders. The integration of Session Initiation Protocol (SIP) and Public Switched Telephone Network (PSTN) gateways is a vital feature. This allows the AI voice agent to make and receive calls from standard phone numbers, extending its reach beyond the digital-only world. VideoSDK's support for SIP/PSTN ensures that businesses can deploy AI agents into their existing telephony workflows, providing a seamless, telephony-grade quality experience for every user, regardless of how they connect.

  Key Steps to Build Your Own Voice Agent

  Here’s a step-by-step guide to creating your own AI voice agent using VideoSDK:

  Step 1: Choose the Voice Model (TTS + STT)

  The first step is to select the text-to-speech and speech-to-text models that best suit your application. Consider factors like language support, accuracy, and the desired vocal characteristics of your agent.

  Select providers based on:

  • Latency requirements (e.g., <300ms for real-time calls)
  • Language coverage (multi-lingual support for global deployments)
  • Voice customization (brand-aligned tone & gender)

  Here is the example of the OpenAI TTS model.

  from videosdk.plugins.openai import OpenAITTS
  from videosdk.agents import CascadingPipeline
  
  # Initialize the OpenAI TTS model
  tts = OpenAITTS(
      # When OPENAI_API_KEY is set in .env - DON'T pass api_key parameter
      api_key="your-openai-api-key",
      model="tts-1",
      voice="alloy",
      speed=1.0,
      response_format="pcm"
  )
  
  #  Add tts to cascading pipeline
  pipeline = CascadingPipeline(tts=tts)

  Alternatively you can try Google Gemini and AWS Nova Sonice

  Here is the example of the OpenAI STT model.

  from videosdk.plugins.openai import OpenAISTT
  from videosdk.agents import CascadingPipeline
  
  # Initialize the OpenAI STT model
  stt = OpenAISTT(
      # When OPENAI_API_KEY is set in .env - DON'T pass api_key parameter
      api_key="your-openai-api-key",
      model="whisper-1",
      language="en",
      prompt="Transcribe this audio with proper punctuation and formatting."
  )
  
  #  Add stt to cascading pipeline
  pipeline = CascadingPipeline(stt=stt)

  Step 2: Configure VideoSDK for real-time transport

  Next, you'll need to set up your VideoSDK environment to handle the real-time transport of audio data. This involves configuring your authentication tokens and meeting IDs to enable the AI agent to join a communication session. You will need to set up a .env file to securely store your API keys and tokens.

  Here is the OpenAI API key to Configure VideoSDK for real-time transport 

  VIDEOSDK_AUTH_TOKEN = your_videosdk_auth_token;
  OPENAI_API_KEY = your_openai_api_key;
  

  If you are using gemini or aws nova sonic you will need to provide their respective api key

  Step 3: Create prompt-based flows

  Define the conversational logic of your AI agent by creating prompt-based flows. This involves scripting the agent's initial greetings, questions, and responses based on potential user inputs. You can create a custom agent by inheriting from the base Agent class. 

  from videosdk.agents import Agent, AgentSession, WorkerJob, RoomOptions, JobContext
  import asyncio
  
  class VoiceAgent(Agent):
      def __init__(self):
          super().__init__(
              instructions="You are a helpful voice assistant that can answer questions and help with tasks."
          )
  
      async def on_enter(self) -> None:
          """Called when the agent first joins the meeting"""
          await self.session.say("Hi there! How can I help you today?")
  
      async def on_exit(self) -> None:
          """Called when the agent exits the meeting"""
          await self.session.say("Goodbye!")

  Step 4: Add fallback and escalation logic

  It's crucial to account for scenarios where the AI agent may not understand a user's request or when an error occurs. Implementing fallback logic to provide a helpful response and, if necessary, a mechanism to escalate the conversation to a human agent is a best practice.

  The VideoSDK AI Agent SDK allows you to handle these situations by overriding specific methods in your custom agent class.

  Handling Unrecognized Intents (Fallback)

  If the Large Language Model (LLM) cannot determine the user's intent or if the user's speech is unclear, you can define a fallback behavior. In this example, if the agent doesn't understand, it will ask the user to rephrase their request.

  from videosdk.agents import Agent, AgentSession
  from videosdk.llm import LLM
  from videosdk.stt import STT
  from videosdk.tts import TTS
  
  class VoiceAgent(Agent):
      def __init__(
          self,
          llm: LLM,
          stt: STT,
          tts: TTS,
      ):
          super().__init__(
              llm=llm,
              stt=stt,
              tts=tts,
              instructions="You are a helpful voice assistant that can answer questions and help with tasks."
          )
  
      async def on_enter(self) -> None:
          """Called when the agent first joins the meeting"""
          await self.session.say("Hi there! How can I help you today?")
  
      async def on_fallback(self) -> None:
          """Called when the agent cannot understand the user's intent."""
          await self.session.say("I'm sorry, I didn't quite catch that. Could you please rephrase?")
  
      async def on_exit(self) -> None:
          """Called when the agent exits the meeting"""
          await self.session.say("Goodbye!")

  Handling Errors and Escalation

  For more critical errors, or if the user explicitly asks to speak to a human, you can implement an escalation path. This could involve triggering a notification, transferring the call, or providing the user with contact information for human support.

  The on_error method can be used to catch exceptions that occur during the agent's operation.

  import logging
  
  # ... (previous imports)
  
  class VoiceAgent(Agent):
      # ... (__init__ and on_enter methods)
  
      async def on_fallback(self) -> None:
          """Called when the agent cannot understand the user's intent."""
          await self.session.say("I'm sorry, I didn't quite catch that. Could you please rephrase?")
  
      async def on_error(self, error: Exception) -> None:
          """Called when an error occurs."""
          logging.error(f"An error occurred: {error}")
          # Simple escalation: inform the user and provide a support email.
          await self.session.say("It seems I've run into a technical issue. Please contact our support team at support@example.com for assistance.")
          # In a more advanced scenario, you could trigger an API call
          # to a human handoff service here.
  
      async def on_exit(self) -> None:
          """Called when the agent exits the meeting"""
          await self.session.say("Goodbye!")

  In a real-world application, the on_error or a custom function tool could be used to initiate a more sophisticated escalation process, such as:

  • Human Handoff: Triggering a workflow in a CRM or helpdesk system to alert a human agent to join the call.
  • Ticket Creation: Automatically creating a support ticket with the conversation transcript.
  • SIP Transfer: If using SIP integration, transferring the call to a pre-defined human agent's phone number.

  By implementing these fallback and escalation mechanisms, you ensure that your AI voice agent provides a reliable and helpful experience, even when faced with ambiguity or errors.

  Step 5: Deploy to production

  Once you have thoroughly tested your AI voice agent, you can deploy it. The VideoSDK CLI allows you to run your agent locally for testing and then deploy it to the VideoSDK Cloud.

  # Run the AI Deployment locally
  videosdk run
  
  # Deploy the AI Deployment
  videosdk deploy

  Best Practices for Scale and Accuracy

  To ensure your AI voice agent performs optimally and delivers a high-quality user experience as your user base grows, consider these best practices:

  Use context-aware agents (via MCP or A2A Protocols)

  A truly intelligent agent understands the flow of conversation. Instead of treating each user query as an isolated event, a context-aware agent maintains a memory of the dialogue. This allows for more natural and efficient interactions.

  VideoSDK facilitates this through Agent-to-Agent (A2A) communication protocols. For example, a general-purpose AI voice agent could handle initial user queries and then, upon identifying a specialized need (like a technical support issue), can seamlessly forward the query and the conversation history to a specialist agent. This ensures the user doesn't have to repeat themselves, creating a smoother experience.

  Cache common responses

  Many businesses find that a significant portion of their customer inquiries are repetitive. For these frequently asked questions (e.g., "What are your business hours?" or "How do I reset my password?"), caching the generated audio response can significantly improve performance.

  By storing the pre-rendered TTS audio for common answers, you can:

  • Reduce Latency: Deliver answers almost instantaneously, as you're bypassing the real-time TTS generation step.
  • Lower Costs: Minimize the number of API calls to TTS services, leading to direct cost savings, especially at scale.
  • Increase Consistency: Ensure the answer to a common question is always delivered in the same clear and consistent manner.

  Personalize with user metadata

  Personalization is key to transforming a generic interaction into a memorable customer experience. By leveraging user metadata—such as their name, past purchase history, or support ticket status—your AI voice agent can provide tailored and empathetic responses.

  For instance, an e-commerce voice agent could greet a returning customer with:

  "Welcome back, [Customer Name]! I see your recent order for the [Product Name] has been shipped. Are you calling about that, or is there something else I can help you with today?"

  This level of personalization, achievable by integrating your AI agent with your CRM or user database, makes the interaction feel more human and significantly improves customer satisfaction.

  Use multi-turn dialogs via LLM

  Early voice bots were often limited to simple, one-off commands. Modern AI voice agents, powered by sophisticated Large Language Models (LLMs), excel at handling multi-turn dialogues. This means the agent can manage complex, evolving conversations where the user's intent might be clarified over several exchanges.

  For example, a user might start by saying, "I need a flight to New York." The agent can then ask clarifying questions like, "Which airport in New York?", "What date would you like to travel?", and "Are you looking for a one-way or round-trip ticket?" The LLM's ability to maintain context throughout this back-and-forth is what makes a truly conversational and useful AI possible. VideoSDK’s architecture is designed to support these stateful, long-running conversations seamlessly.

  Definitions Glossary

  AI voice agent: A software system that conducts spoken conversations by combining speech recognition, language understanding, and speech synthesis in real time.

  Speech-to-text (STT): Technology that converts spoken audio into text so language models can process user intent.

  Text-to-speech (TTS): Technology that converts text responses into synthetic speech streamed back to the caller.

  SIP/PSTN integration: Connectivity that lets voice agents place and receive calls on traditional phone networks, not just internet apps.

  Containment rate: The percentage of conversations an AI agent resolves without transferring to a human operator.

  Voice biometrics: Authentication that verifies identity from unique vocal characteristics before discussing sensitive account information.

  Key Takeaways

  • AI voice agent use cases deliver the highest ROI on repetitive, high-volume calls such as scheduling, order tracking, payment reminders, and tier-one support.
  • Sub-300ms round-trip latency and reliable SIP/PSTN connectivity separate production-grade agents from demo-quality prototypes.
  • Regulated industries require compliance planning for HIPAA, PCI, GDPR, and call recording consent before launch, not after transcripts accumulate.
  • Voice agents should always include fast human escalation with full conversation context for complex, emotional, or legally sensitive interactions.
  • VideoSDK gives developer teams a single SDK for real-time audio, pluggable AI providers, telephony integration, and optional video handoff.

  Conclusion

  AI voice agent use cases are no longer experimental. Healthcare clinics, banks, retailers, and insurers deploy them to answer calls instantly, reduce operational cost, and pass rich context to humans when conversations outgrow automation.

  Start with one high-volume workflow, measure containment and satisfaction for 90 days, and expand only where data supports it. VideoSDK provides the real-time audio infrastructure, AI provider integrations, and telephony reach to move from prototype to production without rebuilding your stack. Sign up for a free developer account and ship your first agent this week.

  Frequently Asked Questions

  What are the most common AI voice agent use cases?

  The most common AI voice agent use cases are appointment scheduling, customer support tier-one resolution, order status inquiries, payment and policy reminders, restaurant order taking, and insurance First Notice of Loss intake. These workflows share high call volume, predictable dialog structure, and clear escalation rules when automation reaches its limits.

  How do AI voice agents differ from traditional IVR?

  AI voice agents differ from traditional IVR because they understand natural spoken language instead of forcing callers through fixed keypad menus. IVR systems follow pre-recorded trees, while voice agents use speech-to-text and large language models to manage multi-turn conversations and adapt responses based on caller intent.

  Which industries benefit most from AI voice agents in 2026?

  Healthcare, financial services, e-commerce, restaurants, insurance, and telecommunications benefit most because they combine high inbound call volume with repetitive intents that automation handles well. Industries with heavy compliance requirements still benefit when platforms provide encrypted transport, audit logging, and human escalation paths.

  Are AI voice agents HIPAA compliant?

  AI voice agents can support HIPAA-compliant workflows when deployed on infrastructure with encrypted audio transport, access controls, audit logs, and signed business associate agreements covering all AI and telephony vendors. Compliance depends on implementation choices, not the voice technology alone.

  How much does it cost to run an AI voice agent?

  Running an AI voice agent typically costs a blend of per-minute telephony fees, speech-to-text and text-to-speech API usage, LLM inference, and platform infrastructure charges. Total cost ranges from roughly $0.05 to $0.20 per contained minute for optimized deployments at scale, depending on provider selection and caching strategy [UPDATE: verify date].

  Can AI voice agents transfer calls to human agents?

  AI voice agents can transfer calls to human agents through SIP transfer, conference join, or CRM-triggered callbacks while attaching transcripts and detected intent. Warm handoff preserves customer trust because the human agent continues the conversation without asking the caller to repeat information.

  What should developers look for in a voice agent SDK?

  Developers should look for sub-300ms audio latency, bring-your-own STT/TTS/LLM support, SIP/PSTN telephony integration, built-in orchestration with fallback handling, and optional video escalation. VideoSDK provides these capabilities in a single SDK so teams ship production agents faster than assembling multiple point solutions.

  ]]>https://www.videosdk.live/blog/build-ai-voice-agent6899f0a564df6f042b4d41b5Mon, 11 Aug 2025 13:49:00 GMT

  Imagine a world where your customers can interact with an AI-powered agent that understands their needs, responds instantly, and provides a seamless experience across channels. This is the reality of AI Voice Agents in 2026.

  In today’s fast-paced digital landscape, businesses are striving for ways to automate customer service and improve user experience. With AI-powered solutions like voice agents, SaaS companies can offer cutting-edge support that scales effortlessly. These intelligent systems are no longer a futuristic concept but a present-day reality, transforming how businesses engage with their customers.

  In this blog, we’ll guide you through building an AI Voice Agent in minutes using VideoSDK’s powerful tools and features, focusing on ease of integration, scalability, and customization. By the end, you'll understand the core components and be ready to deploy a sophisticated voice agent that can handle customer interactions, schedule appointments, and much more.

  What Are AI Voice Agents, and Why Build One in 2026?

  An AI voice agent is an advanced software program designed to understand and respond to human speech in a natural, conversational manner. Unlike traditional, rigid IVR systems that rely on keypad inputs, AI voice agents leverage technologies like natural language processing to engage in human-like dialogue, automating both inbound and outbound calls without direct human oversight.

  While off-the-shelf voice agent solutions exist, building your own provides unparalleled advantages in customization, brand identity, and data control. A custom voice agent allows you to create a unique and consistent auditory presence that aligns with your brand's personality, a critical differentiator in a crowded market. This ensures that every customer interaction, from a simple query to a complex transaction, reinforces your brand identity.

  The advantages of integrating a bespoke AI voice agent into your operations are substantial:

  • Enhanced Customer Experience: Provide instant, 24/7 support and eliminate frustrating wait times, leading to higher customer satisfaction.
  • Increased Operational Efficiency: Automate routine tasks like appointment scheduling and order updates, freeing up human agents to handle more complex issues.
  • Cost Reduction: Scale your customer communication capabilities to handle high call volumes without a proportional increase in operational costs.
  • Scalability and Consistency: Deliver standardized, high-quality responses that align with your brand's tone, ensuring a consistent experience for every customer.
  • Data-Driven Insights: Gain valuable insights from customer interactions to further refine your products and services.

  Getting Started with Your AI Voice Agent: The Essential Toolkit

  Building a powerful AI voice agent involves the seamless integration of several core technologies. Here’s a breakdown of the essential components and their roles in creating a conversational AI experience.

  

  Automatic Speech Recognition (ASR)

  Automatic Speech Recognition (ASR), also known as speech-to-text, is the foundational technology that converts spoken language into written text. It is the "ears" of your AI agent, enabling it to listen to and understand human speech in real-time. 

  The accuracy and speed of your ASR system are critical for a smooth conversational flow. A high-quality ASR engine ensures that the user's words are transcribed precisely, which is the first and most crucial step for the agent to comprehend the request. Modern ASR can handle various languages, accents, and even operate in noisy environments, making it a robust solution for global businesses. For any developer looking to build a voice-driven application, integrating a powerful real-time transcription API is non-negotiable.

  Natural Language Processing (NLP) & Large Language Models (LLMs)

  Natural Language Processing (NLP) is a field of AI that gives machines the ability to understand, interpret, and generate human language. Large Language Models (LLMs) are an advanced application of NLP, trained on massive datasets to understand context, nuance, and intent, enabling them to generate coherent and relevant responses.

  NLP and LLMs are the "brain" of your AI voice agent. While ASR transcribes what is said, NLP/LLMs figure out what is meant.  They analyze the transcribed text to identify user intent, extract key information, and formulate a contextually appropriate response. This combination allows for dynamic, human-like conversations that go far beyond scripted, rule-based interactions, leading to more satisfying and effective user engagement.

  Text-to-Speech (TTS)

  Text-to-Speech (TTS) technology converts written text back into natural-sounding human speech. This is the "voice" of your AI agent, giving it the ability to communicate its responses audibly.

  The quality of the TTS voice is paramount for a positive user experience. A robotic, unnatural voice can be off-putting, while a clear, expressive, and human-like voice builds trust and keeps users engaged. Modern TTS systems allow for customization of voice, tone, gender, and accent, enabling you to create a voice personality that perfectly reflects your brand. This consistency across touchpoints strengthens brand recall and fosters a more personal connection with your audience.

  Real-Time Communication (WebRTC)

  WebRTC (Web Real-Time Communication) is an open-source framework that enables real-time voice, video, and data communication directly between web browsers and devices without requiring plugins.

  WebRTC is the "nervous system" that transmits the audio data between the user and the AI agent instantly. It provides the low-latency, secure, and reliable infrastructure necessary for seamless, real-time conversations. Whether you are building for the web, iOS, or Android, leveraging a robust Voice Calling API SDK powered by WebRTC is essential. VideoSDK offers highly reliable, cross-platform SDKs that allow you to deploy AI voice agents and other real-time communication features with just a few lines of code, ensuring a scalable and secure implementation anywhere in the world. By combining these powerful technologies on a flexible platform like VideoSDK, you can build and deploy a sophisticated AI voice agent that not only meets but exceeds customer expectations in 2026.

  How to Build an AI Voice Agent in 6 Simple Steps with VideoSDK

  Building a state-of-the-art AI voice agent is no longer a monumental task reserved for large corporations. With VideoSDK's open-source AI Agent SDK, you can create and deploy a powerful, conversational AI agent in minutes. Our Python-based framework is designed for flexibility, allowing you to either use integrated real-time pipelines or build custom agents by combining your preferred STT, LLM, and TTS providers.

  Here’s a step-by-step guide to bringing your voice agent to life.

  Step 1: Set Up Your Development Environment

  First, ensure you have the necessary prerequisites in place. Your backend will host the AI agent, while a client application will connect users to the agent in a VideoSDK meeting room.

  Prerequisites:

  • Python 3.12 or higher.
  • A VideoSDK Auth Token. You can generate these from the VideoSDK dashboard.
  • API keys for your chosen third-party services (e.g., OpenAI for LLM, Deepgram for STT, ElevenLabs for TTS).

  Installation:Create a Python virtual environment and install the VideoSDK Agents package along with any provider plugins you need.

  # Create and activate virtual environment
  python3.12 -m venv venv
  source venv/bin/activate
  
  # Install the core VideoSDK AI Agent package
  pip install videosdk-agents
  
  # Example: Install plugins for OpenAI, Google, and AWS
  pip install "videosdk-plugins-openai"
  pip install "videosdk-plugins-google"
  pip install "videosdk-plugins-aws"

  Next, create a .env file in your project's root to securely store your API keys and tokens.

  VIDEOSDK_AUTH_TOKEN=YOUR_VIDEOSDK_AUTH_TOKEN
  OPENAI_API_KEY=YOUR_OPENAI_API_KEY
  ELEVENLABS_API_KEY=YOUR_ELEVENLABS_API_KEY

  Step 2: Integrating WebRTC for Real-Time Communication

  WebRTC is the cornerstone of real-time voice experiences, enabling ultra-low-latency audio streaming directly between your users and the AI agent. While traditional methods like WebSockets can introduce noticeable lag, especially on unstable networks, WebRTC is specifically designed for reliable, real-time voice in any environment.

  VideoSDK abstracts the complexities of WebRTC. The AI Agent SDK handles all the underlying communication, allowing your agent to join a meeting room just like any other participant. Your backend simply needs to initiate the session. The client-side application can be built using any of VideoSDK's SDKs, including React, React Native, Android, and iOS.

  Here’s how your Python server instructs the agent to join a meeting:

  # main.py
  from fastapi import FastAPI
  from dotenv import load_dotenv
  import os
  from your_agent_file import VoiceAgent, create_agent_session # Your custom agent logic
  
  load_dotenv()
  
  app = FastAPI()
  
  @app.post("/join-agent")
  async def join_agent(request_data: dict):
      meeting_id = request_data.get("meeting_id")
      auth_token = os.getenv("VIDEOSDK_AUTH_TOKEN")
  
      if not meeting_id or not auth_token:
          return {"error": "Meeting ID and auth token are required."}
  
      # Create and start the agent session in the background
      session = await create_agent_session(meeting_id, auth_token)
      await session.start()
  
      return {"message": "Agent is joining the meeting."}

  Step 3: Configuring Speech-to-Text and NLP Capabilities

  Accurate and fast transcription is non-negotiable for a voice agent to understand users correctly. Similarly, a high-quality, natural-sounding voice is essential for user engagement. VideoSDK’s CascadingPipeline offers a modular approach, giving you complete control to mix and match the best STT and TTS providers for your needs. You can choose from providers like Deepgram, Google, and more.

  Here is how you can configure a pipeline using Google for STT and ElevenLabs for TTS:

  # agent_logic.py
  from videosdk.agents import Agent, AgentSession, CascadingPipeline
  from videosdk.plugins.google import GoogleSTT
  from videosdk.plugins.elevenlabs import ElevenLabsTTS
  
  # Define your agent's personality and tools
  class VoiceAgent(Agent):
      def __init__(self):
          super().__init__(
              instructions="You are a friendly and helpful assistant."
          )
  
  # Configure the pipeline with your chosen STT and TTS providers
  async def create_agent_session(meeting_id: str, auth_token: str):
      stt_config = {"model": "long", "language_code": "en-US"}
      stt_provider = GoogleSTT(config=stt_config)
  
  # Configure OpenAI LLM below
  
  llm_provider = OpenAILLM(
     model="gpt-4o",
     # When OPENAI_API_KEY is set in .env - DON'T pass api_key parameter
     api_key="your-openai-api-key",
     temperature=0.7,
     tool_choice="auto",
     max_completion_tokens=1000
  )
  
  
      tts_config = {"model_id": "eleven_multilingual_v2"}
      tts_provider = ElevenLabsTTS(config=tts_config)
      
      pipeline = CascadingPipeline(
          stt=stt_provider,
          tts=tts_provider,
          llm=llm_provider
      )
  
      agent = VoiceAgent()
      session = AgentSession(agent=agent, pipeline=pipeline, meeting_id=meeting_id, auth_token=auth_token)
      return session

  Step 4: Adding AI for Contextual Conversations

  The Large Language Model (LLM) is the brain of your agent. This is where you integrate models like GPT-4 to process the transcribed text and generate intelligent, context-aware responses. VideoSDK’s framework seamlessly connects your chosen LLM into the conversation flow.

  Let's complete the pipeline from the previous step by adding OpenAI's GPT-4 as the LLM.

  # agent_logic.py (continued from Step 3)
  from videosdk.plugins.openai import OpenAILLM
  
  # ... (VoiceAgent class and other imports) ...
  
  # Configure the pipeline with STT, TTS, and now LLM
  async def create_agent_session(meeting_id: str, auth_token: str):
      # ... (STT and TTS provider setup) ...
  
      llm_config = {"model": "gpt-4"}
      llm_provider = OpenAILLM(config=llm_config)
      
      pipeline = CascadingPipeline(
          stt=stt_provider,
          llm=llm_provider,
          tts=tts_provider,
      )
  
      agent = VoiceAgent()
      session = AgentSession(agent=agent, pipeline=pipeline, meeting_id=meeting_id, auth_token=auth_token)
  
      # Add a greeting when the agent joins
      @agent.on_enter
      async def on_enter(session):
          await session.say("Hello! I'm your AI assistant. How can I help you today?")
  
      return session

  With this setup, the audio pipeline is complete: VideoSDK captures user audio, Google transcribes it, OpenAI processes the text and generates a response, and ElevenLabs converts that text back into speech for the user to hear.

  Step 5: Deploy and Test Your AI Voice Agent

  Your AI agent's backend logic, housed in a web server like FastAPI, can be deployed to any modern cloud service.  Popular choices include serverless platforms like Vercel or AWS Lambda for scalability, or containerized applications on services like AWS Fargate or Google Cloud Run.

  Create a videosdk.ymal file with following structure :

  version: "1.0"
  deployment:
    id: your_ai_deployment_id
    entry:
      path: entry_point_for_deployment
  env: # Optional to run your agent locally
    path: "./.env"
  secrets:
    VIDEOSDK_AUTH_TOKEN: your_auth_token
  deploy:
   cloud: true

  Deploy voice agent :

  1. Run AI Development Locally:

  videosdk run

  2. Deploy Voice Agent

  videosdk deploy

  Testing focuses on two key metrics:

  • Latency: Measure the time from when a user finishes speaking to when they hear the agent's response. An ideal response time is under one second to feel natural. VideoSDK's infrastructure is optimized for sub-80ms latency, giving you a strong foundation.
  • Accuracy: Review call transcripts and agent responses to check for errors in transcription or intent recognition. Use this data to refine your agent's instructions (prompts) and improve its conversational abilities.

  Step 6: Optimize for Scale and Performance

  As your user base grows, you'll need to ensure your voice agent can handle high traffic without degrading performance.

  Best Practices for Optimization:

  • Load Balancing: Deploy your backend server across multiple instances and use a load balancer to distribute incoming requests. This prevents any single server from becoming a bottleneck.
  • Efficient Prompt Engineering: Optimize your LLM prompts for speed and clarity. Well-structured prompts reduce the computation required by the model, leading to faster response generation.
  • Asynchronous Processing: Leverage asynchronous tasks for non-blocking operations. For instance, if your agent needs to fetch data from an external API, do it asynchronously so it can still handle other conversational turns. VideoSDK's Python SDK is built with async support to facilitate this.
  • Monitor Performance Metrics: Continuously track metrics like first-call resolution, average response time, and user sentiment. Use these insights to iteratively improve your agent's logic and performance.

  Best Practices for an Exceptional AI Voice Agent Experience

  Building an agent that can talk is one thing; creating an experience that feels natural and intelligent is another. Here are the core principles to elevate your AI voice agent from functional to exceptional.

  Low Latency is Key

  In human conversation, the natural pause between turns is often just a few hundred milliseconds. Any delay beyond that feels awkward and breaks the conversational flow, making the AI feel slow or robotic. This is why ultra-low latency is not just a technical metric but a fundamental requirement for a great user experience. VideoSDK is built for this, with a global mesh network optimized for real-time communication and an AI Agent SDK engineered to minimize delays, typically achieving end-to-end latency of under 600ms. This ensures that interactions are snappy, responsive, and feel as natural as talking to a person.

  Contextual Memory

  A conversation without memory is just a series of disconnected questions and answers. For an agent to be truly helpful, it must remember what was said earlier in the conversation (short-term memory) and even recall information from past interactions (long-term memory). VideoSDK's framework is designed to support this, allowing you to build agents that maintain context. This enables more coherent, personalized, and intelligent dialogues where the agent can reference past details, understand follow-up questions, and provide truly relevant responses.

  Handling Interruptions

  Natural conversations are not always perfectly linear; people interrupt each other. A great AI voice agent must handle this gracefully. This is achieved through advanced Voice Activity Detection (VAD), which detects when a user starts speaking and signals the agent to stop talking immediately. VideoSDK’s Agent SDK manages this complex process, allowing for fluid turn-taking where users can jump in, change the topic, or ask for clarification without waiting for the agent to finish its sentence. This capability is crucial for making the interaction feel collaborative rather than scripted.

  Conclusion

  In 2026, building a powerful, conversational AI voice agent is no longer a futuristic vision but an accessible reality. By combining the core technologies of ASR, NLP/LLMs, and TTS with a robust real-time communication backbone like WebRTC, developers can create truly intelligent systems.

  As we've seen, VideoSDK provides a comprehensive, open-source framework that abstracts away the complexities of real-time transport and multi-provider integration. With just a few lines of Python, you can deploy a scalable, low-latency agent equipped with contextual memory, function-calling capabilities, and natural interruption handling.

  Whether you're looking to revolutionize customer support, automate sales qualification, or create interactive educational experiences, the tools are at your fingertips. The future of human-computer interaction is here, and with VideoSDK, you have everything you need to build it.

  ]]>https://www.videosdk.live/blog/deploy-telephony-agents-on-cerebrium688b61f964df6f042b4d40ecTue, 05 Aug 2025 06:46:18 GMT

  AI-powered telephony solutions are revolutionizing customer service, sales, and communication workflows. This comprehensive guide shows you how to build a sophisticated AI telephony agent using VideoSDK's powerful voice agent capabilities, deployed seamlessly on Cerebrium's cloud platform.

  What We're Building

  We'll create a complete AI telephony system that can:

  • Handle both inbound and outbound voice calls
  • Integrate with SIP providers like Twilio
  • Leverage Google's Gemini AI for intelligent conversations
  • Deploy automatically on Cerebrium's scalable infrastructure
  • Provide real-time voice processing with minimal latency

  

  Architecture Overview

  Our AI telephony agent combines several powerful technologies:

  • VideoSDK Agents: The core voice agent framework
  • SIP Integration: For telephony connectivity via Twilio
  • Gemini AI: Real-time conversational intelligence
  • Cerebrium: Cloud deployment and scaling platform

  Prerequisites

  Before we start, you'll need accounts and credentials for the following services. Here are the links to get you started:

  • VideoSDK Auth Token
  • Twilio SIP trunking setup
  • Google API key for Gemini
  • Cerebrium account, follow docs here

  Project Structure

  Our project follows a clean, modular structure:

  ├── cerebrium.toml
  ├── main.py
  ├── requirements.txt
  └── README.md
  

  Initialize Your Project

  Let's begin by setting up our project directory and basic configuration using the Cerebrium Command Line Interface (CLI).

  pip install cerebrium
  cerebrium login
  cerebrium init videosdk-telephony-agent
  

  Configure Cerebrium Deployment

  First, let's set up our cerebrium.toml configuration file for optimal deployment:

  [cerebrium.deployment]
  name = "sip-ai-agent"
  python_version = "3.12"
  include = ["./*", "main.py", "cerebrium.toml"]
  exclude = [".venv"]
  disable_auth = true
  
  [cerebrium.hardware]
  region = "us-east-1"
  provider = "aws"
  compute = "CPU"
  cpu = 2
  memory = 4.0
  gpu_count = 0
  
  [cerebrium.runtime.custom]
  port = 8000
  entrypoint = ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
  healthcheck_endpoint = "/"
  
  [cerebrium.scaling]
  min_replicas = 1
  max_replicas = 2
  cooldown = 30
  replica_concurrency = 4
  scaling_metric = "concurrency_utilization"
  scaling_target = 80
  
  [cerebrium.dependencies.paths]
  pip = "requirements.txt"
  

  This configuration ensures:

  • Scalability: Auto-scaling between 1-2 replicas based on concurrency, ensuring the app can handle fluctuating call volumes.
  • Performance: Optimized CPU and memory allocation for real-time voice processing.
  • Dependency Management: It clearly points to our requirements.txt file, keeping our dependencies separate and organized.

  Define Project Dependencies

  Create a requirements.txt file in your project directory, you can also view or download the complete file directly from the project's official GitHub repository

  Build the Core AI Agent

  Now, let's create our main application in main.py. Here's the complete implementation:

  import asyncio
  import os
  import logging
  from contextlib import asynccontextmanager
  from typing import Optional
  from dotenv import load_dotenv
  from fastapi import FastAPI, Request, Response
  import uvicorn
  from pyngrok import ngrok
  from videosdk.plugins.sip import create_sip_manager
  from videosdk.agents import Agent, JobContext, function_tool, RealTimePipeline
  from videosdk.plugins.google import GeminiRealtime, GeminiLiveConfig
  
  load_dotenv()
  
  logging.basicConfig(level=os.getenv("LOG_LEVEL", "INFO"))
  logger = logging.getLogger(__name__)
  
  def create_agent_pipeline():
      """Function to create the specific pipeline for our agent."""
      model = GeminiRealtime(
          api_key=os.getenv("GOOGLE_API_KEY"),
          model="gemini-2.0-flash-live-001",
          config=GeminiLiveConfig(
              voice="Leda", # type: ignore
              response_modalities=["AUDIO"], # type: ignore
          ),
      )
      return RealTimePipeline(model=model)
  
  class SIPAIAgent(Agent):
      """A AI agent for handling voice calls."""
  
      def __init__(self, ctx: Optional[JobContext] = None):
          super().__init__(
              instructions=(
               "You are a helpful voice assistant that can answer questions and help with tasks. Be friendly and concise."
               "Talk to the user as if you are a human and not a robot."
               ),
              tools=[self.end_call], # type: ignore
          )
          self.ctx = ctx
          self.greeting_message = "Hello! Thank you for calling. How can I assist you today?"
          logger.info(f"SIPAIAgent created")
  
      async def on_enter(self) -> None:
          pass
  
      async def greet_user(self) -> None:
          await self.session.say(self.greeting_message) # type: ignore
  
      async def on_exit(self) -> None:
          pass
  
      @function_tool
      async def end_call(self) -> str:
          """End the current call gracefully"""
          await self.session.say("Thank you for calling. Have a great day!") # type: ignore
          await asyncio.sleep(1)
          await self.session.leave() # type: ignore
          return "Call ended gracefully"
  
  sip_manager = create_sip_manager(
      provider=os.getenv("SIP_PROVIDER", "twilio"),
      videosdk_token=os.getenv("VIDEOSDK_AUTH_TOKEN"),
      provider_config={
          # Twilio config
          "account_sid": os.getenv("TWILIO_ACCOUNT_SID"),
          "auth_token": os.getenv("TWILIO_AUTH_TOKEN"),
          "phone_number": os.getenv("TWILIO_PHONE_NUMBER"),
      }
  )
  
  @asynccontextmanager
  async def lifespan(app: FastAPI):
      """Lifespan manager for FastAPI app startup and shutdown."""
      port = int(os.getenv("PORT", 8000))
      try:
          ngrok.kill()
          ngrok_auth_token = os.getenv("NGROK_AUTHTOKEN")
          if ngrok_auth_token:
              ngrok.set_auth_token(ngrok_auth_token)
          tunnel = ngrok.connect(str(port), "http")
          sip_manager.set_base_url(tunnel.public_url) # type: ignore
          logger.info(f"Ngrok tunnel created: {tunnel.public_url}")
      except Exception as e:
          logger.error(f"Failed to start ngrok tunnel: {e}")
      yield
      try:
          ngrok.kill()
          logger.info("Ngrok tunnel closed")
      except Exception as e:
          logger.error(f"Error closing ngrok tunnel: {e}")
  
  app = FastAPI(title="SIP AI Agent", lifespan=lifespan)
  
  @app.post("/call/make")
  async def make_call(to_number: str):
      if not sip_manager.base_url:
          return {"status": "error", "message": "Service not ready (no base URL)."}
      agent_config = {"room_name": "Call", "enable_pubsub": True}
      details = await sip_manager.make_call(
          to_number=to_number,
          agent_class=SIPAIAgent,
          pipeline=create_agent_pipeline,
          agent_config=agent_config
      )
      return {"status": "success", "details": details}
  
  @app.post("/sip/answer/{room_id}")
  async def answer_webhook(room_id: str):
      logger.info(f"Answering call for room: {room_id}")
      body, status_code, headers = sip_manager.get_sip_response_for_room(room_id)
      return Response(content=body, status_code=status_code, media_type=headers.get("Content-Type"))
  
  @app.post("/webhook/incoming")
  async def incoming_webhook(request: Request):
      try:
          content_type = request.headers.get("Content-Type", "")
          if "x-www-form-urlencoded" in content_type:
              webhook_data = dict(await request.form())
          else:
              webhook_data = await request.json()
          logger.info(f"Received incoming webhook: {webhook_data}")
  
          agent_config = {"room_name": "Incoming Call", "enable_pubsub": True}
          body, status_code, headers = await sip_manager.handle_incoming_call(
              webhook_data=webhook_data,
              agent_class=SIPAIAgent,
              pipeline=create_agent_pipeline,
              agent_config=agent_config
          )
          return Response(content=body, status_code=status_code, media_type=headers.get("Content-Type"))
      except Exception as e:
          logger.error(f"Error in incoming webhook: {e}", exc_info=True)
          return Response(content="Error processing request", status_code=500)
  
  @app.get("/sessions")
  async def get_sessions():
      return {"sessions": sip_manager.get_active_sessions()}
  
  @app.get("/")
  async def root():
      return {"message": "SIP AI Agent"}
  
  if __name__ == "__main__":
      port = int(os.getenv("PORT", 8000))
      logger.info(f"Starting SIP AI Agent on port {port}")
      uvicorn.run(app, host="0.0.0.0", port=port)
  

  Key Components Explained

  1. Agent Pipeline Creation

  The create_agent_pipeline() function sets up our Gemini AI model with specific configurations:

  • Model: gemini-2.0-flash-live-001 for real-time processing
  • Voice: "Leda" for natural-sounding responses
  • Modalities: Audio-only responses for telephony

  2. SIPAIAgent Class

  Our custom agent class inherits from VideoSDK's Agent base class and includes:

  • Instructions: Clear behavioral guidelines for the AI
  • Tools: Built-in function tools like end_call()
  • Lifecycle methods: on_enter(), greet_user(), on_exit()

  3. SIP Manager Integration

  The SIP manager handles all telephony operations:

  • Provider Configuration: Twilio credentials and settings
  • Call Management: Both inbound and outbound call handling
  • Session Tracking: Active session monitoring

  API Endpoints

  Our API provides several key endpoints:

  • POST /call/make: Initiate outbound calls
  • POST /webhook/incoming: Handle incoming call webhooks
  • POST /sip/answer/{room_id}: Process SIP responses
  • GET /sessions:Monitor active sessions

  Environment Configuration

  Set up these environment variables for your deployment by adding them to a .env file

  # VideoSDK Configuration
  VIDEOSDK_AUTH_TOKEN=your_videosdk_token
  
  # AI Configuration
  GOOGLE_API_KEY=your_google_api_key
  
  # Twilio Configuration
  TWILIO_ACCOUNT_SID=your_account_sid
  TWILIO_AUTH_TOKEN=your_auth_token
  TWILIO_PHONE_NUMBER=your_phone_number
  
  # Optional
  NGROK_AUTHTOKEN=your_ngrok_token
  SIP_PROVIDER=twilio
  

  To make these available to your deployment, add them to the Secrets view on your Cerebrium dashboard.

  Deploying on Cerebrium

  Cerebrium makes deployment incredibly straightforward:

  1. Install Cerebrium CLI:

  pip install cerebrium
  

  1. Deploy your application:

  cerebrium deploy
  

  After the deployment finishes, you will receive a public URL to interact with your application. It will have a format similar to this:

  <https://api.aws.us-east-1.cerebrium.ai/v4/p-xxxxxxxx/sip-ai-agent/>
  

  VideoSDK Documentation Resources

  For deeper integration and customization, explore these VideoSDK resources:

  • AI Agents Introduction
  • Quick Start Guide
  • Core Components Overview
  • Agent Implementation
  • Real-time Pipeline
  • SIP Integration
  • Multiple Agents

  Conclusion

  Building an AI telephony agent with VideoSDK and deploying on Cerebrium creates a powerful, scalable communication solution. This architecture provides:

  • Rapid Development: Get started in minutes with pre-built components
  • Enterprise Scale: Handle thousands of concurrent calls
  • Cost Efficiency: Pay-per-use pricing model
  • Global Reach: Deploy worldwide with minimal latency

  The combination of VideoSDK's robust voice agent framework and Cerebrium's intelligent cloud platform creates an ideal environment for modern AI-powered telephony solutions.

  Ready to get started? Check out the VideoSDK AI Agents documentation and deploy your first agent on Cerebrium today!

  ]]>https://www.videosdk.live/blog/flutter-ios-callkit677e5cf8dbc171042ac708d9Mon, 21 Jul 2025 11:46:00 GMT

  
  Creating a native-like call experience on iOS using Flutter can be challenging—but with the right tools, it becomes seamless. In this guide, we’ll show you how to build a video calling feature for iOS that feels just like a regular phone call.
  

  Using the flutter_callkit_incoming package, you’ll be able to trigger native iOS call screens, manage VoIP-style incoming call notifications, and handle user interactions like answering or rejecting a call—all without writing any Swift code

  Explore the full source code to see how everything fits together.

  Introduction to Flutter CallKit Incoming

  The flutter_callkit_incoming package simplifies the integration of iOS CallKit into Flutter applications. This package allows developers to:

  • Present native iOS call screens.
  • Manage incoming call notifications.
  • Enhance the overall call experience.

  It abstracts the complexities of implementing CallKit in Swift, enabling quick and efficient call handling for iOS users.

  Overview of the Application in IOS
  

  Initiating a Call:

  • The caller (e.g., Pavan) enters the recipient's unique ID (e.g., Jay) and presses the call button.
  • This action triggers a server request to initiate the call, which sends a Firebase Cloud Messaging (FCM) notification to the recipient's device through API, and a fake call is being triggered on the Receiver Caller Side.

  Receiving the Call:

  • If the app is in the foreground: If the App is in the foreground no need to handle the notification, Firebase Messaging Package looks into it. Firebase Messaging Package provided by Firebase in Flutter helps one to manage the Foreground. 
  • If the app is in the background: When it comes to background Firebase provides you the feature using Firebase_background_handler on a method called onBackgroundMessage. Using that we can achieve the concept of handling the background message(Basically the message which is received when the app is closed/Minimised).

  Handling Call Status:

  • Upon answering or rejecting the call, the recipient's action triggers a server request (update call). This request updates the call status and is communicated back to the caller (Pavan) to reflect the recipient's response.

  This structured interaction between client, server, and platform-specific components ensures a smooth and intuitive experience for both the caller and the recipient, providing reliable video communication across devices.

  Core Components of the App

  The app integrates several core components to manage video calling and notifications seamlessly across iOS platforms:

  flutter_callkit_incoming

  • Purpose: Provides a native call interface on iOS.
  • Function: Displays the incoming call UI and handles actions like answering or rejecting calls, leveraging iOS’s native CallKit functionality without requiring native Swift code.

  Push Notifications

  • Purpose: Notifies the recipient about incoming calls.
  • Function: Uses Firebase Cloud Messaging (FCM) and Apple Push Notification Service (APNs) for iOS to send VoIP notifications, triggering the appropriate call UI even if the app is in the background or inactive.

  Firebase Realtime Database

  • Purpose: Stores user tokens and caller IDs.
  • Function: Ensures secure and real-time communication between users, facilitating the initiation of calls and status updates.

  VideoSDK

  • Purpose: Powers the video calling functionality.
  • Function: Enables real-time video and audio communication, providing high-quality video conferencing features for both Android and iOS users.

  Node.js Server

  • Purpose: Acts as the backend service for call initiation and status updates.
  • Function: Sends push notifications via Firebase and APNs, and updates call statuses (e.g., accepted, rejected) between users.

  These core components work together to ensure a smooth, feature-rich experience for both video calling and real-time notifications across devices and platforms.

  Prerequisites

  Before starting with the development of the video calling app, ensure that the following prerequisites are met:

  1. Flutter Development Environment:
     • Install Flutter SDK and set up your development environment. Follow the official Flutter installation guide.
  2. Firebase Project:
     • Create a Firebase project in the Firebase Console.
     • Set up Firebase Cloud Messaging (FCM) for push notifications.
  3. VideoSDK Account:
     • Sign up for a VideoSDK account at VideoSDK and obtain the required credentials (API keys and authentication tokens).
  4. Node.js Server:
     • Set up a Node.js server to manage API requests, handle call initiation, and send push notifications. This server will also interact with Firebase to send call-trigger notifications.

  With these prerequisites in place, you'll be ready to begin implementing the video calling functionality, leveraging both Firebase and VideoSDK for a seamless cross-platform experience.

  Connecting Firebase to Flutter Using FlutterFire CLI

  To easily set up Firebase with your Flutter app, follow these steps:

  Step 1: Create a Firebase Project

  1. Go to the Firebase Console.
  2. Create a new project by following the prompts.

  

  Step 2: Select the Flutter Option

  • Once your project is created, navigate to the "Add app" section.
  • Choose the Flutter option to proceed.

  

  Step 3: Install Firebase CLI

  • Use npm to globally install the Firebase CLI. Run the following command in your terminal:

  npm install -g firebase-tools

  Step 4: Login to Firebase

  • Log in to your Firebase account using the Firebase CLI by running:

  firebase login

  

  • This will open a browser window prompting you to sign in with your Google account.

  Step 5: Configure Firebase in Flutter Using FlutterFire CLI

  • Use the FlutterFire CLI to connect your Flutter project to Firebase
  • Select an existing Firebase project or create a new one.
  • Choose the platforms (e.g., Android, iOS) you want to integrate Firebase with.
  • This will add new dart file named as firebase_option.dart in your project .

  

  Step 6: Add dependency in pubspec.yaml file

  • add firebase_core dependency in your pubspec.yaml file to resolve errors

  iOS Setup: Enabling PushKit and CallKit

  To enable PushKit notifications in your application, it is essential to acquire the necessary certificates from your Apple Developer Program account and set them up for your iOS VoIP application. Follow the steps below:

  Step 1: Request a Certificate Using Keychain

  1. Open the Keychain Access application on your Mac.

  2. Select Certificate Assistant -> Request a Certificate From a Certificate Authority.

  

  3. Enter your email and common name, then click Continue.

  

  4. Modify the certificate’s name and save it.

  

  Step 2: Create an App ID in the Apple Developer Account

  This process requires an active Apple Developer Program account. Follow these steps:

  1. Log into your Apple Developer account.

  2. Navigate to Certificates, Identifiers & Profiles and select Identifiers.

  

  3. Click the + icon to add a new identifier.

  

  4. Add a description, specify your bundle ID, check PushKit under Capabilities, and click Continue.

  

  The image below shows the finished App ID.

  

  Step 3: Create a New VoIP Services Certificate

  
  1. Go to the Certificates section in your Apple Developer Program account.
  

  2. Click to add a new certificate.

  

  3. Select the VoIP Services Certificate and choose the App ID you created.

  

  4. Use the private certificate generated in Keychain Access and click Continue.

  

  5. Download the voip_services.cer file provided by your VoIP service provider.

  Step 4: Convert .cer to .p12

  1. Double-click the voip_services.cer file to open it in Keychain Access.
  2. Locate the certificate titled "VoIP Services: YourProductName".
  3. Right-click on it and select Export to save it as a .p12 file.
  4. Create a strong password when prompted and save the file securely.

  Step 5: Convert .p12 to .pem

  1. Open a terminal and navigate to the directory where the .p12 file is saved.
  2. Enter the password created earlier.
  3. A new file named Certificates.pem will be created in the same directory.

  Run the following command:

  openssl pkcs12 -in YourFileName.p12 -out Certificates.pem -nodes -clcerts -legacy

  Note: The bundle ID of your VoIP services will influence the exact certificate name. This .pem file is now ready for use in your push notification implementation.

  PushKit Setup

  PushKit will allow us to send notifications to the iOS device. To implement push notifications, you must upload an APN Auth Key. The following details about the app are needed when sending push notifications via an APN Auth Key:

  • Auth Key file
  • Team ID
  • Key ID
  • Your app’s bundle ID

  Creating an APN Auth Key

  1. Visit the Apple Developer Member Center

  

  2. Click on Certificates, Identifiers & Profiles. Go to Keys from the left side. Create a new Auth Key by clicking on the plus button on the top right side.

  

  3. On the following page, add a Key Name, and select APNs.

  

  4. Click on the Register button.

  

  5. You can download your auth key file from this page and upload this file to the Firebase dashboard without changing its name.

  

  6. In your Firebase project, go to Settings and select the Cloud Messaging tab. Scroll down the iOS app configuration and click Upload under APNs Authentication Key

  

  7. Enter Key ID and Team ID. Key ID is in the file name AuthKey\_{Key ID}.p8 and is 10 characters. Your Team ID is in the Apple Member Center under the membership tab or displayed always under your account name in the top right corner.

  

  8 . Enable Push Notifications in Capabilities

  

  

  9. Enable selected permission in Background Modes

  

  Project Structure

  Client
  │
  ├── ios
  │   └── Runner
  │       ├── AppDelegate.swift
  │       ├── Info.plist
  │       └── GoogleService-Info.plist
  │
  ├── lib
  │   |
  │   ├── meeting
  │   |   ├── api_call.dart
  │   |   ├── join_screen.dart
  │   |   ├── meeting_controls.dart
  │   |   ├── meeting_screen.dart
  │   |   └── participant_tile.dart
  │   |
  │   ├── firebase_options.dart
  │   ├── home.dart
  │   └── main.dart
  │
  ├── .env
  ├── pubspec.yaml
  └── README.md
  

  Now go to the server side and setup our server

  Step 1 : Create a New Project Directory

  mkdir server
  cd server 
  npm init -y 

  Step 2 : Install required dependencies

  npm install express cors morgan firebase-admin uuid

  Step 3 : Set Up Firebase 

  Enable Realtime Database and set the rule true and Firebase Cloud messaging (FCM).

  

  Step 4 :

  Download the private key by navigating to Project Settings > Service Accounts, selecting Node.js, and then clicking Download.

  

  Step 5 :

  Place this File into your server side and your server side structure look like this.

  Server
    ├── node_modules/
    ├── server.js
    ├── callkit-3ec73-firebase-adminsdk-ghfto-9d9fc7a362.json
    ├── package-lock.json
    └── package.json

  Step 6 : now go to server.js we have to add several API in our server.js

  1. GET /: Returns a "Hello Coding" message to confirm the server is running.
  2. POST /register-device: Registers a device by storing its unique ID and FCM token in Firebase.
  3. POST /api/add: Stores call details (callerId, roomId, and calleeId) in memory.
  4. GET /api/getByCallee/:calleeId: Retrieves call details for a given callee ID from memory.
  5. POST /send-call-status: Sends a notification to the caller about the status of a call (e.g., accepted, rejected, ended).
  6. POST /send-notification: Sends a notification to a caller with details about an incoming call, including room ID and VideoSDK token.

  Step 7 : Now we add below code to run the server at your ip 

  const PORT = process.env.PORT || 9000; 
  const LOCAL_IP = '10.0.0.161'; // Replace with your actual local IP address 
  app.listen(PORT, LOCAL_IP, () => {
  console.log(Server running on http://${LOCAL_IP}:${PORT});
   }); 

  Step 8 :

  we setup ngrok and using this below command we redirect our local ip to temporary public URL that can you share outside to your local network ngrok http http://10.0.0.161:9000

  

  It looks like that we can use this https://8190-115-246-20-252.ngrok-free.app as our server url.

  Refer the complete code of server,js here.

  VideoSDK is a cutting-edge platform that enables seamless audio and video calling with low latency and robust performance. Start by signing up at VideoSDK, generate your API token from the dashboard, and integrate real-time calling into your app effortlessly!

  

  Now add this token into your .env file

  

  Now add all permission for iOS

  
  For iOS we have to add all the permission in info.plist file
  

  <?xml version="1.0" encoding="UTF-8"?>
  <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
  <plist version="1.0">
  <dict>
  	<key>CADisableMinimumFrameDurationOnPhone</key>
  	<true/>
  	<key>CFBundleDevelopmentRegion</key>
  	<string>$(DEVELOPMENT_LANGUAGE)</string>
  	<key>CFBundleExecutable</key>
  	<string>$(EXECUTABLE_NAME)</string>
  	<key>CFBundleIdentifier</key>
  	<string>$(PRODUCT_BUNDLE_IDENTIFIER)</string>
  	<key>CFBundleInfoDictionaryVersion</key>
  	<string>6.0</string>
  	<key>CFBundleName</key>
  	<string>Flutter VideoSDK App</string>
  	<key>CFBundlePackageType</key>
  	<string>APPL</string>
  	<key>CFBundleShortVersionString</key>
  	<string>$(FLUTTER_BUILD_NAME)</string>
  	<key>CFBundleSignature</key>
  	<string>????</string>
  	<key>CFBundleVersion</key>
  	<string>$(FLUTTER_BUILD_NUMBER)</string>
  	<key>LSRequiresIPhoneOS</key>
  	<true/>
  	<key>NSCameraUsageDescription</key>
  	<string>$(PRODUCT_NAME) Camera Usage!</string>
  	<key>NSMicrophoneUsageDescription</key>
  	<string>$(PRODUCT_NAME) Microphone Usage!</string>
  	<key>RTCAppGroupIdentifier</key>
  	<string>group.com.example.broadcastScreen</string>
  	<key>RTCScreenSharingExtension</key>
  	<string>live.videosdk.flutter.example.FlutterBroadcast</string>
  	<key>UIApplicationSupportsIndirectInputEvents</key>
  	<true/>
  	<key>UIBackgroundModes</key>
  	<array>
  		<string>voip</string>
  		<string>fetch</string>
  		<string>processing</string>
  		<string>remote-notification</string>
  	</array>
  	<key>BGTaskSchedulerPermittedIdentifiers</key>
  	<array>
  		<string>dev.flutter.background.refresh</string>
  	</array>
  	<key>UILaunchStoryboardName</key>
  	<string>LaunchScreen</string>
  	<key>UIMainStoryboardFile</key>
  	<string>Main</string>
  	<key>UISupportedInterfaceOrientations</key>
  	<array>
  		<string>UIInterfaceOrientationPortrait</string>
  		<string>UIInterfaceOrientationLandscapeLeft</string>
  		<string>UIInterfaceOrientationLandscapeRight</string>
  	</array>
  	<key>UISupportedInterfaceOrientations~ipad</key>
  	<array>
  		<string>UIInterfaceOrientationPortrait</string>
  		<string>UIInterfaceOrientationPortraitUpsideDown</string>
  		<string>UIInterfaceOrientationLandscapeLeft</string>
  		<string>UIInterfaceOrientationLandscapeRight</string>
  	</array>
  	<key>UIViewControllerBasedStatusBarAppearance</key>
  	<false/>
  	
  </dict>
  </plist>

  Along with info.plist file also check the AppDelegate.swift file

  import UIKit
  import Flutter
  
  @main
  @objc class AppDelegate: FlutterAppDelegate {
    override func application(
      _ application: UIApplication,
      didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?
    ) -> Bool {
      GeneratedPluginRegistrant.register(with: self)
        application.registerForRemoteNotifications()
      return super.application(application, didFinishLaunchingWithOptions: launchOptions)
    }
  }

  Okay, let's move forward with integrating the VideoSDK into our project. Please refer to the Quickstart documentation provided in the link and follow the initial steps to build the necessary steps till step 4 which is creating a paticipant_tile.dart screen.

  Now, after adding the other files to the Meeting folder, lets create a meeting_screen.dart file through which our meeting will be created, rendered and managed.

  import 'dart:convert';
  
  import 'package:flutter/foundation.dart';
  import 'package:flutter/material.dart';
  import 'package:videosdk/videosdk.dart';
  import 'package:videosdk_flutter_example/home.dart';
  
  import 'package:videosdk_flutter_example/meeting/meeting_controls.dart';
  import './participant_tile.dart';
  import 'package:http/http.dart' as http;
  
  class MeetingScreen extends StatefulWidget {
    final String meetingId;
    final String token;
    final String url;
    final String callerId;
    String? source;
    MeetingScreen(
        {Key? key,
        required this.meetingId,
        required this.token,
        required this.callerId,
        required this.url,
        this.source})
        : super(key: key);
  
    @override
    State<MeetingScreen> createState() => _MeetingScreenState();
  }
  
  class _MeetingScreenState extends State<MeetingScreen> {
    late Room _room;
    var micEnabled = true;
    var camEnabled = true;
  
    Map<String, Participant> participants = {};
  
    @override
    void initState() {
      // create room
      if (widget.source == "true") {
        sendnotification(
            widget.url, widget.callerId, "Call Accepted", widget.meetingId);
      }
      _room = VideoSDK.createRoom(
          roomId: widget.meetingId,
          token: widget.token,
          displayName: "John Doe",
          micEnabled: micEnabled,
          camEnabled: camEnabled,
          defaultCameraIndex: kIsWeb
              ? 0
              : 1 // Index of MediaDevices will be used to set default camera
          );
  
      setMeetingEventListener();
  
      // Join room
      _room.join();
  
      super.initState();
    }
  
    Future<void> sendnotification(String api, callerId, status, roomId) async {
      await sendCallStatus(
          serverUrl: api, callerId: callerId, status: status, roomId: roomId);
    }
  
    @override
    void setState(fn) {
      if (mounted) {
        super.setState(fn);
      }
    }
  
    // listening to meeting events
    void setMeetingEventListener() {
      _room.on(Events.roomJoined, () {
        setState(() {
          participants.putIfAbsent(
              _room.localParticipant.id, () => _room.localParticipant);
        });
      });
  
      _room.on(
        Events.participantJoined,
        (Participant participant) {
          setState(
            () => participants.putIfAbsent(participant.id, () => participant),
          );
        },
      );
  
      _room.on(Events.participantLeft, (String participantId) {
        if (participants.containsKey(participantId)) {
          setState(
            () => participants.remove(participantId),
          );
        }
      });
  
      _room.on(Events.roomLeft, () {
        participants.clear();
        Navigator.pushAndRemoveUntil(
          context,
          MaterialPageRoute(
              builder: (context) => Home(
                    callerID: widget.callerId,
                  )),
          (route) => false, // Removes all previous routes
        );
      });
    }
  
    // onbackButton pressed leave the room
    Future<bool> _onWillPop() async {
      _room.leave();
      return true;
    }
  
    Future<void> sendCallStatus({
      required String serverUrl,
      required String callerId,
      required String status,
      required String roomId,
    }) async {
      final url = Uri.parse('$serverUrl/send-call-status');
     
      try {
        // Request payload
        final body = jsonEncode({
          'callerId': callerId,
          'status': status,
          'roomId': roomId,
        });
  
        // Sending the POST request
        final response = await http.post(
          url,
          headers: {
            'Content-Type': 'application/json',
          },
          body: body,
        );
  
        // Handling the response
        if (response.statusCode == 200) {
          print("Notification sent successfully: ${response.body}");
        } else {
          print("Failed to send notification: ${response.statusCode}");
    
        }
      } catch (e) {
        print("Error sending call status: $e");
      }
    }
  
    // This widget is the root of your application.
    @override
    Widget build(BuildContext context) {
      // ignore: deprecated_member_use
      return WillPopScope(
        onWillPop: () => _onWillPop(),
        child: Scaffold(
          appBar: AppBar(
            title: const Text('VideoSDK QuickStart'),
          ),
          body: Padding(
            padding: const EdgeInsets.all(8.0),
            child: Column(
              children: [
                Text(widget.meetingId),
                //render all participant
                Expanded(
                  child: Padding(
                    padding: const EdgeInsets.all(8.0),
                    child: GridView.builder(
                      gridDelegate:
                          const SliverGridDelegateWithFixedCrossAxisCount(
                        crossAxisCount: 2,
                        crossAxisSpacing: 10,
                        mainAxisSpacing: 10,
                        mainAxisExtent: 300,
                      ),
                      itemBuilder: (context, index) {
                        return ParticipantTile(
                            key: Key(participants.values.elementAt(index).id),
                            participant: participants.values.elementAt(index));
                      },
                      itemCount: participants.length,
                    ),
                  ),
                ),
                MeetingControls(
                  micEnabled: micEnabled,
                  camEnabled: camEnabled,
                  onToggleMicButtonPressed: () {
                    setState(() {
                      micEnabled = !micEnabled;
                    });
                    micEnabled ? _room.unmuteMic() : _room.muteMic();
                  },
                  onToggleCameraButtonPressed: () {
                    setState(() {
                      camEnabled = !camEnabled;
                    });
                    camEnabled ? _room.enableCam() : _room.disableCam();
                  },
                  onLeaveButtonPressed: () {
                    _room.leave();
                  },
                ),
              ],
            ),
          ),
        ),
        //home: JoinScreen(),
      );
    }
  }
  

  Great, we have successfully added our videosdk meeting screens. Lets deep dive into main.dart file and home.dart file.
  
  main.dart
  
  

  import 'package:firebase_core/firebase_core.dart';
  import 'package:firebase_messaging/firebase_messaging.dart';
  import 'package:flutter/material.dart';
  import 'package:flutter_dotenv/flutter_dotenv.dart';
  import 'package:videosdk_flutter_example/home.dart';
  
  void main() async {
    WidgetsFlutterBinding.ensureInitialized();
    await dotenv.load(fileName: ".env");
    runApp(MyApp());
  }
  
  class MyApp extends StatelessWidget {
    @override
    Widget build(BuildContext context) {
      return MaterialApp(
        debugShowCheckedModeBanner: false,
        home: Home(),
      );
    }
  }
  

  After creating the main.dart file, now its time to create a home.dart file. In home.dart file we will be working on managing the notification and also the call triggering logic is present in this file.

  Future<void> sendNotification({
      required String callerId,
      required String callerInfo,
      required String roomId,
      required String token,
    }) async {
      // Prepare the request payload
  
      final Map<String, dynamic> payload = {
        'callerId': callerId,
        'callerInfo': {'id': callerInfo},
        'videoSDKInfo': {'roomId': roomId, 'token': token},
      };
  
      try {
        // Send POST request to the API
        final response = await http.post(
          Uri.parse("${apiUrl!}/send-notification"),
          headers: {'Content-Type': 'application/json'},
          body: jsonEncode(payload),
        );
  
        // Handle the response from the API
        if (response.statusCode == 200) {
          print('Notification sent successfully');
          if (mounted) {
            ScaffoldMessenger.of(context).showSnackBar(
              const SnackBar(content: Text("Message sent successfully")),
            );
          }
          print(response.body);
        } else {
          print('Failed to send notification: ${response.body}');
        }
      } catch (e) {
        print('Error occurred while sending notification: $e');
      }
    }
  
  
  Future<void> makeFakeCallInComing(String callerId) async {
  
    {
      _currentUuid = const Uuid().v4();
  
      final params = CallKitParams(
        id: _currentUuid,
        appName: 'VideoSdk',
        avatar: 'https://i.pravatar.cc/100',
        handle: "VideoSdk",
        type: 0,
        duration: 30000,
        textAccept: 'Accept',
        textDecline: 'Decline',
        missedCallNotification: const NotificationParams(
          showNotification: true,
          isShowCallback: true,
          subtitle: 'Missed call',
          callbackText: 'Call back',
        ),
        extra: <String, dynamic>{'userId': '1a2b3c4d'},
        headers: <String, dynamic>{'apiKey': 'Abc@123!', 'platform': 'flutter'},
        ios: const IOSParams(
          iconName: 'CallKitLogo',
          handleType: '',
          supportsVideo: true,
          maximumCallGroups: 2,
          maximumCallsPerCallGroup: 1,
          audioSessionMode: 'default',
          audioSessionActive: true,
          audioSessionPreferredSampleRate: 44100.0,
          audioSessionPreferredIOBufferDuration: 0.005,
          supportsDTMF: true,
          supportsHolding: true,
          supportsGrouping: false,
          supportsUngrouping: false,
          ringtonePath: 'system_ringtone_default',
        ),
      );
      await FlutterCallkitIncoming.showCallkitIncoming(params);
      Future.delayed(const Duration(seconds: 10), () async {
        await FlutterCallkitIncoming.endAllCalls();
        // _initializePhoneAccount();
      });
    }
  }

  To get entire code for home.dart file you can refer here.

  Note: The above code is only applicable when app is in Foreground state or when app is open in Background, yet working on the case where app is totally terminated

  ---

  App Reference

  Conclusion

  With this, we've successfully built the Flutter iOS video calling app using the Flutter_callkit_incoming package, VideoSDK, and Firebase, Node js. For additional features like chat messaging and screen sharing, feel free to refer to our documentation. If you encounter any issues with the implementation, don’t hesitate to reach out to us through our Discord community.

  ]]>https://www.videosdk.live/blog/flutter-android-calltrigger6777c913dbc171042ac7075aMon, 21 Jul 2025 06:39:00 GMT

  In today’s connected world, building a seamless video calling experience is more important than ever. If you're looking to create a cross-platform video calling app with native call-trigger functionality—just like a regular phone call—you’re in the right place.

  In this tutorial, we’ll walk through building a powerful video calling app using:

  • Flutter for the cross-platform UI
  • Node.js for handling server-side token generation
  • Firebase for real-time user status and call event synchronization
  • VideoSDK for high-quality, low-latency audio and video communication
  • Android’s Telecom Framework to provide native call UI and call management behavior

  By the end, you’ll have a production-ready video calling experience complete with incoming/outgoing call screens, call controls, and real-time updates. Be sure to check out the sample code and demo video to see the final result in action.