Large Language Models (LLM)

Overview

In Piopiy, the LLM is your agent's reasoning layer. It receives transcribed user speech (from STT), uses your instructions and context, and streams responses for TTS playback.

How Piopiy Handles LLM

Inside VoiceAgent.Action(...), Piopiy places the LLM between context aggregation and TTS:

1. STT provides user text.
2. Piopiy appends turns to session context.
3. LLM generates streaming response tokens.
4. TTS converts those tokens into spoken audio.

The LLM is session-scoped, so each live call keeps isolated conversation state.

Supported LLM Providers (SDK)

The list below is based on Piopiy SDK LLM integrations.

OpenAI

GPT family models for fast and high-quality conversational reasoning.

Anthropic

Claude models with strong reasoning and tool-use capabilities.

Google

Gemini-based LLM integrations for multimodal and realtime use cases.

Azure OpenAI

Enterprise OpenAI deployments through Azure infrastructure.

AWS Bedrock

AWS-backed LLM options for production-grade cloud deployments.

Groq

Low-latency inference for realtime voice interactions.

Mistral

Mistral-hosted models with strong throughput and quality.

Cerebras

High-speed inference optimized for low-latency serving.

DeepSeek

Cost-effective reasoning models for general-purpose assistants.

Fireworks

Managed inference for open-source and custom model stacks.

Together AI

Broad model catalog with flexible deployment options.

OpenRouter

Unified routing layer across multiple LLM providers.

Perplexity

Search-aware conversational models for knowledge tasks.

Ollama

Local/open-source LLM runtime for privacy-first deployments.

SambaNova

Enterprise model serving for large-scale, performance-focused workloads.

NVIDIA NIM

NVIDIA-hosted inference stack for accelerated model serving.

Qwen

Qwen model support for multilingual and reasoning-heavy tasks.

OpenPipe

Workflow-oriented LLM integration for specialized production pipelines.

xAI Grok

Grok model support for realtime and conversational workloads.

Ultravox

Audio-native realtime LLM optimized for low-latency voice AI.

Implementation Example

import os
from piopiy.voice_agent import VoiceAgent
from piopiy.services.openai.llm import OpenAILLMService

async def on_new_session(agent_id, call_id, from_number, to_number, metadata=None):
    voice_agent = VoiceAgent(
        instructions="You are a concise voice assistant.",
        greeting="Hello! How can I help you?",
    )

    llm = OpenAILLMService(
        api_key=os.getenv("OPENAI_API_KEY"),
        model="gpt-4.1",
    )

    # initialize stt / tts as usual
    await voice_agent.Action(stt=stt, llm=llm, tts=tts)
    await voice_agent.start()

Best Practices

• Keep prompts short and voice-friendly.
• Prefer low-latency models for live calls.
• Use tool calling for precise backend actions instead of hallucinated data.
• Keep a fallback provider for reliability.

What's Next

• Function Calling: Connect the LLM to real backend actions using tool schemas.
• Context Management: Learn how conversation memory is maintained per session.
• Telephony: Deploy LLM-driven flows to real phone sessions.