ai-agents-tutorial

Learn AI Agents step by step, from scratch - from function calling to agent loops to multi-agent systems, orchestration, and evaluation.

Automatically checked · GitHub checked 2026-11-14

ai-agents-tutorial

Overview

AI Agents Tutorial

AI Agents Tutorial

Learn AI Agents step by step, from scratch - from function calling to agent loops to multi-agent systems, orchestration, and evaluation.


Prepared and maintained by the Founder of Outcome School: Amit Shekhar


Note: This series will continue to grow as I write more blogs and create more videos on new topics. Keep learning.


AI Engineering (LLM, RAG, MCP, Agent)

Before diving into AI Agents, it's a good idea to first understand the foundations that agents are built on.

In this video, we will cover the following:

  • LLM
  • RAG
  • MCP
  • Agent
  • Fine-tuning
  • Quantization

Let's get started: AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, Quantization


AI Agent Explained

In this blog, we will learn about the AI Agent - what it is, how it is different from a plain LLM, its five core parts, how it works end to end, the main types, and when to use one.

We will cover the following:

  • The Big Picture
  • What is an AI Agent
  • AI Agent vs Plain LLM vs Chatbot
  • The Five Core Parts
  • How an AI Agent Works End to End
  • A Concrete Example: Research Agent
  • Types of AI Agents
  • What AI Agents Can Do Today
  • When to Use an AI Agent
  • Common Failure Modes
  • Quick Summary

Let's get started: AI Agent Explained


How does Function Calling work in LLMs?

In this blog, we will learn about how Function Calling works in LLMs. We will see what it is, why we need it, the key insight behind it, and how it powers AI agents and assistants step by step.

We will cover the following:

  • What is Function Calling
  • Why We Need Function Calling
  • The Key Insight: The Model Does Not Run the Function
  • How Function Calling Works Step by Step
  • A Concrete Example: get_weather(city)
  • The Conversation Loop
  • Multi-Step and Parallel Function Calling
  • Relation to Structured Outputs and JSON Mode
  • Real-World Use: The Backbone of AI Agents
  • Quick Summary

Let's get started: How does Function Calling work in LLMs?


AI Agent Loop

In this blog, we will learn about the AI Agent Loop - what it is, why an AI Agent needs it, the think-act-observe cycle that powers it, how the loop knows when to stop, and the common ways the loop fails.

We will cover the following:

  • The Big Picture
  • What is the AI Agent Loop
  • Why an AI Agent Needs a Loop
  • The Think-Act-Observe Cycle
  • The Loop Step by Step
  • The Loop in Real Code
  • Parallel Tool Calls in One Turn
  • How the Loop Knows When to Stop
  • Common Loop Failures
  • Quick Summary

Let's get started: AI Agent Loop


ReAct Agent

In this blog, we will learn about the ReAct Agent - what it is, how it is built, its anatomy, how it thinks and acts, and how to handle its common failure modes.

When we hear ReAct Agent, it sounds complex. But do not worry. If we break it down into its individual parts, every single piece is simple.

We will cover the following:

  • What is a ReAct Agent
  • ReAct Agent vs AI Agent
  • Anatomy of a ReAct Agent
  • The ReAct Prompt Template
  • How a ReAct Agent Thinks and Acts
  • A Full Trace Example
  • Implementing a ReAct Agent
  • Common Failure Modes and How to Fix Them
  • Quick Summary

Let's get started: ReAct Agent


Reflection Agent

In this blog, we will learn about the Reflection Agent - what it is, how it is built, its anatomy, how it generates, critiques, and revises its own work, and how to handle its common failure modes.

We will cover the following:

  • What is a Reflection Agent
  • Reflection Agent vs AI Agent
  • Anatomy of a Reflection Agent
  • How a Reflection Agent Works
  • A Full Trace Example
  • Reflection Agent vs ReAct Agent
  • Common Failure Modes and How to Fix Them
  • Quick Summary

Let's get started: Reflection Agent


Plan-and-Execute Agent

In this blog, we will learn about the Plan-and-Execute Agent - what it is, its anatomy, how it plans and runs the steps, how it differs from a ReAct Agent, and how to handle its common failure modes.

We will cover the following:

  • What is a Plan-and-Execute Agent
  • Plan-and-Execute Agent vs AI Agent
  • Anatomy of a Plan-and-Execute Agent
  • How a Plan-and-Execute Agent Works
  • A Full Trace Example
  • Plan-and-Execute Agent vs ReAct Agent
  • Common Failure Modes and How to Fix Them
  • Quick Summary

Let's get started: Plan-and-Execute Agent


AI Agent Memory

In this blog, we will learn about AI Agent Memory - why agents need it, the memory stack, the four core operations (write, read, update, forget), how memory flows at runtime, and the common mistakes.

We will cover the following:

  • The Big Picture
  • Why AI Agents Need Memory
  • The Memory Stack
  • The Four Core Operations
  • How Memory Flows at Runtime
  • What to Store and What Not to Store
  • Common Mistakes and How to Fix Them
  • Quick Summary

Let's get started: AI Agent Memory


How does ChatGPT remember you?

In this blog, we will learn how ChatGPT remembers us across chats even though the model itself remembers nothing. We will see why LLMs forget by default, how saved memories, chat history, and Custom Instructions work, and how these memories are added to the prompt before every reply.

We will cover the following:

  • Why LLMs forget by default
  • The context window
  • Memory within one chat
  • What ChatGPT Memory is
  • Saved memories
  • Reference chat history
  • Custom Instructions
  • How memories enter the prompt
  • An end-to-end example
  • Viewing and deleting memories
  • Temporary Chat
  • Where memory helps and where it fails

Let's get started: How does ChatGPT remember you?


Context Engineering

In this blog, we will learn about Context Engineering - what it is, why it has become the most important skill for building reliable AI applications, how it differs from Prompt Engineering, the components that make up the context, common patterns like RAG, few-shot examples, tools, and memory, and the best practices and common mistakes to keep in mind.

We will cover the following:

  • What is Context Engineering?
  • The Big Picture
  • Why Context Engineering matters
  • Prompt Engineering vs Context Engineering
  • The components of the context
  • Common patterns in Context Engineering
  • Common mistakes to avoid
  • Best practices
  • Quick Summary

Let's get started: Context Engineering


Agentic RAG

In this blog, we will learn about Agentic RAG - what it is, why standard RAG falls short, the agentic RAG loop, the three building blocks, the common patterns, when to use it, and the limitations to keep in mind.

A hard question often needs more than one search. Some of those searches need different sources. Some depend on what the previous search found. Standard RAG cannot do any of this. Agentic RAG can.

Agentic RAG = Agentic + RAG

We will cover the following in this blog:

  • The Big Picture
  • A Quick Recap of RAG
  • A Quick Recap of AI Agent
  • Why Standard RAG Falls Short
  • What is Agentic RAG
  • The Agentic RAG Loop
  • The Three Building Blocks
  • A Walkthrough with a Real Example
  • Common Patterns of Agentic RAG
  • Standard RAG vs Agentic RAG
  • When to Use Agentic RAG
  • Limitations of Agentic RAG
  • Quick Summary

Let's get started: Agentic RAG


GraphRAG

In this blog, we will learn about GraphRAG and how it improves retrieval by using a knowledge graph along with vector search.

We will cover the following:

  • What is GraphRAG?
  • Why normal RAG is not enough
  • The big picture of GraphRAG
  • How GraphRAG builds the knowledge graph
  • How GraphRAG answers a question
  • Local search vs Global search
  • When to use GraphRAG
  • Trade-offs of GraphRAG
  • Quick Summary

Let's get started: GraphRAG


Multi-Agent Systems

In this blog, we will learn about Multi-Agent Systems - what they are, the three pillars that hold them together, the common agent roles, how agents communicate and coordinate, the trade-offs, and when to use them.

We will cover the following:

  • The Big Picture
  • What is a Multi-Agent System
  • The Three Pillars
  • Common Agent Roles
  • How Agents Communicate
  • How Agents Coordinate
  • Multi-Agent vs Single Agent - The Trade-offs
  • Common Mistakes
  • When to Use a Multi-Agent System
  • Quick Summary

Let's get started: Multi-Agent Systems


AI SubAgents

In this blog, we will learn about AI SubAgents. We will understand what they are, why we need them, how they work, and how to use them to build AI systems that can handle big and complex tasks.

We will cover the following:

  • What is an AI Agent?
  • What are AI SubAgents?
  • Why do we need SubAgents?
  • How do SubAgents work?
  • Example use case
  • Benefits of using SubAgents
  • Challenges with SubAgents
  • Best practices

Let's get started: AI SubAgents


What is MCP (Model Context Protocol)?

In this blog, we will learn about MCP (Model Context Protocol). We will also see the problem it solves, the pieces it is made of, how a request travels from the AI model all the way to a tool and back, and what we must be careful about while using it.

We will cover the following:

  • The problem before MCP
  • What is MCP?
  • MCP = Model + Context + Protocol
  • The USB-C analogy
  • How it is different from a normal API
  • The three parts of MCP
  • How it all works step by step
  • How the connection happens
  • A real example
  • Importance of MCP
  • Things we must be careful about
  • Summary

Let's get started: What is MCP (Model Context Protocol)?


AI Orchestration

In this blog, we will learn about AI Orchestration. We will understand what it is, why we need it, how it is different from AI Agents, and the common patterns we use to coordinate multiple LLMs, tools, and steps together to build real AI products.

We will cover the following:

  • What is AI Orchestration?
  • Why do we need AI Orchestration?
  • AI Orchestration vs AI Agents
  • Components of AI Orchestration
  • How AI Orchestration works
  • Patterns of AI Orchestration
  • Sequential Pattern
  • Parallel Pattern
  • Conditional Pattern
  • Loop Pattern
  • Orchestrator-Worker Pattern
  • Tools for AI Orchestration
  • Challenges in AI Orchestration
  • Best Practices

Let's get started: AI Orchestration


How AI Agents Communicate

In this blog, we will learn about how AI agents communicate. We will understand why agents need to communicate, the main ways they talk to each other, the message format, and the protocols that make agents work together to finish complex tasks.

We will cover the following:

  • What is agent communication?
  • Why do agents need to communicate?
  • What agents need in order to communicate
  • How a message flows between agents
  • The ways AI agents communicate
  • Direct Communication
  • Centralized Communication
  • Broadcast Communication
  • Shared Memory Communication
  • What a message looks like
  • The rules agents follow to talk
  • Challenges when agents communicate
  • Best Practices

Let's get started: How AI Agents Communicate


How do Computer-Use Agents work?

In this blog, we will learn about how computer-use agents work.

We will cover the following:

  • What is a computer-use agent?
  • Why do we need a computer-use agent?
  • The perceive, think, act loop
  • How does the agent see the screen?
  • How does the agent decide what to do?
  • How does the agent take actions?
  • A step-by-step walkthrough with an example
  • The system prompt and tools
  • Safety and guardrails
  • Limitations of computer-use agents
  • Conclusion

Let's get started: How do Computer-Use Agents work?


How do Browser Agents work?

In this blog, we will learn about how browser agents work. We will also see how a browser agent sees a web page, how it takes actions, and how the observe-think-act loop completes a task from start to finish.

We will cover the following:

  • What is an AI agent?
  • Why do we need Browser Agents?
  • What is a Browser Agent?
  • The parts of a Browser Agent
  • How does a Browser Agent see a web page?
  • How does a Browser Agent take actions?
  • The observe-think-act loop
  • A complete example: ordering a book
  • A simple code example
  • Screenshots vs page structure
  • Problems and how they are handled
  • Where it works well and where it fails

Let's get started: How do Browser Agents work?


How does LangChain work?

In this blog, we will learn about how LangChain works. We will also see why we need it, what chains, prompts, memory, and output parsers are, how retrieval and agents fit in, and how the full flow works together in the real world.

We will cover the following:

  • What is LangChain?
  • Why do we need LangChain?
  • The core idea behind LangChain
  • LLM and Prompt Template
  • What is a Chain?
  • Output Parser
  • Memory
  • Retrieval and RAG
  • Tools and Agents
  • A complete flow of how LangChain works

Let's get started: How does LangChain work?


How does LangGraph work?

In this blog, we will learn about how LangGraph works. We will also see why we need it, what graphs, state, nodes, and edges are, how tools work and who actually calls them, how memory and human-in-the-loop fit in through a complete example, and when to use it in the real world.

We will cover the following:

  • What is LangGraph?
  • Why do we need LangGraph?
  • What is a Graph in LangGraph?
  • What is State in LangGraph?
  • Nodes and Edges
  • Conditional Edges
  • A complete example
  • Tools and who calls them
  • Memory and persistence
  • Human-in-the-loop
  • When to use LangGraph

Let's get started: How does LangGraph work?


How do Human-in-the-Loop agents work?

In this blog, we will learn about how Human-in-the-Loop agents work. We will also see why a fully automatic agent is risky, the points where a human can step in, how an agent pauses and continues after the human replies, how to decide which actions need a human, and where this approach works well and where it slows us down.

We will cover the following:

  • The problem with fully automatic agents
  • What is a Human-in-the-Loop agent?
  • Where can a human step in?
  • How does a Human-in-the-Loop agent work?
  • How does the agent pause and continue?
  • A simple code example
  • Which actions need a human?
  • Common mistakes
  • Advantages and disadvantages

Let's get started: How do Human-in-the-Loop agents work?


Harness Engineering in AI

In this blog, we will learn about Harness Engineering in AI. We will understand what a harness is, why we need it, and how it is used in AI Agents and evaluation systems.

We will cover the following:

  • What is a Harness in AI?
  • Why do we need Harness Engineering?
  • Components of an AI Harness
  • Harness Engineering for AI Agents
  • Harness Engineering for Evaluation
  • Best Practices in Harness Engineering
  • Putting It All Together

Let's get started: Harness Engineering in AI


AI Agent Observability

In this blog, we will learn about AI Agent Observability. We will also see why we need it, how it is different from normal software monitoring, what we must observe inside an agent, the key concepts like traces and spans, the metrics we must track, the tools we can use, and the best practices to follow.

We will cover the following:

  • What is an AI Agent?
  • What is Observability?
  • What is AI Agent Observability?
  • Why do we need AI Agent Observability?
  • How is AI Agent Observability different from traditional Observability?
  • The Three Pillars of Observability
  • Traces and Spans
  • What should we observe inside an AI Agent?
  • Key Metrics for AI Agent Observability
  • How AI Agent Observability works
  • Tools and Frameworks for AI Agent Observability
  • Observability vs Evaluation
  • Challenges in AI Agent Observability
  • Best Practices

Let's get started: AI Agent Observability


AI Agent Evaluation

In this blog, we will learn about AI Agent Evaluation. We will also see why it is different from LLM Evaluation, the types of evaluation we can do, the key metrics we must track, the methods we can use, and the best practices to follow.

We will cover the following:

  • What is an AI Agent?
  • What is AI Agent Evaluation?
  • Why do we need AI Agent Evaluation?
  • How is AI Agent Evaluation different from LLM Evaluation?
  • Types of AI Agent Evaluation
  • Outcome Evaluation
  • Trajectory Evaluation
  • Tool Use Evaluation
  • Planning Evaluation
  • Key Metrics for AI Agents
  • Agent Benchmarks
  • Methods to Evaluate AI Agents
  • Frameworks and Tools for AI Agent Evaluation
  • Challenges in AI Agent Evaluation
  • Best Practices

Let's get started: AI Agent Evaluation


Decoding Sakana Fugu Technical Report

In this blog, we are going to learn about Sakana Fugu, a family of AI models that work like a conductor for a team of other AI models.

We will cover the following:

  • What is Sakana Fugu?
  • Why Fugu was needed
  • The big picture: what Fugu does
  • Collective Intelligence
  • The two Fugus - Fugu and Fugu-Ultra
  • How Fugu picks the right model - the lightweight selection head
  • Teaching Fugu who is best - supervised fine-tuning
  • Polishing Fugu on real tasks - evolutionary strategies
  • How Fugu-Ultra conducts an orchestra - the Conductor
  • Teaching Fugu-Ultra to conduct - GRPO
  • Stopping the agents from copying each other
  • How well does Fugu perform?
  • The clever strategies Fugu discovered on its own
  • Quick Summary

Let's get started: Decoding Sakana Fugu Technical Report


What are Agent Skills?

In this blog, we will learn about Agent Skills. We will also see why we need them, what is inside a Skill, how the description makes an agent pick up the right Skill by itself, how progressive disclosure keeps the context window free, how Agent Skills differ from MCP, and how we can create our own Skill.

We will cover the following:

  • The problem before Agent Skills
  • What are Agent Skills?
  • What is inside a Skill?
  • The description is the trigger
  • Progressive disclosure, the main idea
  • A Skill can carry real code
  • Where Skills live
  • How do we create our own Skill?
  • Agent Skills vs MCP
  • A real example
  • Importance of Agent Skills
  • Things we must be careful about
  • Summary

Let's get started: What are Agent Skills?


Design a Real-Time Voice AI Agent

In this blog, we will learn about how to design a Real-Time Voice AI Agent, a system that listens to a person speaking, understands what they said, thinks about it, takes actions if needed, and talks back in a natural human-like voice, all within a fraction of a second. We will also see why voice is much harder than a text chatbot, the two big ways to build it (the cascaded pipeline of Speech-to-Text, LLM, and Text-to-Speech versus the end-to-end Speech-to-Speech model), how the agent knows when the user has stopped talking, how we handle interruptions, how tools and memory fit in, how we scale it to thousands of calls, the edge cases that break a voice agent in production, the pros and cons of every approach, and when to use which one.

We will cover the following:

  • What is a Voice AI Agent?
  • Why is Real-Time Voice hard?
  • Requirements
  • Back-of-the-envelope estimation
  • High-Level Architecture
  • Component 1: Audio Transport
  • Component 2: Voice Activity Detection and Turn Detection
  • Component 3: Speech-to-Text (STT)
  • Component 4: The Brain - LLM with Tools
  • Component 5: Text-to-Speech (TTS)
  • Approach 1: Cascaded Pipeline (STT -> LLM -> TTS)
  • Approach 2: Speech-to-Speech Model
  • Approach 3: Hybrid Approach
  • Cascaded vs Speech-to-Speech: Comparison
  • Latency Budget: Where every millisecond goes
  • Handling Interruptions (Barge-in)
  • Tool Calling in a Voice Agent
  • Memory and Context
  • Telephony: Connecting to real phone calls
  • Scaling the system
  • Edge Cases and how to handle them
  • Observability and Evaluation
  • Safety, Security, and Privacy
  • Cost
  • How to present this design in an interview

Let's get started: Design a Real-Time Voice AI Agent


What is turn-taking in a voice AI agent?

In this blog, we will learn about turn-taking in a voice AI agent, how the agent decides when the user has finished speaking, when it is its turn to talk, and what to do when the user starts speaking again while the agent is talking. We will also see why silence alone is not enough, how semantic turn detection reads the meaning of the words, how barge-in and backchannels are handled, and how half-duplex agents differ from full-duplex agents.

We will cover the following:

  • What is turn-taking?
  • How humans take turns
  • How a voice AI agent works
  • Why turn-taking is hard for a machine
  • The two jobs of turn-taking
  • Approach 1: Wait for silence
  • Approach 2: Understand the meaning
  • Handling interruptions
  • Not every sound is a new turn
  • Half-duplex vs full-duplex agents
  • A complete walkthrough

Let's get started: What is turn-taking in a voice AI agent?


How does endpointing decide when the user has finished speaking?

In this blog, we will learn about how endpointing decides when the user has finished speaking. We will also see why this decision is so hard, how a simple silence timer works, how voice activity detection finds speech and silence, how the words and the tone of the voice help, how all these signals come together, and how a real voice assistant balances speed with not cutting off the user.

We will cover the following:

  • What is Endpointing?
  • Why is Endpointing Hard?
  • Approach 1: Fixed Silence Timer
  • How Voice Activity Detection Works
  • Approach 2: Reading the Words
  • Approach 3: Listening to the Voice
  • Approach 4: Combining All the Signals
  • Waiting Too Long vs Cutting Off Too Early
  • Extra Steps Around the Endpoint

Let's get started: How does endpointing decide when the user has finished speaking?


How do silence-based and model-based turn detection differ?

In this blog, we will learn about how silence-based and model-based turn detection differ. We will also see what turn detection is, how silence-based turn detection works and why a fixed pause fails, how a model predicts the end of a turn from words and from sound, how both are combined in a voice pipeline, and when to use which one.

We will cover the following:

  • What is Turn Detection?
  • How Silence-Based Turn Detection Works
  • The Problems with Silence-Based Turn Detection
  • How Model-Based Turn Detection Works
  • Text-Based and Audio-Based Turn Detection Models
  • How Both Work Together in a Voice Pipeline
  • Silence-Based vs Model-Based Turn Detection
  • When to Use Which One

Let's get started: How do silence-based and model-based turn detection differ?


How does dynamic endpointing adapt to how fast a user talks?

In this blog, we will learn about how dynamic endpointing adapts to how fast a user talks. We will also see what endpointing is, why a fixed silence timer fails for fast and slow talkers, how the system learns the pauses of a user and turns them into a waiting time, how the meaning of the sentence changes that waiting time, and where it works well and where it fails.

We will cover the following:

  • What is endpointing?
  • The problem with a fixed silence timer
  • What is dynamic endpointing?
  • How does the system measure how fast a user talks?
  • Turning pauses into a waiting time
  • A simple dynamic endpointer in code
  • A step-by-step walkthrough
  • Using the meaning of the sentence as well
  • Where it works well and where it fails

Let's get started: How does dynamic endpointing adapt to how fast a user talks?


How does a voice agent handle long pauses when the user is thinking?

In this blog, we will learn about how a voice agent handles long pauses when the user is thinking. We will also see why silence is confusing for a voice agent, why a simple silence timer fails, how the agent uses the words and the voice to decide, how it changes its waiting time, what it does when it speaks too early or the user stays silent for very long, and where it works well and where it fails.

We will cover the following:

  • What is a thinking pause?
  • Why is a long pause a hard problem?
  • How does a voice agent hear silence?
  • Approach 1: A fixed silence timer
  • Approach 2: A longer silence timer
  • Approach 3: Looking at the words
  • Approach 4: Listening to how the words are said
  • Changing the waiting time dynamically
  • What if the agent starts speaking too early?
  • What if the user stays silent for a very long time?
  • The complete flow
  • Where it works well and where it fails

Let's get started: How does a voice agent handle long pauses when the user is thinking?


How does a voice agent know a phone number or email is not finished yet?

In this blog, we will learn about how a voice agent knows a phone number or email is not finished yet. We will also see why people pause in the middle of numbers and emails, why waiting for silence alone fails, how the agent uses the question it just asked, how it checks whether the value looks complete, how a turn detection model reads the words, and where this works well and where it fails.

We will cover the following:

  • The problem: people pause in the middle
  • What is endpointing?
  • Approach 1: Wait for a fixed silence
  • Approach 2: Wait longer for everyone
  • Approach 3: Use the question the agent just asked
  • Approach 4: Check if the value looks complete
  • Approach 5: A turn detection model that reads the words
  • How all the signals work together
  • What the agent does when it is still unsure
  • Where it works well and where it fails

Let's get started: How does a voice agent know a phone number or email is not finished yet?


Why does a voice agent cut people off mid-sentence?

In this blog, we will learn about why a voice agent cuts people off mid-sentence. We will also see how a voice agent listens and decides when to reply, why a simple silence timer gets it wrong, the common situations that cause it, why making the agent wait longer creates a new problem, how modern voice agents fix it by understanding the meaning of what we said, and how they recover when they still get it wrong.

We will cover the following:

  • How a voice agent decides it is time to reply
  • The silence timer
  • Why a pause is not the end of a sentence
  • Common reasons a voice agent cuts us off
  • Why not just wait longer?
  • How does semantic turn detection fix it?
  • Recovering when the agent still gets it wrong

Let's get started: Why does a voice agent cut people off mid-sentence?


What is barge-in in a voice AI agent?

In this blog, we will learn about barge-in in a voice AI agent, what happens when the user starts speaking while the agent is still talking, and how the agent stops so it can listen. We will also see why the agent can hear its own voice and how echo cancellation fixes it, how the agent tells a real interruption apart from "uh-huh", a cough, or background noise, what it does after a barge-in, and why it must remember only the part of the reply the user actually heard.

We will cover the following:

  • What is barge-in?
  • Where the word barge-in comes from
  • How a voice AI agent speaks and listens
  • Problem 1: The agent hears itself
  • Problem 2: Not every sound is an interruption
  • How the agent detects barge-in
  • What the agent does after barge-in
  • Remembering what the user actually heard
  • Why barge-in must be fast
  • A complete walkthrough

Let's get started: What is barge-in in a voice AI agent?


Cascaded vs Speech-to-Speech: How to choose the right voice agent architecture?

In this blog, we will learn about Cascaded vs Speech-to-Speech and how to choose the right voice agent architecture. We will also see how a cascaded voice agent works, how a speech-to-speech voice agent works, how the two differ in delay, tone, control, and cost, what a half-cascade in the middle looks like, and when to use which one.

We will cover the following:

  • What is a voice agent architecture?
  • What is a cascaded voice agent?
  • Strengths and weaknesses of a cascaded voice agent
  • What is a speech-to-speech voice agent?
  • Strengths and weaknesses of a speech-to-speech voice agent
  • The middle path: Half-Cascade
  • Differences between Cascaded, Half-Cascade, and Speech-to-Speech
  • How to choose the right voice agent architecture

Let's get started: Cascaded vs Speech-to-Speech: How to choose the right voice agent architecture?


How do half-duplex and full-duplex voice agents differ?

In this blog, we will learn about how half-duplex and full-duplex voice agents differ. We will also see what the word duplex means, how a half-duplex voice agent works step by step, why it feels slow and unnatural, how barge-in tries to fix it, how a full-duplex voice agent listens and speaks together, and when to use which one.

We will cover the following:

  • What does duplex mean?
  • What is a half-duplex voice agent?
  • How a half-duplex voice agent works step by step
  • The problems with half-duplex
  • Barge-in on top of half-duplex
  • What is a full-duplex voice agent?
  • How a full-duplex voice agent works
  • The same conversation in both agents
  • Half-duplex vs full-duplex
  • When to use which one

Let's get started: How do half-duplex and full-duplex voice agents differ?


How does a voice agent tell a real interruption from a cough or background noise?

In this blog, we will learn about how a voice agent tells a real interruption from a cough or background noise. We will also see why this is hard, how the agent cleans the sound before judging it, how it checks for human speech, length, and real words, how it pauses first and resumes when it was wrong, how a trained model helps, and where it works well and where it fails.

We will cover the following:

  • What is an interruption in a voice agent?
  • Why is it hard to tell a real interruption from noise?
  • Step 1: Remove the agent's own voice
  • Step 2: Remove the background noise
  • Step 3: Is it human speech?
  • Step 4: Did it last long enough?
  • Step 5: Did the user say real words?
  • Step 6: Pause first, then decide
  • Using a trained interruption model
  • Putting it all together
  • Where it works well and where it fails

Let's get started: How does a voice agent tell a real interruption from a cough or background noise?


What are backchannels, and why should "mm-hmm" not stop the agent?

In this blog, we will learn about what backchannels are and why "mm-hmm" should not stop the agent. We will also see how backchannels work in a normal human conversation, what goes wrong when a voice agent treats them as interruptions, how an agent tells a backchannel apart from a real interruption, what the agent does while it is still unsure, and where this works well and where it fails.

We will cover the following:

  • What is a backchannel?
  • Backchannels in a normal conversation
  • What goes wrong when "mm-hmm" stops the agent
  • Backchannel vs real interruption
  • How a voice agent tells them apart
  • What the agent does while it is unsure
  • A complete example
  • Where it works well and where it fails

Let's get started: What are backchannels, and why should "mm-hmm" not stop the agent?


How does a voice agent stop speaking instantly when interrupted?

In this blog, we will learn about how a voice agent stops speaking instantly when interrupted. We will also see where the agent's audio waits before we hear it, why stopping only the LLM is not enough, how the agent clears every waiting area in the right order, how it ignores late audio that is still on the way, how it remembers only what we actually heard, and where it works well and where it fails.

We will cover the following:

  • What does stopping instantly mean?
  • How a voice agent speaks
  • Where the audio waits before we hear it
  • Why stopping the LLM is not enough
  • Step 1: Stop the sound at our ear
  • Step 2: Clear the server buffer
  • Step 3: Cancel the work that makes new audio
  • Step 4: Ignore late audio with a reply number
  • Step 5: Remember only what was heard
  • Keeping the waiting areas small
  • A complete example
  • Where it works well and where it fails

Let's get started: How does a voice agent stop speaking instantly when interrupted?


Why does TTS audio buffering make interruptions slow?

In this blog, we will learn about why TTS audio buffering makes interruptions slow. We will also see how a TTS model creates audio faster than real time, where the extra audio waits before we hear it, how much delay a full buffer adds when we interrupt, why we cannot simply remove all the buffers, how pacing the audio keeps the buffers small, and where this works well and where it fails.

We will cover the following:

  • What is TTS audio buffering?
  • TTS creates audio faster than we can hear it
  • Where the extra audio waits
  • What happens when we interrupt
  • How much delay does the buffer add?
  • Why we cannot remove the buffers
  • The fix: send audio at the speed of speech
  • Clearing what is left
  • A complete example
  • Where it works well and where it fails

Let's get started: Why does TTS audio buffering make interruptions slow?


How does a voice agent truncate its memory to only what the user actually heard?

In this blog, we will learn about how a voice agent truncates its memory to only what the user actually heard. We will also see what the memory of a voice agent is, why the agent prepares its reply faster than it speaks it, what goes wrong when the memory is not cut, how the agent measures what was played and turns that time into words, how a speech-to-speech API does the same thing, and which edge cases the agent must handle.

We will cover the following:

  • What is the memory of a voice agent?
  • The problem: the agent remembers more than the user heard
  • Why the reply is ready before it is spoken
  • What goes wrong if we do not truncate
  • How the agent knows what the user actually heard
  • Turning played time into words
  • Truncating the memory step by step
  • A complete walkthrough
  • How a speech-to-speech API does this
  • Edge cases

Let's get started: How does a voice agent truncate its memory to only what the user actually heard?


How does a voice agent recover from a false interruption?

In this blog, we will learn about how a voice agent recovers from a false interruption. We will also see what a false interruption is, why pausing works better than stopping, how the agent decides that an interruption was false, how it continues speaking from the right place, how it keeps the conversation history correct, and how to tune the waiting time.

We will cover the following:

  • What is a false interruption?
  • What goes wrong without recovery
  • Pause, do not stop
  • How does the agent decide that an interruption was false?
  • How does the agent continue speaking?
  • When the interruption turns out to be real
  • Keeping the conversation history correct
  • A complete example from start to end
  • Tuning the waiting time

Let's get started: How does a voice agent recover from a false interruption?


How does a voice agent handle the user and agent talking at the same time?

In this blog, we will learn about how a voice agent handles the user and agent talking at the same time. We will also see why overlapping speech is a hard problem, the different kinds of overlap, how the agent hears the user over its own voice and decides whether to stop or continue, how it keeps the conversation history correct after it stops, how full-duplex models handle overlap in a different way, and where it works well and where it fails.

We will cover the following:

  • What is overlapping speech?
  • Why is overlapping speech a hard problem for a voice agent?
  • The different kinds of overlap
  • Step 1: Keep listening while speaking
  • Step 2: Remove the agent's own voice
  • Step 3: Decide what kind of overlap it is
  • Step 4: Act on the decision
  • Step 5: Keep the conversation history correct
  • What if both start speaking at the same moment?
  • How full-duplex models handle overlap
  • A complete example
  • Where it works well and where it fails

Let's get started: How does a voice agent handle the user and agent talking at the same time?


How does a voice agent add its own backchannels like "okay" and "got it"?

In this blog, we will learn about how a voice agent adds its own backchannels like "okay" and "got it". We will also see why a silent agent feels broken, the two kinds of backchannels an agent uses, how the agent decides the right moment and the right sound, how it plays the sound without interrupting itself, how full-duplex models do it, and where it works well and where it fails.

We will cover the following:

  • What is a backchannel?
  • Why does a silent voice agent feel broken?
  • The two kinds of backchannels a voice agent uses
  • The parts of a voice agent
  • How the agent decides when to say "mm-hmm"
  • How the agent picks the right sound
  • How the agent plays the sound without stopping its listening
  • How the agent avoids interrupting itself
  • How the agent says "got it" after the user finishes
  • How full-duplex models do it in a different way
  • Where it works well and where it fails

Let's get started: How does a voice agent add its own backchannels like "okay" and "got it"?


How do you build a latency budget for a voice AI agent?

In this blog, we will learn about how to build a latency budget for a voice AI agent, a plan that splits the total waiting time for a reply into smaller pieces, one for each step of the agent. We will also see why a voice AI agent needs one, how to pick the total target and split it across every stage, how to measure the real numbers and cut the biggest delays first, how streaming and tool calls change the budget, and what trade-off we must accept.

We will cover the following:

  • What is a latency budget?
  • Why does a voice AI agent need a latency budget?
  • Step 1: Pick the total target
  • Step 2: List every stage of the pipeline
  • Step 3: Give each stage its share of the time
  • Step 4: Measure the real numbers
  • Step 5: Cut the biggest delay first
  • How streaming changes the budget
  • How tool calls fit into the budget
  • The trade-off we must accept

Let's get started: How do you build a latency budget for a voice AI agent?


How does the voice-to-voice latency of a voice agent get measured?

In this blog, we will learn about how the voice-to-voice latency of a voice agent gets measured. We will also see where the time goes inside a voice agent, why the server logs give the wrong answer, how a two-channel recording gives the right answer, how we automate it over many calls, why we look at percentiles instead of the average, and where it works well and where it fails.

We will cover the following:

  • What is voice-to-voice latency?
  • Why is this latency important?
  • Where does the time go?
  • Approach 1: Using the logs on the server
  • Approach 2: Adding timestamps at every step
  • Approach 3: Recording the call on one timeline
  • Automating the measurement over many calls
  • Why do we look at percentiles and not the average?
  • Where it works well and where it fails

Let's get started: How does the voice-to-voice latency of a voice agent get measured?


Why does the time-to-first-audio matter more than total response time?

In this blog, we will learn about why the time-to-first-audio matters more than total response time. We will also see what each of these two times measures, how people experience silence in a conversation, how streaming lets an agent start speaking before the full reply is ready, which steps decide the time-to-first-audio, and when the total response time still matters.

We will cover the following:

  • What is time-to-first-audio?
  • What is total response time?
  • How do people experience waiting on a voice call?
  • Why does the time-to-first-audio matter more?
  • An example with two voice agents
  • How does streaming bring the first audio early?
  • What decides the time-to-first-audio?
  • The rest of the reply must keep up
  • When does the total response time still matter?

Let's get started: Why does the time-to-first-audio matter more than total response time?


How does sentence chunking feed LLM output into TTS faster?

In this blog, we will learn about how sentence chunking feeds LLM output into TTS faster. We will also see why waiting for the full reply makes a voice agent slow, how a buffer collects tokens until a sentence is complete, how the chunker finds where a sentence ends, how it handles tricky cases like "Dr." and "3.5", how the first chunk is made smaller to start speaking even sooner, and where it works well and where it fails.

We will cover the following:

  • The problem: the voice waits for the full reply
  • What is sentence chunking?
  • Why not send every token to the TTS?
  • How does the chunker work?
  • Finding the end of a sentence
  • The tricky cases
  • Making the first chunk smaller
  • A walkthrough with numbers
  • What happens when the user interrupts?
  • Where it works well and where it fails

Let's get started: How does sentence chunking feed LLM output into TTS faster?


How does a TTS model start speaking before the full sentence arrives?

In this blog, we will learn about how a TTS model starts speaking before the full sentence arrives. We will also see why waiting for the full sentence makes a voice assistant feel slow, why a TTS model wants to see the future words, how speaking sentence by sentence and word by word compare, how a small lookahead of a few words solves the problem, how newer models read text and speak at the same time, and where it works well and where it fails.

We will cover the following:

  • The problem: the text arrives slowly
  • Why does a TTS model want the full sentence?
  • Approach 1: Wait for the full reply
  • Approach 2: Speak sentence by sentence
  • Approach 3: Speak word by word
  • Approach 4: Speak with a small lookahead
  • Approach 5: Models that read and speak at the same time
  • A walkthrough with numbers
  • Where it works well and where it fails

Let's get started: How does a TTS model start speaking before the full sentence arrives?


How does speculative LLM generation start replying before the user finishes?

In this blog, we will learn about how speculative LLM generation starts replying before the user finishes. We will also see where the waiting time goes in a normal voice agent, how the agent makes an early guess from the words it has so far, what happens when the guess is right or wrong, how it can prepare the voice early too, what it costs, and when to use it.

We will cover the following:

  • The normal way a voice agent replies
  • Where the waiting time goes
  • What is speculative LLM generation?
  • How speculative LLM generation works step by step
  • What happens when the guess is right
  • What happens when the guess is wrong
  • How does the agent know that the words changed?
  • Preparing the voice early too
  • The cost of speculative LLM generation
  • When to use it and when not to

Let's get started: How does speculative LLM generation start replying before the user finishes?


How do filler phrases hide latency in a voice agent?

In this blog, we will learn about how filler phrases hide latency in a voice agent. We will also see why silence feels so bad on a voice call, what a filler phrase is, when the agent decides to say one, the different ways to produce it, how it fits into the conversation memory, what to do when the user interrupts it, and where it helps and where it hurts.

We will cover the following:

  • The problem: silence on a voice call
  • What is a filler phrase?
  • How a filler phrase hides latency
  • When should the agent say a filler phrase?
  • Ways to produce a filler phrase
  • Adding the filler phrase to the conversation memory
  • Handling an interruption during a filler phrase
  • Where filler phrases help and where they hurt

Let's get started: How do filler phrases hide latency in a voice agent?


How does a voice agent hide the delay of a slow tool call?

In this blog, we will learn about how a voice agent hides the delay of a slow tool call. We will also see why a slow tool call hurts so much on a voice call, how the agent speaks a short line right away, how it gives progress updates during a long wait, how it keeps talking while the tool runs in the background, how it starts the tool early to cut the delay itself, and what it says when the tool takes too long.

We will cover the following:

  • The problem: a slow tool call on a voice call
  • What is a tool call in a voice agent?
  • Technique 1: Speak a short line right away
  • Technique 2: Progress updates during a long wait
  • Technique 3: Keep talking while the tool runs in the background
  • Technique 4: Start the tool early
  • Technique 5: Make the tool itself faster
  • What if the tool takes too long?
  • Putting it all together in code

Let's get started: How does a voice agent hide the delay of a slow tool call?


What are audio frames and chunk sizes in a real-time voice pipeline?

In this blog, we will learn about audio frames and chunk sizes in a real-time voice pipeline. We will also see how sound becomes numbers, how those numbers are grouped into frames, how to calculate the size of a frame, how chunk size affects both delay and overhead, why every part of the pipeline wants a different size, and how to pick the right chunk size for our use case.

We will cover the following:

  • How sound becomes numbers
  • What is an audio frame?
  • How to calculate the size of a frame
  • What is a chunk size?
  • How chunk size affects delay
  • The cost of very small chunks
  • Different parts want different sizes
  • How to pick the right chunk size

Let's get started: What are audio frames and chunk sizes in a real-time voice pipeline?


Why do sample rates like 8 kHz, 16 kHz, and 24 kHz matter for voice agents?

In this blog, we will learn about why sample rates like 8 kHz, 16 kHz, and 24 kHz matter for voice agents. We will also see how a computer turns sound into numbers, what a sample rate really is, how much of our voice each sample rate keeps, where 8 kHz, 16 kHz, and 24 kHz are used inside a voice agent, what goes wrong when the sample rates do not match, and how to pick the right one for each part of the agent.

We will cover the following:

  • How Does a Computer Store Sound?
  • What is a Sample Rate?
  • How Much of Our Voice Does a Sample Rate Keep?
  • 8 kHz: The Phone Call Sample Rate
  • 16 kHz: The Speech Recognition Sample Rate
  • 24 kHz: The Voice Output Sample Rate
  • How Sample Rates Flow Through a Voice Agent
  • What Happens When Sample Rates Do Not Match?
  • Sample Rate and the Size of the Audio
  • How to Pick the Right Sample Rate

Let's get started: Why do sample rates like 8 kHz, 16 kHz, and 24 kHz matter for voice agents?


How do WebRTC and WebSocket differ for voice AI agents?

In this blog, we will learn about how WebRTC and WebSocket differ for voice AI agents. We will also see what a voice AI agent needs from the network, how WebSocket carries audio, how WebRTC carries audio, why a lost packet hurts one of them more than the other, what extra work WebRTC does for voice, and when to use which one.

We will cover the following:

  • What a voice AI agent needs from the network
  • How audio travels over the internet
  • What is WebSocket?
  • How a voice agent uses WebSocket
  • What is WebRTC?
  • How a voice agent uses WebRTC
  • What happens when a packet is lost
  • The extra work WebRTC does for voice
  • WebRTC vs WebSocket
  • When to use which one

Let's get started: How do WebRTC and WebSocket differ for voice AI agents?


How does the Opus codec compress voice for real-time calls?

In this blog, we will learn about how the Opus codec compresses voice for real-time calls. We will also see why raw voice is too big to send, what a codec does, how Opus cuts the voice into small frames, how its two engines SILK and CELT shrink the sound, how it handles silence and lost packets, and why it is used in WebRTC and voice AI agents.

We will cover the following:

  • Why raw voice is too big for a call
  • What is a codec?
  • What is the Opus codec?
  • Step 1: Cutting the voice into small frames
  • Step 2: Keeping only what our ears can hear
  • Step 3: SILK, the speech engine
  • Step 4: CELT, the music engine
  • Step 5: Hybrid mode, using both engines together
  • How Opus saves data during silence
  • How Opus handles lost packets
  • Why Opus is used for real-time calls and voice agents

Let's get started: How does the Opus codec compress voice for real-time calls?


How does a jitter buffer keep voice audio smooth over a bad network?

In this blog, we will learn about how a jitter buffer keeps voice audio smooth over a bad network. We will also see how voice travels over the internet in small packets, what jitter is and why it breaks the audio, how the jitter buffer fixes it step by step, how a fixed jitter buffer differs from an adaptive one, what happens when a packet arrives too late or never arrives, and where the jitter buffer sits in a voice AI agent.

We will cover the following:

  • How voice travels over the internet
  • What is jitter?
  • Why uneven arrival breaks the audio
  • What is a jitter buffer?
  • How a jitter buffer works step by step
  • The trade-off: smoothness vs delay
  • Fixed jitter buffer vs adaptive jitter buffer
  • What happens when a packet is late or lost
  • Where the jitter buffer sits in a voice AI agent

Let's get started: How does a jitter buffer keep voice audio smooth over a bad network?


How does packet loss concealment hide missing audio?

In this blog, we will learn about how packet loss concealment hides missing audio. We will also see what packet loss is, why a gap in audio sounds so bad, how the approaches evolved from playing silence to repeating the pitch of the voice, how codecs like Opus and neural networks make a better guess, where it fits in a voice AI agent, and where it works well and where it fails.

We will cover the following:

  • What is packet loss?
  • Why does a gap in audio sound so bad?
  • What is packet loss concealment?
  • Approach 1: Play silence
  • Approach 2: Repeat the last packet
  • Approach 3: Repeat one pitch cycle and fade out
  • Approach 4: Let the codec guess
  • Approach 5: Let a neural network guess
  • Where does packet loss concealment fit in a voice AI agent?
  • Where it works well and where it fails

Let's get started: How does packet loss concealment hide missing audio?


How does a voice AI agent answer a phone call?

In this blog, we will learn about how a voice AI agent answers a phone call. We will also see how a call travels from the caller's phone to our server, what the phone number and the telephony provider do, how the caller's voice reaches our agent as a stream of audio chunks, how the agent listens, thinks, and speaks back into the call, how it handles interruptions, and how the call ends or gets transferred to a human.

We will cover the following:

  • What is a voice AI agent on a phone call?
  • The journey of a phone call
  • Step 1: The caller dials a phone number
  • Step 2: The telephony provider receives the call
  • Step 3: Our server decides what to do with the call
  • Step 4: The caller's voice reaches our agent
  • Step 5: The agent listens, thinks, and speaks
  • Step 6: The agent's voice goes back into the call
  • Handling interruptions on a phone call
  • How the call ends

Let's get started: How does a voice AI agent answer a phone call?


How do SIP and RTP carry a phone call to a voice agent?

In this blog, we will learn about how SIP and RTP carry a phone call to a voice agent. We will also see how a normal phone call reaches the internet, how SIP rings the voice agent and agrees on the audio format, how RTP packs the voice into small packets, what happens inside the voice agent when those packets arrive, how the agent speaks back, and how the call finally ends.

We will cover the following:

  • The journey of a phone call to a voice agent
  • What is SIP?
  • What does a SIP call setup look like?
  • What is SDP?
  • What is RTP?
  • What is inside an RTP packet?
  • What the voice agent does with the RTP packets
  • How the voice agent speaks back
  • How the call ends
  • SIP vs RTP

Let's get started: How do SIP and RTP carry a phone call to a voice agent?


Why does phone audio make speech recognition worse?

In this blog, we will learn about why phone audio makes speech recognition worse. We will also see how our voice travels over a phone call, why the phone cuts off the high sounds of our voice, how compression and lost packets damage the audio further, how noise and echo add to the problem, why a model trained on clean audio struggles with phone audio, and what we can do to make speech recognition work better on phone calls.

We will cover the following:

  • What is Speech Recognition?
  • How Does Our Voice Travel Over a Phone Call?
  • Reason 1: The Phone Cuts Off the High Sounds
  • Reason 2: Compression Throws Away More Detail
  • Reason 3: Lost Packets Leave Holes in the Audio
  • Reason 4: Noise, Echo, and Small Microphones
  • Reason 5: The Model Learned From Different Audio
  • How All the Problems Stack Up
  • How to Make Speech Recognition Work Better on Phone Audio

Let's get started: Why does phone audio make speech recognition worse?


More blogs and videos coming soon!

License

   Copyright (C) 2026 Outcome School

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.