Key Moments
Workshop: Building and optimizing dictation features
AssemblyAIWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
AssemblyAI's Sync API enables dictation under one second by combining audio processing with LLM cleanup, but the real magic is warming the connection while recording.
Key Insights
The Sync API is crucial for dictation as it offers a sub-second turnaround for short audio bursts (typically 10-12 seconds), unlike Async (seconds) or Streaming (WebSocket overhead).
Warming the API connection by making a `client.sync.warm` call while recording significantly reduces latency by handling DNS, TCP, and TLS handshakes upfront.
LLM Gateway, specifically the Quinn 3.54B fast model, is recommended for a cleanup step to remove conversational noise, false starts, and rephrasing, achieving a total pipeline time under one second.
The new Dictation API, launching soon, will simplify this pipeline by offering a single `client.dictation.describe` call, providing both verbatim and cleaned text, and supporting 19 languages.
For custom cleanup, users can leverage LLM Gateway or integrate with external providers or locally hosted LLMs, with OpenAI schema compatibility facilitating easy plugging and playing.
The Sync API has a 120-second (2-minute) maximum duration and an 80-millisecond floor for transcription, while LLM context limits depend on the specific model used (e.g., Quinn).
The limitations of existing transcription APIs for dictation
Historically, AssemblyAI offered two primary transcription endpoints: Async and Streaming. Async is designed for pre-recorded audio files, requiring users to upload files and poll for results, which introduces a several-second delay. Streaming, while real-time, necessitates establishing and maintaining a WebSocket connection for each session. This overhead is particularly problematic for dictation use cases where audio bursts are typically short, averaging around 10-12 seconds. Constantly setting up and tearing down WebSockets, even if each connection is only a few hundred milliseconds, accumulates into significant latency for frequent, short dictation inputs. The need for a faster, lower-overhead solution that bridges the gap between Async and Streaming led to the development of the Sync API.
Introducing the Sync API for rapid dictation
The Sync API is engineered to provide near real-time transcription for shorter audio inputs, aiming for a turnaround time under one second. Unlike Async, it processes audio in a single POST request, delivering the transcript directly. Compared to Streaming, it avoids the overhead of persistent WebSocket connections, making it ideal for discrete dictation events. The core principle of the Sync API is its speed for smaller files. While it can handle longer audio, its primary advantage lies in its efficiency for the short, frequent bursts characteristic of dictation. This makes it a better fit than Async, which has a built-in latency floor, and Streaming, which has connection setup costs that are disproportionate for brief utterances.
Optimizing for speed: The power of connection warming
To further reduce latency and achieve sub-second dictation, a crucial technique is 'warming' the API connection. This involves making an initial, often throwaway, `client.sync.warm` call to the API *while* the user is still recording. This preemptive step handles the necessary network handshakes, such as DNS resolution, TCP connection establishment, and TLS encryption. By paying this networking debt upfront, the subsequent actual transcription request experiences significantly reduced latency because the connection is already established and ready. The first transcription request, without warming, can take hundreds of milliseconds due to these initial network overheads. However, after warming, requests can be processed much faster, often falling into the 100-200 millisecond range, enabling a total dictation-to-screen time of under a second.
Ensuring accuracy with key terms prompting
Basic transcription can sometimes struggle with specific terminology, names, or jargon. To address this, the AssemblyAI API supports 'key terms prompting,' also referred to as vocabulary. By providing an explicit list of terms that the model should recognize, users can significantly improve accuracy. For example, if a name is consistently misspelled, adding the correct spelling to the key terms list ensures it's transcribed accurately. This feature is particularly useful for proper nouns, technical terms, or domain-specific language, ensuring that the initial transcription is as close to the desired output as possible before any cleanup steps are applied.
Refining output with LLM Gateway for cleanup
Dictation often includes filler words, false starts, repetitions, and conversational nuances that are not ideal for final text output. To clean this up, AssemblyAI leverages its LLM Gateway, which provides access to various large language models. For dictation cleanup, the recommended model is Quinn 3.54B fast, chosen for its speed and efficiency on short utterances. The cleanup process involves passing the transcribed text to the LLM with a specific prompt. This prompt instructs the model to maintain the original intent of the message while removing conversational elements, ensuring a clean, coherent final output. By using prompt engineering, including few-shot examples and defining what *not* to change, the model can effectively transform raw dictation into polished text. This cleanup step, when combined with a warmed connection, can result in the entire dictation pipeline delivering text to the screen in under a second.
The upcoming Dictation API for out-of-the-box fluency
AssemblyAI is launching a dedicated Dictation API to further simplify the process. This new API aims to provide an out-of-the-box solution for dictation, handling the complexities of fluency, false starts, and cleanup with a single API call. Users will be able to use a `client.dictation.describe` function, passing in audio and optional vocabulary. The API will return both the verbatim transcription and the LLM-processed, cleaned text. This allows users to see how their prompts perform and make adjustments. The Dictation API will also support 19 languages and will include a default prompt for basic cleanup if no custom instruction is provided, making it incredibly easy to implement high-quality dictation features.
Flexibility in LLM integration and language support
While AssemblyAI recommends its LLM Gateway and specific models like Quinn 3.54B for cleanup, users have flexibility. The LLM Gateway utilizes an OpenAI schema, allowing integration with external LLM providers or locally hosted models. This means users can opt for different providers or self-host models if they have specific requirements or existing infrastructure. Regarding language support, the underlying 3.5 Pro model family used for Sync and other services is unified and performs exceptionally well across 19 different languages. This means that the dictation feature can handle code-switching within a single utterance, seamlessly transcribing English, Spanish, or other supported languages as they are spoken. Benchmarks for language performance are available and can be shared upon request.
Key considerations and limitations
When implementing dictation, several factors influence performance. On the server-side, shorter clips and removing silence generally lead to faster transcription. On the client-side, warming the connection, staying in the same geographic region as the API servers, and using smaller, faster LLM models for cleanup are crucial for minimizing latency. The Sync API itself has a maximum duration of 120 seconds (2 minutes) and an 80-millisecond floor. For LLM context limits, these depend on the specific model used via the LLM Gateway. For example, if using Quinn, its context window will dictate how much text can be processed for cleanup at once. While cleanup on longer dictations might push the total time closer to or over a second, for typical short dictation bursts, the goal of achieving a response under one second is consistently attainable with this optimized pipeline.
Mentioned in This Episode
●Software & Apps
●Companies
Building and Optimizing Dictation Features
Practical takeaways from this episode
Do This
Avoid This
Dictation Pipeline Latency Breakdown (Estimated)
Data extracted from this episode
| Component | Estimated Latency (Milliseconds) | Notes |
|---|---|---|
| Assembly AI Inference (Sync, 12s+ file) | ~140 | Base processing time for transcription. |
| Transport (Network Upload/Download) | ~100 | Varies by user location relative to servers. |
| Connection Warm-up (DNS, TCP, TLS Handshake) | Paid upfront during recording (network overhead) | Avoids cold request latency. |
| LLM Gateway Cleanup (Quinn 3.5 4B) | <600 | Highly dependent on prompt complexity and audio length; under 1 second for typical short dictations. |
| Total (Typical Short Dictation) | <1000 | Goal for a seamless user experience. |
| Total (Longer dictations with complex cleanup) | Can exceed 1 second | Latency increases with monologue length and processing needs. |
Sync API vs. Async API vs. Streaming API for Dictation Use Cases
Data extracted from this episode
| Feature | Sync API | Async API | Streaming (WebSocket) |
|---|---|---|---|
| Primary Use Case | Short, quick dictations (voicemails, IVR, quick notes) | Pre-recorded audio files, batch processing | Real-time captions, voice agents, continuous audio streams |
| Latency | Low (ideal for immediate feedback) | High (not suitable for real-time dictation) | Low (but with WebSocket overhead) |
| Overhead | Minimal (single POST request) | Moderate (upload, polling/webhooks) | High (maintaining WebSocket connection) |
| Setup Complexity | Simple | Moderate | High |
| Suitability for Dictation Bursts (<15s) | Excellent | Poor | Moderate (if not for connection overhead) |
Common Questions
The Sync API is designed for quick, small transcription requests, making it ideal for dictation. Unlike Async which has higher latency for immediate results, or Streaming which has overhead from maintaining WebSockets for short bursts, Sync offers a balance of speed and low overhead by using a single POST request for fast transcription.
Topics
Mentioned in this video
A specific model used for sync transcription, noted for its speed and capabilities across multiple languages.
A suite of models offered by Assembly AI that provides LLMs based on the OpenAI schema, used for tasks like prompt-based cleanup.
A specific LLM model hosted by Assembly AI, known for its speed and efficiency, particularly for cleanup tasks.
A recommended model for dictation tasks due to its speed and cost-efficiency, suitable for short utterances.
An alternative model that can be experimented with for dictation, offering speed comparable to other leading models.
A more intelligent model from Quinn that can be used for dictation, though it may come with increased latency.
An upcoming API from Assembly AI designed to simplify the process of building dictation features out-of-the-box.
An LLM model that has performed well in streaming applications, also considered for translation tasks.
More from AssemblyAI
View all 54 summaries
47 minBuild Smarter Voice Agents
60 minBuild a Voice Agent in an Hour with Claude Code | AssemblyAI Workshop
56 minVoice AI: Beyond Transcription with Granola, CoLoop & EdgeTier
53 minYour Ground Truth Is Wrong: Evaluating STT with truth files & semantic WER | AssemblyAI Workshop
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free