Do captions influence what people take away from charts?

Yes, captions can influence takeaways, especially when they highlight features that are not the most visually salient. If a caption aligns with a prominent visual feature, people tend to focus on that feature, but if it highlights a less prominent one, the caption can guide attention.

What are the implications of text vs. visualization preference for accessibility?

Research suggests that preferences differ between sighted individuals and those who are blind or visually impaired (BLV). Sighted individuals may prefer higher-level descriptions, while BLV individuals might favor lower-level details, highlighting the need for coequal consideration of natural language and visualizations in design.

Does more text on a chart lead to better user preference?

Studies indicate that 'more text is better,' challenging minimalist design principles. Participants generally preferred charts with more annotations and text, suggesting that relevant textual information enhances understanding and preference.

How does text affect viewer predictions on visualizations?

Viewer predictions based on visualizations appear largely unaffected by accompanying text annotations. While text can signal potential author bias, it doesn't seem to sway the visual perception or the prediction made from the visual data itself.

What do cognitive models suggest about text and visualization interaction?

Cognitive models offer competing explanations. Dual coding theory suggests independent processing that enhances memory, while cognitive load theory posits that simultaneous input can overload working memory. The interaction is not definitively understood, with evidence supporting both synergy and competition depending on the task and design.

How do multimodal LLMs process text and visuals?

Multimodal LLMs typically use architectures like unified embedding or cross-modality embedding. These models process visual and textual inputs, often using attention mechanisms to integrate information and learn relationships between the modalities, aiming to understand and generate responses based on both.

Can icons or visualizations within text paragraphs improve reading?

While research is ongoing, initial findings suggest that inserting non-alphanumeric visuals like icons into paragraphs may impede fluid reading by distracting the eye and interrupting the reading flow. Inline charts between paragraphs are suggested as a potentially better alternative.

Key Moments

Stanford CS547 HCI Seminar | Spring 2026 | Show It or Tell It? Text, Visualization, and Combination

Stanford Online

Education5 min read57 min video

Jul 31, 2026|403 views|15

Stanford Stanford Online LLM Human-AI Interaction

Save to Pod

Want to know something specific about what's covered?

We've already dissected every moment. Ask and we will deliver (with timestamps).

Key Moments

On this page

TL;DR

More text on visualizations is preferred, even over minimalism, but the impact of text on predictions is minimal, while its potential for signaling bias is significant.

Key Insights

Titles on charts receive the longest fixations and are most likely to be recalled, indicating the crucial role of text in information visualization.

For blind and low-vision individuals, higher-level text descriptions of charts are opposed, while lower-level descriptions are favored, contrasting with sighted readers' preferences.

Contrary to minimalist design principles, studies show participants significantly prefer visualizations with more accompanying text, challenging ideas like Tufte's minimalism.

Viewer predictions based on ambiguous visualizations are largely unaffected by text annotations, suggesting visual perception can dominate textual influence in such contexts.

Text annotations on visualizations provide a strong signal of potential author bias, even if they don't influence predictions.

Modern multimodal language models (MLMs) process text and visuals through architectures like unified embedding or cross-modality attention, mirroring some aspects of human cognitive processing but with unknown implications for human comprehension.

Text is a critical, yet historically underestimated, component of visualization

Information visualization aims to provide insight into data through systematic spatial representations. However, the role of language alongside visualizations has been largely overlooked by the visualization community, as evidenced by early design guidelines that omitted text elements. Research, such as the 'Beyond Memorability' study, has shown that titles and labels receive significant attention during chart encoding and are crucial for recall, directly contradicting the notion that text is secondary to visuals. This underscores that language is not merely supplementary but a fundamental part of how users understand and remember information presented in visualizations.

Balancing text and visuals: attention, salience, and takeaway

Deciding what information to convey through text versus visualization is complex. Studies indicate that when text captions align with the most visually salient parts of a chart, users focus on those prominent features. However, when captions highlight less salient areas, the text can override visual salience, influencing user takeaways more strongly than the visual elements themselves. Conversely, if the text is completely unaligned with visual features, the visual elements regain dominance. This dynamic interplay suggests a sophisticated interaction where text and visuals compete and cooperate to shape user understanding, and their relative influence depends on alignment and salience.

Accessibility needs reveal differing preferences for text augmentation

Research on the accessibility of visualizations for blind and low-vision (BLV) individuals has highlighted divergent preferences for text augmentation compared to sighted readers. BLV participants often oppose high-level text descriptions that provide external context, favoring instead lower-level descriptions that detail specific data points. Sighted readers, conversely, tend to prefer high-level interpretations. This finding is significant because it suggests that what might seem helpful for sighted users could be detrimental or irrelevant for users with different sensory needs, emphasizing the importance of considering diverse user groups in design. The conclusion drawn from this work is that natural language should be considered coequal with visualizations in design.

More text is generally preferred, challenging minimalist design

Experiments involving varying amounts of text on charts have revealed a surprising preference for more text, rather than less. Contrary to minimalist design principles, such as those advocated by Edward Tufte, participants consistently favored visualizations that included titles, annotations, and subtitles over those with minimal text. While a small minority expressed a preference for no text or very little text, the majority indicated that 'more text is better.' This challenges the prevailing wisdom of minimizing visual elements and suggests that ample, relevant textual information can significantly enhance user engagement and comprehension.

Text annotations don't sway predictions but signal author bias

Interestingly, when presented with ambiguous visualizations designed for prediction tasks, text annotations had a minimal impact on viewers' predictions. For example, adding text that described a trend did not significantly change whether participants believed a blue or green data line would be higher in the future. However, these same text annotations strongly signaled potential author bias. This suggests that while text may not alter objective interpretations of data in certain contexts, it plays a crucial role in shaping perceptions of credibility and intent.

Cognitive models offer competing explanations for text-visual interaction

Understanding the cognitive underpinnings of text-visualization interaction involves examining theories like Dual Coding Theory and Cognitive Load Theory. Dual Coding Theory posits that separate visual and verbal systems process information independently and simultaneously, enhancing memory and understanding. In contrast, Cognitive Load Theory suggests that limited working memory capacity can be overloaded by presenting information in multiple modalities, negatively impacting comprehension. The effectiveness of combining text and visuals may depend on task complexity, individual abilities, and design choices, with research still being definitive.

System one vs. system two thinking and visualization interpretation

Applying dual process theory, particularly Kahneman and Tversky's System 1 (automatic, fast) and System 2 (deliberate, effortful) thinking, provides a framework for understanding visualization interpretation. System 1 processing relies on quick, visually-based decisions, which might explain why text annotations don't influence predictions in ambiguous visual scenarios. System 2, requiring more cognitive effort and top-down attention, is necessary for accurate, calculated interpretations. This framework suggests that visualizations might predominantly trigger System 1, making them less susceptible to textual influence unless System 2 is explicitly engaged, a process that may require specific design or user intent.

Multimodal language models process text and visuals through complex architectures

Modern multimodal language models (MLMs) integrate text and visual information using advanced architectures like transformers. These models employ strategies such as unified embedding, where visual and text embeddings are concatenated, or cross-modality attention, which allows information flow between different modalities. Architectures like Flamingo use masked cross-attention to selectively link text to relevant image segments. Research into the internal workings of these models, like the Lava models, suggests that early layers process general visual information, middle layers handle semantic processing relevant to the task, and top layers consolidate the answer. While these MLMs show remarkable capabilities in tasks involving both text and visuals, their complex processing mechanisms and their relationship to human cognitive processing remain areas of active research and speculation.

Mentioned in This Episode

●Software & Apps

●Companies

●Organizations

●Books

●Studies Cited

●Concepts

●People Referenced

Common Questions

The main challenge is understanding how to optimally balance and integrate text and visualizations for a given message, as the optimal approach is complex and context-dependent. The interaction between language and visual elements is not yet fully understood.

Topics

Mindset & Self-Improvement Technology & Innovation Science & Mathematics Human-computer Interaction Reading Comprehension Multimodal LLMs Information Visualization Text Visualization Cognitive Models Data Presentation

Mentioned in this video

People

Terry Winograd

Mentioned as the long-standing figure associated with the seminar series, indicating a legacy of HCI research.

Marie Agrawala

Co-author on work related to chart attention and caption influence.

Maryanne Wolf

Author of 'Proust and the Squid', an expert on reading who discusses the acquired nature of reading and the role of parafoveal processing.

Richard Mayer

A prominent researcher in multimedia learning, known for the dual coding theory and dual channel model, which suggests separate processing for visual and verbal information.

Kahneman and Tversky

Known for developing the dual process theory, distinguishing between System 1 (automatic) and System 2 (effortful) thinking.

Companies

Apple

Released design guidelines for web pages, which initially overlooked text in charts.

Books

Beyond Memorability

A paper that found titles received the longest fixations and were most likely to be recalled during chart encoding.

Proust and the Squid

A book recommended for understanding the basics of reading acquisition and its cognitive underpinnings.

Concepts

Tufte's minimalism

A design principle emphasizing simplicity and minimizing non-essential elements, which the speaker's findings challenge.

Organizations

Tableau Research

Collaborated on a tool for asking questions of data and displaying visualizations as answers, leading to studies on text-viz combinations.

New York Times

Mentioned for their 'You Draw It' visualization feature, which explored how people recall data from text and visualizations.

Studies & Research

Ottley et al.

A study comparing Bayesian reasoning outcomes with text-only versus visualization-only conditions, finding no significant accuracy difference but noting participants often focused on one modality.

Software & Apps

Flamingo model

A multimodal language model architecture used as an example to explain how visual and text inputs are processed and integrated.

Lava models

Multimodal large language models whose internal workings, regarding the role of text and visuals across layers, were schematically presented.

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free