What are common mistakes L&D teams make when measuring learning programs?

L&D teams often measure the wrong things, failing to align metrics with the training's actual purpose. They also tend to over-rely on subjective surveys instead of objective assessments, and sometimes fall into the trap of focusing on likability over measurable skill improvement.

Should anonymous surveys be used for measuring learning effectiveness?

No, Eric Grant advises against anonymous surveys. While they might seem to encourage honesty, anonymity prevents the crucial ability to filter, sort, and categorize results, which is vital for understanding why different groups may have varying responses.

What are the main barriers to L&D teams adopting data-driven approaches?

A primary barrier is the desire to be a 'likable' team, positioning learning as a perk rather than a strategic driver of job performance. This focus on engagement and retention can overshadow the objective measurement of skill development and business impact.

How can L&D measure soft skills and company culture effectively?

Measuring soft skills and culture is difficult and often requires looking at how AI can objectively assess communication or other nuanced behaviors in the future. For now, focusing on measurable outcomes and clear communication of value, even if not perfectly quantifiable, is key.

Can qualitative data, like confidence levels, change the interpretation of quantitative metrics?

Yes, qualitative data like confidence, when combined with assessments, provides a richer picture. For example, knowing if someone is 'confidently incorrect' highlights a critical learning need that simple right/wrong answers might miss.

How can L&D partner with data teams or HR to get the data they need?

L&D should actively partner with stakeholders and data teams, clearly articulating their needs and understanding limitations. Creative approaches, like anonymized data sharing for aggregate analysis, can help overcome data access barriers.

What are the best tools or resources for L&D leaders wanting to be more data-driven?

Practice is more important than specific tools. Use AI to generate practice datasets and visualize them in tools like Google Sheets or Excel. Focus on building data fluency through reps rather than just acquiring new software.

The learning metrics that matter: A data masterclass with Figma's Eric Grant

Big Think

Education5 min read48 min video

Nov 7, 2024|2,696 views|57

Big Think BigThink Big Think+Big Think Plus business training learning and development L&D coaching

Save to Pod

Key Moments

On this page

TL;DR

Data fluency in L&D is about practice, not just tools. Focus on actionable insights for better learning and leadership.

Key Insights

Data fluency in L&D is crucial for advocating resources, driving program success, and career advancement.

The hardest problems in L&D, like effectively measuring learning impact, offer the greatest potential for growth.

Measuring learning requires understanding the 'three rights': the right learning, for the right person, at the right time.

Avoid measuring the wrong things; focus assessments on knowledge retention and application, not just subjective feelings.

Shift from likability-focused metrics (NPS, survey scores) to objective measures of skill improvement and job performance.

AI is rapidly evolving to help quantify previously hard-to-measure skills like communication and to personalize learning.

THE EVOLUTION OF L&D AND DATA FLUENCY

Eric Grant's journey into Learning and Development (L&D) began serendipitously, first at Epic Systems and later at a fast-growing startup. His path to data analytics was spurred by observations at Uber, where successful colleagues effectively used data to advocate for resources, secure tools, and achieve promotions. This highlighted a personal gap: a lack of data to support learning initiatives, which often hindered advocacy for investment in time, money, and resources. To bridge this, he pursued a Master's in Data Science, believing that tackling complex problems like embedding data into L&D yields significant dividends for both individuals and organizations.

FIGMA'S LEARNING CULTURE AND DATA ADVANTAGES

At Figma, Eric leads the learning program for the support function, focusing on equipping specialists with the knowledge needed to address customer inquiries about the tool, billing, and account management. A unique challenge at Figma is that customers often possess deeper knowledge of the product than support specialists, who are focused on day-to-day customer interactions. This requires targeted training to build confidence and expertise. The support environment offers a significant data advantage, with direct performance data like ticket resolution speed, customer satisfaction, and reopen rates providing near-direct attribution for the impact of learning programs.

THE FUNDAMENTAL IMPORTANCE OF DATA IN LEARNING MEASUREMENT

Data is critical for L&D because it helps hone in on the 'three rights': the right learning, for the right person, at the right time. Each of these variables involves complex contexts and nuances that data helps clarify. Grant uses a fundamental equation: Is it more valuable for a person to spend the next 30 minutes learning or working? Data is the tool to objectively answer this, proving the value of learning over immediate task completion. This involves understanding the value proposition of training versus the value derived from day-to-day work, a complex equation best solved through data analysis.

COMMON PITFALLS IN MEASURING LEARNING PROGRAMS

L&D teams often err by measuring the wrong things or over-relying on subjective measures like surveys. For instance, tying sales training solely to revenue increases might be a false equivalency if the training focuses on methodology adoption. Instead, measure the understanding and application of that methodology. Surveys, while easy to administer, provide subjective sentiment rather than objective learning. Grant advocates for assessments over surveys, utilizing questions that gauge knowledge, capability, and confidence, offering more robust insights into learning effectiveness. He emphasizes that anonymity in surveys is often detrimental, sacrificing valuable categorizable data for perceived honesty.

OVERCOMING ORGANIZATIONAL BARRIERS TO DATA-DRIVEN L&D

A significant barrier for L&D teams is the desire to be perceived as a likable, beneficial function, often positioning learning as a perk for employee engagement and retention. While positive, this can impede objective measurement. Historically, research linked good learning programs to better engagement, leading to increased investment. However, this framing can shift focus from developing job skills to simply making employees happier or more likely to stay. Grant argues that the true value lies in making individuals better at their jobs. This requires a strategic shift from a perk-based approach to a change-agent mindset, focusing on measurable skill enhancement.

KEY PERFORMANCE INDICATORS AND THE FUTURE OF MEASUREMENT

Key indicators for performance measurement include change and application of learned skills. Confidence is highlighted as an underrated yet crucial metric, providing insight into an individual's willingness to apply new knowledge. Looking ahead, advancements in AI and communication platforms (like Slack and Teams) will offer quantifiable data on employee interactions, response times, and communication styles. This will enable L&D to define and measure performance more granularly. Companies may leverage AI to set specific communication standards, turning quantitative data into subjective performance metrics, fundamentally changing how learning's impact is assessed.

MEASURING INTANGIBLE SKILLS AND FOSTERING CULTURE

Measuring soft skills and company culture remains challenging. While concepts like communication can be conceptually understood, AI's future ability to objectively assess communication effectiveness is anticipated. For culture, engagement surveys provide some insight, but translating stated learning desires (e.g., for AI or data skills) into actual program utilization is difficult. The core issue is demonstrating the value of learning time over work time. Grant suggests not every aspect needs quantification; sometimes, leadership understanding of intent, like fostering a 'playful' culture at Figma, is sufficient when direct measurement is impractical.

THE ROLE OF PARTNERSHIP AND CREATIVE DATA SOLUTIONS

Partnership is vital for successful learning measurement. Collaborating with stakeholders helps translate qualitative requests into measurable outcomes. For data acquisition, L&D professionals should actively ask for data, even if it requires creative solutions. This might involve anonymizing data sets with HR or IT departments to identify aggregate trends, such as correlations between training participation and promotions. The goal is to foster a collaborative approach where data teams assist L&D in understanding and leveraging available information to demonstrate learning's impact, overcoming limitations through ingenuity.

SUCCESSFUL INITIATIVES AND PRACTICAL TOOLS

A significant win involved improving First Contact Resolution (FCR) at Coinbase by 5-6%. This initiative, framed as a key metric for agent success and customer satisfaction, involved a workshop focused on psychology and effective customer education. By emphasizing FCR's importance and demonstrating its significant cost savings (saving the company ~$800,000), the program showed a 20x ROI. For tools, Grant recommends practicing data visualization with readily available platforms like Google Sheets or Excel, using AI for generating practice datasets. True data fluency comes from practice and repetition, not just tools.

AI'S EMERGING ROLE IN LEARNING AND DEVELOPMENT

AI is revolutionizing L&D, moving beyond generic tools to specialized applications. Platforms like SAA and Articulate are enabling easier course creation from existing content. More significantly, adaptive learning technologies, similar to Duolingo's model, are emerging. These platforms personalize learning paths based on individual performance, showing content when needed and reinforcing it over time. AI is also facilitating data analysis, personalization, and the creation of customized learning pathways, promising a future where learning is more responsive, effective, and tailored to individual needs.

Mentioned in This Episode

●Software & Apps

●Companies

●Organizations

●Concepts

Key Performance Indicators for Learning Programs

Data extracted from this episode

Indicator	Description	Measurement Challenge
Change	Observing if individuals are doing anything differently after training.	Requires tracking and analysis of new behaviors.
Application	Seeing learned knowledge translate into action.	Can happen immediately or over time; measurement is challenging.
Confidence	Self-reported level of assurance in performing a task or applying knowledge.	Subjective, but can be a powerful signal for performance and culture.

Common Questions

Data is essential for honing in on the 'right' learning, for the 'right' person, at the 'right' time. It helps target the most effective interventions and provides a way to measure the value and impact of learning versus the value of work.