Jeremy Howard: fast.ai Deep Learning Courses and Research | Lex Fridman Podcast #35

A platform for machine learning competitions, where Jeremy Howard was formerly president and a top-ranking competitor.

An organization where doctors who completed fast.ai courses are now publishing papers and creating journal reading groups.

The institution that ran the DawnBench competition.

A suite of cloud computing services, highlighted by Jeremy Howard for offering a fantastic fast.ai and PyTorch-ready instance, and a $300 free credit.

Where Jeremy Howard holds an office for students to work on deep learning during his courses.

A healthcare provider mentioned as one of the sources from which users can download their medical data via the Doc.ai app.

Where Jeremy Howard had his first job, at McKinsey & Company.

The top computational linguistics conference where the ULMFiT paper was published.

A high-level programming language used by Jeremy Howard on his Commodore 64.

A modern tool described as a smaller subset of what Microsoft Access offered in terms of data management and application creation.

A modern, object-oriented programming language developed by Microsoft for the .NET framework, mentioned for comparison with Delphi.

Apple's declarative UI framework for all Apple platforms, mentioned in relation to Swift's potential.

NVIDIA's parallel computing platform and programming model for GPUs. Key deep learning components, like recurrent neural networks and sparse convolutional neural networks, often require rewriting in CUDA C for performance, posing a barrier to innovation.

A domain-specific language for image processing and computational photography, serving as a foundational research project for many modern compiler and tensor computation efforts like MLIR.

IBM's AI platform, mentioned as an example of a system that pushes the idea of needing more data and computation.

A large language model developed by OpenAI, following GPT, also exemplifying transfer learning in NLP.

A cloud computing platform used by fast.ai for renting machines with multiple GPUs.

Amazon's AI-powered virtual assistant, mentioned in the context of audio processing challenges and opportunities.

TensorFlow's API for building input pipelines, described by Jeremy Howard as a 'big mess' and incredibly inefficient.

A tool used by fast.ai students to scrape images and create custom datasets for fine-tuning models.

A research institute dedicated to making deep learning more accessible, founded by Jeremy Howard. It focuses on practical application and hands-on exploration of deep learning.

An object-oriented programming language and integrated development environment previously developed by Anders Hejlsberg. Jeremy Howard loved it for being a fast, compiled language as easy to use as Visual Basic.

A free, cross-platform, open source developer platform for building many different types of applications, including desktop, web, cloud, mobile, gaming, IoT, and AI.

A strongly typed programming language that builds on JavaScript, developed by Anders Hejlsberg.

An open-source Pascal compiler, used by the Lazarus project to recreate Delphi.

A compiler infrastructure project led by Chris Lattner, aimed at optimizing domain-specific languages for tensor computations and deep learning.

A collection of modular and reusable compiler and toolchain technologies, which Swift builds upon, offering potential for optimized deep learning.

A functional programming language, whose team, led by Don Syme, has made efforts to integrate database concepts directly into the type system.

A scripting language for macOS that Jeremy Howard used for inter-application communication.

An array-oriented programming language, originating from Ken Iverson's mathematical notation for 'notation as a tool for thought' in the late 1950s/early 1960s.

A transformer-based machine learning technique for natural language processing pre-training developed by Google, demonstrating transfer learning.

An open-source machine learning framework developed by Google. Fast.ai used it for a time but found it difficult for teaching and research due to its computational graph approach and later, the slowness of TF Eager and the inefficiency of TF Data.

A learning system and software that implements spaced repetition, developed by Piotr Wozniak.

A database program from Microsoft Office that Jeremy Howard loved for its ability to create user interfaces, tie data, and generate reports graphically using Visual Basic for Applications (VBA).

The programming language within Microsoft Access, which Jeremy Howard found powerful for creating applications.

An early Pascal compiler and IDE created by Anders Hejlsberg.

A high-level, class-based, object-oriented programming language, mentioned for comparison with Delphi.

A scripting language that Jeremy Howard found flexible and used extensively for his email company FastMail in the late 90s/early 2000s.

An open deep learning compiler stack for CPUs, GPUs, and specialized accelerators, mentioned as a project focused on optimizing tensor computations.

A commonly used deep residual network architecture for image classification, mentioned in the context of ImageNet accuracy thresholds.

Microsoft's spreadsheet program, mentioned in comparison to Access for its data management capabilities.

Apple's programming language for building apps across all Apple platforms. Jeremy Howard hopes it will achieve the level of ease and compilation speed that Delphi once offered, especially with Swift UI and cross-platform development.

A Free Pascal-based IDE used to recreate Delphi, making good progress.

A high-level, general-purpose programming language widely used in data science and machine learning, which Jeremy Howard uses pragmatically due to its data science libraries, despite finding it less elegant and 'unhackable' due to its slowness.

An email company founded by Jeremy Howard, with its entire backend written in Perl.

Apple's machine learning framework for integrating trained models into apps. Jeremy Howard considers its low-level libraries 'rubbish' and not sufficiently 'swifty'.

A project created by Jason Antic using fast.ai techniques to colorize old black-and-white movies and photos efficiently on a single GPU.

A computational photography feature in Google Pixel phones that produces high-quality images in near darkness.

A popular optimization algorithm for training deep learning models, mentioned in the context of hyperparameter tuning.

A family of large language models known for their pre-training and transfer learning capabilities in NLP.

An open-source machine learning library primarily developed by Facebook's AI Research lab. Jeremy Howard praises it for its interactive Pythonic approach, making it easier for research and teaching, though it requires more boilerplate for beginners.

An open-source web application that allows you to create and share documents containing live code, equations, visualizations, and narrative text. Jeremy Howard recommends it for data-driven research.

An early deep learning framework that fast.ai initially used, but found harder to teach and research with due to its computational graph definition.

A high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano. Fast.ai used it with Theano and TensorFlow.

A free and open-source flashcard program that uses spaced repetition, highly recommended by Jeremy Howard for learning, especially for languages.

An imperative execution environment for TensorFlow that evaluates operations immediately, making development and debugging easier. Jeremy Howard found it syntax-sugar but 10x slower than PyTorch.

Jeremy Howard points to advancements in compiler technology, particularly MLIR, as key to enabling easier GPU programming for deep learning.

Google's approach to automating machine learning model design and hyperparameter tuning, which Jeremy Howard views as inferior to truly understanding model mechanics.

A core concept in deep learning, referring to computational models inspired by biological neural networks, capable of learning from data.

A regularization technique for neural networks that prevents overfitting, cited as a breakthrough not requiring multiple GPUs.

A regularization technique used in neural networks to prevent overfitting by adding a penalty to the loss function that is proportional to the magnitude of the weights.

A technique used for training deep learning models, particularly important for transfer learning, where different layers of the model are trained at different learning rates.

A system for automatically generating high-performance GPU code for tensor operations.

The use of computational techniques to improve digital photography, an area Jeremy Howard noted as underdeveloped but now standard in phones.

A machine learning technique where the algorithm interactively queries a user to label new data points. Jeremy Howard advocates for it as a powerful, under-researched area.

An algorithm introduced by Jeremy Howard and Sebastian Ruder, demonstrating successful transfer learning in NLP for the first time and smashing state-of-the-art results.

A class of AI algorithms used in generative modeling. Jeremy Howard mentions fast.ai has achieved GAN-level outcomes without needing GANs and with transfer learning on a single GPU.

Mentioned as an area with similar regulatory and data challenges to AI in medicine.

Discussed as an application of AI where individual user data is valuable, but Jeremy Howard argues that many approaches to data collection for them are unnecessarily invasive.

A critical technique highlighted by Jeremy Howard for deep learning that allows achieving state-of-the-art results with orders of magnitude less data; he criticizes its under-study in academia.

A technique for improving the training of deep neural networks, cited as a major breakthrough that didn't require multiple GPUs.

A technique discovered by Leslie Smith that allows certain neural networks to be trained 10 times faster with a 10 times higher learning rate and better accuracy, but was initially rejected by academic publishers.

A foundational textbook on deep learning, often read by fast.ai students.

Founder of fast.ai, distinguished research scientist at the University of San Francisco, former president of Kaggle, and a successful entrepreneur and educator in the AI community.

Creator of Turbo Pascal and Delphi, who later went on to create .NET and TypeScript.

Google's lead of AI, mentioned for stating Google's goal to require 'a thousand times more computation but less people' for AI, which Jeremy Howard disagrees with.

A psychologist who created the concept of spaced repetition by studying his own memory of random letter sequences.

The creator of the Perl programming language, whose reduced involvement led to Perl's decline.

A friend of Jeremy Howard who, before joining Google, spent a year training ImageNet in 10 days.

Leader of the F# team, recognized for integrating database concepts into the type system.

The creator of APL, who developed a new type of mathematical notation that became a programming language.

Jeremy Howard's colleague who co-authored the ULMFiT paper, introducing transfer learning to NLP.

A student of fast.ai who created DeOldify, using transfer learning to colorize black-and-white movies on a single GPU.

The developer of Swift, advocating for its 'infinitely hackable' nature, and currently working at Google on Swift for TensorFlow.

A researcher who has recently started using spaced repetition for concepts and ideas in papers, and written about it.

A researcher focused on practical neural network training, who discovered 'super-convergence', but struggled to publish his findings in academia.

The naturalist who traveled and observed different species, used as an analogy for how a data scientist should engage with data.

A political figure running on a platform addressing labor force displacement due to AI.

The developer of SuperMemo, a software and theory for spaced repetition who has built his life around its principles to become a 'Renaissance man'.

Currently the dominant hardware for deep learning, though Jeremy Howard notes they are overpriced due to lack of competition and better software programmability compared to alternatives like TPUs.

An 8-bit home computer on which Jeremy Howard wrote his first program in BASIC to analyze musical scales.

Google's custom-designed ASICs for accelerating machine learning workloads. Jeremy Howard criticizes them for being almost entirely unprogrammable due to Google's desire to protect IP.

A social media platform mentioned in the context of data privacy concerns.

A major AI player whose approach to deep learning Jeremy Howard criticizes for requiring excessive data and computation rather than optimizing existing resources, and also for its TPU strategy.

A social media platform mentioned in the context of data privacy concerns.

A competitor in the GPU market, which Jeremy Howard hopes will improve its software act to provide competition to NVIDIA.

Jeremy Howard's previous startup, the first company to focus on deep learning for medicine, founded in 2014.

A startup helped by Jeremy Howard, which provides an app for users to download and control sharing of their medical data from various providers.

A global management consulting firm where Jeremy Howard spent 10 years, gaining experience with domain experts.

A startup focused on active learning, co-founded by Jeremy Howard.

A company developing Intelligence Processing Units (IPUs) for machine learning, mentioned as an alternative tensor computation device.

A major technology company that also competed in DawnBench against fast.ai, with fast.ai surprisingly outperforming them.

A video-sharing platform mentioned in the context of data privacy concerns.

A cloud computing platform offering virtual machines for various computing tasks, including deep learning, similar to Salamander in ease of use for launching Jupyter notebooks.

A major AI player whose approach to deep learning Jeremy Howard criticizes for requiring excessive data and computation.

Streaming service mentioned for its past and present approaches to user data collection for recommendations.

A pharmacy chain mentioned as one of the sources from which users can download their medical data via the Doc.ai app.

A platform (owned by fast.ai's creators) that allows users to launch a Jupyter Notebook with fast.ai courses pre-installed with a single click, making GPU access easier.

Mentioned as a market with a large population and small numbers of doctors where AI in medicine could have a significant impact.

Specified as the only country in Africa with a notable number of pediatric radiologists.

A continent with very few pediatric radiologists, highlighting the need for AI-driven diagnostic solutions.

A location whose national bird, hummingbirds, was the subject of a classification project by a fast.ai student.

Mentioned as a market with a large population and small numbers of doctors where AI in medicine could have a significant impact.

Mentioned as a market with a large population and small numbers of doctors where AI in medicine could have a significant impact.

Referencing the academic hub where Kinane has students who are both data science and medical experts.

US legislation meant for data portability, which lawyers often misinterpret as primarily restricting data sharing, thus hindering medical AI progress.