The 10,000x Yolo Researcher Metagame — with Yi Tay of Reka

Yi Tay was the architecture co-lead on Palm 2 at Google Brain, a significant company-wide effort in large language model development.

Yi Tay was a core contributor to Flan at Google Brain, a project mainly led by Hieu Pham and Sharan Narang focusing on instruction tuning.

Mentioned as an example of models that researchers were fine-tuning in late 2019, before the focus shifted entirely to foundational models.

Models released by Reka AI, which achieved state-of-the-art results even against larger models from bigger labs, showcasing the team's ability to achieve high performance with efficient cycles.

Yi Tay contributed to Palm 1 and was a co-lead for the modeling workstream of Palm 2.

Hypothetical example of a Transformer alternative that may show good performance at small scales but whose implications for larger models are unknown.

Yi Tay is the inventor of UL2, which he started as a personal project during a break and became the largest encoder-decoder model released by Google at the time.

The launch of ChatGPT in November 2022 drastically changed the AI research landscape, making previous task-specific work largely obsolete and accelerating the focus on general-purpose models.

Used by Reka AI for some orchestration tasks, but Yi Tay notes that generalized orchestration tools for ML experimentation are still lacking in open source.

GPT-3's release marked the emergence of few-shot and in-context learning, changing the paradigm of large language model research.

A series of small language models from Microsoft Research that aims to achieve strong performance with significantly less data, though Yi Tay questions whether they can truly 'cheat' scaling laws.

A model whose architecture is speculated to have pluggable experts, particularly for vision, suggesting a modular approach to multimodal capabilities.

A pre-print server where Yi Tay used to frequently browse new papers at 9:30 AM Singapore time to stay updated on research.

An early general-purpose model from Google Brain, mentioned by Yi Tay as an example of the shift towards universal foundation models, even before the general public caught up.

A coding benchmark noted as being saturated and contaminated, similar to GSM8K, making it less effective for truly evaluating new model capabilities.

Discussed as a platform for researchers to market their work and a place where misunderstandings or undue scrutiny can arise from tweets.

A company mentioned for its contributions to Mixture-of-Experts (MoE) models.

Yi Tay is the Chief Scientist at Reka, an AI company that announced a $58 million Series A funding round in June 2023, with stated goals including universal intelligence, multimodal, and multilingual agents, self-improving AI, and model efficiency.

Reka AI's co-founders, including Yi Tay, collaborated at DeepMind before starting Reka AI.

Meta has developed a strong stack for training large language models, evident in the success of the LLaMA series, particularly LLaMA 3.

A company mentioned for its contributions to Mixture-of-Experts (MoE) models.

The Hugging Face leaderboard is cited as contributing to the proliferation of lightweight, derivative open-source models that often struggle to achieve significant advancements.

A company mentioned for releasing GPT-1, which significantly influenced the direction of AI research towards general-purpose models.

A position embedding method used in Transformer architectures, appreciated for its extrapolation properties in long context models.

A feed-forward activation function from a single-author paper by Noam Shazeer that was initially obscure but gained popularity after its use in T5 1.1, and which Yi Tay considers very effective.

An architectural approach that Yi Tay is bullish on, believing it to be a promising direction in terms of flop-to-parameter ratio, offering a way to 'cheat' traditional scaling laws by having more parameters at a low flop cost. He questions its tradeoffs in capabilities.

Yi Tay's early research at Google brain focused on efficient Transformers, exploring alternatives to attention mechanisms and their broader applications.

A benchmark for mathematical reasoning that, along with others like HumanEval, is considered 'contaminated' or saturated, meaning high scores are no longer truly indicative of breakthrough performance.

A multimodal benchmark recently released, seen as a next step beyond MMLU but likely to have a limited lifespan before saturation.

A normalization technique used in Transformer architectures, noted as being non-controversial and a default choice in many models.

A paper on which Yi Tay advised, focusing on vision with some T5 experiments, exploring the early stages of sparse upcycling which later became relevant for MoE models.

An influential paper co-authored by Yi Tay and Jason Wei, whose central idea was primarily Jason Wei's thesis. It explored the unexpected capabilities that arise in large language models as they scale.

A paper that contested the claims of 'Emergent Abilities of Large Language Models,' arguing that some supposed emergent properties were artifacts of evaluation metrics. Yi Tay believes in emergence despite this critique.

A science fiction novel mentioned as an inspiration for describing the 'chaotic and stable phases' of compute providers during Reka AI's early training runs.

Yi Tay's alma mater, where he pursued his PhD, noting that the research community there is more focused on publishing papers than real-world impact.

Yi Tay previously worked at Google Brain for three years, where he was involved in significant AI research projects like Palm 2, UL2, and Flan.

A close collaborator and friend of Yi Tay at Google, co-author on the 'Emerging Abilities' paper, known for his marketing prowess and influential ideas in AI research.

The first author of the Flan paper, described by Yi Tay as a great engineer, and someone from whom Yi Tay learned about systematic thinking and life perspectives.

Credited as the single author of the SWIGLU paper and a co-author on a paper exploring Transformer modifications, influencing Transformer architecture choices.

Yi Tay's former manager at Google, described as a research-oriented person with good intuition and taste in identifying impactful research, and more of a friend figure than a corporate manager.

An academic mentioned by Yi Tay as someone who has discussed the shift in academic research from application-specific tasks to universal models.

Co-founder and CEO of Adept, whose discussion on screen modality and its value highly impressed the host.

The H100 GPU series experienced major delays and supply chain issues, which significantly impacted Reka AI's initial training runs due to unreliable hardware.

Google's proprietary AI accelerator. Yi Tay mentions the significant difference in TPU experience inside versus outside Google due to infrastructure and deprecated codebases.

Yi Tay's home country, whose research community he sees as less impact-driven compared to the US, and for which he offers advice on fostering AI talent.

Contrasted with Singapore, the US research community is seen as impact-driven, influencing Yi Tay's perspective on research culture.