cover of episode OpenAI's Noam Brown, Ilge Akkaya and Hunter Lightman on o1 and Teaching LLMs to Reason Better

OpenAI's Noam Brown, Ilge Akkaya and Hunter Lightman on o1 and Teaching LLMs to Reason Better

2024/10/2
logo of podcast Training Data

Training Data
AI Deep Dive AI Insights AI Chapters Transcript
People
H
Hunter Lightman
I
Ilge Akkaya
N
Noam Brown
P
Pat Grady
S
Sonya Huang
Topics
Noam Brown：O1模型结合了大型语言模型和深度强化学习的优势，能够进行更长时间的思考，从而提升推理能力。这与人类的系统一和系统二思维类似，有些问题需要更长时间的思考才能获得更准确的结果。O1模型的推理方式更通用，可以应用于多个领域，例如数学、编程等。在AlphaGo的启发下，O1模型也受益于更长的思考时间，但其思考方式更通用，可以应用于更多领域。O1模型的推理过程是人类可理解的，可以观察其思考过程。 Ilge Akkaya：最初对O1模型的成功信心不足，但在看到模型以不同方式解决问题后，信心增强。OpenAI采用经验数据驱动的方法，当数据开始显示出想要的结果时，团队会全力以赴。ChatGPT的出现是一个范式转变，O1模型则代表了在推理能力上的新方向。O1模型被医生和研究人员用作头脑风暴伙伴，帮助他们进行癌症研究和基因治疗等方面的研究。 Hunter Lightman：O1模型系列使用Aral训练，能够进行思考和推理，与传统的LLM有根本区别，并且在多个推理领域都实现了泛化。O1模型的推理方法在不同的问题类型上保持一致，无论生成答案和验证答案的难易程度如何。在O1的训练过程中，模型在数学评估中的得分超过了之前的任何尝试，并且其思维链显示出回溯和自我纠正的能力。这让我对O1模型的成功充满信心。

Deep Dive

Key Insights

Why is O1 better at STEM problems than previous models?

O1 is better at STEM problems because these tasks often benefit from longer reasoning, which O1 is designed to do. STEM problems tend to fall into the category of hard reasoning tasks where there is a clear benefit from considering more options and thinking for longer.

Why did the team choose to hide the chain of thought in O1?

The team chose to hide the chain of thought in O1 partly for competitive reasons and to mitigate risks associated with sharing the thinking process behind the model, similar to the decision not to share model weights.

What is the significance of inference-time scaling laws in O1?

The inference-time scaling laws in O1 are significant because they suggest that the model's capabilities can improve dramatically with more compute time, potentially leading to breakthroughs in areas like math research and software engineering.

What are the main bottlenecks to scaling test-time compute?

The main bottlenecks to scaling test-time compute include the need for significant engineering work to build and run large-scale systems, the availability of appropriate benchmarks, and the diminishing returns as compute time increases.

What is the biggest misunderstanding about O1?

The biggest misunderstanding about O1 is the origin of its codename, 'Strawberry.' It was simply named because someone in the room was eating strawberries at the time, not because of the popular question about counting Rs in 'strawberry.'

How should founders think about using GPT-4 versus O1?

Founders should use GPT-4 for general tasks and O1 for tasks that benefit from longer reasoning, such as STEM and coding. O1 is designed to think for longer and can be more effective in these domains.

Chapters
The researchers discuss their initial convictions and aha moments leading to the development of O1.
  • Initial conviction was promising but the path was unclear.
  • Aha moments came when methods started to work and outputs showed different problem-solving approaches.
  • OpenAI's empirical, data-driven approach when trends aligned.

Shownotes Transcript

Combining LLMs with AlphaGo-style deep reinforcement learning has been a holy grail for many leading AI labs, and with o1 (aka Strawberry) we are seeing the most general merging of the two modes to date. o1 is admittedly better at math than essay writing, but it has already achieved SOTA on a number of math, coding and reasoning benchmarks.

Deep RL legend and now OpenAI researcher Noam Brown and teammates Ilge Akkaya and Hunter Lightman discuss the ah-ha moments on the way to the release of o1, how it uses chains of thought and backtracking to think through problems, the discovery of strong test-time compute scaling laws and what to expect as the model gets better. 

Hosted by: Sonya Huang and Pat Grady, Sequoia Capital 

Mentioned in this episode:

- Learning to Reason with LLMs): Technical report accompanying the launch of OpenAI o1.

- Generator verifier gap: Concept Noam explains in terms of what kinds of problems benefit from more inference-time compute.

- Agent57: Outperforming the human Atari benchmark), 2020 paper where DeepMind demonstrated “the first deep reinforcement learning agent to obtain a score that is above the human baseline on all 57 Atari 2600 games.”

- Move 37): Pivotal move in AlphaGo’s second game against Lee Sedol where it made a move so surprising that Sedol thought it must be a mistake, and only later discovered he had lost the game to a superhuman move.

- IOI competition): OpenAI entered o1 into the International Olympiad in Informatics and received a Silver Medal.

- System 1, System 2: The thesis if Danial Khaneman’s pivotal book of behavioral economics, Thinking, Fast and Slow), that positied two distinct modes of thought, with System 1 being fast and instinctive and System 2 being slow and rational.

- AlphaZero): The predecessor to AlphaGo which learned a variety of games completely from scratch through self-play. Interestingly, self-play doesn’t seem to have a role in o1.

- Solving Rubik’s Cube with a robot hand): Early OpenAI robotics paper that Ilge Akkaya worked on.

- The Last Question): Science fiction story by Isaac Asimov with interesting parallels to scaling inference-time compute.

- Strawberry: Why?

- O1-mini): A smaller, more efficient version of 1 for applications that require reasoning without broad world knowledge.

00:00 - Introduction

01:33 - Conviction in o1

04:24 - How o1 works

05:04 - What is reasoning?

07:02 - Lessons from gameplay

09:14 - Generation vs verification

10:31 - What is surprising about o1 so far

11:37 - The trough of disillusionment

14:03 - Applying deep RL

14:45 - o1’s AlphaGo moment?

17:38 - A-ha moments

21:10 - Why is o1 good at STEM?

24:10 - Capabilities vs usefulness

25:29 - Defining AGI

26:13 - The importance of reasoning

28:39 - Chain of thought

30:41 - Implication of inference-time scaling laws

35:10 - Bottlenecks to scaling test-time compute

38:46 - Biggest misunderstanding about o1?

41:13 - o1-mini

42:15 - How should founders think about o1?