AI的下一步：强化学习是正确的AGI解法吗？ - 强化学习圆桌讨论 Silicon Valley 101

type

date

slug

summary

01. 开场：AI时代的转折点与嘉宾介绍

(AI Era Turning Point: Intro & Panelists)

Bill:

So we're going to focus on RL today and we all know that reinforcement learning serve is core to today's AI advancements. From earlier this year of DeepSeek release of R1 where they have reinforcement fine tuning to really have some sort of self-improvement on coding and math. To later this year where we see a lot of applications on math, coding, chemistry, physics, even embody AI.

我们要聚焦于强化学习（RL），众所周知，它是当今 AI 进步的核心。从今年早些时候 DeepSeek 发布 R1，利用强化微调在代码和数学上实现自我提升；到今年晚些时候，我们看到 RL 在数学、代码、化学、物理，甚至具身智能上的大量应用。

Lihong (Amazon):

Hi everyone. My name is Lihong from Amazon. I work on reinforcement learning and language models for a while.

大家好，我是来自亚马逊的 Lihong。我从事强化学习和语言模型的研究有一段时间了。

Jiahui (OpenAI):

Hi my name is Jiahui. So I'm currently working at OpenAI and before that I was at DeepMind. So I also work on reinforcement learning and actually I'm a big fan of Lihong's paper.

大家好，我是 Jiahui。目前在 OpenAI 工作，之前在 DeepMind。我也从事强化学习研究，实际上我是 Lihong 论文的忠实粉丝。

Alborz (LinkedIn):

Good morning everyone. I'm Alborz. I'm a distinguished scientist at LinkedIn. Mostly focusing on agental. Before coming to LinkedIn, I was at Meta where I was actually working with Bill closely.

大家早上好，我是 Alborz，领英的杰出科学家。主要关注 Agent（智能体）领域。来领英之前我在 Meta 工作，实际上曾与 Bill 密切合作过。

02. RLVR的潜力与局限：可验证奖励

(Potentials & Limits of RLVR: Can it go beyond Math/Code?)

Bill:

One of the trends is RLVR (Reinforcement from Verifiable Rewards). The idea is that you can literally use rules and verifier systems from math to verify whether or not the outputs are correct. Where do you guys think RLVR can go beyond just current reinforcement fine tuning on math and coding?

目前的趋势之一是 RLVR（基于可验证奖励的强化学习）。其核心理念是利用规则和数学验证系统来判断输出是否正确。你们认为除了目前的数学和代码微调，RLVR 还能往哪里发展？

Lihong:

It definitely has a lot of great impact. But on the other hand, we also see that many problems are not verifiable. An example is that when we are children in school, we have tests that are verifiable (ABCD choose one). But when we actually go out to work, there's a lot of high judgment call and non-verifiable outcomes. So that's a much fuzzier place.

它肯定有巨大的影响力。但另一方面，我们也看到许多问题是“不可验证”的。举个例子，我们在学校时的考试是可验证的（做选择题）。但当我们步入社会工作，很多决策需要高度的判断力，且结果不可验证。那是一个更模糊的领域。

Jiahui:

Usually it's much easier to verify than generate. It can go beyond fine-tuning. For example, you can use this approach to develop stronger reasoning skills. You can also use it in somewhat different framework like reinforcement learning based on debate or self-critic.

通常验证比生成容易得多。它绝对可以超越微调。例如，你可以用这种方法开发更强的推理能力。或者将其用于不同的框架，如基于辩论（Debate）或自我批评（Self-critic）的强化学习。

Alborz:

RL kind of give you the capability to go beyond what a tutor basically shows you. Verifiable rewards are definitely a way to go because it's very crystal clear. But many tasks require a combination. There are things we don't know exactly how to define, like writing a summary or providing an innovative idea. That's where we need to get inspiration from humans to guide us to shape this reward function.

RL 赋予了你超越导师传授内容的能力。可验证奖励是一条必经之路，因为它们非常清晰。但许多任务需要结合两者。有些东西我们不知道如何定义，比如写摘要或提出创新点子。这时我们需要从人类那里获得灵感，来指导我们设计奖励函数。

03. 黄金标准难题：平衡数据成本与质量

(The Golden Standard: Balancing Low-quality Human Data vs. Expensive Experts)

Bill:

For very complex tasks like chemistry Olympics, a single data annotation takes about $20 from a PhD. How do we handle this mixture of verifiable and non-verifiable reward without breaking the bank?

对于像化学奥赛这样的复杂任务，博士级专家标注一条数据大约需要 20 美元。我们如何在不破产的情况下，处理这种混合了可验证和不可验证奖励的情况？

Alborz:

We want to be cost aware. Think of it as an active learning situation. You can get like four of us here decide about a small set of datasets. If we all agree, then this can be a rubric to then go and check LLM judges. If they comply, then it can be a blessed low-cost non-verifiable reward applied on the masses.

我们需要有成本意识。可以把它看作“主动学习”。比如让我们四个人制定一个小数据集的标准。如果我们达成一致，这就可以作为准则去测试 LLM 裁判。如果 LLM 通过测试，它就可以成为一种低成本的奖励机制，应用到海量数据上。

Jiahui:

We might try to decompose the problem. Maybe we can decompose it into some subtasks and for some of the subtasks it's verifiable. Also, exploration is crucial here. Because data is costly, you want to identify the right expert to ask the right question.

我们可以尝试分解问题。也许将其分解为子任务，其中部分子任务是可验证的。此外，探索（Exploration）在这里至关重要。因为数据昂贵，你需要识别出“对的专家”去问“对的问题”。

Lihong:

RL is limited by how good a simulator data is. At the end of the day, we need to make trade-off on how to make sure the environment is of high quality.

RL 的上限取决于模拟器数据的质量。归根结底，我们需要在确保环境（Environment）的高质量上做出权衡。

04. 探索的平衡：预训练 vs 后训练

(Exploration Trade-off: Pre-training vs. Post-training)

Bill:

Where do you think is the balance between pre-training and post-training when you actually really want to induce strong exploration behavior?

当你想通过 RL 激发真正的探索行为时，如何在预训练和后训练之间取得平衡？

Jiahui:

The main role of the pre-training is to have a model that has a good understanding of fundamentals. If you start seeing that you're getting better in some tasks but worse in others, you're going too far. Post-training is where you fine-tune on a specific task.

预训练的主要作用是让模型对基础知识有良好的理解。如果你开始发现模型在某些任务变强，但在其他任务变弱，那就过头了。后训练才是针对特定任务进行微调的阶段。

Lihong:

When pre-training is done, the space of exploration is already very limited. For more aggressive exploration, it might be interesting to look at beyond the token level, or something that has more high level. That may allow the model to think out of the box.

预训练完成后，探索空间其实已经很受限了。为了进行更激进的探索，我们需要超越 Token（词元）层级，在更高层级进行探索。这能让模型跳出固有思维框架。

05. 强化学习之父的OaK架构：分层与抽象

(Hierarchical RL & Abstraction: Breaking Complex Tasks)

Bill:

Rich Sutton proposed "Oak Architecture". Two questions: 1. How much should we go beyond token level? 2. How important is hierarchical structure?

Rich Sutton 提出了 Oak 架构。两个问题：1. 我们需要在多大程度上超越 Token 层级？2. 分层结构有多重要？

Jiahui:

It's very important. Currently most signals are just at token level and it's so low level. This causes a lot of problems of credit assignment and data efficiency. We need to go to a higher abstraction like subtasks, plans.

这非常重要。目前的信号大多停留在 Token 层级，太低级了。这导致了信用分配（Credit Assignment）和数据效率的问题。我们需要上升到更高的抽象层级，如子任务、计划。

Alborz:

We as humans, when we plan to go to Japan, we don't think about the little muscle twitches (go through the door, take a cab). We actually think on a higher level of abstraction. Currently LLMs are missing this. We need a policy that thinks in a higher level to unlock the whole new era of reasoning.

我们人类计划去日本时，不会去思考肌肉的每一次抽动（比如怎么走过门、怎么打车）。我们是在更高的抽象层面上思考的。目前的 LLM 缺失了这一点。我们需要一个能在高层级思考的策略（Policy），以开启推理的新纪元。

06. 过程奖励与模型大小

(Process-based Rewards & Model Size)

Bill:

Is it really hard to derive process-based rewards?

推导基于过程的奖励真的很难吗？

Lihong:

It's a promising idea. Intermediate rewards can be used as part of a curriculum training. We use them at the beginning to guide exploration and then eventually take them away.

这是个有前景的想法。中间奖励可以作为课程学习的一部分。我们在开始时使用它们来引导探索，最终再把它们撤掉。

Alborz:

It depends on the model size. Bigger models might find the workable solution easier. But for smaller models, if they cannot see that good cookie at the end at all, then they can't learn. If you shape rewards well for smaller models, you can get about 3-4 times compute saving.

这取决于模型大小。大模型可能更容易找到可行的方案。但对于小模型，如果它完全看不到最后的“甜头（Cookie/奖励）”，它就学不到东西。如果你给小模型设计好奖励，可以节省约 3-4 倍的算力成本。

07. 创新：RL能否创造新知识？

(Innovation: Can RL create new knowledge?)

Bill:

Can pure RL eventually give us the possibility of generating new knowledge and being an innovator?

纯粹的 RL 最终能否让我们创造新知识，成为创新者？

Lihong:

It's possible (e.g., in math). But RL is limited by the simulator. If the model we provide is just based on human knowledge, RL can only explore within this framework. It won't create new physics theory unless the simulator allows it.

是可能的（比如在数学领域）。但 RL 受限于模拟器。如果我们提供的模型仅基于人类知识，RL 只能在这个框架内探索。除非模拟器允许，否则它无法创造新的物理理论。

Alborz:

AI has shown us already. AlphaGo Move 37. Everyone thought it was a mistake, but it turned out to be the winning move. If you have a good simulator and a rich reward function, it can go beyond what every human is capable of doing.

AI 已经向我们证明了这一点。AlphaGo 的第 37 手棋，所有人都以为是失误，结果却是制胜一手。只要你有好的模拟器和丰富的奖励函数，它就能超越人类的能力。

08. 未来展望

(Future Outlook)

Bill:

Where do you see the most near-term advancement from RL in the next year?

你们认为明年 RL 最近期的突破会在哪里？

Alborz:

Abstraction. It's going to give us a lot of goodies.

抽象（Abstraction）。它会给我们带来很多好处。

Jiahui:

Hierarchical reinforcement learning and abstraction is the way to go.

分层强化学习和抽象是必经之路。

Lihong:

How we can learn from more diverse data, not beyond verifiable and LLM-as-a-judge signals.

如何从更多样化的数据中学习，不仅仅是局限于可验证数据或 LLM 裁判的信号。