// HACKER NEWS — CYBERSECURITY
Learning to solve hard problems in RL for LLMs by never giving up
This is a blog post for my recent paper on RL post-training of LLMs: introducing the Matthew Effect and proposing to solve it with Never Give Up. It is presented interactively and less formally, more like how I give the talk. For a deeper, more technical dive, check out the paper on arxiv and code on github.
Every good RL practitioner has no doubt seen an eval curve go up. Here is the AIME 2025 eval during our RL training of Olmo 3.1 RL-Zero Math(1)
(1)see Olmo 3.1 blog post and arxiv
Our eval is an average over 30 AIME questions. Let’s break those 30 questions down into 3 levels of difficulty.
Every question that our initial, pre-RL model gets 0 for pass@32 will be labelled “hard”. The other questions we’ll divide evenly by into “medium” and “easy” based on their pass-rates. So our initial pass@1 averages will be 0%, 3.8%, 22.7% for our subsets. How do you think performance on each subset will evolve?
Averaging our AIME eval was hiding something important: the majority of our improvements are coming from the easiest problems going from somewhat solved to mostly solved. The hardest problems are barely improving. This is clearly visible if you look at how each example’s solve rate changes over time (see plot in the margin).
⊕
Accuracy of each AIME eval example over training. We order examples by difficulty from top (initial model pass@32=0) to bottom (initial model pass@1 > 30%). The hardest examples (top rows) barely improve over training. The model mainly learns to better solve easy and medium-difficulty examples that were already reasonably-well solved. We call this discrepancy the Matthew Effect.
But this is for math RL on LLMs. What about other domains?
We evaluate code RL and agentic RL using Deepcoder and DeepSWE, two nice open-source projects that released models and logs.
We can use the initial model to split each benchmark into difficulty buckets (Deepseek-R1-Distilled-Qwen-14B on LCBv6) or we can use existing task length/difficulty labels (SWEBench).
The gains from RL are proportional to how easy the problems are. We connect this bias to a similar phenomenon in network science and economics, the Matthew Effect(2)
(2)Merton (1968), also see Wikipedia
, generally summarized as “the rich get richer”.
We therefore propose The Matthew Effect in RL for LLMs
RL improves performance on a task in proportion to a model’s initial competence—making easy tasks easier while hard tasks often remain difficult.
You might assume the issue has to do with GRPO. If we just don’t get a correct answer to our problem in our $k$ sampled completions, then we don’t get any gradient and can’t improve on this problem.(3)
(3)Xiong et al (2025) call this signal loss
One possible answer is to sample more completions i.e. larger $k$.(4)
(4)Other approaches include using priveleged information and curriculum learning. These are generally complimentary to our approach.
To test this out, we train Qwen 2.5 0.5B Instruct with GRPO on GSM8k platinum and test on the same. We split our dataset into difficulty levels using initial pass@1: easy (25%), medium (10%), hard (5%), and extra-hard (0%) problems. We vary $k \in \{4, 8, 16, 32\}$ but keep batch size fixed.