Computation and Language - QuestA Expanding Reasoning Capacity in LLMs via Question Augmentation

Hey PaperLedge crew, Ernis here, ready to dive into some seriously cool AI research! Today, we're tackling a paper about how to make those brainy language models, the kind that can reason and solve problems, even better at thinking things through. Think of it like this: we're trying to train a student to ace a tough math test, not just pass it.

The paper kicks off by pointing out that reinforcement learning, or RL, which is like training an AI with rewards and punishments – a digital carrot and stick – is a popular way to boost these language models. RL is used to train models to improve multi-step reasoning – but recent studies are questioning if RL is really effective on the most difficult problems. It's like trying to teach your dog a super complex trick; sometimes, the usual treats just don't cut it.

So, what's the solution? Well, the researchers propose something called Question Augmentation, or QuestA for short. Imagine you're helping that student with their math homework. Instead of just giving them the problem and saying, "Good luck!", you give them hints, right? Maybe a partial solution, or a step-by-step breakdown. That's essentially what QuestA does. It feeds the language model partial solutions during training to make the problems a little easier and give it more helpful clues along the way.

Think of it like this: If you are training a model to bake a cake, you might give it the first few steps of the recipe completed, or a picture of what the batter should look like.

The result? The researchers found that QuestA significantly improved the language model's ability to solve math problems, not only getting the answer right in the first try (pass@1) but also improving the chances of getting the answer correct after multiple tries (pass@k). This is especially true for those super tricky problems where regular RL struggles.

"Our method, QuestA, when applied during RL training on math reasoning tasks, not only improves pass@1 but also pass@k-particularly on problems where standard RL struggles to make progress."

But here's where it gets really exciting. They used QuestA to train some already powerful open-source language models, and they saw even more improvement. These models, with about 1.5 billion parameters (that's a LOT of brainpower!), achieved state-of-the-art results on challenging math benchmarks. We're talking about significant jumps in accuracy on exams like AIME24, AIME25, and HMMT25.

To give you some stats, they got a 67.1% (+5.3%) on AIME24, 59.5% (+10.0%) on AIME25, and 35.5% (+4.0%) on HMMT25. To put it in perspective, that’s like going from a C to a solid B, or even an A-, just by giving the model a little help during practice!

So, why does this matter?

• For AI developers: This provides a practical way to enhance the reasoning abilities of existing language models without drastically increasing their size or complexity. It means we can get more out of the models we already have.
• For educators: The concept of providing partial solutions mirrors effective teaching strategies. It reinforces the idea that scaffolding and guidance are crucial for learning complex skills.
• For everyone else: As AI becomes more integrated into our lives, improving its reasoning abilities is essential. Better reasoning leads to more accurate and reliable AI systems that can assist us in various tasks, from research to problem-solving.

The paper even delves into the theory behind why QuestA works, suggesting that it improves sample efficiency. This means the model learns faster and more effectively because it's getting more informative signals during training. It's like learning to ride a bike with training wheels first – you gain confidence and balance before tackling the real thing.

So, what are the big takeaways?

• QuestA is a simple but powerful technique for improving the reason