PTS: Pivotal Token Search

This paper introduces Pivotal Token Search (PTS), a novel method for improving the performance of large language models by focusing on critical decision points in their output sequences. Unlike traditional methods that treat all generated tokens equally, PTS identifies "pivotal tokens" that significantly influence the probability of a successful generation. By using a binary search algorithm to pinpoint these key tokens, PTS generates preference pairs specifically centered on these critical decisions, leading to a more efficient learning signal during training. The release includes an open-source implementation, datasets of pivotal tokens and preference pairs, and fine-tuned models demonstrating the technique's effectiveness. This approach has potential applications in improving reasoning abilities, agent trajectories, and model interpretability.