The Free Transformer: VAE Extension for Decoders

The October 20, 2025 Meta FAIR paper introduces the **Free Transformer**, an innovative extension of the decoder-only Transformer architecture, which addresses the limitations of purely autoregressive language modeling by integrating **random latent variables** into the generative process. This new model is structured as a **conditional Variational Autoencoder (VAE)**, where an encoder learns the latent variables unsupervised, and a decoder conditions its token generation on these variables. The implementation requires only a minor computational overhead due to sharing half of the decoder's blocks with the encoder. Experimental results with 1.5B and 8B parameter models demonstrate that this conditioning leads to **substantial performance improvements** on reasoning and coding benchmarks like HumanEval+ and GSM8K. The authors conclude that the Free Transformer significantly **improves the inductive bias** of the vanilla Transformer.

Source:

https://arxiv.org/pdf/2510.17558v1