Sequential Diagnosis with Language Models

The provided text introduces a Sequential Diagnosis Benchmark (SDBench), a novel method for evaluating AI and human diagnostic abilities using 304 complex medical cases from the New England Journal of Medicine. Unlike traditional static evaluations, SDBench simulates real-world clinical practice by requiring a diagnostic agent to iteratively request information and tests, with performance measured by diagnostic accuracy and associated costs. To complement this, the text presents the MAI Diagnostic Orchestrator (MAI-DxO), an AI system that outperforms both individual physicians and off-the-shelf language models in diagnostic accuracy while simultaneously reducing medical costs. The MAI-DxO achieves this through a multi-agent orchestration strategy that mimics a panel of specialized doctors, demonstrating the potential for AI to enhance both diagnostic precision and cost-effectiveness in healthcare.