Computer Vision - Agent-X Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks

Hey everyone, Ernis here, and welcome back to PaperLedge! Today, we're diving into a fascinating paper about how well AI agents can really reason, especially when they have to use their "eyes" – meaning, understanding what they see.

Think about it like this: You're trying to bake a cake. You need to read the recipe (text), look at pictures of what the cake should look like (images), maybe even watch a video of someone making it (video). And then, step-by-step, you use different tools – measuring cups, a mixer, an oven – to get the job done. That's multi-step, multimodal reasoning in action!

The problem is, a lot of AI benchmarks – the tests we use to see how smart AI is – are kind of like asking an AI to just identify a picture of a cake, not actually bake one. They're often simple, single-step tasks in a perfect, artificial world.

That's where Agent-X comes in. This paper introduces a brand new, much tougher benchmark for testing AI agents. It's designed to see if they can truly understand the world through their "eyes" and reason their way through complex tasks.

Imagine giving an AI agent tasks like:

• Helping you choose the best outfit from a bunch of pictures (general visual reasoning)
• Browsing a website to find the cheapest flight (web browsing)
• Monitoring security camera footage to spot something suspicious (security and surveillance)
• Navigating a virtual car through a busy street (autonomous driving)
• Analyzing a sports game to predict the next play (sports)
• Solving a geometry problem with diagrams (math reasoning)

Agent-X contains a whopping 828 of these kinds of tasks! These tasks involve real-world images, videos, and text instructions. It's like throwing the AI into the deep end!

The key thing is that Agent-X forces the AI to break down these tasks into smaller, logical steps and use virtual "tools" along the way. It's not enough to just get the right answer; the AI has to show how it got there, step-by-step.

So, how did the AI agents do? Well, even the best ones – models like GPT, Gemini, and Qwen – struggled! They got less than 50% of the full tasks right. That's like failing half your baking attempts, even with a recipe!

This tells us something important: current AI models still have a long way to go when it comes to truly understanding the visual world and reasoning their way through complex, multi-step tasks. They might be good at recognizing objects, but they aren't great at using that information to solve problems like humans do.

The researchers also came up with a really detailed way to grade each step of the AI's reasoning. This helps us pinpoint exactly where the AI is getting stuck – is it misunderstanding the image? Is it making a logical leap that doesn't make sense? Is it using the virtual tools effectively?

Why does this research matter? Well, think about the future:

• For self-driving cars, this means improving their ability to understand complex traffic situations and make safe decisions.
• For healthcare, it could lead to AI that can analyze medical images and assist doctors in diagnosing diseases.
• For everyday life, it could mean AI assistants that can truly understand your needs and help you with complex tasks.

Ultimately, Agent-X is helping us push the boundaries of AI and build systems that can truly see, understand, and reason about the world around us.

The research team has made all their data and code publicly available (you can find the link at https://github.com/mbzuai-oryx/Agent-X), so other researchers can build on their work and improve AI reasoning even further.

Now, here are a few things that popped into my head while reading this paper:

• How much does the type of "tool" available to the AI impact its performance? For example,