Beyond Arc-Aagi: Gaia and searching for a real intelligence test

Intelligence is ubiquitous, but its measurement seems subjective. At best, we present its measure through tests and references. Think about entrance exams in college: every year, countless students register, remember tricks imposed on tests, and sometimes go away with excellent results. Does one number, say 100%, mean those who got it, divide the same intelligence – or that somehow eliminated their intelligence maximally? Of course not. Benchmarks are approximations, not correct measurements of someone else’s – or something – real possibilities.

AI generative community has long been about references Mmlu (A huge understanding of the multi -purpose language) to assess the model’s capabilities through multiple choice questions in academic disciplines. This format allows plain comparisons, but it is not possible to really capture clever possibilities.

For example, both Claude 3.5 Sonnet and GPT-4.5 achieve similar results at this reference point. On paper, this suggests equivalent possibilities. However, people who work with these models know that there are significant differences in their actual performance.

What does it mean to measure “intelligence” in artificial intelligence?

On the heels of the up-to-date one ARC-AGI Benchmark-test edition designed to push models in the direction of general reasoning and original problem solving-a concentrated debate on what it means to measure “intelligence” in artificial intelligence. Although not everyone has yet tested the ARC-AGI reference point, the industry with satisfaction accepts other efforts to the evolution of the test framework. Each reference point has its own advantages, and the Arc-agi is a promising step in this wider conversation.

Another significant recent development of AI is “The last exam of humanity“A comprehensive reference point containing 3000 reviewed, multi-stage questions in various disciplines. While this test is an ambitious attempt to challenge AI at expert levels, early results show quick progress-Openai reportedly achieved a result of 26.6% within a month of its release. However,, however, like other classic comparative tests, it primarily assesses knowledge and reasoning in isolation, without testing, Practical possibilities of using tools that are increasingly crucial for the actual applications of AI.

In one example, many of the most contemporary models do not improve the number of “R” in the word strawberries. In another, they incorrectly identify 3.8 as less than 3.1111. Such failures-in tasks that even a tiny child or a basic calculator can solve the mismatch between progress based on the pattern and reliability in the real world, reminding us that the intelligence does not only apply to passing exams, but reliably moving on everyday logic.

A up-to-date standard of measuring AI’s ability

As the models develop, these classic comparative tests have shown their limitations-GPT-4 with tools, it achieves only about 15% on more elaborate, real tasks in the world Gaia benchmarkDespite the impressive results in multiple choice tests.

This disconnection between reference efficiency and practical ability is becoming more and more problematic, because AI systems are moving from research environments to business applications. Classic test research Recall knowledge, but there are no key aspects of intelligence: the ability to collect information, perform code, data analysis and synthesis of solutions in many domains.

Gaia is a necessary change in AI assessment methodology. Created through cooperation between Meta-Fair, Meta-Genai, Huggingface and AutoGPT teams, the reference point includes 466 carefully made questions at three levels of difficulty. These questions are tested by browsing websites, multimodal understanding, code performance, file support and elaborate reasoning-possibilities necessary for real AI applications.

Questions at level 1 require about 5 steps and one tool for people to solve. Level 2 questions require 5 to 10 steps and many tools, and questions at level 3 may require up to 50 prudent steps and any number of tools. This structure reflects the actual complexity of business problems in which solutions rarely come from one activity or tool.

The priority is flexibility before complexity, the AI ​​model reached 75%accuracy compared to Gaia-Przewerza Magnetic-1 (38%) and agent Langfun Google (49%). Their success results from the utilize of combinations of specialized models for understanding and audiovisual reasoning, with the Anthropica Sonnet 3.5 model.

This evolution in AI’s assessment reflects a wider change in the industry: we move from independent SAAS applications to AI agents, which can organize many tools and work flows. Because companies are increasingly relying on AI systems to support elaborate, multi -stage tasks, references such as GAIA provide a more significant measure of ability than classic multiple choice tests.

The future of AI assessment is not in isolated knowledge tests, but in a comprehensive assessment of problems solving. Gaia establishes a up-to-date standard of measuring AI-Taka’s ability, which better reflects the challenges and possibilities of realization of real.

Sri Ambati is the founder and general director H2O.Ai.

Daily observations in matters of business utilize with VB daily

If you want to impress your boss, VB Daily is covered by you. We give you an internal measure about what companies do with generative artificial intelligence, from regulatory changes to practical implementation, so you can share insights for the maximum roi.

Read our Privacy Policy

Thanks for the subscription. Check out more VB newsletter here.

There was a mistake.