To understand AI’s capabilities in these cognitive abilities, we propose a three-step evaluation protocol that compares system performance relative to human capabilities:
- Evaluate AI systems across a broad set of cognitive tasks across every ability, using test suites to prevent data contamination
- Collect human baselines for the same tasks from a demographically representative sample of adults
- Map the performance of each AI system against the human performance distribution for each capability
Moving from theory to practice
Defining these cognitive abilities is a crucial first step, but we need more than just a framework to measure progress. To put this theory into practice, we’re launching a up-to-date Kaggle hackathon — “Measuring progress towards AGI: Cognitive abilitiesThe hackathon encourages the community to design assessments for five cognitive abilities where the assessment gap is greatest: learning, metacognition, attention, executive function, and social cognition.
Participants can apply the newly launched Kaggle software Community benchmarks a platform to create and test your assessments against a range of pioneering models.
We are offering a total prize pool of $200,000: $10,000 prizes for the top two entries in each of the five songs, and $25,000 grand prizes for the four absolute best entries overall. Applications can be submitted from March 17 to April 16, and the results will be announced on June 1. Go to the website Kaggle website start building.