Measuring Progress In the direction of AGI: A Cognitive Framework

To know AI capabilities throughout these cognitive talents, we suggest a three-stage analysis protocol that benchmarks system efficiency in relation to human capabilities:

1. Consider AI methods throughout a broad suite of cognitive duties overlaying every capability, utilizing held-out take a look at units to stop information contamination
2. Acquire human baselines for a similar duties from a demographically consultant pattern of adults
3. Map every AI system’s efficiency relative to the distribution of human efficiency in every capability

Going from idea to follow

Defining these cognitive talents is an important first step, however we want greater than a framework to measure progress. To place this idea into follow, we’re launching a brand new Kaggle hackathon — “Measuring progress toward AGI: Cognitive abilities”. The hackathon encourages the group to design evaluations for 5 cognitive talents the place the analysis hole is the biggest: studying, metacognition, consideration, govt features and social cognition.

Contributors can use Kaggle’s newly launched Community Benchmarks platform to construct and take a look at their evaluations towards a lineup of frontier fashions.

We’re providing a complete prize pool of $200,000: $10,000 awards for the highest two submissions in every of the 5 tracks, and $25,000 grand prizes for the 4 best possible general submissions. Submissions are open March 17 via April 16, and we’ll announce the outcomes June 1. Head over to the Kaggle website to start out constructing.