ML//benchmark//BIG-Bench

BIG-Bench is a collaborative benchmark containing more than 200 tasks across reasoning, language, knowledge, mathematics and social understanding. BIG-Bench Hard (BBH) is the subset on which models initially struggled, with a focus on multi-step reasoning.

BIG-Bench is a collaborative benchmark containing more than 200 tasks across reasoning, language, knowledge, mathematics and social understanding. BIG-Bench Hard (BBH) is the subset on which models initially struggled, with a focus on multi-step reasoning.

Its breadth is the point and the problem: in an evaluation, a broad suite reveals an uneven capability profile that the headline average compresses away, and an aggregate score can hide catastrophic failure on a narrow task. It is more useful as a capability microscope than as a single-number leaderboard.