METR, an AI evaluation nonprofit, released an update to its time horizon plot. The report finds that some AI capabilities continue to improve along an exponential trajectory. It also finds that models perform worse on tasks that require judgment than on coding tasks.

The original plot measured how long a task took human testers to complete, then rated models by the longest tasks they could finish. It showed the length of tasks frontier models could complete doubling about every four months. The update adds tasks that humans would take far longer to finish, along with more open-ended tasks.

On a task where models designed new AI evaluations, none of the AI-designed evaluations was good enough for METR employees to find useful for their research. On a task where models scored philosophical arguments, models also fell short of human performance. METR writes that its employees disagree about how relevant this task is to automating AI research and development.

Sayash Kapoor, a computer scientist and incoming professor at UC Berkeley, said, "We're at the point where AI evaluations need to get more creative." Najoung Kim, a linguist and computer scientist at Boston University, said, "The improvement we observe, I think that does tell us something for the current moment," and added, "I'm not sure just yet how to think about how it would extrapolate."

The report says some newer models, including Opus 5.5, reach what METR calls "borderline above the bar" on the evaluation design tasks. METR's assessment is that these results are not yet usable for its research but are promising. The source says that a year ago, every model scored in the "clearly below the bar" category.

METR includes few open-ended and judgment-heavy tasks in the report because they are difficult to design and score. The source says this makes it hard to predict whether models will keep improving in these areas.