AI's reasoning ability deteriorates when faced with complex problems.

The stronger the model, the weaker the "thinking"?

In a newly published report, Apple researchers evaluated the performance of Large Reasoning Models (LRMs) in handling logic problems of increasing difficulty, such as the Tower of Hanoi or the River Crossing problem.

The results were shocking: when faced with highly complex problems, the accuracy of advanced AI models not only deteriorated, but "collapsed completely."

What's even more concerning is that before the performance plummeted, the models began to... reduce their reasoning effort, a behavior contrary to intuition, as more thought should be needed when dealing with a difficult problem.

In many cases, even when given the correct algorithm, the models still fail to provide a solution. This reveals profound limitations in their ability to adapt and apply rules in new environments.

The challenge of "general theory"

Responding to this research, American scholar Gary Marcus, one of the voices skeptical about the true capabilities of AI, called Apple's findings "quite devastating."

In his personal Substack newsletter, he stated: "Anyone who thinks that large language models (LLMs) are a direct path to AGI is deceiving themselves."

Concurring with this view, Andrew Rogoyski, an expert at the Human-Centered AI Institute (University of Surrey, UK), believes this finding points to the possibility that the technology industry is heading into a "dead end": "When models only perform well with simple and medium-difficulty problems, but completely fail at increasing difficulty, it's clear there's a problem with the current approach."

One particular point highlighted by Apple is the lack of "general reasoning" ability, that is, the ability to extend understanding from a specific situation to similar situations.

When knowledge cannot be transferred in the way humans typically do, current models easily fall into a state of "rote learning": strong in repetitive patterns, but weak in logical thinking or deduction.

Furthermore, research has found that large-scale reasoning models consume computational resources by repeatedly performing the correct steps for simple problems, but choosing the wrong approach from the outset for slightly more complex problems.

The report tested a range of leading models, including OpenAI's o3, Google's Gemini Thinking, Claude 3.7 Sonnet-Thinking, and DeepSeek-R1. While Anthropic, Google, and DeepSeek have not yet responded, OpenAI declined to comment.

Apple's research doesn't deny AI's achievements in language, imagery, or big data. However, it highlights a blind spot that's being overlooked: the ability to reason genuinely, which is at the core of achieving true intelligence.

Source: https://baovanhoa.vn/nhip-song-so/ai-suy-luan-kem-dan-khi-gap-bai-toan-phuc-tap-141602.html