r/singularity • u/TMWNN • 1d ago

AI Is 'Attention' the Essence of Intelligence?

https://www.quantamagazine.org/is-attention-the-essence-of-intelligence-20261012/
120 Upvotes

95% Upvoted

36 comments sorted by

29

u/TMWNN 1d ago

From the article:

From the ability to translate the word “it,” we somehow got a revolution.

“That’s quite a leap, isn’t it?” said Llion Jones, one of the authors of “Attention Is All You Need,” who is now the chief technology officer of Sakana AI.

Jones and others suspect that predicting the next word, which is known as language modeling, is “AI-complete”: a task so difficult that solving it means you have arguably solved AI in general. The reason it may be AI-complete is that this task includes, as a subtask, virtually anything humans do in written form.

24

u/KnackeHackeWurst 20h ago

Imagine you have a criminal case and feed all text you have about into a AI/LLM input context. The task is just to predict one word: the culprit behind the case. Assuming there is no clear confession in the case documents then "predicting" the correct name means you understand the case: person relations, timelines, evidence, whatever.

Predicting even just a single word correct can mean everything right down to the deepest understanding possible. Current LLMs are certainly not yet perfect in that regard, but to say in a derogatory way they are "just predicting words" is flawed.

So I very much agree with your posted statement.

5

u/ProfessionalHefty349 17h ago

More like, based on your search history, purchase history, movements (flock camera), relationship to the victim, etc an AI will be able to generate a list of probable culprits and assign some kind of probability score to them. Eventually it’ll probably be good enough for LE to obtain warrants or bring you into custody.

1

u/deviantbono 14h ago

Of course real world racial bias will also be built into it and skew the results (this is a real thing that has already happened).

2

u/manubfr AGI 2028 13h ago

True, the quesiotn is whether you can get the model to self-correct in post training (eg Anthropic's RLAI) or at test-time with live monitoring. Bias can be subtle so I would expect post-training is the way to go ?

2

u/deviantbono 14h ago

A system that inserts the next word to determine the culprit is great if it's right! But just naming a name becuase it understands the "context" of picking a culprit is a distopian nightmare. This already exists to a degree. Parole boards use computer scores to assess release decisions, but the real world training data is biased by existing racial patterns and silently discriminates against minority candidates.

4

u/KnackeHackeWurst 14h ago

I also wrote in another comment that my example was not meant as a argument for LLM usage in law enforcement. I see it was a poor choice. I just wanted to discuss general modelling capacities for word prediction.

-3

u/hippydipster 17h ago

Except that's just predicting the next word. It does not mean solving the case correctly. It's just the next word mostly likely to be written. They move the models well beyond this with the RLHF training.

3

u/KnackeHackeWurst 17h ago

RLHF has nothing to do with the fact that LLMs output token, one at a time.

Maybe I constructed a poor example. I don't mean to say that LLMs should do law enforcement but instead that giving the (likely) correct answer even with just one word means a modelling/abstraction capability that is on par with humans.

0

u/hippydipster 16h ago

RLHF has nothing to do with the fact that LLMs output token, one at a time.

No one said it did.

I agree, the LLMs get a modelling/abstraction capability on par with humans, but what they are modeling and abstracting isn't quite the same thing humans are modeling and abstracting. The models learn writings - books, internet, magazines, etc, whereas humans mostly model the physical world.

2

u/KnackeHackeWurst 15h ago

It's just the next word mostly likely to be written. They move the models well beyond this with the RLHF training.

Sorry, I understood that sentence that you mean that LLMs moved beyond next word prediction via RLHF.

The models learn writings - books, internet, magazines, etc, whereas humans mostly model the physical world.

In principle you are right of course, but we don't know how exactly humans do modelling and don't understand the inner worlds of LLMs in that regard.

At work Opus 5.5 or GPT-6 Astra have no problem building complex physical 3D simulations especially compared to older models. However that is possible, they seem to have some "understanding" (i.e. modelling) of the physical world.

See the interesting Othello-GPT study that shows that LLMs are capable to construct world models from text alone.

3

u/hippydipster 14h ago

Yeah, familiar with the Othello study. It is fascinating that they really do learn something of the physical world just from words, in contrast to LeCunn's assertions, but it does also seem clear to me that their ability when it comes to spatial/physical reasoning lags very much behind their abilities to do logical/mathematical/software reasoning, and I do hypothesize it's due to how difficult it is to learn physical world modeling just from texts.

I understood that sentence that you mean that LLMs moved beyond next word prediction via RLHF.

Yes, but it doesn't change that they output one token at a time. What is meant is that RLHF teaches them to generate output that is not what they would generate if strictly predicting next token based on the prompt. The point of RLHF is to create better responses from the LLMs than just what their pre-training gets. When predicting the perpetrator is "Carl", they may very well just be predicting what an Agatha Christie story would say in the given case, whereas the RLHF post-training can help move the models toward doing more "real-world" reasoning, if you will.

2

u/First_Grapefruit_265 15h ago

This is a great article, well worth reading if you want to go from having little idea of what the "attention mechanism" is, to having a very basic idea of it.

39

u/Pyros-SD-Models 1d ago

IMHO, the ability to predict stuff is basically THE thing that makes something intelligent, and I'd put it on two axes:

How general are the things you can predict on one axis, and how accurate are your predictions on the other?

The more you max out both axes at the same time, the more intelligent you are.

And attention just seems to be reaaaaally good at maxing both.

Which is also why I never understood the whole "LLMs are JUST next-token predictors" argument. Like, yeah? That's the point? Being able to predict what comes next across basically any domain, anything language can describe, is a pretty fcking impressive capability, don't you think?

Like Hinton says, to predict someting accurately you must have an accurate model of it, so to be able to predict language you need to have a model that accurately models everything language is able to describe

Funnily, Tao came to the same conclusion, before the bots backstabbed him lol

https://www.reddit.com/r/AI_India/comments/1qocpqz/terence_tao_says_the_era_of_ai_is_proving_that/

6

u/sir_racho 23h ago

At some point our language has to accept limitations and concede “understanding” is a better way to go (though that carries consciousness baggage so is not ideal either). Make a list of books and ask an LLM to rank them best to worst and explain WHY it chooses that ranking. It’s not a predictive exercise because there is no right answer. It’s a reasoning, knowledge-heavy task which turns on “understanding”. 

3

u/genshiryoku AI specialist 19h ago

We're pretty sure that from a mathematical standpoint prediction = compression = intelligence.

1

u/Kitchen-Jicama8715 17h ago

The Mayans could predict the stars but that’s not a intelligent as Newtons laws

3

u/genshiryoku AI specialist 16h ago

Newtons laws are a better prediction and a better compression of physical laws as well. This relationship always holds.

2

u/Kitchen-Jicama8715 15h ago

Kind of. But Newtons gravitational laws require estimating parameters like the distance and mass of all surrounding bodies which isn’t always available. While the Mayan charts could predict without needing that just by looking at past history. Yet even if charts of trajectories can be more accurate than Newtons laws given the same information, we still see Newtons laws as the more intelligent representation.

1

u/Zealousideal7801 23h ago

I wonder if there could be another axis to those two ? Based on other intelligent systems that don't use tokens per se, nor act in consciousness per se.

LLMs do that with words, diffusion models do that with latent/pixels, and it's always the same global process that is also seen in neurogenesis in axons/synapses systems : the emergent capability (what's called intelligence but the word is so loaded I can't even use it) is what the learning in (old) context + perceiving the current (new) context + remembering in (new) context + aligning to (new) context + expressing prediction in (new) context amounts to. That last part we call it "reaction" or "behavior" or "resulting action", and whether we're doing a physical or intellectual task, all of those are constantly aimed at the latter. We just don't have any conscious ATTENTION (the exact word used in neuropsychology) of that process just like we don't have any conscious attention of how we beat our hearts or color our eyes or grow our hair. Yet all of that is highly intelligent too, remembers and predicts and adapts and expresses.

And that's the fascinating part I think, that right now this article and many research papers are getting a new (which is not so new but hadn't been perdues/accepted before) vision of what intelligence could mean for us and for everything alive (and not alive...?).

2

u/alwaysbeblepping 11h ago

LLMs do that with words, diffusion models do that with latent/pixels

Inside the model, it's pretty much always a sequence of tokens. Flow/diffusion models flatten the spatial dimensions into patches so the internal shape is going to be something like batch, sequence, channels (or you can call channels the model's hidden size or dimension). A video model might add a time dimension. sequence is going to be tokens either way.

1

u/JoelMahon 23h ago

fast and cheap compression is the engine behind that though imo, not as glamorous "predicting the future" but for any computer not the size of our solar system it's necessary to compete with a human brain in real time.

It's why a data centre can't play cornhole as well or fast as a child can, because it's processing too much noise.

3

u/sir_racho 1d ago

Good article. Thanks for posting OP.

2

u/davesmith001 20h ago

No, it’s only step one. Saying it’s “ai complete” sounds a bit silly.

1

u/TheOriginalAcidtech 14h ago

Prediction without ground = dead. There is a reason humans dont just dream all day long. Dreaming is pure prediction on prediction on prediction.

1

u/catsRfriends 1d ago

No, but it's close.

10

u/Davorian 1d ago

Quick you should go tell them! 

0

u/SkaldCrypto 17h ago

As someone with adhd I can say maybe.

We do get the hyperfocus as well so it nets out when on stimulants

0

u/aegersz 23h ago

no, it's only a third of it !

-2

u/sceadwian 16h ago

No, it seems like a silly statement suggesting something without any actual sensible link present.

Why do you think the essence of intelligence is attention?

If you need an "essence of intelligence" it is this.

Intelligence is compression.