I’m definitely late to the party, but I think I just grok’ed how awesome GPUs are.

In his 2024 interview with Lex Fridman, Yann LeCun claims that LLMs “are doomed” due to their autoregressive nature. He states that if we assume that the prediction errors from token to token are independent then it will eventually lead to a sequence being ‘incorrect’. He does state that this is a strong assumption, but how strong is it?

2024 was a great year. What interested me:

A commonly held belief is that reasoning is locked behind larger models. The idea is that larger models have more layers and wider layers which help them to better reason about relationships in text. This feels like a hand-wavy solution. For example, in a 2-hop reasoning problem, do you need 10 layers, but then for 4 hop, you need 20 layers? If your reasoning problem is more complex, do you need wider layers? As far as I know, none of these questions have good answers yet.

Being reasonable can be hard. For LLMs it’s even harder.

TL;DR There are eight parts of speech: noun, pronoun, adjective, verb, adverb, preposition, conjunction and interjection. You should know this, and probably more.