2024 was a great year. What interested me:

A commonly held belief is that reasoning is locked behind larger models. The idea is that larger models have more layers and wider layers which help them to better reason about relationships in text. This feels like a hand-wavy solution. For example, in a 2-hop reasoning problem, do you need 10 layers, but then for 4 hop, you need 20 layers? If your reasoning problem is more complex, do you need wider layers? As far as I know, none of these questions have good answers yet.

Being reasonable can be hard. For LLMs it’s even harder.

TL;DR There are eight parts of speech: noun, pronoun, adjective, verb, adverb, preposition, conjunction and interjection. You should know this, and probably more.

Merry Christmas from OpenAI: “o3 is coming, and it’s better than all your models!”

I recently came across Chollet’s mental model for LLMs and it really helped me (quoted from his blog):