Intelligence = predictability
Information is even more fundamental than matter: an electron and a proton are fundamentally different particles, but the spin of either of them can be used to carry a single qubit of information. From the point of view of information, it is natural to relate AI, or more generally the development of intelligence, to information and complexity. Why do we feel we are more intelligent than dogs? Dogs are better than us on many physical capabilities and we are better than them on some other capabilities, but the fundamental difference is that we can predict certain things such as weather tomorrow, much better than dogs. Intelligence is fundamentally about the capability of prediction. The world has less uncertainty for a more intelligent agent.
Predictability = compression
Where does predictability comes from? If I want to store the trajectory of a tennis ball, and I do not understand Newton mechanics, I will have to record the position of the ball for all time instances. This leads to no predictability. If I want to record the ball's position on time instances, the number of parameter is . On the contrary, if I understand Newton mechanics, (let's ignore air friction) then the trajectory is completely fixed by the initial velocity, initial position and gravitational force. The number of parameter to reproduce the position of the ball is (the three vectors that determines the trajectory, and time). Comparing the raw description and the equation of motion description, we see that the latter can be viewed as a compression: there is a cheaper way to predict the future compared with directly doing an experiment. All Nature's laws can be viewed as such data compression. Each law is a shortcut that allows to predict the future with a simulation that is much simpler than Nature itself. Considering that the computer that is used to run the code (or human with paper and pen) that makes the prediction is also part of Nature, we can say that compression means there exists two physical process in Nature which has a one-to-one correspondence to each other.
Komogorov complexity
In what sense the prediction from equation of motion is simpler than the original experiment? To measure that it is helpful to introduce Komogorov complexity. Roughly speaking, Komogorov complexity is the length of shortest code that achieves certain task. Therefore running a for loop that iterates the same equation of motion times do not increase the Komogorov complexity, although the time of computation could increase linearly with time. If a ball is driven by an external force that we don't know, simulating its motion will have a Komogorov complexity that grows linearly with time, while in the case of a fixed equation of motion it does not grow with time.
Thinking this way, language models are essentially extremely complicated "equation of motion": the model itself stays static during inference, which generates a probablistic dynamics in the world of tokens. A more intelligent model is better at predicting the right output, which means it could correctly predict things such as correct answer of math problems, or generating the correct code that achieves certain goal. The Komogorov complexity of an LLM is basically the length of the code that run it, which is more or less determined by the number of parameters of the model. Therefore in this point of view, the LLM is just a "effective theory" that has trillions of parameters, rather than the kind of theory that we are familiar with in physics which only has a dozen parameters.
We are used of the simpler kinds of "theories" because they are most efficient and they can be understood by human brain. In general, the simpler theories are not sufficient in handling complex systems such as human society. We can view the progress of AI as an adventure towards the land of "intermediate Komogorov complexity" which is filling the gap between the simple theories with dozens of parameters and the actual world that contains huge number of parameters (parameters in the initial state of the universe, proportional to the number of atoms in the universe). Of course the Komogorov complexity of an LLM could be smaller than its number of parameters because there might be a way to find a smaller model that performs equally well. More precisely the Komogorov complexity is roughly the size of the minimal LLM with the same performance. It is the "irreducible complexity" of the LLM.