AI models with more parameters don't always produce better results. Parameters influence calculations but don't directly encode knowledge, and their impact depends on training and architecture.
Parameters and AI capabilities
A model of artificial intelligence can have ten times more parameters than another without producing better responses. These small coefficients influence its calculations, sometimes mobilize considerable memory, and ultimately say less about its capabilities than one might think.What is a parameter
A parameter is first of all a number. More precisely, an internal and persistent coefficient of the model, which influences how it processes data and calculates its predictions. Whether it is set to 0.018, -0.42 or 1.27 matters little in itself: extracting a parameter from the model does not allow one to deduce much.How parameters influence predictions
In a language model generating text token by token, this succession of calculations ultimately assigns a score to each token in its vocabulary. These scores are converted into probabilities, and then one of the tokens is selected. After "Paris is the capital of the ", for example, the joint action of billions of parameters will give a high probability to the token likely to start a sequence such as "France", and much lower to those that do not fit well with the context.Limitations of individual parameters
It is therefore impossible to isolate the parameter that contains "Paris", the conjugation of a verb or the date of an event: none contains information that one could read or extract as is. What the model has learned depends on the combined effects of a very large number of these coefficients. The same knowledge can thus mobilize a large number of parameters, and a single parameter can participate in useful calculations in very different contexts.Structure of parameters in AI models
Technically, these billions of parameters are grouped into number tables called tensors. Of different sizes, they are distributed in the model according to the operations to which they participate. A large part of these tables contains weights, used in the model's calculations. According to the architecture, there are also biases, which, unlike weights, add a value to the result of a calculation rather than multiply the values received.How a model learns
The model's learning, a matter of billions of parameters to adjust. When you use a model, the values of its parameters are already fixed: they do not change with each new request. But before reaching these billions of coefficients that will allow it to produce good predictions, it was necessary to find their values. That is the role of training. To avoid plunging too quickly into too technical considerations, one can imagine the process as a gigantic equalizer with billions of sliders.Learning through adjustments
During training, their positions are refined by a multitude of micro-adjustments, calculated from the gap between what the model produces and what it should have produced, in order to gradually correct its errors. These adjustments are not made one by one. Each prediction depends on a very large number of parameters, and the measured error allows to calculate how to simultaneously adjust their values. Once training is complete, the obtained adjustments are fixed and constitute a sort of huge "preset" digital that the model will then use to generate its responses.How models store knowledge
It is thus that a model learns. It does not gradually fill a database in which it would store the information encountered during its training. Through adjustments, it learns regularities of the language, associations and relations between concepts, which it can then exploit when faced with texts it has never encountered. This does not prevent it from memorizing sometimes certain sequences from its training corpus.More parameters, more performance?
More parameters, necessarily a more performant AI? At comparable architecture and training, a model with 70 billion parameters obviously has a greater margin of maneuver than one that only counts 7 billion. These additional coefficients indeed offer it more possibilities to take into account the links between words and ideas, distinguish the different meanings of a term according to its context or combine several pieces of information in a single response.Limitations of parameter quantity
But ten times more parameters do not mean ten times more knowledge, and even less ten times better answers. Because having more parameters does not guarantee that they will be better exploited. The quantity and quality of the data, the computing resources mobilized, the network architecture or even the duration and methods of training can make a considerable difference. That is how in 2022, the researchers from DeepMind succeeded in doing better with four times fewer parameters.Total and active parameters
Total parameters and active parameters, not all at once. According to the architecture used, not all the announced parameters necessarily participate in each prediction. In a so-called "dense" model, almost all the parameters intervene in the calculations necessary for the generation of each new token. But other models adopt an architecture called Mixture of Experts (MoE), or "mixture of experts".Efficient parameter usage
A part of the network is organized into several groups of parameters, the famous experts, among which a routing system selects those it will solicit. One then distinguishes the total parameters, which correspond to the entire set of coefficients of the model, and the active parameters, actually mobilized to process a token. Despite their name, these experts are not small specialized AI each in a domain, such as mathematics, translation or programming.Memory and precision
Billions of parameters to hold in memory. Because all these parameters, whether active or not, take up space. And whether a coefficient is set to 0.018 or to 1.27, it does not change anything. It is not its value that determines its memory occupancy, but the number of bits used to represent it. These bits, zeros and ones, serve to code the number in memory: the more one uses, the less one needs to round its value.Quantization for efficiency
This degree of exactness is called numerical precision. With 16 bits, or two bytes per parameter, a model of 7 billion parameters thus occupies about 14 Go for its coefficients alone. Pass to 70 billion, and you already reach 140 Go, without even counting the additional memory needed for the model's calculations. You see the problem. And yet, such voluminous models are running today locally on personal computers. How? Because all these parameters do not need to be coded with as much precision. That is the whole interest of quantization, a technique that allows using fewer bits to represent their values, even if it introduces some approximations.
