Pretraining is the first and most extensive learning phase of an AI model, in which it learns how language works from huge amounts of text.
Example
During pretraining a language model reads billions of sentences and practises predicting the next word.
For pros
Pretraining produces the base model. Post-training follows, with fine-tuning and RLHF. Pretraining is usually the most compute-intensive step and sets the knowledge cutoff.
