Tuning explained for noobs (how pre-trained models learn up-to-date skills)

# Introduction

# What is pre-training?

If you start with a freshly created model that has millions or billions of parameters assigned randomly, and you try to directly teach it a very specific task—say, how to classify movies into different categories—it will have to learn the entire English language from scratch at the same time, which is impossible, especially with the circumscribed set of data you have at your disposal. It’s like teaching a petite child biology before they first understand the language or basic scientific concepts.

Pre-training solves this problem by learning arduous and general things one time from a huge amount of data. At this stage, the computational and data requirements are quite high. But once you train it, you will have a model that already understands the language. At this stage, you are teaching your child a very uncomplicated skill: anticipating the next word. You show the model a piece of text with the next word hidden and it has to guess what comes next. Good guesses result in a petite loss, bad guesses result in a immense loss, and the model adjusts.

For example, in the diagram above, if we give a sentence “The Cat Sat on ____”the model learns this “mat” is much more likely than that “car”. Repeating this training over billions of sentences, books, and articles makes the model a very good predictor of the next word and forces it to absorb grammar, facts, reasoning patterns, and more. After initial training, you have a model that already understands the language. Every task you build afterward will build on this foundation, rather than starting from scratch. That’s why they are often called foundation models.

You almost never train anything alone. You download the ready result – a pre-trained model, e.g Lama, MistralOr Qwen – and start there. This brings us to the actual topic of tuning.

# What is tuning?

Many beginners believe that once a model is trained, the weights are frozen forever. In fact, having a pre-trained model means that the weights have been set to “good values” that encode intelligence and perform well on general tasks. Once you have this model, you can tailor this intelligence to your specific needs using task-specific data – and this is called “tuning.” The data requirements at this stage are also much lower than for initial training, as you only need examples for the task you are interested in.

This is similar to a situation where different chefs train at the same culinary school and when they join a restaurant, they learn restaurant-specific skills. Since we’re not building something from scratch here, it’s cheaper – similar to the assumption that training a brand up-to-date person for a restaurant takes a lot more effort than training someone who’s already gone to culinary school. The diagram below summarizes the difference between pre-training and fine-tuning.

# How does tuning work?

We discussed next token prediction and the pre-training process. Now let’s look at the tuning loop.

You show the model an example of task-specific data—say, a movie—ask it to categorize the movie and make a guess, then compare the answer to the ideal answer, shift the weights slightly, and repeat the process until it performs better on a further task. There are also two main things that are done differently in the tuning process than in pre-training:

1. Data → Miniature, high-quality, task-specific data instead of the entire Internet.
2. Learning speed → Low learning rate and few passes because we want the model to adapt without overwriting its overall skills.

# Two common types of tuning

Although you can find various definitions online, based on the number of model parameters you want to tune or adjust, tuning can generally be divided into two categories:

1. Full tuning: In this setting, every parameter in the model can be changed freely. You will run the loop above and all the billions of numbers will move slightly towards your task. The main problem with this approach is memory – you need enough of it to store and update the entire model, which in the case of a immense LLM means sedate hardware. There is also a greater risk of catastrophic forgetting, which simply means that the model becomes good at a specific task but loses its general ability at everything else.
2. Parameter Effective Tuning (PEFT): Instead of updating every weight in the network, PEFT techniques freeze the base model – each original number remains locked – and introduce a petite set of completely up-to-date, trainable numbers. I only train like that. This can be achieved by various techniques such as LoRA, QLoRA, and speedy tuning, but their details are beyond the scope of this article. PEFT requires less memory and training time, with a lower risk of forgetting knowledge already learned. For most LLM tune-ups, this is the default choice.

# Is tuning always the answer?

Tuning is great for teaching models up-to-date skills, styles, behaviors or tasks, but it’s not the only tool – and often not the first one worth using. A better hint can sometimes solve a problem without any training. Similarly, when it makes more sense to search for information on the Internet or in a database at query time, search-assisted generation (RAG) will be a better option, especially when facts are plentiful or change frequently. These approaches are not competitors; in practice, most systems apply them together. It’s worth keeping this in mind before committing to a full tune-up.

# Additional resources

If you want to practice tuning specifically in LoRA, here are some recommended resources:

• PEFT face hugging: The open source standard library for LoRA, QLoRA, quick tuning and more. Start with your documents and repository.
• TRL Hugging Face: Combines with PEFT and gives the finished product SFTTrainer for a supervised tuning loop.
• Laziness: The most beginner-friendly path to LoRA/QLoRA, with free Colab and Kaggle notebooks, ~2x faster training, and significantly less VRAM.
• Axolotl: Once you’re comfortable, a popular configuration-based (YAML) tool for running tuning pipelines without writing a lot of code.
• Original LoRA paper: “LoRA: Adapting Low-Level Large-Language Models.”
• QLoRA article: “QLoRA: Efficient LLM Tuning.”

To make a good first draft, take a petite instructor model (something like Llama 8B, Qwen, or Gemma), open the Unsloth QLoRA notebook, fine-tune it on a few hundred tidy examples of your task, and watch your training loss decrease. Once you do this once, each term in this article will seem much more specific.

Kanwal Mehreen is a machine learning engineer and technical writer with a deep passion for data science and the intersection of artificial intelligence and medicine. She is co-author of the e-book “Maximizing Productivity with ChatGPT”. As a 2022 Google Generation Scholar for APAC, she promotes diversity and academic excellence. She is also recognized as a Teradata Diversity in Tech Scholar, a Mitacs Globalink Research Scholar, and a Harvard WeCode Scholar. Kanwal is a staunch supporter of change and founded FEMCodes to empower women in STEM fields.