Theoretical Guide: Fine-Tuning LLMs

Theoretical Guide: Fine-Tuning LLMs

1. Fine-Tuning Fundamentals

1.1 What Is Fine-Tuning and Why Do We Need It?

Fine-tuning is the process of adapting a pretrained model to a specific task or domain. Think of a base model as someone who graduated from university with broad general knowledge. Fine-tuning is like the professional specialization that turns them into an expert in a specific area.

Why not use the base model directly?

Base models are trained on general text from the internet, Wikipedia, books, and so on. This means they do not know your industry's specific terminology, do not understand the nuances of your use case, may generate generic rather than specialized answers, and do not follow the format or style your business requires.

A concrete example for collections:

Given “The customer says they already paid,” a base model might respond with generic information about payment disputes. A model fine-tuned on your operational data would know to check the system, offer to send a receipt, and follow your company's specific protocol.

1.2 The Adaptation Spectrum

There are different levels of adaptation, ordered from the least to the most intervention.

Prompt Engineering does not modify the model; it only optimizes how we ask for things. It is fast and has no training cost, but its capacity for specialization is limited.

Few-Shot Learning includes examples in the prompt to guide the model. It also does not modify the model, but consumes tokens from the available context.

Fine-Tuning modifies the model's weights with specific data. It requires data and compute, but achieves deep and permanent specialization.

Pretraining from scratch trains a completely new model. It requires massive resources and is rarely necessary.

For most enterprise use cases, fine-tuning is the optimal balance between cost and benefit.

2. Fine-Tuning Techniques

2.1 Full Fine-Tuning (Complete Training)

Concept:

In full fine-tuning, we update ALL of the model's parameters during training. If the model has 7 billion parameters, all 7 billion are modified.

The mathematical process:

During training, for each example we calculate the loss (how far the prediction is from the target), then calculate gradients (how each parameter contributes to the error), and finally update all parameters in the direction that reduces the error.

The update follows this formula: θ_nuevo = θ_anterior - learning_rate × gradiente

Where θ represents all of the model's parameters.

Advantages:

The model can adapt completely to your domain, achieves the maximum potential for specialization, and has no restrictions on what it can learn.

Disadvantages:

Memory requirements are enormous. For a 7B-parameter model, you need approximately 28 GB just for model weights in FP32, plus 28 GB for gradients, plus 56 GB for Adam optimizer states. This totals around 112 GB of VRAM, which is impractical for most people.

There is also a high risk of “catastrophic forgetting,” in which the model forgets general capabilities by specializing too much. Training is slow and expensive, and you need to save a complete copy of the model for every version.

When to Use Full Fine-Tuning:

Use it when you have significant compute resources (multiple A100 GPUs), need maximum domain adaptation, the domain is very different from the original training, or want to create an internal base model for the organization.

2.2 LoRA (Low-Rank Adaptation)

The Intuition:

LoRA starts from a key research observation: when you fine-tune a model, the weight changes have “low intrinsic rank,” meaning the new information the model needs to learn can be represented in a much lower-dimensional space.

Imagine you have a 4096 × 4096 matrix (16 million parameters). LoRA says that the changes needed to adapt that matrix can be approximated with two much smaller matrices: one of 4096 × 16 and another of 16 × 4096 (only 131 thousand parameters, less than 1%).

The Mathematics:

In the original model, a linear layer computes: h = W₀x

Where W₀ is the pretrained weight matrix of dimension d × k.

In full fine-tuning, we would update W₀ directly: h = (W₀ + ΔW)x

LoRA decomposes ΔW into two low-rank matrices: ΔW = BA

Where B has dimension d × r and A has dimension r × k, with r << min(d, k).

The forward pass becomes: h = W₀x + BAx

The parameter r (rank):

Rank r controls adaptation capacity. A lower r means fewer parameters and faster training, but less expressive capacity. A higher r enables more complex changes but requires more memory.

Typical values range from r=8 for simple tasks such as classification to r=64 for complex adaptations such as a complete domain change. The value r=16 is a good general starting point.

Which layers should LoRA be applied to?

Not all Transformer layers contribute equally to adaptation.

The Query and Value layers (q_proj, v_proj) in attention are the most important and are almost always included. The Key and Output layers (k_proj, o_proj) are secondary; include them if you have the parameter budget. Feed-Forward layers (mlp) capture factual knowledge and are useful for domain adaptation. Embedding layers are rarely modified with LoRA.

A common configuration is to apply LoRA to q_proj, v_proj, k_proj, o_proj with r=16.

Advantages of LoRA:

The memory reduction is dramatic, typically 10-100x fewer trainable parameters. The base model remains intact, avoiding catastrophic forgetting. You can have multiple LoRA adapters for different tasks. It is easy to merge the adapter into the base model for deployment.

Disadvantages of LoRA:

There is a performance ceiling, since very complex tasks may require full fine-tuning. Experimentation is required to find the optimal rank. Not all frameworks support it equally well.

2.3 QLoRA (Quantized LoRA)

The Problem It Solves:

LoRA reduces trainable parameters, but you still need to load the complete base model into memory for the forward pass. A 7B model in FP16 still needs 14 GB of VRAM.

QLoRA combines base model quantization with LoRA to dramatically reduce memory requirements.

Key Components of QLoRA:

The first component is 4-bit quantization (NF4). Instead of FP16 (16 bits per parameter), QLoRA uses a special format called NormalFloat 4-bit. This format is optimized for the typical weight distribution in neural networks, which tends to be normal (Gaussian). A 7B model goes from 14 GB to approximately 3.5 GB.

The second component is Double Quantization. Quantization parameters are also quantized, saving additional memory (approximately 0.5 GB for a 7B model).

The third component is Paged Optimizers. It uses paged CPU memory to handle memory spikes during training, enabling training with less VRAM.

The fourth component is LoRA on a quantized model. Only the LoRA adapters are in full precision (FP16 or BF16), and they are the only ones trained.

The Computation Flow:

During the forward pass, quantized weights are dequantized on the fly for computation. The result is combined with the output of the LoRA adapters. During the backward pass, gradients are calculated only for LoRA parameters. The quantized base model is not modified.

Performance Impact:

4-bit quantization introduces some error, but studies show that the quality loss is minimal (1-2% on benchmarks) compared with LoRA in FP16. For most practical applications, the difference is imperceptible.

Hardware Requirements:

With QLoRA, you can fine-tune a 7B model with 6-8 GB of VRAM (an RTX 3060 or a Colab T4). A 13B model needs approximately 10-12 GB. A 70B model requires approximately 48 GB, which is feasible with an A100.

This democratizes fine-tuning, enabling teams without massive infrastructure to adapt large models.

2.4 Technique Comparison

General Recommendation:

For most enterprise use cases, QLoRA is the best option. It offers the best balance of quality, cost, and accessibility. Reserve full fine-tuning for cases where QLoRA does not achieve the required performance and you have the resources available.

3. Preparing Datasets for Training

3.1 The Importance of the Dataset

The dataset is the most critical factor in fine-tuning. A mediocre model with excellent data outperforms an excellent model with mediocre data. The “garbage in, garbage out” rule applies strongly.

3.2 Dataset Formats

Instruction-Response Format (the most common):

This format has three components. The instruction or prompt is what the user asks for. The optional input provides additional context. The response or output is what the model should generate.

An example for intent classification would use the instruction “Classify the customer's intent into one of these categories: payment, complaint, negotiation, evasion,” the input “Customer: ‘Tomorrow I will deposit the 2000 soles I owe,’” and the output “payment.”

Conversational Format (for chatbots):

This format consists of message turns with system, user, and assistant roles. It is ideal for maintaining conversational context and training specific chat behaviors.

Completion Format (for generation):

This format has only a prompt and a completion. It is simpler but less structured. Useful for pure generation tasks without explicit instructions.

3.3 Amount of Data Needed

General Rules:

For simple classification with few categories, between 200 and 500 examples per category is usually sufficient. For complex classification with many categories or subtle nuances, you need between 500 and 2000 examples per category. For generation in a specific style, at least 1000 high-quality examples. For technical domain adaptation, between 5000 and 10000 diverse examples.

The Learning Curve:

Improvement is not linear with more data. Typically, there is a dramatic improvement with the first 500-1000 examples, then diminishing improvements up to 5000-10000, followed by a plateau where more data helps only marginally. It is better to have 1000 high-quality examples than 10000 low-quality ones.

3.4 Dataset Quality

Quality Criteria:

The accuracy criterion means that labels and answers must be correct. A single error can teach the model the wrong behavior.

The consistency criterion means similar examples should have similar answers. Inconsistency confuses the model during training.

The diversity criterion requires covering the variety of cases the model will encounter, not just easy or common cases.

The representativeness criterion means the dataset's distribution should reflect the actual usage distribution.

The completeness criterion means answers must be complete, not truncated or partial.

Common Mistakes to Avoid:

The extreme imbalance problem occurs when 90% of your examples belong to one class and 10% to others, biasing the model toward the majority class. The solution is balancing through oversampling, undersampling, or class weights.

The noisy labels are labeling errors that confuse learning. Even a 5% labeling error rate can significantly degrade performance.

The data leakage problem occurs when information from the test set contaminates training. Always split the data BEFORE any preprocessing.

The problem of insufficient difficult examples occurs when you include only clear cases and the model does not learn to handle ambiguity. Deliberately include edge cases and ambiguous cases.

3.5 Practical Preparation

Preparation Pipeline:

The first step is collection where you define data sources, whether conversation logs, support tickets, or synthetic data.

The second step is cleaning where you remove duplicates, correct obvious errors, and normalize formats.

The third step is labeling where you define clear annotation guidelines and use multiple annotators if possible (inter-annotator agreement).

The fourth step is validation where you manually review a sample and calculate dataset quality metrics.

The fifth step is splitting where you separate into train (80%), validation (10%), and test (10%). Stratification maintains class proportions in each split.

The sixth step is formatting where you convert to the format required by your training framework.

Stratified Split Example for Collections:

If your dataset has 40% payment, 25% complaint, 20% negotiation, and 15% evasion, each split should maintain approximately these proportions. Do not accidentally leave all “evasion” examples in the test set.

3.6 Data Augmentation for NLP

Applicable Techniques:

The paraphrasing technique uses a model to generate variations of the same message. For example, “I will pay tomorrow” can become “Tomorrow I will make the payment” or “I will make the deposit tomorrow.”

The back-translation technique translates into another language and back, generating natural variations.

The synonym substitution technique replaces words with synonyms while preserving meaning.

The synthetic data approach uses a larger LLM to generate additional examples. Useful, but be careful about amplifying biases in the generating model.

When to Use Augmentation:

Use it when you have few samples in a category, for edge cases that are difficult to collect naturally, and to improve robustness to input variations. Avoid it if you already have enough real data and for categories where synthetic variation could introduce noise.

4. Key Hyperparameters

4.1 Learning Rate

The Most Important:

The learning rate controls how large weight update steps are. It is the most critical hyperparameter and the one with the greatest impact on training.

If it is too high, training diverges, loss oscillates or increases, and the model “jumps” over the optimal minimum. If it is too low, training is very slow, may get stuck in local minima, and never achieves good performance.

Typical Values:

For full fine-tuning, values between 1e-5 and 5e-5 are used. For LoRA/QLoRA, values between 1e-4 and 3e-4 are used, higher because there are fewer parameters to move. The value 2e-4 is a good starting point for LoRA.

Learning Rate Schedulers:

The constant scheduler keeps the LR fixed throughout training. It is simple but not optimal.

The linear decay scheduler starts high and decreases linearly to zero. It allows initial exploration and final refinement.

The cosine scheduler follows a cosine curve, decreasing smoothly. Very popular, it works well in practice.

The warmup + decay scheduler starts low, rises gradually during the first steps (warmup), then decreases. It stabilizes initial training. A warmup of 3-10% of total steps is recommended.

4.2 Batch Size

Concept:

Batch size determines how many examples you process before updating weights. It affects both training stability and memory requirements.

Trade-offs:

A large batch provides more stable gradients (averaging more examples) and better hardware utilization (parallelism), but requires more memory. A small batch means more gradient noise (which can help escape local minima) and less memory required, but potentially less stable training.

Gradient Accumulation:

If you do not have enough memory for a large batch size, simulate it with gradient accumulation. Instead of updating weights after 1 batch of 32, accumulate gradients from 8 batches of 4 and then update. Mathematically equivalent, but it fits in memory.

Effective batch size is calculated as: batch_size × gradient_accumulation_steps × num_gpus

Recommended Values:

Typical effective batch size is between 16 and 64 for fine-tuning. For LoRA/QLoRA, use the maximum that fits in memory with gradient accumulation.

4.3 Epochs

Definition:

An epoch is a complete pass through the entire training dataset. The number of epochs determines how many times the model sees each example.

How Many Epochs?

For fine-tuning LLMs, 1-5 epochs are typically sufficient. More can cause overfitting. The optimum depends on dataset size: with small datasets (fewer than 1000 examples), use 3-5 epochs; with medium datasets (1000-10000 examples), use 2-3 epochs; and with large datasets (more than 10000 examples), use 1-2 epochs.

Early Stopping:

Do not set epochs arbitrarily. Use early stopping to stop training when validation loss stops improving. Typical patience is 2-3 evaluations without improvement.

4.4 LoRA-Specific Parameters

Rank (r):

Controls the dimensionality of adaptation. With r=8, you have high parameter efficiency but lower capacity. r=16 offers a good general balance. With r=32 or r=64, there is greater capacity but more parameters. Start with r=16 and adjust based on results.

Alpha (lora_alpha):

Scaling factor for LoRA weights. The effective update is: ΔW = (alpha/r) × BA. Common values are alpha equal to r (1x scaling), alpha equal to 2r (2x scaling, more aggressive), or alpha equal to 16 with r=16 as a common default.

Dropout:

Applied to LoRA layers during training. Helps prevent overfitting. Typical values are between 0.05 and 0.1. With small datasets, use higher values (0.1-0.2).

Target Modules:

Which layers receive LoRA adapters. The effective minimum is q_proj and v_proj. The recommended configuration also includes k_proj and o_proj. For more capacity, add MLP layers (gate_proj, up_proj, down_proj).

4.5 Other Hyperparameters

Weight Decay:

L2 regularization that penalizes large weights. Helps prevent overfitting. Typical values are between 0.01 and 0.1.

Max Sequence Length:

Maximum input sequence length. It affects memory quadratically because of attention. Use the minimum needed for your task. For classification, 256-512 is usually sufficient. For generation, 1024-2048 may be needed.

Warmup Ratio:

Proportion of initial steps with an increasing learning rate. Stabilizes early training. Typical values are between 0.03 and 0.1.

4.6 Hyperparameter Search Strategy

Practical Approach:

Start with reasonable defaults based on your technique (LoRA, QLoRA) and model size. Run a complete initial training run to establish a baseline. Adjust the learning rate first, since it has the greatest impact. Then adjust epochs and batch size. Finally, optimize LoRA-specific parameters if applicable.

Recommended Starting Values for QLoRA on a 7B Model:

Learning rate of 2e-4, batch size of 4 with gradient accumulation of 4 for an effective batch size of 16, 3 epochs with early stopping, LoRA rank of 16, LoRA alpha of 32, LoRA dropout of 0.05, and target modules q_proj, v_proj, k_proj, o_proj.

5. Evaluation and Overfitting Prevention

5.1 What Is Overfitting?

Overfitting occurs when the model memorizes training data instead of learning generalizable patterns. The classic symptom is falling training loss while validation loss rises.

In the context of LLMs, overfitting can manifest as answers that reproduce training examples verbatim, inability to handle variations of previously seen inputs, degradation of the model's general capabilities (catastrophic forgetting), and excessive confidence in incorrect predictions.

5.2 Evaluation Metrics

For Classification:

The Accuracy metric is the proportion of correct predictions. It is simple but misleading with imbalanced classes.

Precision answers the question “Of those I predicted as positive, how many actually are?” It is calculated as TP/(TP+FP). It matters when the cost of false positives is high.

Recall answers the question “Of those that are actually positive, how many did I identify?” It is calculated as TP/(TP+FN). It matters when the cost of false negatives is high.

F1-Score is the harmonic mean of precision and recall, calculated as 2 × (P×R)/(P+R). It balances both metrics.

Macro F1 averages F1 across classes, giving equal weight to every class.

Weighted F1 averages F1 weighted by the number of examples in each class.

For Generation:

Perplexity measures how “surprised” the model is by the text. It is calculated as exp(loss). A lower value is better and indicates that the model predicts the next token well.

BLEU/ROUGE measure overlap with reference answers. Useful, but limited because they do not capture semantic equivalence.

BERTScore uses embeddings to measure semantic similarity. Better than BLEU at capturing paraphrases.

Human Evaluation remains the gold standard for generation quality. It considers clarity, relevance, factuality, and usefulness.

5.3 Evaluation Strategy

Data Splitting:

The Training Set (80%) is used to train the model. The Validation Set (10%) is used to monitor training, adjust hyperparameters, and decide on early stopping. The Test Set (10%) is the final evaluation, must remain untouched until the end, and simulates production performance.

The Importance of the Test Set:

Never use the test set to make training decisions. If you do, it is no longer a valid estimate of performance on new data. It is easy to fall into “overfitting to the test set” by adjusting hyperparameters until test results improve.

Cross-Validation for Small Datasets:

With fewer than 1000 examples, use k-fold cross-validation (typically k=5). You train k models, each with a different fold as validation. Report the mean and standard deviation of metrics.

5.4 Signs of Overfitting

In Training Curves:

Mild overfitting is observed when validation loss flattens while training loss keeps falling.

Moderate overfitting appears when validation loss starts rising while training loss falls.

Severe overfitting manifests as a large gap between training and validation, with training loss almost at zero.

In Model Behavior:

There is memorization when it generates exact answers from the dataset for similar inputs. There is a lack of generalization when it fails on minor input variations. There is a loss of capabilities when tasks the base model handled well now fail. There is overconfidence when it assigns high probability to incorrect predictions.

5.5 Prevention Techniques

Regularization:

The Weight Decay regularization method adds an L2 penalty to weights, preventing them from growing too large. A value of 0.01-0.1 is used.

The Dropout technique randomly switches neurons off during training, forcing redundancy. For LoRA, use 0.05-0.1.

Early Stopping:

Monitor validation loss. Stop when it does not improve for N consecutive evaluations, where N is typically 2-3. Save the checkpoint with the best validation result, not the last one.

Reducing Model Capacity:

For LoRA, use a lower rank. For full fine-tuning, freeze more layers. Fewer trainable parameters means less capacity to memorize.

Data Augmentation:

More variety in the data makes memorization harder. Even simple augmentation such as paraphrasing helps.

Reducing Epochs:

Overfitting gets worse with more passes through the data. Often, 2-3 epochs are sufficient for LLMs.

5.6 Evaluating Catastrophic Forgetting

The Problem:

During fine-tuning, the model can “forget” capabilities it previously had. This is more severe with full fine-tuning than with LoRA.

How to Detect It:

Maintain an evaluation set of general tasks. Compare the base model's performance against the fine-tuned model on these tasks. Significant degradation indicates forgetting.

Examples of General Tasks to Evaluate:

Basic reading comprehension, simple arithmetic, general factual knowledge, following basic instructions, and simple translation.

Mitigation:

LoRA/QLoRA help because they do not modify the base model. You can also mix general-task data into fine-tuning (typically 5-10% of the dataset), or use regularization to keep weights close to their original values.

5.7 Evaluation in Production

It Does Not End with the Test Set:

Real performance may differ from the test set because of a different distribution of real data, users interacting in unanticipated ways, and temporal drift (language and patterns change).

Continuous Monitoring:

Log predictions and metrics in production. Implement feedback loops where users can flag errors. Review samples manually on a regular basis. Set up alerts for metric degradation.

A/B Testing:

Before replacing a model, run an A/B test. Compare business metrics, not just ML metrics. The technically “better” model is not always better for the business.

6. Summary: Fine-Tuning Checklist

Before You Start

• First, is fine-tuning necessary? Have you tried prompt engineering and few-shot?
• Second, do you have enough quality data? A minimum of 500-1000 examples.
• Third, is the data correctly labeled and consistent?
• Fourth, do you have the necessary hardware? For QLoRA on 7B, you need at least 8GB VRAM.

During Training

• First, monitor train loss AND validation loss.
• Second, use early stopping with patience of 2-3.
• Third, save checkpoints regularly.
• Fourth, verify that there is no data leakage.

After Training

• First, evaluate on a test set that has never been used before.
• Second, test manually with varied real examples.
• Third, compare against a baseline (base model, simple rules).
• Fourth, evaluate catastrophic forgetting on general tasks.
• Fifth, document hyperparameters and results for reproducibility.

7. Key Points for Your Training Sessions

Key Message on Techniques:

QLoRA democratized fine-tuning. You no longer need a supercomputer. With a gaming GPU, you can adapt 7B-parameter models to your specific domain.

Key Message on Data:

Dataset quality matters more than the fine-tuning technique. Invest time in clean, well-labeled, diverse data. “Garbage in, garbage out” applies strongly.

Key Message on Evaluation:

Overfitting is the main enemy of fine-tuning. Monitor obsessively, use early stopping, and never trust training metrics alone. Real performance is measured on data the model has never seen.

For Executives:

Fine-tuning enables models specialized in the company's domain with a reasonable investment of resources. ROI comes from greater accuracy on specific tasks and reduced API costs through using your own models.

Next chapter: Everything You Need to Know About RAGs