📊 Correlation is not causation—and not every trend is significant.
More study hours may seem linked to higher scores, but statistical testing matters. Here, r = 0.234, which is not significant at the 5% level, so we fail to reject H₀.
#Statistics#DataScience#Correlation#STEM
Everyone is fine-tuning LLMs. Almost nobody understands what is actually being updated inside the model.
That distinction matters because LoRA, QLoRA, LoRA-FA, VeRA, Delta-LoRA and LoRA+ are usually discussed as if they were small variations of the same method. They are not. Some reduce trainable parameters, some reduce activation memory, some reduce the memory occupied by the frozen model, and some change the way the adapter itself is optimized.
I'm actually sharing 6 techniques.
1/ LoRA
Suppose a layer contains a pretrained weight matrix W. Full fine-tuning would update W directly. LoRA leaves W frozen and represents the update using two much smaller matrices, A and B, so that ΔW = BA.
For a square d × d weight matrix, full fine-tuning has d² parameters available to update. A rank-r LoRA adapter has roughly 2dr trainable parameters instead, where r is normally much smaller than d. This is the basic reason LoRA can adapt very large models without training every parameter in them.
2/ LoRA-FA
Standard LoRA trains both A and B. LoRA-FA freezes A and trains only B.
There is a useful reason for doing this beyond simply reducing the number of trainable parameters. Computing the gradient for A requires retaining the layer input activation. If A is fixed, that gradient is no longer required, which allows LoRA-FA to reduce activation memory as the LoRA rank grows.
3/ QLoRA
QLoRA attacks a different part of the memory problem. LoRA makes the adapter small, but the frozen base model can still occupy tens of gigabytes.
QLoRA keeps the base model frozen in 4-bit form and trains LoRA adapters through it. The original work used NF4, double quantization and paged optimizers, and demonstrated fine-tuning a 65B model on a single 48GB GPU.
This is an important distinction: QLoRA is not simply "LoRA with smaller adapters." The large memory saving comes from quantizing the frozen base model.
4/ VeRA
VeRA reduces the adapter itself further. Instead of learning a separate A and B for every adapted layer, it uses frozen random low-rank matrices that can be shared across layers, while learning much smaller scaling vectors.
The low-rank basis is therefore fixed. Training mainly determines how strongly different parts of that basis should contribute. This is why VeRA can use considerably fewer trainable parameters than ordinary LoRA.
5/ Delta-LoRA
Ordinary LoRA treats W as fixed throughout training. Delta-LoRA relaxes that constraint.
A and B are still trained, but the change in their product from one training step to the next is also used to update W. The base weights can therefore move without maintaining the ordinary gradients and optimizer states that full fine-tuning would require for W.
6/ LoRA+
LoRA+ does not introduce another adapter structure. It keeps W frozen and still trains A and B.
Its change is in the optimizer: A and B use different learning rates, with B receiving a larger rate. The motivation is that the two LoRA matrices do not behave identically during optimization, so forcing them to use the same learning rate is not necessarily the best choice.
Once these are separated by what they actually change, the family becomes much easier to understand.
LoRA reduces the number of weights being trained. LoRA-FA also targets activation memory. QLoRA compresses the frozen base model. VeRA reduces the learned adapter parameters further. Delta-LoRA allows the pretrained weights themselves to evolve through low-rank changes, while LoRA+ keeps the LoRA structure and changes its optimization.
That is really what PEFT is about => deciding which parts of a very large model actually need to move during adaptation, and which parts can remain fixed.
If I need to pick ONLY FIVE courses to learn AI & ML from scratch, I'll pick:
❯ CS221 - Artificial Intelligence
❯ CS229 - Machine Learning
❯ CS230 - DL
❯ CS234 - RL
❯ CS336 - LLM
These courses could cost $100K. But Stanford is offering for FREE on YouTube:
A = UΣVᵀ
Every real matrix (square or rectangular, full-rank or singular) factors into a pair of orthogonal matrices flanking a rectangular diagonal of nonnegative singular values. The left and right singular vectors rotate the domain and codomain so that the linear map becomes a pure stretch along aligned axes; the singular values themselves are those stretches.
When the values decay rapidly the map is essentially low-dimensional, and the partial sum formed by the first k terms is the nearest rank-k matrix in every unitarily invariant norm. Beltrami obtained the square case in 1873, Jordan independently a year later.
One factorization therefore exposes geometry, numerical rank, and the most economical compressions of the data the matrix represents.
The map
[ xₙ₊₁ ] [ 2 1 ] [ xₙ ]
[ yₙ₊₁ ] = [ 1 1 ] [ yₙ ] (mod 1)
has determinant 1, so it preserves area while remaining invertible. Reduced modulo 1 it becomes an automorphism of the torus: opposite edges of the unit square are identified, the linear stretch is folded back, and every region keeps its measure.
The eigenvalues (3 ± √5)/2 drive exponential separation along one irrational eigendirection and contraction along the other. Vladimir Arnold illustrated the process in 1967 with a sketch of a cat; after only a few iterations the figure dissolves into stripes, a deterministic scramble that never changes total area.