📢 Postdoc opening! 📢
@n_keriven and I are hiring a postdoc to work on the theoretical analysis of large-context Transformers (infinite-token limits, depth, looped Transformers, reasoning traces).
📍Location: Rennes (IRISA) or Nice (LJAD), France
The alpha version of my new book "Optimal Transport
for Machine Learners" is out, with in particular an online version with interactive figures
https://t.co/xEdZpMXgjx
Of course you can always use complex step approximation if you don't like roundoff error and you have stable complex arithmetic :)
https://t.co/47PJEflEAC
Complex step approximation is a numerical method to approximate the derivative from a single function evaluation using complex arithmetic. It is some kind of “poor man” automatic differentiation. https://t.co/A4wc9JmJci
A quick remark:
When you use centered finite difference instead of forward or backward difference, the truncation error decreases faster, but I never realized that the roundoff error grows before!
🧵 New preprint! "Towards Understanding Steering Strength" with @MagamedTm and D. Garreau
Activation steering is a popular way to control LLM behavior at inference. But how much should you steer? We provide the first theoretical analysis of the steering strength α.
1/7
TL;DR: Steering strength α has precise, predictable effects on LLM outputs. Understanding these laws may help design better steering strategies.
Paper: https://t.co/qwZAHqoN6L
Code: https://t.co/W0IbUQ7AZe
7/7
🧵 New preprint! "Towards Understanding Steering Strength" with @MagamedTm and D. Garreau
Activation steering is a popular way to control LLM behavior at inference. But how much should you steer? We provide the first theoretical analysis of the steering strength α.
1/7