Tech with Mak

AI, coding, software, and whatever’s on my mind.

[Tech with Mak @techNmak]

Everyone is fine-tuning LLMs.
Almost nobody understands what is actually being updated inside the model.

Here are 5 techniques that change how you think about model adaptation, and what each one is actually doing to the weights:

1./ LoRA - Learn the update, not the weights
The pretrained weight W is frozen. Completely untouched.
Instead of updating W directly, two small matrices are trained =>
A ∈ ℝʳˣᵈ and B ∈ ℝᵈˣʳ, where r ≪ d

The weight update is: ΔW = BA Effective weight: W' = W + BA

The entire adaptation happens in a tiny low-rank space. W never changes.

2./ LoRA-FA - What if we freeze even more?
Same structure as LoRA. One change.
A is frozen alongside W. Only B is trained. Effective weight: W' = W + BA (A is fixed)

Half the trainable matrices of LoRA. Same core idea. Fewer parameters.

3./ VeRA - What if the matrices don't need to be learned at all?
This is where it gets interesting.

A and B are both frozen, and randomly initialized. What gets trained are just two tiny scaling vectors =>
b ∈ ℝʳ and d ∈ ℝʳ

Instead of learning the low-rank matrices themselves, VeRA keeps them frozen and learns small scaling vectors that modulate their contribution.

Initialization => b = 0, d = 1

You're not learning matrices. You're learning how to scale them.
One of the most parameter-efficient techniques on this list.

4./ Delta-LoRA - What if W itself learns from the low-rank updates?
This one is fundamentally different.

Unlike standard LoRA, the base weight W is not fully frozen. It is updated through low-rank delta propagation at every step =>
W^(t+1) = W^t + c(B_(t+1)A_(t+1) − B_t A_t)

Where c is a scaling factor.

A and B are trainable. W evolves, but guided entirely by low-rank changes.

5./ LoRA+ - Same structure. Smarter learning rates.
Identical to LoRA, freeze W, train A and B.

One change => B is assigned a larger learning rate than A. η_B > η_A
A ← A − η_A · ∂J/∂A B ← B − η_B · ∂J/∂B

A small optimization change that can make LoRA training more effective.

The core idea running through all five:
You do not always need full fine-tuning to adapt a model.
LoRA updates two matrices.
LoRA-FA updates one.
LoRA+ updates two at different speeds.
Delta-LoRA lets W evolve - guided by low-rank deltas. VeRA updates two vectors.

Same goal. Five different answers to the same question:
=> What is the minimum we actually need to learn?

That is the core idea behind parameter-efficient fine-tuning.
And now you know what is actually happening inside the model.

Happy Learning!!!

[Tech with Mak @techNmak]

Meta open-sourced the design system behind 13,000+ internal apps. It took eight years to build!

It's called Astryx, a 24K-star GitHub repo containing more than 150 accessible React components, ready-made templates, customizable themes, and developer tooling.

Built on React 19 and StyleX, it includes:
→ Tables, forms, navigation, and complex UI components
→ Ready-to-use page templates and patterns
→ Seven customizable themes and dark mode
→ Precompiled CSS without a StyleX build plugin
→ Full TypeScript support
→ CLI tooling designed for AI coding agents

The interesting part is its support for AI-assisted development.

Coding agents can discover component APIs, search templates, and find suitable building blocks through the Astryx CLI instead of generating every interface from scratch.

You can also eject a component's source code for deeper customization or override styles using Tailwind, CSS Modules, or plain CSS.

Eight years of production experience, now available under the MIT license.

GitHub: https://t.co/FjIXDjsOmZ

[Tech with Mak @techNmak]

@devops_nk Cool! I got a new paper to explore this weekend.

[Tech with Mak @techNmak]

This is insane. You can literally type a sentence and watch a Transformer process it.

This interactive explainer runs a GPT-style model in the browser and lets you follow the whole path:

tokenization → embeddings → Q/K/V → attention → MLP → logits → next-token probabilities

You can inspect attention maps, intermediate values, and change temperature, top-k and top-p to see how generation changes.

If Transformers still feel like boxes connected by arrows, this is worth opening.

https://t.co/vkyxOqDVll

[Tech with Mak @techNmak]

@techyoutbe @GlbGPT That's nice!! Keep playing

[Tech with Mak @techNmak]

You can make a wrong AI prediction look 99% certain without changing a single model weight.

This is a surprising consequence of softmax temperature, and it reveals why a neural network's reported confidence should never be confused with its actual accuracy.

Neural network classifiers commonly produce numerical scores called logits. Softmax converts these scores into probabilities, while a temperature parameter controls how strongly the distribution favors the highest-scoring options.

pᵢ = exp(zᵢ/T) / Σⱼ exp(zⱼ/T)

Consider three logits, (2, 1, 0). At temperature T = 2, the highest-scoring class receives 50.65% probability. Reduce the temperature to 0.2, and that probability becomes 99.33%.

The model has learned nothing new. Its input, weights, and predicted class are unchanged. For any positive temperature, scaling every logit by the same factor preserves their ordering, so the model's top-1 classification accuracy remains identical.

This distinction matters because confidence and correctness are separate properties.

In a 2017 ICML paper, researchers showed that temperature scaling fitted on held-out data could substantially improve probability calibration in modern neural networks. The objective is to make reported probabilities better reflect actual correctness rates, rather than simply making predictions appear more certain.

The same mathematics appears in large language models, where temperature influences which token is sampled next. Lower temperatures concentrate probability on higher-scoring tokens, while higher temperatures give alternatives relatively more chance. Different sampled tokens can lead to different generated responses, even though the underlying model parameters remain unchanged.

The important distinction is that changing a probability distribution does not add new knowledge to a model.

A 99% softmax probability tells us how strongly one class is favored under a particular temperature. Whether that probability is a reliable measure of correctness must be established through evaluation.

[Tech with Mak @techNmak]

@devops_nk Hosting, recovery, integrations, and your own branding under one roof.

A pretty compelling direction. ;)

[Tech with Mak @techNmak]

@moltedhq Getting an agent to work is one challenge. Keeping it running, preserving its state, and recovering when things break is another.

Love that you're addressing the less glamorous engineering that actually matters in production.

[Tech with Mak @techNmak]

You can now keep an AI agent running 24/7 for $5.25 a month in hosting. The interesting engineering is everything that keeps it working after the first day.

Molted brings together the runtime, persistent workspace, tools, memory services, and automatic recovery in one environment.

An always-on AI agent needs three distinct layers:

→ The model interprets the task and determines the next step.

→ The runtime executes tools, processes results, and advances the workflow.

→ The environment provides compute, persistent files, connected services, monitoring, and recovery.

These layers become important when an agent needs to work across multiple sessions, execute scheduled tasks, remember previous results, or recover from failures.

Molted provides this infrastructure for agents such as Hermes, OpenClaw, and Mue, with versioned workspaces and automatic repair.

Its Large configuration costs $5.25/month for up to 3 vCPU, 3 GB RAM, and 10 GB NVMe. Model access and usage-based services are billed separately.

I've put together a visual breakdown of how always-on agents work, including the architecture, task execution, memory, recovery, and infrastructure costs.

[Molted @moltedhq]

We just changed the agent hosting game.

Build your own agent product under your own brand. Your own Grok Bot, your own Muse, your own Instinct.

No technical skills needed. Your mom could do it. Really.

Just want hosting? We're the cheapest and fastest.
From $2.88/month, always on.

Everyone makes you choose between cheap and fast. We couldn't pick, so we built both.

Pick Mue, our own runtime on Molted and your first 5 days of hosting are on us.

-> Comment "molted" and I'll send you $10 of add-on credits.

PS: comment "nails" for my nail tech's contact ;)

[Tech with Mak @techNmak]

@NovaXCode @CopilotKit 🔥

[Tech with Mak @techNmak]

OpenBot 1.0 just killed every personal AI coworker

GrokBot, Muse, Dots are all closed, locked to one model, and running on other servers.

@CopilotKit open-sourced the best alternative, and it’s free to run on your own infra.

Here’s what you get:

1/ Run it locally or download the app
3/ Use any model, any agent
4/ Each Bot gets its own computer
5/ Full Generative UI
6/ Active learning: turn real tasks into skills
7/ Every Bot can spin up subagents

It also has their viral open-source versions of Muse, Dots and Duet as selectable "Skins", if you like those UI's!

OpenBot already has 11K+ GitHub stars.

Built on top of CopilotKit building blocks, which are already used in production by many Fortune 500 companies.

OpenBot for Business that takes it next level.

[Atai Barkai @ataiiam]

Introducing OpenBot 1.0 🚀

The customizable, open source AI coworker platform your company can actually own and self-host.

It has everything you'd expect and want:

1. Great UX
2. Computer-use
3. Web search and integrations
4. Routines
5. Security

AND so much more, available only with us:

⚙️ Works on your infra + stack
🔌 Plug in any agent (via AG-UI)
👔 Enterprise-ready
🧠 Continual Learning
🔮 Generative UI

1.0 brings everything we've worked on into one comprehensive product, now available on Mac and Windows!

Try OpenBot now: https://t.co/sPNP6quzgn

Need this for business? Talk to us and we'll support you from day-0.

Follow @0penBot for more.

[Tech with Mak @techNmak]

@devops_nk @CopilotKit Of course!!!

[Tech with Mak @techNmak]

@CopilotKit Here’s where you can check it out: https://t.co/l1oFTJfdiG

[Tech with Mak @techNmak]

Btw, free 50 credits for the first 20 people: https://t.co/hyRyejq4Kg

[Tech with Mak @techNmak]

Lottie AI just made it possible to write UI animations in plain English & edit the actual motion code afterward.

This is a particularly useful development for developers building websites and applications.

Here’s how it works:
→ Lottie AI generates a motion script using Frolic, a programming language developed by @LottieFiles

→ A compiler transforms that script into a standard Lottie animation with vector shapes and motion instructions

→ Every animation comes with editable layers, its underlying code, and a control panel tailored to that animation

→ You can adjust from the control panel directly, or use follow-up prompts to refine the motion

→ It supports interactive animations - describe behaviors like play on click or hover and define different states without manually constructing keyframes

→ Export as dotLottie, Lottie JSON, GIF or video

Best suited for loading indicators, animated icons, buttons, onboarding flows and small interface illustrations.

I created the attached animation with Lottie AI to demonstrate the process, including the generated motion, its underlying code and the export workflow.

What interests me most is the possibility of treating motion as part of the software implementation itself, with source code that remains accessible and editable after generation.

[LottieFiles @LottieFiles]

Day 1 is in motion: Lottie AI is live. 🚀

Describe an animation. Get a real Lottie file you can refine with a follow-up, read as code and drop straight into your product.

Start with 10 free generation.

[Tech with Mak @techNmak]

You can create it too, worth exploring :
https://t.co/NzPtpyuwSv

[Tech with Mak @techNmak]

@LottieFiles This is a really good direction of developer tools. Motion has always been one of those things that's easy to imagine but surprisingly difficult to implement properly. Making it accessible through code and natural language is a meaningful improvement.

Great work!!

[Tech with Mak @techNmak]

Richard Feynman lied to a librarian to borrow a calculus book. He was only about 13 years old.

In the early 1930s, trigonometry was the highest mathematics course offered at Far Rockaway High School in New York. But Feynman wanted to learn calculus, so he turned to a book by J. E. Thompson called Calculus for the Practical Man.

When he tried to borrow it, the librarian questioned why a child wanted such a book. Feynman told her it was for his father.

He took it home and taught himself calculus.

What makes this story remarkable is that his original notebook still exists.

Preserved at the Niels Bohr Library & Archives, the small green notebook contains handwritten explanations, carefully copied diagrams, formulas, and an index he created himself. His notes cover the entire book.

There was also a moment he never forgot.

When his father tried reading the book, he struggled to understand the opening material. Feynman later recalled:

"It was the first time I realized that I could understand what he couldn't understand."

That teenager went on to study at MIT, pursue graduate research at Princeton, work on the Manhattan Project at Los Alamos, and share the 1965 Nobel Prize in Physics for his contributions to quantum electrodynamics.

He also developed his own mathematical notation while young, experimenting with different ways of expressing familiar functions.

Decades later, his lectures and diagrams would help generations of physicists understand difficult ideas.

The book in this photograph is the 1946 second edition. Feynman learned from an earlier edition published in the 1930s.

His surviving notebook is a remarkable reminder of what independent study can look like. He didn't simply collect formulas. He worked through the mathematics carefully enough to make it his own.

[Tech with Mak @techNmak]

@techyoutbe @ParadigmStudy I like the idea!! Great

[Tech with Mak @techNmak]

@NaceAI @ArtificialAnlys Impressive! Enjoyed the paper. I've shared my breakdown :)

[Tech with Mak @techNmak]

AI models can make decisions without generating a single word. The mathematics behind this is fascinating.

Traditional language models generate tokens to produce an answer. Decision models take a different approach. Given a context, a question, and a set of possible choices, they score the options directly and return a probability distribution.

The engineering goes much deeper than classification.

→ Architecture
A language-model backbone processes the input, while a decision head produces scores for the supplied options. Softmax converts these scores into probabilities without autoregressive answer generation.

→ Training
Nace AI's research describes Reinforcement Learning from Adjusted Feedback (RLAF). It combines verified outcomes with probability feedback from a generative reasoning judge. The judge can perform extended reasoning during training, while the decision model learns to produce its answers directly.

→ Calibration
A model predicting 80% confidence should be correct approximately 80% of the time across comparable predictions. High accuracy alone doesn't guarantee reliable confidence.

→ Consistency
Reordering options or changing labels shouldn't change the underlying decision. This matters when AI systems make thousands of decisions across different inputs.

→ Document reliability
Getting individual answers right isn't enough. When a workflow requires 15 correct answers, a single mistake can cause the entire document to fail.

This is what makes Artificial Analysis's new Decision Index v2 interesting. It evaluates decision models across document understanding, reasoning, tool calling, calibration, and consistency.

Nace AI's Drex Max v1.1 has now taken the #1 position among dedicated decision models on the independent Decision Index v2, narrowly ahead of GLiDE v1.2, with both scoring 73 when rounded.

I've put together a visual breakdown of the architecture, RLAF training, probability calibration, and evaluation methods behind these systems.

[Nace AI @NaceAI]

Drex Max v1.1 is #1 on Decision Index @ArtificialAnlys and going open-weight soon!

Both achieving SOTA and Pareto frontier, it’s one of the fastest & most capable Decision models today

Drex API: https://t.co/TELVPW6vAK (~1M free requests)
Paper: https://t.co/qPzUovvwAv

Make better decisions.