catethos

[catethos @catethos]

@trulite007 @yuntiandeng @wtzhang0820 @liliana_hotsko @pyniex @pmphlt @MaximeRivest compile GEPA optimized DSPy workflow into one stand alone neural program huh... so cool wonder how is it compare to fine tuning

[catethos @catethos]

@MaximeRivest probabilistic programming and differential programming are quite well established ... looking forward to see something similar in AI programming :)

[catethos @catethos]

Thinking about writing this up as a blog series:

1. building a tensor language for GPTs
2. writing autodiff from scratch
3. why Rust was not magically faster than Python
4. what optimization actually helped
5. why Burn became the right backend

Would you read this?

[catethos @catethos]

I tried to build a tiny GPT stack from scratch.

Not just the model.

A small tensor language.
A parser + HIR.
A custom autodiff engine.
An interpreter.
Training.
Inference.
KV cache.
Batching.
Fusion.
Benchmarks.

Then I learned why frameworks exist.

[catethos @catethos]

So I optimized.

Dense IDs instead of hash maps.
In-place tensor ops.
KV cache reuse.
Fast paths for sum(lhs * rhs).
Less cloning.
Less allocation.
More precomputed execution plans.

Some changes helped a lot.
Some clever ideas broke correctness immediately.

[catethos @catethos]

Funny result:

my Rust tensor-language interpreter was not automatically faster than the pure Python microGPT.

Rust was fast.
My abstraction stack was not.

Interpreter dispatch, shape metadata, map lookups, small tensor allocation, and generic graph execution ate the advantage.

[catethos @catethos]

Then came autodiff.

Reverse-mode autodiff sounds simple:

run forward,
walk backward,
accumulate gradients.

But once tensors, reductions, softmax, cross entropy, params, targets, and shape checks enter the room, “simple” becomes a very educational swamp.

[catethos @catethos]

So I built Tenscript.

Example vibe:

axis time = 16
axis embed = 16

logits[time, vocab] =
sum embed. x[time, embed] * lm_head[vocab, embed]

The goal was to make GPT architecture readable as math, not buried in framework code.

[catethos @catethos]

The project started from Karpathy microGPT: one tiny Python file where the whole algorithm is visible.

That file is beautiful because nothing is hidden.

But I wanted to ask:

what if the model itself was written as tensor equations, with named axes, and compiled from there?

[catethos @catethos]

@quatermain32 is it possible to use in codex as well ?

[catethos @catethos]

@BlinkDL_AI maybe it can APL for agent lol

[catethos @catethos]

@spacebat @mikehostetler oh nice , any one that catches your eyes ? since Prolog is also homoiconic , it would be interesting to see those languages created for AI is used by AI :)

[catethos @catethos]

@zli11010 is it similar to backtracking ? if so maybe we can get it for free using something like Prolog

[catethos @catethos]

@biantaishabi5 期待用这个库手搓另一个Openclaw

[catethos @catethos]

@jmbollenbacher how about using pythonx as the REPL, https://t.co/2PDrnWwiQ1

[catethos @catethos]

@mikehostetler ah can't wait to see some code :) been trying to do use pythonx to do data analysis in similar way. I am keeping the python process alive so it's kinda look like a REPL to me, is it not making sense ?

[catethos @catethos]

@Jarodxu7 happycapy

[catethos @catethos]

@MaximeRivest to be useful in non data analysis and do "real" literate programming , I really miss the weaving and tangling