Ethan

trying to feel the magic. prev: global research lead at @canva | cofounder at @leonardoai

[Ethan @torchcompiled]

personally I feel like the inflection point was early 2022. The sweet spot where clip-guided diffusion was just taking off, forcing unconditional models to be conditional through strange patchwork of CLIP evaluating slices of the canvas at a time. It was like improv, always trying to riff of mistakes and sitting right at the fine line between interesting and incoherent.

[EPROM @eprombeats]

Image synthesis used to look so good. These are from 2021. I feel like this was an inflection point, and the space has metastasized into something abhorrent today (Grok, etc). Even with no legible representational forms, there was so much possibility in these images.

[Ethan @torchcompiled]

@ptuls Yep that’s why I particularly like the typical sampling paper arguing that we balance information transfer with coherence, length penalties are common and can be hacked just as much, in a way we craft speech to be understood efficiently and making that an objective is desirable

[Ethan @torchcompiled]

The part on “length penalty” has long interested me from perspectives of Kolmogorov/solomonoff encouraging simplicity. Also the paper on typical sampling, arguing humans optimize their speech for information transfer as opposed to choosing tokens of high reward/probability

[lifcc @mylifcc]

为什么现在的顶尖大模型说话越来越“拗口、难懂”了？
NYU 和 UT Austin 刚刚发了一篇非常扎心的心理语言学论文，直接扒开了现代大模型“不好好说人话”的底牌。
研究人员把 Claude Sonnet 5 / Opus 5.5、GPT-5.6、GLM-5、Kimi K2.5 等 9 款前沿模型拉出来做全方位句法解剖，发现了 3 个反直觉真相：
1️⃣ 顶尖 AI 正在滥用 6 种“高认知负荷”句式
相比真实人类写作和同义回译文本，前沿模型极度偏爱：省略 that 的宾语从句、多重名词硬堆叠（如 reader-patience budget）、名词强行作动词（如 calendar it / footnote it）、无生命主语配动作动词，以及超长句法依存距离。这些特征在心理语言学里全是消耗人类短期记忆的“阅读杀手”。
2️⃣ 根源在于“人类大脑”与“推理后训练”的错位
人类语言遵循“均匀信息密度（UID）”——为了照顾听众有限的工作记忆，我们会本能地加入过渡词把信息摊平。而现代大模型经过大量数学、代码和推理（RLVR）后训练，只追求在有限窗口内最大化“信息密度”。
3️⃣ 一个简单的长度惩罚，就能复刻这种“反人类文风”
团队在微调实验中加入长度惩罚（Length Penalty），逼模型少用 Token，结果模型立刻学会了砍掉连接词、堆叠名词，瞬间变成那股熟悉的“高密度 AI 味”。
当模型为了解题效率疯狂压缩 Token 时，你觉得未来的 AI 应该区分“高密度推理模式”和“人类友好沟通模式”吗？

[Ethan @torchcompiled]

One of the places that this first came to mind was the alpha geometry paper which showed that models could solve complex problems but in a brute force manner of many steps when humans often found elegant insights, encouraging brevity may help

[Ethan @torchcompiled]

@sleenyre The MTP-like equivalent I was trying was separate heads for [x0, flow, eps] and/or something like [xt-k, xt-2k, …]

[Ethan @torchcompiled]

have generally addressed this in an unstructured way, handing off original docs on motivations and relevant theories, masking anything around a specific solution direction, this blank slate helps a bunch escaping getting glued to current topic

[Alex Dimakis @AlexGDimakis]

As agents perform AutoResearch they get fixated on their old ideas and do only incremental innovations. We introduce a simple context management trick that forces the agents to forget everything they have done except the best solution they have found so far.
This simple modification in context management allows the agents to get out of their idea basins and come up with more novel solutions.
Closely related to the work 'Fork and Flush' and 'Idea-basins' that @DimitrisPapail posted a few days ago.

[Ethan @torchcompiled]

- most coding languages already vectorize, the matmul is sparse so If anything it’s already more expensive than necessary
- the most interesting thing is designing hardcoded weight networks and optimizing them, but can be done in PyTorch with minimal code all the same

[Ethan @torchcompiled]

I don’t fully get the hype here, it’s neat but

- the observation that if/else trees are neural nets has been around as early as 1940s
- autodiff (differentiable code) is kind of everywhere at this point, writing JAX feels similar

[Cheng Lou @_chenglou]

My dear friends, I’d like to show you my latest toy project

Introducing Handwired: turn your regular JavaScript code into a neural net!
- no training
- exact outputs vs the JS program
- fully differentiable

All runnable in a browser, without GPU

[Ethan @torchcompiled]

@RamonAstudill12 You may appreciate this! We did sampling of Lora’s conditioned on arcface embeddings

https://t.co/4NhoEwEAR0

https://t.co/FuzebexrDI

[Ethan @torchcompiled]

@generatorman_ai I like this perspective, the pretrained model is our any domain of anything and everything. there may be some perfect conditioning or sampling strategy to yield an arbitrary output, one way or another feels that we just need to sharpen around area that matters.

[Ethan @torchcompiled]

It’s really surreal how much RL changed LLMs

[N8 Programs @N8Programs]

Honored to announce my first LessWrong post. It's a weird one, and it may be jarring initially, but I truly believe it is worth a dedicated close-read. It is written in the form of an LLM CoT, but is entirely authored by me. Link:

https://t.co/q3jOymScEO

[Ethan @torchcompiled]

https://t.co/GszOkB2O4F

[Ethan @torchcompiled]

@zhengyaojiang I think there’s a lot of great stuff here borrowing from UT and beyond

[Ethan @torchcompiled]

A few thoughts for looped transformers, both some ideas and reuse of closely-related Universal transformer literature.

- Probably biggest one, and something I contend with a bit. We observe (Remarkable Robustness of LLMs: Stages of Inference) the last few transformer layers in transformers have a very specific role in preparing hidden states for readout at LLM head, acting more as a filter/somewhat reducing richness of representations previously built-up. It could be worth setting aside N transformer layers with entirely unique weights on the output side separate from the loop
- This may also be relevant to having a couple of unique input layers outside the loop as well.
- norm scale/bias params can easily be unique per layer per loop, this is cheap possible additional expression
- on loops past the first, weights can be made slightly unique/adapted by using LoRAs, this has been done with UT
- MoEUT effectively added MoE to universal transformers, doing the same for looped transformers may allow more computtional pathways across easy loop
- the x0 residual addition might be worth some scrutiny, for two reasons. firstly, its a static addition, depending on how helpful its contribution and how much we'd like to weigh it, the residual stream may have to upsize/downsize its norm rather than having control over x0's magnitude. it could be worth either some form of learnable scaling or projection of x0. Secondly, x0 should effectively represent the preceding token. It feels like it'd be more helpful to have a somewhat processed feature, like taking a representation from the middle of the network on the first loop or similar?
- universal transformer originally suggested timestep-style conditioning to tell model which loop its on, which may nicely adapt computation. Could imagine an embedding table per-loop/layer condition or a more complex function
- Universal transformer originally proposed early halting, which I think is what Elastic Looped Transformers is going after
- Experimenting with some weights not being shared across loops, or partially shared. i.e. same MLP up for intervals of 2 loops. Some operations might get away with parameter reuse more-so than others

[Ethan @torchcompiled]

@techno_yoda My baby is not pooping all over the place it’s just following an epsilon greedy policy with really high epsilon

[Ethan @torchcompiled]

I think about this a bunch. We don’t necessarily need to follow biology but I imagine something like:
- pretraining = predictive coding/world calibration
- chat finetuning = social mimicry
- RL = reward following, goal behaviors

I’d think biology is doing all 3 of these simultaneously lifelong, though perhaps with different plasticity/learning rate across life.

Though the part that interests me is that the latter 2 are either biased distributions/subsets of the world and of course reduce variance, and perhaps there is some form of regularization in balancing 1/2 with 3rd, though KL anchors and similar do sort of cover this

[JJ @JosephJacks_]

Pre-Training / Post-Training divide does not exist in biology.

Current AI works almost nothing like biology.

[Ethan @torchcompiled]

@MichaelLee04 @industriaalist The resnet paper went this high and I think gains asymptoted/saturated granted that was image classification and quite different in a lot of ways now

[Ethan @torchcompiled]

@industriaalist My running thought was that depth was not given proper focus due to lack of parallelizability and instead we moved towards width/MoE to saturate gpu, and looped transformers etc for depth gains at least without additional params

[Ethan @torchcompiled]

We’ve reinvented maskgit from first principles

[Anshu @anshuc]

Jev can draw pictures by predicting every pixel in parallel

[Ethan @torchcompiled]

Post

[Ethan @torchcompiled]

New post: Pangram and other AI-text-detection overall serves a good purpose, but understated fallibility and misplaced usage leads to more chaos.

Here, I review research on AI-text-detection along with their claims and company positioning

[Ethan @torchcompiled]

Don’t make me tap the sign

[David Sacks @DavidSacks]

@CharlieBull0ck Now I know these AI detectors are bogus. I run all my posts through Grok to fact-check them and suggest line edits but tell it not to change my writing style. That’s why my posts have edge. AI can’t cook like I do.

[Ethan @torchcompiled]

@EigenGender I wrote pretty extensively on this both at the technical level and the business incentives

[Ethan @torchcompiled]

New post: Pangram and other AI-text-detection overall serves a good purpose, but understated fallibility and misplaced usage leads to more chaos.

Here, I review research on AI-text-detection along with their claims and company positioning