Raamesh Balabhadrapatruni

Exploring Transformer Alternatives.
Class of 2027

[Raamesh Balabhadrapatruni @raameshb_]

Somebody get Gemini 3.1 Pro some help man. Seems to have a mental health crisis every day.

[Raamesh Balabhadrapatruni @raameshb_]

@shaurizard safety razors often give a smoother shave with less irritation if you're skilled enough. being skilled enough is hard.

[Raamesh Balabhadrapatruni @raameshb_]

@fouvy @seanphan wait 64GB A100s exist??

[Raamesh Balabhadrapatruni @raameshb_]

@fouvy @seanphan nice flex

[Raamesh Balabhadrapatruni @raameshb_]

@ChaseBrowe32432 if feel like there’s layers to this

[Raamesh Balabhadrapatruni @raameshb_]

@jpschroeder I can't tell if things like this are sarcasm anymore

[Raamesh Balabhadrapatruni @raameshb_]

@tunguz How??

[Raamesh Balabhadrapatruni @raameshb_]

Maybe the reality is that we should be warning that Bill Gates is more dangerous than the New York Times is willing to admit.

https://t.co/d8djJxKYSH

[Raamesh Balabhadrapatruni @raameshb_]

@menhguin This is true.

[Raamesh Balabhadrapatruni @raameshb_]

Time for all the children to learn to switch to Linux. I did when I was an early teen (I didn't like the parental controls), so it's possible.

[Raamesh Balabhadrapatruni @raameshb_]

@dhh Wait is this the latest Apple Silicon?

[Raamesh Balabhadrapatruni @raameshb_]

@prrthh132 Switching back to 1.1.1.1 now.

[Raamesh Balabhadrapatruni @raameshb_]

The customization of Omarchy is a result of it being essentially an arch skin. Linux has far better user experiences than Omarchy; Omarchy makes absolutely no sense for non-technical users, you can't even change your display resolution without learning neovim and how to config file works.

[Raamesh Balabhadrapatruni @raameshb_]

@elonmusk @p_ferragu Pi as an int is diabolical
The explanation makes sense though.
Though, doesn't NASA use 15 digits?

[Raamesh Balabhadrapatruni @raameshb_]

@elonmusk Aren't SF Bay Area views often moreso auth. left?

[Raamesh Balabhadrapatruni @raameshb_]

@_kiyosh1 The distro wars have begun.
No but probably just use Fedora or Ubuntu (I recommend the former). Widely used, so therefore there is a large community for support. Hence also trusted by a lot of people and maintained and funded well, so it's a more stable experience.

[Raamesh Balabhadrapatruni @raameshb_]

@dustedcodes No, your perspective on direction changes on whether you view the yellow/tan part as the "front" or the blue part as the "front".

[Raamesh Balabhadrapatruni @raameshb_]

I agree that the disagreement is quite small.

1. After reading your argument about fp4 pretraining, I believe I definitely overstated the discrete function issue, as fp4 has quite literally 16 possible values; honestly you're underselling this saying "very rough" lol. I wasn't aware that fp4 pretraining has been accomplished, thank you for bringing it up. Though, just to make sure, you're referring to this: https://t.co/8zzGWMovOg , right?

Though I feel as if the issue still persists when you have to tune things like number of layers, batch size, and other similar hyparams that you can't just easily run a gradient through. I may be unaware of some research in this as well though.

2. Yup, I grant this, under your premise of when KV cache becomes a limiting factor, I agree that ES doesn't benefit transformers when compared to backprop based methods due to the reasons you've listed.

3. I have a question here. Are you saying that I'm likely conflating backprop-based methods with GD? If so, yeah, I think you're right; backprop based methods rely on approximating the loss landscape as smooth and thereby also are approximating true GD on continuous functions, just like how ES algorithms effectively approximates GD.

[Raamesh Balabhadrapatruni @raameshb_]

So about the forward pass being non differentiable
ReLU can easily be approximated as differentiable, int valued functions cannot
ReLU is differentiable everywhere except a single point. Same thing with digital floats, the differences are so relatively small at the scales we typically train at that we can just assume they are continuous when they actually are not.
Int valued functions (especially when the range is small) are non differentiable everywhere and cannot easily be approximated as differentiable. Loss in respect to values in int valued functions is also non differentiable and not easily approximated in most cases.

I yield that ES approximates GD, but we seem to agree that ES doesn’t use directly GD and has specific properties that standard GD doesn’t. So as per my logic, ES != GD but rather ES ~ GD. My point in the original message was the address the post, saying that ES has some advantages over GD, not that they were universal or that it is simply a better algorithm.

Also, I appreciate your cordiality and insight. It’s unfortunately rather rare for debates on the internet to be well mannered.

[Raamesh Balabhadrapatruni @raameshb_]

Unless I’m misunderstanding something, no it is not.
It’s an approximation of GD that can deal perfectly well with non-differentiable landscapes.

The number of layers of a neural network cannot be easily approximated as a differentiable function, so therefore you can’t perform GD on it. ES strategies work just fine.

ES also parallelizes better than GD, you can get pop sizes of over a million going if you want. https://t.co/FMHAkb8YCe