serdarml

Aspiring AI researcher, undergrad student @TU_Muenchen

[roon @tszzl]

I think you can make an equally convincing argument that waymo, claude, and the Ford Motor Company are conscious

[serdarml @cs_serdar]

@raghavdotsol @nikbhintade You learn more from failures, and you still have more than half the year left

[serdarml @cs_serdar]

@beaversteever Unpaid intern role haha

[Andrew Gordon Wilson @andrewgwils]

When someone says "we need theory of deep learning", note that probably nothing will "count" as a theory of deep learning unless it is *their* theory, or unless it is speaking a language, and using techniques, that they already have a bias towards.

[serdarml @cs_serdar]

@anshuc @AlexiGlad

[serdarml @cs_serdar]

@yacineMTB R u having visa issues?

[Tilde @tilderesearch]

https://t.co/rmTk8GMkir

[serdarml @cs_serdar]

@tilderesearch Damn u guys are cooking

[serdarml @cs_serdar]

@HeMuyu0327 So this is why value embeddings/projections work so well in nanochat and adjacent repos.

[serdarml @cs_serdar]

@yacineMTB @ThePrimeagen Have u tried open sourcing ur drone stuff

[serdarml @cs_serdar]

@splitbycomma I'm confused abt the word striver, does it mean larper or somebody actually competent and lacking credentials?

[serdarml @cs_serdar]

Why don't we train LLMs with Q heads? Training a head to predict the models loss during pre-training could be useful later, and even if not, it could be good signal to propagate back like MTP. Is there a reason we don't do this?

[serdarml @cs_serdar]

@itsreallyvivek Huh, i guess my work at SAP may be more relevant than i thought regarding the second point

[serdarml @cs_serdar]

The way i think about it is (which may be totally wrong), clean data does not give crazy grad norms. Grad clipping is more for really bad batches that somehow slipped in, like just repeating tokens and other artifacts of web data. So it’s there for “emergencies”, and if your data is clean it won’t do anything at all. But this may be wrong, an easy way to dive deeper into this would be to just manually examine which batches trigger the grad clipping.

[serdarml @cs_serdar]

@luisgnet @creatine_cycle Drugs maybe?

[serdarml @cs_serdar]

@kalomaze Hm i understand now, thanks!

[serdarml @cs_serdar]

Interesting, the whole multiple specialized experts then OPD thing sounded overengineered when i first read it, ill definitely take a deeper look now.

Seems counterintuitive to the idea that generalized models can transfer skills between tasks, and there potentially being a general but sparse “high iq” signal shared between tasks. Seems not AGI pilled, if that makes sense.

[serdarml @cs_serdar]

@kalomaze Are you suggesting this assumption is holding back algo design, or that we should design different tasks that have denser/continuous reward signals?

[serdarml @cs_serdar]

@0xAX For me it was good to get some exposure to every layer of the stack even if not all together in one project. It'd be good to do mini projects for the lower layers even if you build on a higher level for your main projects.