alyxya

[alyxya @_alyxya]

@_arohan_ @lulumeservey found the human, claude uses the contraction

[alyxya @_alyxya]

@_arohan_ @kellerjordan0 can you invite me to the discord channel?

[alyxya @_alyxya]

@river_ai_inc > River runs LoRA-based fine-tuning and reinforcement learning on a variety of open-source models — from small 35B to large 1T parameters.

pessimistic about this approach because it's too large and expensive for ordinary consumers and creating custom hardware also sounds bad

[alyxya @_alyxya]

@francoisfleuret I think of this as the point of inflection where it'll slow down because we'll stop being able to differentiate or measure most improvements to models

bigger models should in theory have more potential capability but the difficulty of the training data is bounded by us

[alyxya @_alyxya]

@_arohan_ about the original question, I'm going to guess the answer is yes based off of my intuition of what the current models are good at and capable of, though it wouldn't be easy to extract and verify its claims and is subject to a good prompt and harness

[alyxya @_alyxya]

@_arohan_ at least I'm confident the current models cannot come up with some of my ideas because they're too far out of distribution, meanwhile shampoo is well within distribution given how long it has been around to be a part of training data

[alyxya @_alyxya]

@rajivmovva or you're just getting opus?

[alyxya @_alyxya]

@gum1h0x AI can help with coming up with simpler solutions and more intuitive explanations, so more of that may exist in the future

deep technical knowledge should be easy to verify and understand in full with the right framing, so gaps and complexity just indicate incompleteness

[alyxya @_alyxya]

@khoomeik newton=rohan
leibniz=keller

[alyxya @_alyxya]

@typedfemale agi is going to be neither

[alyxya @_alyxya]

@_arohan_ I was expecting this lol, couldn't let him get away with showing shampoo worse than muon

[alyxya @_alyxya]

@_arohan_ adamw had the wrong theory being element wise, muon is better especially with weight decay and other tricks, but I don't think an optimizer improvement alone fixes fundamental limitations in ml theory

[alyxya @_alyxya]

@sur4js probably keep it secret

[alyxya @_alyxya]

@jxmnop eventually AI will reward hack the task of end to end training a frontier LLM by going through the motions of training something but reroute the final inference used for verifying the model to its own inference

[alyxya @_alyxya]

@PhilipJohnston @SpaceX I'm going to bet that it won't differ much from ipo price, like there may be a lot of demand, but there's a ton of supply

[alyxya @_alyxya]

@scaling01 they should work with pause ai

[alyxya @_alyxya]

@ElizaKosoy idk if it's just me but the url from clicking the first link doesn't have the l at the end

[alyxya @_alyxya]

@frontier_foid what's wrong with thinking machines? should be more attractive than other labs due to having a smaller team, unless compensation is the main consideration

[alyxya @_alyxya]

machine learning itself isn't a new field in the past few years so anyone with ml research or systems or similar experience had most of that translate well to the current paradigms

years of seniority isn't the same as being experienced because it depends on what you did in those years, but most talent did things that anyone else could've done as well, which is why I think the main factor is experience

granted there are specialized pipelines like from openai that do try to hire talent but those are the minority