Michael Jupp

17 • working on environmental sci + ML • I also build rockets

[Michael Jupp @michael_jupp4]

@ar0cket1 not when there is no codex 5h limit

[Michael Jupp @michael_jupp4]

@jonkomet my theory is that they reduced limits overall since chatgpt work will be used by a lot of knowledge workers (i.e. before 99% chatgpt plus/pro subscriptions were subsidizing 1% of plans that use codex)

[Michael Jupp @michael_jupp4]

@GShamane love to see it. I think you're onto something quite neat!

[Michael Jupp @michael_jupp4]

@vikhyatk I wonder if the effectiveness of OPD is from token level advantages or just easy signal from distillation

gflownets are dope

[Michael Jupp @michael_jupp4]

@ar0cket1 I would think larger models generalise RL training better for tasks outside training data

ie because knowledge is less of an issue, rl optimises logical thinking/methods rather than directly knowing what to output

But w enough rl 200b can easily beat 1T

[Michael Jupp @michael_jupp4]

@mertunsal2020 @JawadKazi13 tech report no work

[Michael Jupp @michael_jupp4]

@elliotarledge plz share when you find/make it

[Michael Jupp @michael_jupp4]

@thkostolansky is this a metaphor for RSI

[Michael Jupp @michael_jupp4]

@LucasSimula @micka_design @referodesign @mobbin @mattpocockuk @pbakaus link?

[Michael Jupp @michael_jupp4]

@ar0cket1 nomos 1?

[Michael Jupp @michael_jupp4]

@susuinldn @antirez good generalisation?

[Michael Jupp @michael_jupp4]

@ar0cket1 does it scale closer to active or total params?
eg qwen 27b vs 35b a3b
i would expect dense 27b better self teacher but not sure

[Michael Jupp @michael_jupp4]

@snowclipsed - Astrosage (llama 70B)
- aion (clip fine tune i think)
- multimodal universe data is good

nothing overly modern or natively multimodal AR llm. I think aion encoding onto an llm would be interesting

I looked into a lot of this stuff a while ago, happy to chat

[Michael Jupp @michael_jupp4]

@ar0cket1 real ones run on free gpu credits

[Michael Jupp @michael_jupp4]

@thsottiaux can you fix the pasting large text on desktop app

[Michael Jupp @michael_jupp4]

@ar0cket1 ‘parameter optimisation’

[Michael Jupp @michael_jupp4]

@liulicheng10 hi what r your thoughts on sequence level supervision as a middle ground between dense (OPD) and sparse (RLHF) rewards?

could provide decently dense signal but prevent model collapse

eg PPO between teacher-student at every \n

[Michael Jupp @michael_jupp4]

@ritteradam @Sapient_Int the main advantage is small training, so closed source would completely eliminate that