Matthias Hericks

Scaling Post-Training @Cohere | Prev. Statistics @UniofOxford

[Matthias Hericks @MatthiasHericks]

@gabriberton Some people do mid-training, others do mid training.

[Matthias Hericks @MatthiasHericks]

@kalomaze This prompt / answer doesn’t tell you much about whether the model is calibrated or not. It’s totally possible that this was a 1% event.

[Matthias Hericks @MatthiasHericks]

@wtgowers Do you have a recommendation on how departments should prepare for this scenario?

[Matthias Hericks @MatthiasHericks]

@nrehiew_ This is not what your experiment demonstrates. It’s just likely that whitespace and the word ‘hi’ isn’t merged into a single token. This doesn’t mean that whitespace is never merged.

[Matthias Hericks @MatthiasHericks]

@_arohan_ @NandoDF @AnthropicAI @DarioAmodei How does this help accelerate others? Seeing the bar doesn’t tell you how to clear it.

[Matthias Hericks @MatthiasHericks]

@finbarrtimbers What ended up being the issue?

[Matthias Hericks @MatthiasHericks]

@evilmathkid @suchenzang If you don’t include any test data in training, how are the task embeddings for the test samples trained? Not at all?

[Matthias Hericks @MatthiasHericks]

@AdithyaDsilva @evilmathkid I mean the example test outputs, e.g. Ex. 1/2/3 Output on the right side of this figure:

https://t.co/iEkSv2tsY8

[Mithil Vakde @evilmathkid]

In technical terms, this is joint self-supervised compression.

To justify this, I rely on the Minimum Description Length principle (MDL)

I take MDL to its logical conclusion by compressing every possible source of information

[Matthias Hericks @MatthiasHericks]

@AdithyaDsilva @evilmathkid Are you still training on other information of the test set (e.g. the expected completions given as part of the input of a test set sample)?

[Matthias Hericks @MatthiasHericks]

@jsuarez @yacineMTB Why not just learn the initial hidden state?

[Matthias Hericks @MatthiasHericks]

@BrandonLuuMD The study doesn’t make any claims about a causal effect.

People maintaining a daily exercise schedule are likely generally more mindful about their health. This largely confounds the effect of exercise alone.

[Matthias Hericks @MatthiasHericks]

@suchenzang @soldni Would be cool if there was a model acting on & outputting encrypted text though.

[Matthias Hericks @MatthiasHericks]

@francoisfleuret This will differ from regular accuracy ofc as it favors high-variance methods.

Also, if done reactively to not-as-nice looking accuracy curves, there’s a potential problem with multiple testing.

It’s not problematic if the metric is selected upfront though.

[Matthias Hericks @MatthiasHericks]

@khoomeik Do you have a few examples of algorithmic ideas breaking down at scale?

[Katie Everett @_katieeverett]

1. We often observe power laws between loss and compute: loss = a * flops ^ b + c
2. Models are rapidly becoming more efficient, i.e. use less compute to reach the same loss

But: which innovations actually change the exponent in the power law (b) vs change only the constant (a)?

[Matthias Hericks @MatthiasHericks]

@janleike This is very simple, but works for me: In CAncer diagnosis, you‘d aim for high ReCAll.

[Matthias Hericks @MatthiasHericks]

@adad8m The question is not exactly well-defined. However, when putting a prior on theta, then the posterior mean minimises the Bayes integrated risk.

[Matthias Hericks @MatthiasHericks]

@daboigbae I replaced GitHub with ChatGPT from the start. My team exclusively stores code in ChatGPT chats.

[Matthias Hericks @MatthiasHericks]

@paulabartabajo_ Try to overfit a single batch.