Giles Thomas

On sabbatical / founded @PythonAnywhere, which found a home at @anacondainc / XP / Python / PSF Fellow / opinions my own

[Giles Thomas @gpjt]

The appendices in "Build an LLM (from Scratch)" have a lot of really useful stuff. Should I have read them before heading off on my own training runs? I think, on balance, no -- it would have saved time, but I think that by trying, failing, and trying again, I learned more. YMMV!

https://t.co/EdtLC4AQ62

[Giles Thomas @gpjt]

@teortaxesTex It's a good thing that education science has already shown us how teaching smart students should differ from teaching the less-smart ones. Otherwise we'd be completely lost.

[Giles Thomas @gpjt]

@WarrenFahy @agraybee It's Han Houdini!

[Giles Thomas @gpjt]

Hear me out: a model that reasons in liminal space.

[Giles Thomas @gpjt]

@hunvreus Convincing. I've read stuff in corporate environments where I felt my brain cells dying from sentence to sentence. If an AI had rewritten it, I would have been saved that pain. But maybe it was a useful pain, like the one that makes you pull your hand away from a flame.

[Giles Thomas @gpjt]

@kiffboet @mitsuhiko What downside do you see in syncing it with Dropbox?

[Giles Thomas @gpjt]

@valsaven @mitsuhiko I'm thinking maybe file timestamping differences between OSes could trip things up, but yeah, maybe it's Syncthing vs Dropbox. I am planning to switch to Syncthing, will have to keep an eye out for problems like that!

[Giles Thomas @gpjt]

@valsaven @mitsuhiko Maybe related to OS? I'm on Linux, and phone access is from Android

[Giles Thomas @gpjt]

@valsaven @mitsuhiko Interesting! Never been bitten by that, myself.

[Giles Thomas @gpjt]

@heynavtoor Worth noting that LLMs are trained to minimise cross entropy loss, which is (loosely) a measure of how surprising the real next token is when compared to its belief about what it should be.

[Giles Thomas @gpjt]

Seeing double...

[Giles Thomas @gpjt]

@teortaxesTex There's algorithmic overhang as well. After all, we have an existence proof that you can train an AGI for maybe $100k on a constant 100W. Downside is that it takes ~20 years and sometimes (existence proof again) produces Literally Hitler.

[Giles Thomas @gpjt]

@DanielJHannan "Reality" works well too...

[Giles Thomas @gpjt]

I've spent some time learning JAX over the last month, and I have Thoughts: https://t.co/Tukcvl2WT2

[Giles Thomas @gpjt]

@xeophon Weekdays are for gradient descent?

[Giles Thomas @gpjt]

@teortaxesTex If I were to channel Claude, I'd say "the word 'civilian' is doing a lot of heavy lifting in that sentence".

[Giles Thomas @gpjt]

@fredsted @levelsio Thanks! I'll give that a go next time I'm staying at a hotel

[Giles Thomas @gpjt]

@levelsio Anecdotally, it sounds like Portugal spent at least some of its EU funds well -- I've heard that the fibre infrastructure that means that you can get 10G Internet to your home was paid for that way. Impressive if true.