Sid Radcliffe

Tech blog - | deep learning library from scratch -

[Sid Radcliffe @sidradcliffe]

@Yawaru @ID_AA_Carmack Another great nautical series is Master and Commander, Patrick O'Brian

[Sid Radcliffe @sidradcliffe]

I refreshed SmallPebble: a minimalist deep learning library written from scratch in Python. It implements autodiff and dynamic computation graphs (like PyTorch) but strictly on CPU with NumPy. < 1000 lines of core logic.

https://t.co/5R7APRWrK7

[Sid Radcliffe @sidradcliffe]

Take: Papers should drop "et al.," and use "+"

(Bob et al., 2025) ->
(Bob+ 2025)

[Sid Radcliffe @sidradcliffe]

@ID_AA_Carmack An example of this done right, is the Clip paper, (lead author is also lead author of GPT). It has a Python code block in it that makes the idea very clear. https://t.co/WuM4MOqzux

[Sid Radcliffe @sidradcliffe]

@fchollet Or maybe it does fully describe it, just the form of the program is inefficient

[Sid Radcliffe @sidradcliffe]

@karpathy @bryan_johnson ^- this 1000%, I went on holiday to a place that was quiet at night and got the best sleep I've ever recorded.. I was looking into whether quiet sleep pods are a thing, but they don't seem to be (outside of DIY), and there's air quality concerns

[Sid Radcliffe @sidradcliffe]

@AnthropicAI Niice. I attempted this in 2023 with OpenAI vision, but either the model wasn't good enough, or I wasn't surfacing enough information to it.

https://t.co/nc3gwsZJTq

[Sid Radcliffe @sidradcliffe]

Can `gpt-4-vision-preview` play Pokemon Blue? It's hard to say, with only 100 API calls a day to experiment with, (as of now, 200 calls in total).

...so far looks like it has some trouble identifying what things are, and where the player is relative to them.

[Sid Radcliffe @sidradcliffe]

@giffmana 98.8% test error just using least squares in 10 lines of python: https://t.co/W1JrpEQNn4

[ScienceMachine @SciMac]

Why we started ScienceMachine: to accelerate biomedical discovery by scaling the bespoke.

https://t.co/47vh4wvg92

[Sid Radcliffe @sidradcliffe]

@minotauronlucy @francoisfleuret (I probably only respond to <30% of cold messages myself, and I don't get that many)

[Sid Radcliffe @sidradcliffe]

@minotauronlucy @francoisfleuret Haha, me too, this guy dislikes cold emails. On the other hand it's still worth doing because many people are much more open to it, (e.g. I've got into dialogue with some people who've published really interesting stuff). No one's obliged to respond but it's worth a shot

[Sid Radcliffe @sidradcliffe]

@sytelus @kushal1t @sea_snell @typedfemale On the other hand, single files can be more manageable and maintainable. Holding off until theres a good reason to break it up makes good pragmatic sense, similarly to how leaving some code duplication in until a good abstraction emerges makes sense. No imports/modules to arrange

[Sid Radcliffe @sidradcliffe]

@kellerjordan0 Thanks. If I understand correclty, muon is also fastest method of training the conv net here? Neat!

[Sid Radcliffe @sidradcliffe]

@fchollet Simple example of that here, with a token per grid cell used to train a transformer to compute the game of life: https://t.co/Busp8L22nV

[Sid Radcliffe @sidradcliffe]

@DoctorGester @jeremyphoward Haha. 1. In the linked post Python that's been jitted is added. It's still Python. 2. Yes, the reality is that a lot of code doesn't need to be particularly fast. - It does make sense to wait and optimise once you've confirmed it needs to be, unless the speed is completely free

[Sid Radcliffe @sidradcliffe]

New blog post: https://t.co/Busp8L22nV

"We create a simplified transformer neural network, and train it compute Conway’s Game of Life from examples of the game."

"We call the model SingleAttentionNet, because it uses just a single attention block, with single-head attention."

[Sid Radcliffe @sidradcliffe]

@giffmana @jconorgrogan I guess it depends if you don't mind older models and worse results, lol

Here's my lame result, on a mid-range gpu, https://t.co/HBc2HUV6qS

I do feel like better is possible, (if only I had more time..)

[Sid Radcliffe @sidradcliffe]

@StasBekman What about:

for month, _ in enumerate(range(12), 1):
...

(Lol)