driss guessous

bytes and nuggets @pytorch

[driss guessous @drisspg]

Okay that's enough codex for now->

at least now I can finally look at the generation step

[driss guessous @drisspg]

I am trying to make ideogram usable on my spark;
Problem 1. https://t.co/q2LRrDOcFJ

Problem 2. Bitsandbytes is unbelievable slow

[driss guessous @drisspg]

@gazorp5 Okay turns out it was not bnb but acutally safe-tensor using mmap by defualt and no option in 0.7 to break this without explicit clones.

[driss guessous @drisspg]

@gazorp5 I have not even looked at the actual generation step yet. if you try run_inference.py it takes 85 (on my spark) seconds just to start gen. Alot of this is just low hanging fruit

Im currently at 28.6s

[driss guessous @drisspg]

@typedfemale @SkyLi0n https://t.co/sAVOTGfGDJ

[driss guessous @drisspg]

@gaunernst Im just trying `run_inference.py` lols

[driss guessous @drisspg]

@cHHillee https://t.co/j0GeSh2UYH

I have some vibe in here but works great. Export to .pftrace

[driss guessous @drisspg]

"The purpose of abstracting is not to be vague, but to create a new semantic level in which one can be absolutely precise."

This really is a nice quote

[Modal @modal]

Reinforcement learning has exploded on Modal, and we've been cooking.

Here's a review of lessons learned helping teams train at scale, the patterns we kept seeing, and an open-source library to get started with RL on Modal quickly.

[driss guessous @drisspg]

Lol it took 7 hours for it to find GemmUniversal and do some hyper parameter tuning. Can't you feel the AGI!!!

[driss guessous @drisspg]

@_seemethere Dude you don’t even have a manager

[driss guessous @drisspg]

@tenderizzation Bruhhh this is a solved problem

[driss guessous @drisspg]

@KuterDinel Not serious I thought it was just a fun sentence

[driss guessous @drisspg]

To hell with big TMA long live ld/st

[driss guessous @drisspg]

@gaunernst @snowclipsed

[driss guessous @drisspg]

@maharshii > it's unfortunate that torch scaled mm api does not provide a global scale dequantization argument

Can you elaborate here? https://t.co/wa3EYcBZkk

This does support global scales. We should probably expand a lil in the docs but here is gist: https://t.co/rrpwcAlZjh

[driss guessous @drisspg]

@tenderizzation

[driss guessous @drisspg]

@_seemethere @difficultyang Yeah big caveat as that when you first use it’s gunna suck, but if you stick with it and actually muck around with the system prompt +extensions you end up with something that feels very tailored to your preferences

[driss guessous @drisspg]

@difficultyang

[Han Guo @HanGuo97]

LLM training is built on fast MatMuls. But many surrounding ops still run as memory-bound kernels.

CODA reparameterizes them to hide in the matmul’s shadow, fused into its epilogue before results leave the chip.

Bonus: LLMs can write fast CODA kernels too (approaching SoLs).

[driss guessous @drisspg]

Omni looks really cool, everything else is so mehh