pythongiant

open source inference || prev @MSFTResearch

[pythongiant @pyth0ngiant]

new account here :) im an inference engineer and researcher and will be posting about high performance inference systems. if that interests you, do consider a follow :)

[pythongiant @pyth0ngiant]

@shafu0x day 1200 of never using a linked list in my life

[pythongiant @pyth0ngiant]

combined GEMM + Top-K + softmax benchmark showed 7–14% higher throughput with the handwritten PTX on a GH200

[pythongiant @pyth0ngiant]

posting about underappreciated / unconvential techniques with CUDA.

one genuinely underappreciated technique is branchless top-k sorting using PTX predicate registers to avoid loops and branches

[pythongiant @pyth0ngiant]

Suppose you have sorted values in registers, a conventional CUDA implementation inserts the new value into the sorted array by comparing individual elements. with setp.gtu.f32 you can create a predicate based on a comparison and select between two values based on that predicate using selp.f32

[pythongiant @pyth0ngiant]

this paper is lowkey an absolute hidden gem

[Aaron Zhao @aaronzhao123]

@BenKoska Not sure whether u have seen our paper lol https://t.co/vloLMilMYV

[Aaron Zhao @aaronzhao123]

@BenKoska Not sure whether u have seen our paper lol https://t.co/vloLMilMYV

[pythongiant @pyth0ngiant]

the future is open source software

[Ben Koska @BenKoska]

We formalized PTX in Lean so that we can formally prove GPU kernels correct.

This way, we can be confident that the kernels our agents wrote are correct, without ever needing to read them. 🧵

[pythongiant @pyth0ngiant]

@BenKoska i would like to say thank you on behalf of the inference community

[shafu @shafu0x]

the nostalgia for pre-AI swe is so funny to me, yeah man I really enjoyed fixing CI and merge conflicts by hand

[pythongiant @pyth0ngiant]

met the ceo of whatsapp in a random cafe in indrangar this is crazy

[pythongiant @pyth0ngiant]

everytime i see this i wonder what happened in july

[Jordan @justmazer]

been saying if you are non technical and your github looks like this then I want to hire you

[pythongiant @pyth0ngiant]

@prathamgrv lmfaoooo i thought it was kinda neat but it would be cool to switch it up from green

[Unemployed Capital Allocator @atelicinvest]

the software that people who had spent their entire lives thinking about software produced:

[pythongiant @pyth0ngiant]

petition to make this a permanent feature on github

[pythongiant @pyth0ngiant]

@shaurizard the biggest shocker here is why do they think someone who participated in putnam would use reddit

[pythongiant @pyth0ngiant]

im literally standing on a +500% increase in conc streams with a latency gate for an ASR model on a single H100

[pythongiant @pyth0ngiant]

deep learning inference in libraries like tensort is so unoptimized that it virtually guarantees that if you want to host any model outside of an LLM you would have to so some sorts of kernel engineering to get to real production standards

[nick @IAmNickDodson]

Really tired of local LLM benchmarks with 8k/32/64k etc.

Bench it at min. 128k - 256k.