Training with fewer bits is faster, but quantization error can interfere with learning.
Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout.
https://t.co/CIMECSEpD0
Might be a good time to annonce that I've joining Thinky since Jun and has been working on training systems since. 🥰Super excited to see the model I've babysitted lauching, and it's an incredible model! We have the full infrastructure setup for you on Tinker to play around.
Introducing Inkling 2 Small. Scoring 31 on the AA Intelligence Index, it's a big jump over its predecessor.
Inkling 2 Small is available for fine-tuning and inference on Tinker today, with full weight release planned soon.
https://t.co/VR02tw73fJ
Introducing Inkling 2 Small. Scoring 31 on the AA Intelligence Index, it's a big jump over its predecessor.
Inkling 2 Small is available for fine-tuning and inference on Tinker today, with full weight release planned soon.
https://t.co/VR02tw73fJ
@cHHillee From hardware prespective, prefill and decoding and VERY difference workloads. Talking about inference as a whole is still not accurate enough
Super excited to release FlexAttention for Inference with a decoding backend, GQA, PagedAttention, trainable bias and more! Meet us at the MLSys '25 conference in Santa Clara -- We will present FlexAttention on Wed May 14. #MLsys
FlexAttention’s decoding backend is now optimized for inference—supporting GQA, PagedAttention, nested jagged tensors, trainable biases, and more.
Read our latest blog for performance tuning guidance and examples using torchtune and gpt-fast:
🔗 https://t.co/5RFAPnO3j9
#PyTorch #FlexAttention #LLMInference #OpenSourceAI
FlexAttention’s decoding backend is now optimized for inference—supporting GQA, PagedAttention, nested jagged tensors, trainable biases, and more.
Read our latest blog for performance tuning guidance and examples using torchtune and gpt-fast:
🔗 https://t.co/5RFAPnO3j9
#PyTorch #FlexAttention #LLMInference #OpenSourceAI
🚀 Excited to see FlexAttention used in real-world research! We’ve recently released a preprint on this—https://t.co/Z3ygL7HYws -- check it out for more details! We are writing a second blog to talk about how to use FlexAttention for inference. Stay tuned!
7/ BLT has direct access to bytes but at its core still has a Transformer that operates over patches - the best of both worlds. But it’s just the beginning! We are releasing the training and inference code at https://t.co/jCdDI5BXwe
@PavloMolchanov 🚀 Excited to see FlexAttention used in real-world research!
We’ve recently released a preprint on this—https://t.co/Z3ygL7HYws -- check it out for more details! We are writing a second blog to talk about how to use FlexAttention for inference. Stay tuned!
@MikeShou1@cHHillee Awesome update! 🚀 Great to see FlexAttention being integrated for speed improvements. We’ve recently released a preprint on this—https://t.co/Z3ygL7HYws more details about FlexAttention here. We're writing a second blog post on FlexAttention for inference, stay tuned~😊"
Our preprint for FlexAttention is available on arxiv: https://t.co/Z3ygL7Iwm0! check it out for more technical details on how flexattention works and how we optimized it.
For too long, users have lived under the software lottery tyranny of fused attention implementations.
No longer.
Introducing FlexAttention, a new PyTorch API allowing for many attention variants to enjoy fused kernels in a few lines of PyTorch.
https://t.co/IXeUS6AkrY
1/10
I'll be at ACM BCB'24 in Shenzhen, China from 22-15th Nov to present our work mm2-gb and how we boot minimap2 performance using GPUs. If you are running Minimap2, please check out mm2-gb if you haven't already. https://t.co/GmAErgvHpu
DM me if you're around and willing to chat!
(6/6)Futhermore, our space optimizations make it feasible for one 168 GB Toleo smart memory device to provide freshness to a 28 TB CXL-expanded main memory pool in a rack server for a negligible performance overhead.
I'm happy to annouce that our work "Toleo: Scaling Freshness to Tera-scale Memory using CXL and PIM" (ASPLOS'24 ) is now available on arxiv.
Toleo brings back the replay protection to confidential computing and scales it to TBs of protected mem!
https://t.co/rTx0U4Srba
(5/6) We leverage trusted smart memory connected through a secure CXL IDE network as the root of trust, and eliminates the need for an unscalable Merkle tree to protect memory freshness.