Ray

what now ?

[Prateek Dhawan @PattyInnovate]

Faster spool ups are crucial for certain missions where the engine must transition from a dead start to producing thrust within seconds.

In this video from confirmed ignition to idle time in under 15 seconds.

It’s important to note that this is a larger engine with a heavier rotor and significantly more inertia. And despite these challenges, we repeat this repeatedly.

[Peter Steinberger 🦞 @steipete]

The funniest take is that I "failed" 43 times when people look at my GitHub repos and projects.

Uhmm... no? Most of these are part of @openclaw, I had to build an army to make it useful. https://t.co/GLR35USlzu

[J Rosser ✈️ ICML @jrosseruk]

And here's the guide:
https://t.co/FS2ISkvkQe

[Vivek Galatage @vivekgalatage]

Roadmap from Cornell - Introduction to CUDA

https://t.co/Spw4ramTBl

[Alex K @uialexk]

@BraydenWilmoth If it's fully remote, design engineer here 👋
https://t.co/DkWjQ105Fr

[GPU MODE @GPU_MODE]

The most intuitive explanation of floats I've ever come across, courtesy of @fabynou https://t.co/XNiZNZTNlf

[Sebastian Raschka @rasbt]

With the release of the Kimi Linear LLM last week, we can definitely see that efficient, linear attention variants have seen a resurgence in recent months. Here's a brief summary of what happened.

First, linear attention variants have been around for a long time, and I remember seeing tons of papers in the 2020s.

I don't want to dwell too long on these older attempts. But the bottom line was that they reduced both time and memory complexity from O(n^2) to O(n) to making attention much more efficient for long sequences.

However, they never really gained traction as they degraded the model accuracy, and I have never really seen one of these variants applied in an open-weight state-of-the-art LLM.

In the second half of this year, there was a bit of a revival of linear attention variants. The first notable model was MiniMax-M1 with lightning attention, a 456B parameter mixture-of-experts (MoE) model with 46B active parameters, which came out back in June.

Then, in August, the Qwen3 team followed up with Qwen3-Next, which I discussed in more detail above. Then, in September, the DeepSeek Team announced DeepSeek V3.2 with sparse attention.

All three models (MiniMax-M1, Qwen3-Next, DeepSeek V3.2) replace the traditional quadratic attention variants in most or all of their layers with efficient linear variants. (DeepSeek's sparse attention it's not strictly linear but still subquadratic).

Interestingly, there was a recent plot twist, where the MiniMax team released their new 230B parameter M2 model (discussed in section 13) without linear attention, going back to regular attention. The team stated that linear attention is tricky in production LLMs. It seemed to work fine with regular prompts, but it had pure accuracy in reasoning and multi-turn tasks, which are not only important for regular chat sessions but also agentic applications.

This could have been a turning point where linear attention may not be worth pursuing after all. However, it gets more interesting. Last week, the Kimi team released their new Kimi Linear model with linear attention. The tag line is that compared to regular, full attention, it has a 75% KV cache reduction and up to 6x decoding throughput.

Kimi Linear shares several structural similarities with Qwen3-Next. Both models rely on a hybrid attention strategy. Concretely, they combine lightweight linear attention with heavier full attention layers. Specifically, both use a 3:1 ratio, meaning for every three transformer blocks employing the linear Gated DeltaNet variant, there's one block that uses full attention as shown in the figure below.

However, Kimi Linear modifies the linear attention mechanism of Qwen3-Next by the Kimi Delta Attention (KDA) mechanism, which is essentially a refinement of Gated DeltaNet. Interestingly, it also replaces the standard full attention module by multi-head latent attention.

There's no direct comparison to Qwen3-Next in the Kimi Linear paper, but compared to the Gated DeltaNet-H1 model from the Gated DeltaNet paper (which is essentially Gated DeltaNet with sliding-window attention), Kimi Linear achieves higher modeling accuracy while maintaining the same token-generation speed.

Of course, I couldn't resist and added it to my The Big LLM Architecture Comparison article, which has grown to >10,000 words now (basically becoming book!?).

[Cowboy Space Corp. @CowboySpaceCorp]

Let's advance space solar power for America. 🇺🇸

We're hiring → https://t.co/kD1T7XrnNI

[cetusian @cetusian]

full 12 chapters course on algorithms for modern harware

[Andrej Karpathy @karpathy]

Huge congrats to @AIatMeta on the Llama 3.1 release!
Few notes:

Today, with the 405B model release, is the first time that a frontier-capability LLM is available to everyone to work with and build on. The model appears to be GPT-4 / Claude 3.5 Sonnet grade and the weights are open and permissively licensed, including commercial use, synthetic data generation, distillation and finetuning. This is an actual, open, frontier-capability LLM release from Meta. The release includes a lot more, e.g. including a 92-page PDF with a lot of detail about the model:
https://t.co/48e3YJ8Sg9

The philosophy underlying this release is in this longread from Zuck, well worth reading as it nicely covers all the major points and arguments in favor of the open AI ecosystem worldview:
"Open Source AI is the Path Forward"
https://t.co/AdmpadCRM0
I like to say that it is still very early days, that we are back in the ~1980s of computing all over again, that LLMs are a next major computing paradigm, and Meta is clearly positioning itself to be the open ecosystem leader of it.

- People will prompt and RAG the models.
- People will finetune the models.
- People will distill them into smaller expert models for narrow tasks and applications.
- People will study, benchmark, optimize.

Open ecosystems also self-organize in modular ways into products apps and services, where each party can contribute their own unique expertise. One example from this morning is @GroqInc , who built a new chip that inferences LLMs *really fast*. They've already integrated Llama 3.1 models and appear to be able to inference the 8B model ~instantly:
https://t.co/b2kdSsz0fH
And (I can't seem to try it due to server pressure) the 405B running on Groq is probably the highest capability, fastest LLM today (?).

Early model evaluations look good:
https://t.co/RLR5YBpmks https://t.co/ipT4x4wCvy
Pending still is the "vibe check", look out for that on X / r/LocalLlama over the next few days (hours?).

I expect the closed model players (which imo have a role in the ecosystem too) to give chase soon, and I'm looking forward to that.

There's a lot to like on the technical side too, w.r.t. multilingual, context lengths, function calling, multimodal, etc. I'll post about some of the technical notes a bit later, once I make it through all the 92 pages of the paper :)

[Jonathan Ross @JonathanRoss321]

What can you do with Llama quality and Groq speed? You can do Instant. That's what. Try Llama 3.1 8B for instant intelligence on https://t.co/JFfJs01nUJ.

[Soumith Chintala @soumithchintala]

Why do 16k GPU jobs fail?
The Llama3 paper has many cool details -- but notably, has a huge infrastructure section that covers how we parallelize, keep things reliable, etc.
We hit an overall 90% effective-training-time.
https://t.co/5gngOZJHBO

[Shreya Shankar @sh_reya]

wrote a very long piece on data flywheels for LLM applications. hope you like it: https://t.co/EWGE9h6oVc

[ARC Prize @arcprize]

Announcing ARC Prize.

A $1M+ competition to beat the ARC-AGI benchmark and open source the solution.

Hosted by @mikeknoop & @fchollet.

https://t.co/TUr6bhwgz6

[Jeff Clune @jeffclune]

I am thrilled to introduce OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code. Led by @maxencefaldor and @jennyzhangzt, with @CULLYAntoine and myself. 🧵👇

[Sebastian Raschka @rasbt]

Neural architecture search is back! This time for LLMs (of course): "LLaMA-NAS: Efficient Neural Architecture Search for Large Language Models" (https://t.co/puchET0ivJ)
Maybe not super feasible on a default basis, but hey, I think it's an interesting alternative to pruning.

[William Falcon ⚡️ @williamfalcon]

no catch 😃

[DAIR.AI @dair_ai]

Top ML Papers of the Week (Oct 16 - Oct 22):

- Llemma
- Self-RAG
- GenBench
- OpenAgents
- Video Language Planning
- LLMs for Software Engineering
...

----
1/ Llemma - an LLM for mathematics which is based on continued pretraining from Code Llama on the Proof-Pile-2 dataset; the dataset involves scientific paper, web data containing mathematics, and mathematical code; Llemma outperforms open base models and the unreleased Minerva on the MATH benchmark; the model is released, including dataset and code to replicate experiments.

https://t.co/PWYHEWsMdS

2/ LLMs for Software Engineering - a comprehensive survey of LLMs for software engineering, including open research and technical challenges.

https://t.co/Xaf3pYgsvN

3/ Self-RAG - presents a new retrieval-augmented framework that enhances an LM’s quality and factuality through retrieval and self-reflection; trains an LM that adaptively retrieves passages on demand, and generates and reflects on the passages and its own generations using special reflection tokens; it significantly outperforms SoTA LLMs (ChatGPT and retrieval-augmented Llama2-Chat) on open-domain QA, reasoning, and fact verification tasks, including factuality improvements.

https://t.co/ftlwI8smvc

4/ Retrieval-Augmentation for Long-form Question Answering - explores retrieval-augmented language models on long-form question answering; finds that retrieval is an important component but evidence documents should be carefully added to the LLM; finds that attribution error happens more frequently when retrieved documents lack sufficient information/evidence for answering the question.

https://t.co/aqxZzxECSf

5/ GenBench - presents a framework for characterizing and understanding generalization research in NLP; involves a meta-analysis of 543 papers and a set of tools to explore and better understand generalization studies.

https://t.co/R23LIcaHg8

6/ A Study of LLM-Generated Self-Explanations - assesses an LLM's capability to self-generate feature attribution explanations; self-explanation is useful to improve performance and truthfulness in LLMs; this capability can be used together with chain-of-thought prompting.

https://t.co/tV79TaXQqK

7/ OpenAgents - an open platform for using and hosting language agents in the wild; includes three agents, including a Data Agent for data analysis, a Plugins Agent with 200+ daily API tools, and a Web Agent for autonomous web browsing.

https://t.co/mJ5eTM5xYt

8/ Eliciting Human Preferences with LLMs - uses language models to guide the task specification process and a learning framework to help models elicit and infer intended behavior through free-form, language-based interaction with users; shows that by generating open-ended questions, the system generates responses that are more informative than user-written prompts.

https://t.co/6dQHgLubo8

9/ AutoMix - an approach to route queries to LLMs based on the correctness of smaller language models (done via few-shot self-verification); a meta-verifier is introduced to check the verifier's output (typically a smaller model) and route the query to a larger language model if needed. Experiments using LLAMA2-13/70B, on five context-grounded reasoning datasets demonstrate that AutoMix surpasses established baselines, improving the incremental benefit per cost by up to 89%.

https://t.co/f7Tft6hxgi

10/ Video Language Planning - enables synthesizing complex long-horizon video plans across robotics domains; the proposed algorithm involves a tree search procedure that trains vision-language models to serve as policies and value functions, and text-to-video models as dynamic models.

https://t.co/1MCha1QauT

[Goku Mohandas @GokuMohandas]

Excited to share our production guide for building RAG-based LLM applications where we bridge the gap between OSS and closed-source LLMs.

- 💻 Develop a retrieval augmented generation (RAG) based LLM application from scratch.
- 🚀 Scale the major workloads (load, chunk, embed, index, serve, etc.) across multiple workers.
- ✅ Evaluate different configurations of our application to optimize for both per-component (ex. retrieval_score) and overall performance (quality_score).
- 🔀 Implement LLM hybrid routing approach to bridge the gap b/w OSS and closed LLMs.
- 📦 Serve the application in a highly scalable and available manner.
- 💥 Share the 1st order and 2nd order impacts LLM applications have had on our products.

🔗 Links:
- Blog post (45 min. read): https://t.co/QHgOXPT7S0
- GitHub repo: https://t.co/GMNrsHAhpY
- Interactive notebook: https://t.co/UPXSkwDt6h

@pcmoritz and I had a blast developing and productionizing this with the @anyscalecompute team and we're excited to share Part II soon (more details in the blog post).

[Justin Alvey @justLV]

I “jailbroke” a Google Nest Mini so that you can run your own LLM’s, agents and voice models.

Here’s a demo using it to manage all my messages (with help from @onbeeper)

🔊 on, and wait for surprise guest!

I thought hard about how to best tackle this and why, see 🧵