Deepak Narayanan

Research Scientist at @nvidia. Interested in the intersection of Computer Systems and ML. Occasionally tweet about sports. Views are my own.

[Deepak Narayanan @deepakn94]

@sytelus @RajaXg FWIW, GPipe (and our contemporaneous PipeDream) were designed with GPUs in mind. TPUs have much higher all-to-all network bandwidth and so you can"get away" with using higher-communication parallelism strategies like FSDP.

[Deepak Narayanan @deepakn94]

@Big3Tennis Djokovic Nadal 2013 RG semi

[Deepak Narayanan @deepakn94]

@glennko It seems some combination of local attention (e.g., SWA) or Mamba that looks at a compressed representation and global attention is promising. Other work like Gemma 3 also seems to be showing similar types of results.

[Deepak Narayanan @deepakn94]

Excited to share some models we've been training over the last couple of months!

[Bryan Catanzaro @ctnzr]

Nemotron-H: A family of Hybrid Mamba-Transformer LLMs.
* Hybrid architecture means up to 3X faster at the same accuracy
* Trained in FP8
* Great for VLMs
* Weights and instruct versions to come soon.

https://t.co/h3dLuDuiUz

[Deepak Narayanan @deepakn94]

@VictorPontis Seed oils aren’t natural?

[Deepak Narayanan @deepakn94]

@jtwalton01 @StoolGreenie Impossible with 5 teams a group unfortunately :/

[Deepak Narayanan @deepakn94]

@AdamHimmelsbach It’s best-of-3!

[Deepak Narayanan @deepakn94]

Thrilled to announce a new model trained jointly between @MistralAI and @nvidia!

[Bryan Catanzaro @ctnzr]

@MistralAI and @nvidia announce Mistral-NeMo 12B, an awesome bite-size model released under Apache 2.0 that we jointly trained. FP8 aligned checkpoint and 128k context window, great benchmark scores.

https://t.co/gKG3Eewr0p

https://t.co/YNkeQpPLlv

[Deepak Narayanan @deepakn94]

More details here: https://t.co/vRacctHWWb.

[Bryan Catanzaro @ctnzr]

A 8B-3.5T hybrid SSM model gets better accuracy than an 8B-3.5T transformer trained on the same dataset:
* 7% attention, the rest is Mamba2
* MMLU jumps from 50 to 53.6%
* Training efficiency is the same
* Inference cost is much less
https://t.co/x62otbC5uN

[Deepak Narayanan @deepakn94]

@cHHillee We published our initial pipeline parallelism paper at SOSP 2019: https://t.co/GsJd7sxX1Y. Obviously it has found use in a totally different context than what we originally envisioned.

[Deepak Narayanan @deepakn94]

@NikkiAShah Looks cool! :)

[Deepak Narayanan @deepakn94]

@main_horse @_akhaliq This is the commit they compared to: https://t.co/rF7L2Bldiu. It is from January 11, 2023.

[Deepak Narayanan @deepakn94]

@VictorPontis How about telling them the earth is flat?

[Deepak Narayanan @deepakn94]

@robertnishihara @vipulved @anyscalecompute @togethercompute We wrote up some thoughts on our experience measuring inference runtimes of LLM APIs in our recent NeurIPS paper.

https://t.co/aSHCbTmXrK

[Deepak Narayanan @deepakn94]

Seeing a lot of discussion around the fairness of this leaderboard, so figured I would bring up our recent paper that appeared at NeurIPS (https://t.co/l8ajdGNoSZ) where we look into some of the issues that make it hard to compare LLM APIs. (1/n)

[Deepak Narayanan @deepakn94]

@KabirNagrecha We had no visibility, apart from confirmation from OpenAI that `davinci` was the original 175B GPT-3 architecture.

We would have loved to analyze the later (and more relevant) OpenAI models like GPT-4 and GPT-3.5 but we were handcuffed by not knowing anything about those models.

[Deepak Narayanan @deepakn94]

Seeing a lot of discussion around the fairness of this leaderboard, so figured I would bring up our recent paper that appeared at NeurIPS (https://t.co/l8ajdGNoSZ) where we look into some of the issues that make it hard to compare LLM APIs. (1/n)

[Anyscale @anyscalecompute]

📈We’re excited to introduce the LLMPerf leaderboard: the first public and open source leaderboard for benchmarking performance of various LLM inference providers in the market.

Our goal with this leaderboard is to equip users and developers with a clear understanding of the capabilities and limitations of LLM inference solutions, featuring key providers such as @replicate, @awscloud, and @togethercompute!

You can find the leaderboard here: https://t.co/e1dxIwvfAD

The LLMPerf leaderboard tracks three main metrics: time-to-first-token, inter-token latency, and success rate.

- Time-to-first-token (TTFT) measures the time it takes between the query and the first response of the provider. TTFT is especially important for interactive and streaming applications, such as chatbots.

- Inter-token latency measures the average time between consecutive tokens. This is important for applications that require the entirety of the response to be ready, like summarization tasks or agent use cases.

- Finally, success rate measures the number of successful responses where the inference API operates without errors. This measure reflects the reliability and stability of API provider.

Blog announcement: https://t.co/LMFZnKkK6q

(1/2)

[Deepak Narayanan @deepakn94]

In doing this work, we quickly realized comparing the runtime performance of black-box LLM APIs is tricky (but also extremely important).

I'm glad to see conversations around how to do this in the most fair way possible. (7/7)

[Deepak Narayanan @deepakn94]

The best-case runtime is useful to gauge how far off individual runtime measurements are (e.g., what fraction of reported runtime is actually a function of current load on the system, versus fundamental computation?). (6/n)