ParthMannan

Deep Learning Architect @ Nvidia | Making training go brrrr

[ParthMannan @ParthMannan3]

@mervenoyann Do you know if the model does attention compute with the entire 1hr+ video long context or is it chopped up in frames and they are still able to maintain temporal understanding? 😯

[ParthMannan @ParthMannan3]

@drisspg @cHHillee That'll be great! Thanks. Do you have a PR link I can refer to?

[ben hylak @benhylak]

did satya just aquihire openai for $0

[ParthMannan @ParthMannan3]

🤯🔥

[Satya Nadella @satyanadella]

We remain committed to our partnership with OpenAI and have confidence in our product roadmap, our ability to continue to innovate with everything we announced at Microsoft Ignite, and in continuing to support our customers and partners. We look forward to getting to know Emmett Shear and OAI's new leadership team and working with them. And we’re extremely excited to share the news that Sam Altman and Greg Brockman, together with colleagues, will be joining Microsoft to lead a new advanced AI research team. We look forward to moving quickly to provide them with the resources needed for their success.

[Beff (e/acc) @beffjezos]

I wish they made an evening version of caffeine with a shorter half-life

[ThePrimeagen @ThePrimeagen]

best way to describe pointers

[ParthMannan @ParthMannan3]

@mysticaltech @chheplo @theemozilla @yacineMTB @abacaj It supports quite a few Nvidia GPUs, not just A100/H100 but not all. FlashAttention is highly optimized code that relies on using certain architectural features that aren't present in all generations

[ParthMannan @ParthMannan3]

@swyx Absolutely fair haha! Thanks for the reply 😁

[ParthMannan @ParthMannan3]

Is there a good resource documenting the architectural differences between all the top open-source LLMs - Llama2, MPT, Falcon etc. ?
Like does one use RMSNorm instead of LayerNorm or other subtle changes to the Transformer block?

@swyx Any chance you have a pointer? 🙏

[ParthMannan @ParthMannan3]

Long shot but wondering if someone can tell me if this formula for FLOPs should have batch size multiplied as well? And if not, why not?

Maybe @tri_dao if you see this? 🙏

[ParthMannan @ParthMannan3]

@sriramk Just gotta convert @AcquiredFM episodes as chapters in a book! Audiobook is already ready haha xD

I'd buy one @gilbert @djrosent 🔥

[ParthMannan @ParthMannan3]

@sid09_singh @NVIDIAAI Welcome to NVIDIA 😁

[Jim Fan @DrJimFan]

What if we set GPT-4 free in Minecraft? ⛏️

I’m excited to announce Voyager, the first lifelong learning agent that plays Minecraft purely in-context. Voyager continuously improves itself by writing, refining, committing, and retrieving *code* from a skill library.

GPT-4 unlocks a new paradigm: “training” is code execution rather than gradient descent. “Trained model” is a codebase of skills that Voyager iteratively composes, rather than matrices of floats. We are pushing no-gradient architecture to its limit.

Voyager rapidly becomes a seasoned explorer. In Minecraft, it obtains 3.3× more unique items, travels 2.3× longer distances, and unlocks key tech tree milestones up to 15.3× faster than prior methods.

We open-source everything. Let generalist agents emerge in Minecraft! Welcome you all to try today: https://t.co/1d3YocozsI
Paper: https://t.co/JcWEasgtyI
Code: https://t.co/KsvVf7rcl0

Deep dive with me: 🧵

[ParthMannan @ParthMannan3]

@eisokant Not an expert but Isn't generating the right reward quite a challenge for RL still despite whatever policy is used? Probably works well for Minecraft though 🤔
Also, I am assuming the reward would be used to calculate a loss and then update the LLM right?

[Suhail @Suhail]

I once emailed Jensen cold last year and he got a driver bug fixed in 12 hours. 30 years after its founding, this guy still acts like a hardcore founder.

Jensen is the best tech CEO in the game rn.

[ParthMannan @ParthMannan3]

@MichaelBenayou1 Gotcha. Thanks a lot for the detailed response 🙏
Are there any use cases in the wild where folks are using the FX graph for transformations/optimization etc.?

[ParthMannan @ParthMannan3]

Is there a good blog/doc out there explaining what HFTracer/Proxy did differently from the native PyTorch FX and what the use cases it's potentially being used for?

[Shannon Sands @max_paperclips]

Mood