Mike Gunter

CTO and founder, @MatXComputing, designing hardware to make LLMs an order of magnitude smarter.

[Mike Gunter @MikeGunter_]

Very excited to share more about what we've been working on.

[Reiner Pope @reinerpope]

We’re building an LLM chip that delivers much higher throughput than any other chip while also achieving the lowest latency. We call it the MatX One.

The MatX One chip is based on a splittable systolic array, which has the energy and area efficiency that large systolic arrays are famous for, while also getting high utilization on smaller matrices with flexible shapes. The chip combines the low latency of SRAM-first designs with the long-context support of HBM. These elements, plus a fresh take on numerics, deliver higher throughput on LLMs than any announced system, while simultaneously matching the latency of SRAM-first designs. Higher throughput and lower latency give you smarter and faster models for your subscription dollar.

We’ve raised a $500M Series B to wrap up development and quickly scale manufacturing, with tapeout in under a year. The round was led by Jane Street, one of the most tech-savvy Wall Street firms, and Situational Awareness LP, whose founder @leopoldasch wrote the definitive memo on AGI. Participants include @sparkcapital, @danielgross and @natfriedman’s fund, @patrickc and @collision, @TriatomicCap, @HarpoonVentures, @karpathy, @dwarkesh_sp, and others. We’re also welcoming investors across the supply chain, including Marvell and Alchip.

@MikeGunter_ and I started MatX because we felt that the best chip for LLMs should be designed from first principles with a deep understanding of what LLMs need and how they will evolve. We are willing to give up on small-model performance, low-volume workloads, and even ease of programming to deliver on such a chip.

We’re now a 100-person team with people who think about everything from learning rate schedules, to Swing Modulo Scheduling, to guard/round/sticky bits, to blind-mated connections—all in the same building. If you’d like to help us architect, design, and deploy many generations of chips in large volume, consider joining us.

[Mike Gunter @MikeGunter_]

Very excited to share more about want we've been working on.

[Reiner Pope @reinerpope]

We’re building an LLM chip that delivers much higher throughput than any other chip while also achieving the lowest latency. We call it the MatX One.

The MatX One chip is based on a splittable systolic array, which has the energy and area efficiency that large systolic arrays are famous for, while also getting high utilization on smaller matrices with flexible shapes. The chip combines the low latency of SRAM-first designs with the long-context support of HBM. These elements, plus a fresh take on numerics, deliver higher throughput on LLMs than any announced system, while simultaneously matching the latency of SRAM-first designs. Higher throughput and lower latency give you smarter and faster models for your subscription dollar.

We’ve raised a $500M Series B to wrap up development and quickly scale manufacturing, with tapeout in under a year. The round was led by Jane Street, one of the most tech-savvy Wall Street firms, and Situational Awareness LP, whose founder @leopoldasch wrote the definitive memo on AGI. Participants include @sparkcapital, @danielgross and @natfriedman’s fund, @patrickc and @collision, @TriatomicCap, @HarpoonVentures, @karpathy, @dwarkesh_sp, and others. We’re also welcoming investors across the supply chain, including Marvell and Alchip.

@MikeGunter_ and I started MatX because we felt that the best chip for LLMs should be designed from first principles with a deep understanding of what LLMs need and how they will evolve. We are willing to give up on small-model performance, low-volume workloads, and even ease of programming to deliver on such a chip.

We’re now a 100-person team with people who think about everything from learning rate schedules, to Swing Modulo Scheduling, to guard/round/sticky bits, to blind-mated connections—all in the same building. If you’d like to help us architect, design, and deploy many generations of chips in large volume, consider joining us.

[Mike Gunter @MikeGunter_]

The FT's calculation is off by a factor of eight. Rental pricing is per chip, not per eight-chip server.

[Mike Gunter @MikeGunter_]

@eladgil L'Industrie and also get gelato. Boichik is a lot better than one of the NYC places reputed to be best. So, maybe prioritize pizza over bagels?

[Reiner Pope @reinerpope]

MatX is designing chips and systems to 10x the computing power for the world’s largest AI workloads.

Today, we are pleased to announce the closing of a >$100M Series A funding round led by @sparkcapital, with participation from @JaneStreetGroup, @danielgross and @natfriedman, @TriatomicCap, @HarpoonVentures, and @adamdangelo.

In two years, we proved out all our technical bets across ML numerics, chip design and implementation, software, and system design—and secured all the necessary partnerships—to develop our chip.

With this round of investment, we are now sufficiently funded to bring our systems to market.

[Mike Gunter @MikeGunter_]

@wjtweet @TheStalwart @tracyalloway @reinerpope I'm optimistic about ML helping on both the front and back ends of ASIC design. We're not betting the company on it, though. The architecture (eliminated now) and execution risks were enough.

[Mike Gunter @MikeGunter_]

I really enjoyed talking about the process and business of semiconductor design with @tracyalloway and @TheStalwart on the Odd Lots podcast. Joe and Tracy were wonderful hosts: They put me at ease and guided the conversation with the lightest of touch.

We talked about what doing semiconductor design is like, why LLMs are hungry for as many FLOPS/$ as they can get, how @MatXComputing can provide that, and how NVIDIA's moat might be bridged. I particularly liked that @reinerpope got to communicate some of the sense of beauty that I also feel about good design.

Helping lead MatX's engineering team (and meeting with our customers) is a humbling honor: It's talking with people who are the world experts in what we're chatting about. Being on Odd Lots was talking with grandmasters at conversation.

[Joe Weisenthal @TheStalwart]

NEW ODD LOTS:

Two Veteran Chip Designers Have A Plan To Take On Nvidia

@tracyalloway and I talked to @reinerpope and @MikeGunter_, both formerly of Alphabet, about their new company MatX that's aiming to build the ultimate semiconductor just for LLMs https://t.co/xCIMrfByU9

[MatX @MatXComputing]

Introducing MatX: we design hardware tailored for LLMs, to deliver an order of magnitude more computing power so AI labs can make their models an order of magnitude smarter.

Our hardware would make it possible to train GPT-4 and run ChatGPT, but on the budget of a small startup.

Our founding team has designed chips at Google and Amazon, and we’ve built chips with 1/10 the team size typically needed. Here’s how we’re approaching the problem of inefficient and insufficient compute.

While other chips treat all models equally, we dedicate every transistor to maximizing performance on the world’s largest models. Our goal is to make the world’s best AI models run as efficiently as allowed by physics, bringing the world years ahead in AI quality and availability. A world with more widely available intelligence is a happier and more prosperous world—picture people of all socioeconomic levels having access to an AI staff of specialist MDs, tutors, coaches, advisors, and assistants.

Our design focuses on cost efficiency for high-volume pre-training and production inference for large models.

This means:

1/ We’ll support training and inference. Inference first.

2/ We optimize for performance-per-dollar first (we’ll be best by far), and for latency second (we’ll be competitive).

3/ We offer excellent scale-out performance, supporting clusters with hundreds of thousands of chips.

4/ Peak performance is achieved for these workloads: large Transformer-based models (both dense and MoE), ideally 20B+ parameters, and inference having thousands of simultaneous users.

5/ We give you low-level access to the hardware.

We believe that the best hardware is designed jointly by ML hardware experts and LLM experts. Everyone on the MatX team, from new grad to industry veteran, is exceptional. Our industry veterans have built ML chips, ML compilers, and LLMs, at Google or Amazon or various startups.

Our CEO, @reinerpope, was Efficiency Lead for Google PaLM, where he designed and implemented the world’s fastest LLM inference software.

Our CTO, @mikegunter_, was Chief Architect for one of Google’s ML chips (at the time, Google’s fastest) and was an Architect for Google’s TPUs.

Our CDO Silicon, @avinashgmani, has over 25 years of experience in building products and world-class engineering teams in silicon and software at Amazon, Innovium and Broadcom.

We’re backed by $25M of investment from specialist investors and operators who share our vision, including: @danielgross and @natfriedman (lead investors, and experts in the AI space), @rkhemani (CEO at Auradine), @amasad (CEO at Replit), @outsetcap, @homebrew, @svangel. Additionally we have investment from leading AI and LLM researchers including @IrwanBello, @jekbradbury, @achowdhery, @liamfedus, and @hardmaru.

[Mike Gunter @MikeGunter_]

@EricHallahan @reinerpope We should be OK if the trends from the last decade hold!

[Reiner Pope @reinerpope]

I’m excited to announce our new company, MatX, started with @MikeGunter_. We want to make AI better, faster, and cheaper by building more powerful hardware. Read on for a short introduction, or see our full announcement here: https://t.co/1HOx8uYyCA.

[Mike Gunter @MikeGunter_]

@DKThomp It looks like both married and unmarried people were significantly happier in 2018 than 1972. But unmarried people are a *lot* less happy. So, the decrease in happiness over the period has to be caused by a significantly lower percentage of people being married, no?

[Mike Gunter @MikeGunter_]

@satnam6502 I gave one of my workstations the hostname "botrytis". Enjoy!

[Mike Gunter @MikeGunter_]

@TerryMatalas Indeed! The Cruise/Blunt remake is almost as good as the Murray/MacDowell original.

[Mike Gunter @MikeGunter_]

@satnam6502 :). (I like the Ritual gin better.)

[Mike Gunter @MikeGunter_]

@wjarek @_arohan_ Sunrises are great, too! This is the morning of the 18th.

[Mike Gunter @MikeGunter_]

@i_aver @jonpaluga @michelletandler At least in my neighborhood, we make up for our bad-for-fireworks weather with excellent lawlessness.

[Mike Gunter @MikeGunter_]

@satnam6502 Happy birthday, Satnam! Let's share a meal soon.

[Mike Gunter @MikeGunter_]

@DekalbWonk @EverydayBastiat With great power ...