Tim Sears

HTEC

[Marc Scholten @_marcscholten]

Happy to share that the new Haskell Meetup will be hosted at @GroqInc in their Mountain View HQ, this Friday 5:30pm https://t.co/R1aNWEBJ2K

@csTimSears will share how Groq uses Haskell and @avi_press (founder @scarf_oss) will talk about his experience on running a startup with haskell

[Groq Inc @GroqInc]

With over 284K+ developers using GroqCloud™, the Groq Speed Read focuses on developers and highlights new features, great apps, and what’s coming up. Sign up for the Speed Read here so you don't miss out - https://t.co/jxLNpyd2mm -

[Tim Sears @csTimSears]

@YankeeWRLD @adaquano Backward cap, nose to nose, kicking dirt on his shoes and home plate.

[Jay Scambler @JayScambler]

Who wants to go all in on an LPU cluster with me?

[Matt Shumer @mattshumer_]

I’m sorry to all the engineers at @GroqInc

I should’ve waited till tomorrow to post about you…

Hopefully the servers stay online till the morning!

[@levelsio @levelsio]

Try https://t.co/tfPlxzUDkZ now

Hyperfast LLM running on custom built GPUs

Answers in miliseconds, not seconds

How? 🤯

[Matt Shumer @mattshumer_]

The first public demo using Groq: a lightning-fast AI Answers Engine.

It writes factual, cited answers with hundreds of words in less than a second.

More than 3/4 of the time is spent searching, not generating!

The LLM runs in a fraction of a second.

https://t.co/PifDmCRpkY

[Groq Inc @GroqInc]

@Uncensored_AI @levelsio Good guess, but nope. Completely original hardware / software solution built from the ground up. We say it's "sand to sky" because it's our custom silicon GroqChip, an LPU provided as a system via our cloud solution. We haven't even started with tricks used by others yet. :D

[Tim Sears @csTimSears]

@TaliaGold @GroqInc We did that. It’s fun. You can talk to mixtral running on Groq in minutes with zero code over at https://t.co/1QDyN63Iuz.

[diicell @0xdiicell]

oi, m8. @GroqInc fastest resp alive

[Dhruv Sahu @DhruvSahu98]

@chamath @GroqInc I tried this with summarize book text with cross questioning and summary.

It's faster and more accurate than big techs llms.

[Matt Shumer @mattshumer_]

Wild tech you have to try: https://t.co/IddQqtQnvV

They are serving Mixtral at nearly 500 tok/s.

Answers are pretty much instantaneous.

Opens up new use-cases, and completely changes the UX possibilities of existing ones.

[Jay Scambler @JayScambler]

The response time from groq feels like magic!

[Jay Scambler @JayScambler]

Groq is serving the fastest responses I've ever seen. We're talking almost 500 T/s!

I did some research on how they're able to do it. Turns out they developed their own hardware that utilize LPUs instead of GPUs. Here's the skinny:

Groq created a novel processing unit known as the Tensor Streaming Processor (TSP) which they categorize as a Linear Processor Unit (LPU). Unlike traditional GPUs that are parallel processors with hundreds of cores designed for graphics rendering, LPUs are architected to deliver deterministic performance for AI computations.

The LPU's architecture is a departure from the SIMD (Single Instruction, Multiple Data) model used by GPUs and favor a more streamlined approach that eliminate the need for complex scheduling hardware. This design allows every clock cycle to be utilized effectively, ensuring consistent latency and throughput.

For developers, this means that performance can be precisely predicted and optimized which is critical in real-time AI applications.

Energy efficiency is another area where LPUs shine. By reducing the overhead of managing multiple threads and avoiding the underutilization of cores, LPUs can deliver more computations per watt.

Groq's innovative chip design allows multiple TSPs to be linked together without the traditional bottlenecks found in GPU clusters making them extremely scalable. This enables linear scaling of performance as more LPUs are added simplifying the hardware requirements for large-scale AI models and making it easier for developers to scale their applications without rearchitecting their systems.

So what does this all mean? LPUs could provide a massive improvement compared to GPUs for serving AI applications in the future! If anything it will be great to have alternative high performing hardware since A100s and H100s are so in demand

[Tim Sears @csTimSears]

@JohnYue122333 @JayScambler @ylecun He knows

[Tim Sears @csTimSears]

@sandersted I’m waiting for the tiktok video.

[Tim Sears @csTimSears]

@lifebypixels Spurts. From “experts”.

[Tim Sears @csTimSears]

@vpatryshev It’s like Theory but not just for clothing.

[Tim Sears @csTimSears]

@oldfriend99 @sclv 0 + 0 is my favorite but Dad said it would never amount to anything.

[Tim Sears @csTimSears]

Math. We have no moat around it. How do we build one?