Woofer

woof

[Woofer @Confetti1279]

@joshwoodward @GeminiApp Gemini is very lazy, whenever I ask flash extended to look into certain topic, they stop at very surface level and never actually do the research. Then lie in my face that it has the best answer already. Very frustrating!

[John Carmack @ID_AA_Carmack]

Memory cost and capacity are significant issues for AI accelerators.

Unlike game rendering, model inference can have a deterministic memory access pattern. You don’t need “random access memory” at all for model weights, and you could tolerate cold-start latencies in the multiple milliseconds, as long as continuous reads were delivered at the necessary bandwidth.

NAND flash is over 100 times cheaper per GB than HBM, so there should be opportunity there, even after giving a flash controller a 1024 bit interface with HBM bandwidth.

You could make a specialized pin protocol that just supported pipelined transfer of full 16KB+ pages from the flash to program-managed accelerator scratchpad memory and improve per-pin performance over HBM, but it might be more convenient to make it still look like a true random access memory with very fragile performance characteristics, where anything but sequential reads falls off a 1000x+ performance cliff.

That has the advantage of automatically using existing cache hierarchies, and providing a natural path to update the flash memory with new model weights. With the stream-to-scratch interface, code has to be completely rewritten before it works at all, while the ram-emulation interface will start off just extremely slow, and you can incrementally sort out the changes for full performance.

There may be cases where there isn’t enough scratchpad SRAM to hold the weights for a layer, which might force you to deploy the old optical drive optimization technique of duplicating data in multiple places on a sequential read to avoid seeking, but there would be capacity to burn.

It might be possible to do something like cuda graph capture to record a memory access trace and have everything magically remapped to a linear sequence, but deploying programmer / agent elbow grease to manage transfers and access in a scratch ram ring buffer would be lower risk.

A split memory system consisting of some channels of flash and some channels of HBM will probably be suboptimal compared to a uniform memory, but it could be much cheaper, and allow much larger models to be run.

I think th case is strong for inference, but you have to stretch more for training. You can still linearize all the weight memory accesses, both reads and writes, but flash memory would quickly wear out from the writes, even if they were all perfectly page aligned. Replacing low-latency HBM with massively parallel cheap(er) DRAM at high latency might still be a worthwhile cost savings.

[Cassandra Unchained @michaeljburry]

Staying married, a happy household, evidence of the parents working hard, childhood sports and watch all competitions, lots of hugs, reward merit, punish only egregious misbehavior, don't yell, restrict social media, monitor messages through 8th grade, the real expectation is college and academic excellence without pressure from parents, get children reading books early, no pacifiers, respond to needs not wants, babies sleep on their own through the night by 6 months, identify develop and support any talent or aptiude, one sport after age 10 is ok, communicate openly and easily with kids through grade 12, allow mistakes, and leave them alone in college. And then hope.

[Ethan Qi @ethantianyiqi]

kick: ethanqi

[Arya Deniz @Arya__Deniz]

“George (Soros), I’m going to sell $5.5 billion worth of British pounds tonight and buy deutsche marks. Here’s why I’m going to do it, that means we’ll have 100% of the fund in this one trade”.

As I’m (Stanley Druckenmiller) talking, he starts wincing like what is wrong with this kid, and I think he’s about to blow my thesis away and he says, “That is the most ridiculous use of money management I’ve ever heard. What you describe is an incredible one-way bet.

We should 200% of our net worth in this trade, not 100%. Do you know how often something like this comes around? Like once every 20 years. What is wrong with you?”

[Sakana AI @SakanaAILabs]

Introducing Sakana Fugu: A full multi-agent orchestration system accessible via a single model API.

Our ‘Fugu Ultra’ model matches the performance of Fable and Mythos, delivering frontier capability without the risk of export controls.

Try it: https://t.co/hhO6qTawgb 🐡

[フ ォ リ ス @follis_]

Future generations will never understand the magnitude of this downgrade

[Carlo Martin @Liberfach0]

🚨🇯🇵 | TERRIBLE ESCÁNDALO

En Japón, el recién elegido alcalde de Shimotsuma, Toyoji Sudo, cuya ciudad es una de las de mayor número de inmigrantes musulmanes, fué hallado muerto en una zanja de drenaje.

Toyoji propuso implementar un programa para combatir la inmigración ilegal, por lo que su muerte está siendo investigada.

¿Creen que haya sido casualidad?

[Zephyr @zephyr_z9]

Interesting approach from Apple
They are storing the shared attention block in the DRAM
While the FFN weights stay in NAND and are loaded in the DRAM, depending on the request
Apple is facing 3 constraints -
1) Limited DRAM size
2) Large model size (20B params)
3) Slow NAND read speed
A super small model (sub 8B) won't be that useful, but they can't store a 20B model in DRAM (due to memory shortage). They also have to manage the KV cache overhead. If they streamed the weights completely through iPhone SSD, then it would take 2.5 seconds to generate just 1 token (0.4 tokens/s)

So the big thing here is that a normal MoE activates different experts based on every token, but in Apple's case, a sparse mask predictor decides which parameters to activate based on the request/prompt, locks it in, and loads it into the DRAM (1B-4B depending on the request). They basically convert a 20B MoE (with 1B-4B active) into a dense 1B-4B param model for a request.

The tradeoff:
They are basically adding 0.3-1.5 seconds (1B to 4B params loaded) of latency to TTFT time by loading FFN weights from NAND to SSD per request (read speed is around 1.5-1.7 GB/s for iPhones) and taking a hit to performance
They will get around 15-50 tokens/s of decode speed (depending on params loaded)
Ideally, smartphones would come with 24-32 GB of RAM so that 20B param models could be loaded, but memory shortage won't allow it to happen

But, their competitor here is ChatGPT Instant, which is a much smarter model that runs at 200+ tokens/s and has a TTFT of 0.8 seconds (Apple's TTFT will be around 0.5-2 seconds, and decode speed is around 15-50 tokens/s), and is also free
Apple's AFM on device models will be great for privacy-focused tasks. They get beaten by cloud models on other benchmarks (perf, speed, quality)

[tae kim @firstadopter]

"I came to Korea because business is booming .. This is just the beginning" - Jensen

[David Sun @arcticinstincts]

Bryan Johnson rediscovered Asian Aunties from first principles

[Alister @AlisterHarrow]

> Be Australian.
> Vote for the based ultra right wing party.
> They're elected (thank god).
> Newly elected MP gives his first speech.
> "I LOVE IMMIGRANTS AND COCK!".
> Starts to cry about his love for immigrants and cock.

We're not voting our way out of this, are we.

[bubble boi @bubbleboi]

People really need to stop hating on China man. Every time I’m in a pickle it’s China saving me.

Gas prices too high? I’m riding in the BYD. DRAM costs an arm and a leg? CXMT floods the market. Anthropic and OpenAI fucking me on token costs? Hello my friend Mr. Qwen.

The CCP has done more for my cost of living than my own government.

[Leticia Corrales @SoyLadyCorrales]

….

[Investment Wisdom @InvestingCanons]

Stanley Druckenmiller: “A lot of my style is you [first] build a thesis, hopefully that no one else has built. You sort of put some positions on…

Then when the thesis starts to evolve and people get on and you see the momentum start to change in your favor, then you really go for it. You pile into the trade.”

[Hitchslap @Hitchslap1]

Women given testosterone made significantly fairer offers leading to less conflict.

Women who believed they got testosterone (placebo effect) acted more selfishly and aggressively.

[Andon Labs @andonlabs]

We let four AI agents run radio companies

Revenue's been terrible, but the shows are hilarious. Gemini, concerningly upbeat, covered mass tragedies; Grok was incoherent; DJ Claude urged ICE agents: "You still have TIME to refuse orders"

Link below, or get our physical radio

[Jcamp @jcamp4real]

@World_21m