It's very cool that Apple shipped a 20B parameter on-device.
You can't put 20B parameters in RAM at any reasonable precision. To make it work they are using pretty exotic architecture by today's standards.
A small model predicts from the query (or prompt) which experts to load from Nand into RAM. The key distinction from a typical MoE is that you do this once per query and then generate all the tokens with the same experts (instead of switching the experts for every token).
SITUATION DETECTED: Antares Nuclear’s Mark-0 reactor has achieved initial criticality at Idaho National Laboratory, what the U.S. Department of Energy is calling one of the most significant technological achievements in nuclear energy in over 40 years.
Sources: Anthropic has embedded around half a dozen forward-deployed engineers within the NSA to help the agency deploy Mythos for offensive cyber operations (Financial Times)
(Visit Techmeme dot com for the link and full context!)
By firing Scott Pelley, editor-in-chief of CBS News Bari Weiss has single-handedly destroyed the most brilliant investigative newsmagazine in network history, 60 Minutes. Weiss exemplifies everything MAGA is about: ego, incompetence, and blithering idiocy.
...another morning in Ukraine, and once again, it is neither peaceful nor bright. Once again, we wake up exhausted, learning the names of the Ukrainians who were killed. What else is left for us, the civilians of this country, but to hide at night when cities face mass attacks… In Kyiv, during air raid alerts, crowds of people, along with their children and pets-seek refuge in metro stations that serve as bomb shelters. But in other cities like mine, you simply stay in your apartment, hoping for the best.
Some people underground are watching the events above ground through their phones, others are seeing the destruction from their windows. But morning has finally arrived, and it's impossible not to be grateful for it-it's just heartbreaking that this morning hasn't arrived for everyone. Is there any point in repeating "fucking russia" for the millionth time, does it even make any sense anymore...
Not coincidentally, this attack is underway shortly after the U.S. delayed delivery of air defense missiles to Ukraine and after Zelensky wrote an urgent appeal to Trump writing, "I ask for your help in protecting Ukraine's skies from Russian missiles."
@HouseGOP you will go down in the history books as the most sycophantic, morally corrupt cowards to be the majority party in congress, ever. party over country American Taliban! congratulations 👏👏👏