Introducing our first set of Llama 4 models!
We’ve been hard at work doing a complete re-design of the Llama series. I’m so excited to share it with the world today and mark another major milestone for the Llama herd as we release the *first* open source models in the Llama 4 collection 🦙. Here are some highlights:
📌 The Llama series have been re-designed to use state of the art mixture-of-experts (MoE) architecture and natively trained with multimodality. We’re dropping Llama 4 Scout & Llama 4 Maverick, and previewing Llama 4 Behemoth.
📌 Llama 4 Scout is highest performing small model with 17B activated parameters with 16 experts. It’s crazy fast, natively multimodal, and very smart. It achieves an industry leading 10M+ token context window and can also run on a single GPU!
📌 Llama 4 Maverick is the best multimodal model in its class, beating GPT-4o and Gemini 2.0 Flash across a broad range of widely reported benchmarks, while achieving comparable results to the new DeepSeek v3 on reasoning and coding – at less than half the active parameters. It offers a best-in-class performance to cost ratio with an experimental chat version scoring ELO of 1417 on LMArena. It can also run on a single host!
📌 Previewing Llama 4 Behemoth, our most powerful model yet and among the world’s smartest LLMs. Llama 4 Behemoth outperforms GPT4.5, Claude Sonnet 3.7, and Gemini 2.0 Pro on several STEM benchmarks. Llama 4 Behemoth is still training, and we’re excited to share more details about it even while it’s still in flight.
A big thanks to all of our launch partners (full list in blog) for helping us bring Llama 4 to developers everywhere including @huggingface, @togethercompute, @SnowflakeDB, @ollama, @databricks and many others👏 This is just the start, we have more models coming and the team is really cooking – look out for Llama 4 Reasoning 😉
A few weeks ago, we celebrated Llama being downloaded over 1 billion times. Llama 4 demonstrates our long-term commitment to open source AI, the entire open source AI community, and our unwavering belief that open systems will produce the best small, mid-size and soon frontier models. Llama would be nothing without the global open source AI community & we are so ready to begin this next chapter with you. 🦙
Read more about the release here: https://t.co/7mbK3uggjO, and try it in our products today.
🚨Breaking: Microsoft just dropped major updates
- GitHub now has Copilot Agent Mode
- MCP support rolling out to all VS Code users
- New GitHub Copilot Pro+ Plan
Vibe coding with GitHub Copilot.
1/3 🧵
Video Source : GitHub YT
🚨BREAKING: Elon Musk says that there is a literal limestone mine where they store all the US Government's retirement paperwork built in 1950 that they need to go up and down every time they want to retire someone from Federal Government. The speed in which they can retire people is limited by the speed of the elevator shaft. Yes, actually.
You can now scrape any website by just writing a prompt.
Using @firecrawl's /extract feature, just describe what you want to get the web data you need.
This is ChatGPT for web scraping 🤯
I asked DeepSeek how a base model might enact a singularity during training and it started giving me a detailed guide about how to send radio signals via a CUDA kernel??
I asked DeepSeek how a base model might enact a singularity during training and it started giving me a detailed guide about how to send radio signals via a CUDA kernel??
Do people understand what they just did? They released another open source multimodal AI model that outperforms Stable Diffusion and Dalle3 with just 7b parameter.
The NASDAQ has not yet recovered from r1 and DeepSeek is following suit.
OpenAI and everyone else needs to respond now!
Distillation technique in LLM is making waves after DeepSeek-R1
This framework helps measure LLM distillation, promoting diversity and robustness in smaller models.
Revealing potential over-distillation issues.
Proposes a framework to quantify the distillation, or knowledge transfer, from larger to smaller LLMs. It aims to address the issue of over-distillation, which can lead to a lack of diversity and robustness in smaller models.
-----
Original Problem 🤔:
→ Over-reliance on distillation from advanced LLMs can hinder the development of diverse and robust smaller LLMs.
→ Current research lacks clear metrics to quantify the degree of distillation in LLMs.
-----
Solution in this Paper 💡:
→ This paper introduces two metrics: Response Similarity Evaluation (RSE) and Identity Consistency Evaluation (ICE).
→ RSE compares responses from smaller LLMs to a reference LLM (GPT) across style, structure, and content.
→ ICE uses a jailbreaking framework (GPTFuzz) to probe LLMs for inconsistencies in their self-reported identity information, revealing potential distillation from source LLMs.
-----
Key Insights 😲:
→ Base LLMs show higher distillation degrees than aligned LLMs.
→ Many well-known LLMs, both closed and open-source, show high distillation degrees (except Claude, Gemini, and Doubao).
-----
Results 📊:
→ GLM4-Plus, Qwen-Max, and Deepseek-V3 showed the highest suspected distillation degrees based on ICE.
→ GPT series models exhibited the highest response similarity to GPT-4 in RSE (average similarity of 4.240).
→ Claude, Doubao, and Llama3.1 showed lower response similarity, suggesting less distillation (around 3.6-3.7 average similarity).
BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED BASED
This has NEVER happened in recent history:
In a sudden collapse, 30-year interest rates are now LOWER in China than Japan.
China's economy is currently being described as a "deflationary spiral" as seen in Japan in the 1990s.
What does this mean? Let us explain.
(a thread)