Xiao Chen

[World Labs @theworldlabs]

Introducing Atlas:

The world's first multimodal world model that generates image and video frames with pixel-perfect camera control and reconstructs them in 3D.

Model the world, move the camera, and simulate space & time.

[OpenAI @OpenAI]

Now in preview: The ChatGPT desktop app for Linux.

Use ChatGPT, ChatGPT Work, and Codex where you already work and build, with your projects and browser workflows on supported Linux systems.

[Andrej Karpathy @karpathy]

Software horror: litellm PyPI supply chain attack.

Simple `pip install litellm` was enough to exfiltrate SSH keys, AWS/GCP/Azure creds, Kubernetes configs, git credentials, env vars (all your API keys), shell history, crypto wallets, SSL private keys, CI/CD secrets, database passwords.

LiteLLM itself has 97 million downloads per month which is already terrible, but much worse, the contagion spreads to any project that depends on litellm. For example, if you did `pip install dspy` (which depended on litellm>=1.64.0), you'd also be pwnd. Same for any other large project that depended on litellm.

Afaict the poisoned version was up for only less than ~1 hour. The attack had a bug which led to its discovery - Callum McMahon was using an MCP plugin inside Cursor that pulled in litellm as a transitive dependency. When litellm 1.82.8 installed, their machine ran out of RAM and crashed. So if the attacker didn't vibe code this attack it could have been undetected for many days or weeks.

Supply chain attacks like this are basically the scariest thing imaginable in modern software. Every time you install any depedency you could be pulling in a poisoned package anywhere deep inside its entire depedency tree. This is especially risky with large projects that might have lots and lots of dependencies. The credentials that do get stolen in each attack can then be used to take over more accounts and compromise more packages.

Classical software engineering would have you believe that dependencies are good (we're building pyramids from bricks), but imo this has to be re-evaluated, and it's why I've been so growingly averse to them, preferring to use LLMs to "yoink" functionality when it's simple enough and possible.

[Tanishq Kumar @tanishqkumar07]

I've been working on a new LLM inference algorithm.

It's called Speculative Speculative Decoding (SSD) and it's up to 2x faster than the strongest inference engines in the world.

Collab w/ @tri_dao @avnermay. Details in thread.

[Chaofan Shou @shoucccc]

Claude code source code has been leaked via a map file in their npm registry!

Code: https://t.co/jBiMoOzt8G

[Rafael Rafailov @ NeurIPS @rm_rafailov]

We have a new position paper on "inference time compute" and what we have been working on in the last few months! We present some theory on why it is necessary, how does it work, why we need it and what does it mean for "super" intelligence.

[NovaSky @NovaSkyAI]

1/6 🚀
Introducing Sky-T1-32B-Preview, our fully open-source reasoning model that matches o1-preview on popular reasoning and coding benchmarks — trained under $450!

📊Blog: https://t.co/LtuTJeilmv
🏋️‍♀️Model weights: https://t.co/Vn1dmtrHWo

[OpenAI @OpenAI]

We’re starting to roll out the ability to search through your chat history on ChatGPT web.

Now you can quickly & easily bring up a chat to reference, or pick up a chat where you left off.

[AI at Meta @AIatMeta]

We previously shared our research on Layer Skip, an end-to-end solution for accelerating LLMs from researchers at Meta FAIR. It achieves this by executing a subset of an LLM’s layers and utilizing subsequent layers for verification and correction. We’re now releasing inference code and fine-tuned checkpoints for this work.

Model weights on @huggingface ➡️ https://t.co/ZnaD1lNzGg
More details ➡️ https://t.co/hY2q1F1W2x

We hope that releasing this work will open up new areas of experimentation and innovative new research in optimization and interpretability.

[Google AI @GoogleAI]

Building the largest image prompted speech dataset of Indian languages ever! Project Vaani aims to advance #AI capabilities and reduce language barriers by empowering millions of Indian voices. Discover the whole story: https://t.co/x8zzEOn6r9

[Trent McConaghy @trentmc0]

New post:
"bci/acc: A Pragmatic Path to Compete with Artificial Superintelligence: An e/acc zoom-in on brain interfaces, towards human superintelligence"

Strap in: it's long but rewarding 🌞👨‍🎤

The summary:

Artificial superintelligence is coming soon. We need a competitive substrate. BCI is the most pragmatic path. Therefore, we need to accelerate BCI and take it to the masses: bci/acc.

How do we make it happen? We’ll need BCI killer apps like silent messaging to create market demand, which in turn drive BCI device evolution. The net result is *human* superintelligence.

bci/acc draws on today’s technologies without requiring big scientific breakthroughs. It’s e/acc zoomed-in for BCI. It’s solarpunk: optimistic and perhaps a little gonzo. And it could be a grand adventure for Humanity.

https://t.co/EDRD2Ad2xg

[vitalik.eth @VitalikButerin]

The promise and challenges of crypto + AI applications:

https://t.co/ds9mLnshLU

[Google DeepMind @GoogleDeepMind]

Announcing AlphaFold 3: our state-of-the-art AI model for predicting the structure and interactions of all life’s molecules. 🧬

Here’s how we built it with @IsomorphicLabs and what it means for biology. 🧵 https://t.co/gjw6Ip4F2M

[Massimo @Rainmaker1973]

40 years of Boston Dynamics (1983 - 2023)

[📹 World Data Center, HD]

[Sebastian Raschka @rasbt]

Google's Gemma has been the topic of the week for both LLM researchers and users. My colleagues and I just ported the code to LitGPT, and we discovered some interesting surprises and model architecture details along the way:

1) Gemma uses a really large vocabulary and consequently embedding weight matrices.

2) Like OLMo and GPT-2, Gemma also uses weight tying.

3) It even uses GeLU (in the form of GeGLU), similar to GPT-2, unlike Llama 2 and other LLMs.

4) Interestingly, Gemma normalizes the embeddings by the square root of the hidden layer dimension.

5) The RMSNorm layers are in the usual location, different from what was hinted at in the technical paper.

6) However, the RMSNorm implementation comes with the unit offset that wasn't mentioned in the technical paper.

[AK @_akhaliq]

FlashDecoding++: Faster Large Language Model Inference on GPUs

paper page: https://t.co/ZOn8qj5f8p

As the Large Language Model (LLM) becomes increasingly important in various domains. However, the following challenges still remain unsolved in accelerating LLM inference: (1) Synchronized partial softmax update. The softmax operation requires a synchronized update operation among each partial softmax result, leading to ~20% overheads for the attention computation in LLMs. (2) Under-utilized computation of flat GEMM. The shape of matrices performing GEMM in LLM inference is flat, leading to under-utilized computation and >50% performance loss after padding zeros in previous designs. (3) Performance loss due to static dataflow. Kernel performance in LLM depends on varied input data features, hardware configurations, etc. A single and static dataflow may lead to a 50.25% performance loss for GEMMs of different shapes in LLM inference. We present FlashDecoding++, a fast LLM inference engine supporting mainstream LLMs and hardware back-ends. To tackle the above challenges, FlashDecoding++ creatively proposes: (1) Asynchronized softmax with unified max value. FlashDecoding++ introduces a unified max value technique for different partial softmax computations to avoid synchronization. (2) Flat GEMM optimization with double buffering. FlashDecoding++ points out that flat GEMMs with different shapes face varied bottlenecks. Then, techniques like double buffering are introduced. (3) Heuristic dataflow with hardware resource adaptation. FlashDecoding++ heuristically optimizes dataflow using different hardware resource considering input dynamics. Due to the versatility of optimizations in FlashDecoding++, FlashDecoding++ can achieve up to 4.86x and 2.18x speedup on both NVIDIA and AMD GPUs compared to Hugging Face implementations. FlashDecoding++ also achieves an average speedup of 1.37x compared to state-of-the-art LLM inference engines on mainstream LLMs.

[elvis @omarsar0]

Evaluating LLMs

A comprehensive survey (100+ pages) on evaluating LLMs.

We will start to see a lot more focus on LLM evaluation as we continue to apply LLMs in production and real-world use cases. This is a great place to start to learn about the different types of evaluations, datasets, techniques, and more.

Bookmark this one!

https://t.co/XzvQGHx6AR

[Alex Xu @alexxubyte]

How does a Password Manager such as 1Password or Lastpass work? How does it keep our passwords safe?

The diagram below shows how a typical password manager works.

A password manager generates and stores passwords for us. We can use it via application, browser extension, or command line.

Not only does a password manager store passwords for individuals but also it supports password management for teams in small businesses and big enterprises.

Let’s go through the steps.

Step 1: When we sign up for a password manager, we enter our email address and set up an account password. The password manager generates a secret key for us. The 3 fields are used to generate MUK (Master Unlock Key) and SRP-X using the 2SKD algorithm. MUK is used to decrypt vaults that store our passwords. Note that the secret key is stored locally, and will not be sent to the password manager’s server side.

Step 2: The MUK generated in Step 1 is used to generate the encrypted MP key of the primary keyset.

Steps 3-5: The MP key is then used to generate a private key, which can be used to generate AES keys in other keysets. The private key is also used to generate the vault key. Vault stores a collection of items for us on the server side. The items can be passwords notes etc.

Step 6: The vault key is used to encrypt the items in the vault.

Because of the complex process, the password manager has no way to know the encrypted passwords. We only need to remember one account password, and the password manager will remember the rest.

Over to you: Which password manager have you used?
–
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://t.co/uc5M7CdXXC

[Dr Milan Milanović @milan_milanovic]

𝗦𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿 != 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗿

Software Engineer == Problem Solver

Don't attack problems with code. Try first to understand them from a business perspective and then a technical perspective. When you do this, make a plan for how to solve it.

Try to solve without coding. The best code is not code.

If you must code, then do it.

Remember, 𝗰𝗼𝗱𝗲 𝗶𝘀 𝗮 𝗹𝗶𝗮𝗯𝗶𝗹𝗶𝘁𝘆, 𝗻𝗼𝘁 𝗮𝗻 𝗮𝘀𝘀𝗲𝘁.

#technology #softwareengineering #programming #developers #coding

[Jim Fan @DrJimFan]

Hmmm, @OpenAI just acquired a company called "Global Illumination" that makes open-source Minecraft clone.

What's next, multi-agent civilization sim running on GPT-5? Maybe Minecraft is indeed all you need for AGI? I'm intrigued.🤔

Announcement: https://t.co/8MYu2DRukI
Company: https://t.co/o76syilR4q
Game called "Biomes": https://t.co/Ws7PU53ejK