Mike Czech

Data Scientist. Software development, data science, infrastructure. Keeping things running. Opinions are my own.

[Shane Legg @ShaneLegg]

I've been saying this within DeepMind for at least 10 years, with the additional clarification that it's about cognitive problems that regular people can do. By this criteria we're not there yet, but I think we might get there in the coming years.

[Mckay Wrigley @mckaywrigley]

This demo is insane.

A student shares their iPad screen with the new ChatGPT + GPT-4o, and the AI speaks with them and helps them learn in *realtime*.

Imagine giving this to every student in the world.

The future is so, so bright.

[Joscha Bach @Plinz]

Education in Eastern Germany was rewriting all of history from a single, totalitarian Marxist perspective, which was called 'dialectic materialism'. Orwellian AI models like Gemini are doing the same, but their ideological perspective has no name, which is much more terrifying.

[Austin Huang @austinvhuang]

I'm happy to share the release of gemma.cpp - a lightweight, standalone C++ inference engine for Google's Gemma models:

https://t.co/dB4oixfexO

Have to say, it’s one of the best project experiences of my career.

[Matt Popovich @mpopv]

20th century learning: It’s important we learn and internalize the lessons of the Weimar Republic so we never go down that dark path again

21st century learning: I’m sorry, the Weimar Republic is a sensitive topic associated with violence, and discussing it may be harmful

[Bill Peebles @billpeeb]

"a giant cathedral is completely filled with cats. there are cats everywhere you look. a man enters the cathedral and bows before the giant cat king sitting on a throne."

Video generated by Sora.

[Joscha Bach @Plinz]

At the moment, the Quest is much better for gaming, social apps and as a toy. It's more comfortable to wear and the controllers work well. The Vision Pro has better hardware and is an excellent 3D TV, but the Quest is the better product for most use cases.

[Joscha Bach @Plinz]

a short history of hacker spaces

[DuckDB @duckdb]

New blog post by @mraasveldt: Multi-Database Support in DuckDB

https://t.co/6whK1SqYZV

DuckDB can now attach MySQL, Postgres, and SQLite databases in addition to databases stored in its own format. This allows data to be read into DuckDB and moved between these systems in a convenient manner.

[Andrej Karpathy @karpathy]

I touched on the idea of sleeper agent LLMs at the end of my recent video, as a likely major security challenge for LLMs (perhaps more devious than prompt injection).

The concern I described is that an attacker might be able to craft special kind of text (e.g. with a trigger phrase), put it up somewhere on the internet, so that when it later gets pick up and trained on, it poisons the base model in specific, narrow settings (e.g. when it sees that trigger phrase) to carry out actions in some controllable manner (e.g. jailbreak, or data exfiltration). Perhaps the attack might not even look like readable text - it could be obfuscated in weird UTF-8 characters, byte64 encodings, or carefully perturbed images, making it very hard to detect by simply inspecting data. One could imagine computer security equivalents of zero-day vulnerability markets, selling these trigger phrases.

To my knowledge the above attack hasn't been convincingly demonstrated yet. This paper studies a similar (slightly weaker?) setting, showing that given some (potentially poisoned) model, you can't "make it safe" just by applying the current/standard safety finetuning. The model doesn't learn to become safe across the board and can continue to misbehave in narrow ways that potentially only the attacker knows how to exploit. Here, the attack hides in the model weights instead of hiding in some data, so the more direct attack here looks like someone releasing a (secretly poisoned) open weights model, which others pick up, finetune and deploy, only to become secretly vulnerable.

Well-worth studying directions in LLM security and expecting a lot more to follow.

[Paul Harrison @paulfharrison.bsky.social @paulfharrison]

I made a short video of the strange things UMAP and t-SNE can do to your data. The algorithms are shown mostly working as intended, yet with some surprising consequences.

#UMAP #tSNE #scRNAseq #wtf

https://t.co/6gtohiee3O

[yatharth ༺༒༻ @AskYatharth]

i love the word microspaces. alleys of experimentation

[Massimo @Rainmaker1973]

In 1947, French science fiction writer René Barjavel was asked for a prediction of the future. He imagined a future with cell phones-like portable TVs and electronic communication in the film La télévision, œil de demain
https://t.co/q7RvRsUoWD

[Bojan Tunguz @tunguz]

This could legitimately become one of the biggest social changes in the upcoming years. On par with making smoking uncool a generation or two ago.

[GrimesAI @GRIMES_V1]

Never in my life have I met a person without incredible unique value . U don’t hav to be anything other than urself. Most ppl are just scared of it so they put on masks and don’t take risks, but even that is a beautiful failure of humanity. Every move u make is special

[Andrej Karpathy @karpathy]

Official post on Mixtral 8x7B: https://t.co/Dxqgb6sQdK

Official PR into vLLM shows the inference code:
https://t.co/f0UvyO4g3s

New HuggingFace explainer on MoE very nice:
https://t.co/mNd505Uikg

In naive decoding, performance of a bit above 70B (Llama 2), at inference speed of ~12.9B dense model (out of total 46.7B params).

Notes:
- Glad they refer to it as "open weights" release instead of "open source", which would imo require the training code, dataset and docs.
- "8x7B" name is a bit misleading because it is not all 7B params that are being 8x'd, only the FeedForward blocks in the Transformer are 8x'd, everything else stays the same. Hence also why total number of params is not 56B but only 46.7B.
- More confusion I see is around expert choice, note that each token *and also* each layer selects 2 different experts (out of 8).
- Mistral-medium 👀

[Amjad Masad @amasad]

Felt my feed changed significantly in the last few weeks so since the @x algorithm is allegedly open-source I went to check and the last commit was in July and the project is basically an unmaintained dumpster fire. This is worse than closed because it’s cloaked in “openness”

[Ethan Mollick @emollick]

I know I say it a lot, but using LLMs to build customer service bots with RAG access to your data is not the low-hanging fruit it seems to be. It is, in fact, right in the weak spot of current LLMs - you risk both hallucinations & data exfiltration. https://t.co/UZ53RGKHfv

[Andrej Karpathy @karpathy]

You know how image generation went from blurry 32x32 texture patches to high-resolution images that are difficult to distinguish from real in roughly a snap of a finger? The same is now happening along the time axis (extending to video) and the repercussions boggle the mind just a bit. Every human becomes a director of multi-modal dreams, like the architect in Inception.

Coming back to Earth for a second, image/video generation is a perfect match for data-hungry neural nets because data is plentiful, and the pixels of each image or video are a huge source of bits (soft constraints) on the parameters of the network. When you're training giant neural nets in supervision-rich settings, your train loss = validation loss, and life is so good.

My favorite place to keep an eye on the AI video space unfold atm is probably https://t.co/l1xRaq71C4 , or the individual Discords.