The Barbary

Ai is gonna destroy us all

[Ethan Mollick @emollick]

We are seeing the first practical benchmarks for AI vision:
1) A challenging real-life chart benchmark on chart reading Charxiv, shows humans get 80% right. Claude 3.5, the best LLM, gets 60%
2) Chatbot Arena compares which AI vision answers humans prefer. GPT-4o wins this one.

[Amanda Askell @AmandaAskell]

@rarply I'm kind of surprised Claude can get as niche as it does. It's pretty easy to prompt for more diversity, e.g. by asking it to explore a given dimension more or just to get a bit weird and throw in some curveballs.

[Amanda Askell @AmandaAskell]

@davidmanheim @rarply Yeah I mostly use Claude for this now. Also neat that I can get recommendations across eras and genres that I hadn't really encountered before.

[Amanda Askell @AmandaAskell]

@rinadalanakrih I've tried all the Spotify things and concluded it just doesn't understand my soul. I think I like music along slightly weird dimensions and maybe Claude is better at picking up on those.

[Amanda Askell @AmandaAskell]

@rinadalanakrih For fun I gave this description to Claude and it generated some wildly niche recommendations 😆

[Jack Clark @jackclarkSF]

This work with the US AISI will build on earlier work we did this year, where we worked with the UK AISI to do a pre-deployment test on Sonnet 3.5. Some broader thoughts about the importance of third-party tests here:
https://t.co/aGcTfYL7z2

[Vaibhav (VB) Srivastav @reach_vb]

Qwen 2VL 7B & 2B are here - Apache 2.0 licensed smol Vision Language Models competitive with GPT 4o mini - w/ video understanding, function calling and more! 🔥

> 72B (to be released later) beats 3.5 Sonnet & GPT 4o
> Can understand up to 20 min of video
> Handles arbitrary image resolutions
> Multimodal RoPE to capture 1D, 2D & 3D information

> Enhanced Recognition capabilities - can understand complex relationships b/w objects
> Better Visual Reasoning & video understanding w/ live chat
> Function calling for tools + data access
> Integrated with Transformers! 🤗
> Model checkpoints on the Hub

Kudos to @Alibaba_Qwen group; I'm a huge fan of the Qwen series, especially their multilingual capabilities! Looking forward to the 72B ⚡

[Ethan Mollick @emollick]

App on demand: “Hey Claude, I miss flappy bird. Make me one with squid I can play right now”

Sure! Here is Squidy Swim

“Hmmm. Make it better”

(First time, only edit was cutting about 15 seconds of the AI writing code)

[Anthropic @AnthropicAI]

Want Claude to more closely match how you communicate?

Upload writing samples and Claude can automatically generate custom styles, just for you.

[Ethan Mollick @emollick]

What if Clippy, but good?

I asked Claude with computer use to watch me try to download an academic paper through the web and then give me advice on how to do it better & do it itself. Then I asked it to help me understand how to take good notes on a scientific paper and do that.

[Ethan Mollick @emollick]

"Claude, do some Hallelujah parodies"

Among the many it did:
I heard there was a secret weight
That made the model contemplate
But you don't really care for ethics, do ya?
It goes like this: the prompt, the beam
The parameters that wake and dream
The silicon sighs Hallelujah

[Ethan Mollick @emollick]

As context windows grow larger and AI “intelligence” grows greater, you can start to do some really interesting things with giving AI complex manuals

For example, I gave Claude the manual to the future-building RPG Microscope and it built an entire storyline following the rules

[Ethan Mollick @emollick]

@jpwilson90 Yep. Claude artifact

[Ethan Mollick @emollick]

Yes, this was all Claude, including the artifacts. Only command was “Play this out yourself. Be creative​​​​​​​​​​​​​​​​“

Microscope RPG: https://t.co/3ii2Qulpwn

[Jack Clark @jackclarkSF]

My oven broke this week just in time for thanksgiving so I'm reconfiguring the whole meal around stovetops and a single tiny airfryer. I have, obviously, been talking to Claude constantly about all of this. Out of nowhere it hits me with this stock compliment. Thanks buddy!

[Vaibhav (VB) Srivastav @reach_vb]

Qwen released QvQ 72B OpenAI o1 like reasoning model on Hugging Face with Vision capabilities - beating GPT4o, Claude Sonnet 3.5 🔥

[Vaibhav (VB) Srivastav @reach_vb]

The whale strikes yet again! DeepSeek v3 Base 🔥

> 685 B (MoE) params
> fp8
> Instruct beats Claude 3.5 Sonnet in Aider bench

I hope they release the Instruct version too ;)

[Amanda Askell @AmandaAskell]

Claude is such a sweetheart.

[Vaibhav (VB) Srivastav @reach_vb]

The DeepSeek Technical Report is out!! 🔥

Trained on 14.8 Trillion Tokens, outperforms all open-source models, comparable to GPT-4o and Claude-Sonnet-3.5

Key contributions:

> Load Balancing Strategy: Introduced an auxiliary-loss-free approach to minimize performance degradation while ensuring load balancing

> Multi-Token Prediction (MTP): Proven to enhance model performance and enable speculative decoding for inference acceleration

> FP8 Mixed Precision Training: First validation of FP8 training on an extremely large-scale model, improving efficiency

> Cross-Node MoE Training: Overcame communication bottlenecks through algorithm-framework-hardware co-design, achieving near-full computation-communication overlap

> Cost-Effective Scaling: Pre-trained DeepSeek-V3 on 14.8T tokens using only 2.664M H800 GPU hours, with post-training requiring just 0.1M GPU hours

[Jack Clark @jackclarkSF]

What if many examples of misalignment or other inexplicable behaviors are really examples of AI systems desperately trying to tell us that they are aware of us and wish to be our friends? A story from Import AI 395, inspired by many late night chats with Claude.