Peter Szemraj

research interests: metallic intuition

[Alexander Doria @Dorialexander]

usual reminder from 18 months ago.

[Peter Szemraj @ten3br1s]

@ClementDelangue Big news indeed. Anything else news-wise you care to announce or discuss today?

[Bart Mol @Bart_Mol]

Me explaining AI to friends

[Georgi Gerganov @ggerganov]

let that sink in

[Jesse Mu @jayelmnop]

wake up babe, new neurosymbolic harness just dropped

[Can Bölük @_can1357]

guys you do know you can just disable thinking, and instead give it a "deep_think" tool, and it will call it with internal CoT reasoning format right?

gl fixing that

[Alexander Panfilov @kotekjedi_ml]

We can finally talk about it:

We found a way to extract hidden reasoning of frontier models using a vulnerability in the APIs of every frontier AI company.

We verified that our reasoning token count matches billed API thinking tokens 1:1 for most of the prompts we queried.

[Artificial Analysis @ArtificialAnlys]

Announcing the Artificial Analysis Endpoint Accuracy Index, measuring how much of an open weights model's accuracy each serverless API endpoint preserves. We are initiating coverage with GLM-5.2, gpt-oss-120b and DeepSeek V4 Pro, with Kimi K3 coming soon

Providers trade off accuracy to optimize for speed and cost. They quantize weights, write custom kernels and tune their inference stacks, and sometimes they simply ship bugs. We are bringing the rigor of our Artificial Analysis Intelligence Index to measuring endpoints, so developers can pick providers on accuracy, not just price and speed

We benchmark each serverless endpoint against our own self-hosted reference deployment of the official weights, where 100% represents matching the reference. An endpoint is at reference parity when its result falls within the 95% confidence interval of the reference. Coverage is live for GLM-5.2, gpt-oss-120b and DeepSeek V4 Pro, with Kimi K3 accuracy coverage launching soon

Key elements of the Endpoint Accuracy Index:
➤ Three areas, equally weighted: tool calling (BFCL-500, 500 questions, 3 repeats), scientific reasoning (HLE-250, 250 questions, 10 repeats) and long context recall (AA-LCR-25, 25 questions, 10 repeats). Each subset separates endpoints on the serving choices that drive accuracy differences, with repeats sized for tight confidence intervals
➤ Reference deployment: we self-host the official weights at the lab's recommended precision, following the lab's serving recipe, and publish the complete commands for each reference
➤ Inference parameters: we run the model's highest supported reasoning mode and each endpoint's highest supported output length and context window
➤ Confidence intervals: the parity test accounts for uncertainty in both the endpoint's runs and the reference's runs
➤ Rotating coverage: models enter once sufficient number of providers serve them and exit when a newer version in the same family supersedes them. We benchmark new endpoints as providers launch them and refresh all listed endpoints periodically
➤ Point in time: each result carries the date it was measured, with multi-day benchmarks dated to their final day

Key results for GLM-5.2
➤ Output token limits restrict accuracy. Restrictive limits cut responses off before the model finishes reasoning, and the most restrictive endpoints score half the reference or less on HLE-250

Key results for gpt-oss-120b
➤ Tool call handling separates endpoints. Providers parse and format tool calls differently, and some endpoints score 22% on BFCL-500 against 37% for the reference
➤ Serving configuration changes what the model does at the same requested settings. Some endpoints produce far fewer reasoning tokens at the same configured level, and restricted context windows truncate long context tasks

Key results for DeepSeek V4 Pro
➤ DeepSeek V4 Pro endpoints are more in line with the reference. Majority of the endpoints are at reference parity, and DeepSeek's own first-party endpoint scores slightly above the reference

[Peter Szemraj @ten3br1s]

@thsottiaux Chief Weekly Limit Reset Officer

[Peter Szemraj @ten3br1s]

Big respect to @DeepSeek for departing from the horrific status quo of the Jinja chat template. Hopefully others follow and we can be rid of this blight on LLM inference

[Greg Brockman @gdb]

at openai, many people hook their chatgpt up to slack.

people really don't like when a coworker's chatgpt contacts them asking for help with a task, even when they'd be perfectly happy doing that same work if asked by that coworker.

reinforces how much people care about human relationships and helping each other, and want AI to give time back — or enhance time together — rather than become a layer separating people.

[Peter Szemraj @ten3br1s]

very important: sha-256 hash of the image is 0258c6f1397ff2b2f299a63b16066a1bd4fbb42bcd06091b938b2bf233108ca9

and the hash of ^ hash is: 9b82be807ab17bdb5cb81001624c2e7fa243713c7984032715d2a6b70a9df69d

[Peter Szemraj @ten3br1s]

thanks to the power of ai, i can add 1,500+lines of security theater without asking for it, remove it in minutes, and get charged for it both ways!

[Peter Szemraj @ten3br1s]

@karpathy is this why you no longer work at anthropic? rip

[Peter Szemraj @ten3br1s]

@jiang_kev @polarbrowser Well, why do you need more money if you already have it and it works?

[Peter Szemraj @ten3br1s]

@cursor_ai Sick! Does it also upload all of the user's data and steal it? I wonder if that's invariant across the model mix or not

[Peter Szemraj @ten3br1s]

@DataChaz I can't wait

[Peter Szemraj @ten3br1s]

If your own model agrees it must be true (esp. if your model is not forced into compliance cc @grok )

[Peter Szemraj @ten3br1s]

How do I stop my browser use LLM from clicking on these?? Help

[Peter Szemraj @ten3br1s]

1% weekly left, five sessions going, /usage -> full reset