Tim Neumann

search @p0, prev MIT, @trykino

[Tim Neumann @TimNeumann84165]

@sayshrey @CompleteSkeptic @p0 Yes definitely 👍

[Tim Neumann @TimNeumann84165]

Typesafe's CEO @CompleteSkeptic does not trust public benchmarks.

Neither do I, Diogo.

So @p0 and I wrote a blog evaluating 8 decision models on real company training datasets we have personally audited and iterated on.

For what it's worth though, Jev is still on top 👑
https://t.co/Qd2g0Hwp4V

[Vlad @everythingmeta]

https://t.co/zLFxEX5xXR

We spent two weeks extensively testing decision models on real production use cases. It's been super fun, we learned a lot, and we are sharing our findings! 👇

[Tim Neumann @TimNeumann84165]

@CompleteSkeptic I despise public evals as well. A non trivial part of @ArtificialAnlys's success is that they present their results in cute and readable colors...
@p0 and I made a blog post detailing how we go about evaling System One models on our own internal datasets!
https://t.co/IscIqTi7d6

[Ethan Kam @ethank_6]

https://t.co/6OJbOnkydN

[Tim Neumann @TimNeumann84165]

@SophieLWang This is sick Sophie

[Artificial Analysis @ArtificialAnlys]

Announcing the Artificial Analysis Search Index, benchmarking how search API providers perform on quality, cost, and speed when used by an agent. We are initiating coverage with Parallel, Exa, Firecrawl, You (dot) com, Tavily, Keenable, and Brave

Search is one of the most important tools for agents. Search providers make different choices about how they search, rank, and package results, and those choices change what the model reads and how it acts. We are expanding our benchmarking coverage to search APIs, so developers can pick a search provider on measured quality, cost, and speed.

Each provider result pairs a search API provider with the same model, GPT-5.6 Luna (medium). The model runs inside Stirrup, our open-source agent harness, with tools for searching and fetching pages from the web - only the search provider behind the search tool changes.

At launch, the leaderboard covers 11 results across 7 search providers, and we’ll keep expanding coverage as we look to provide the most accurate and comprehensive benchmarking of search providers for AI agent usage.

Key elements of the Artificial Analysis Search Index:

➤ Three equally weighted benchmarks: the Search Index is the average of DeepSearchQA (900 broad research questions that need many searches, graded with an F1 score over answer items), BrowseComp (a 200-sample hard subset of facts that need multi-hop browsing), and AA-Omniscience (a 600 question private subset, balanced across 6 domains)

➤ Same agent, different search provider: the agent has 25 turns available to complete each task. Its web search tool returns the search provider's native response payload (with content modes standardized to snippets), with a maximum of 10 results and contamination sources filtered out

➤ Model-only baseline: we compare search agent results to the same model answering single-shot without tools, showing how much each provider lifts the model above its internal knowledge

➤ Cost and Time per Task: we aggregate the time and cost spent on both model inference and search. This is key - search APIs have different cost and latency structures, but these can be offset where they help an agent use fewer turns and save on costly language model inference

Key results:

➤ Parallel, Exa, and Firecrawl have the strongest overall performance, with Artificial Analysis Search Index scores of 75, 74, and 73 respectively at launch

➤ All search providers tested substantially improve knowledge-based benchmark performance: the model only baseline scores 33 on the Search Index, while search-included provider results score between 65 and 75

➤ Focused search results reduce spend on model inference: Parallel Search (advanced) search costs more per task than Parallel Search (basic) but less per task in total ($0.084 vs $0.11). Higher quality results cut the model's token use by over 40% in this case, more than offsetting increased search costs while reaching higher benchmark scores

➤ Fast search calls do not guarantee fast tasks: Parallel Search (turbo) has the fastest average search calls among Parallel's tiers (0.51s per query vs 1.03s for Parallel Search (basic)) but the basic tier scores higher on quality (73 vs 67) and the two land close on total time per task

[Tim Neumann @TimNeumann84165]

@willdepue I think I can. Unless this superintelligence can perfectly simulate all human feelings, people will still have weird, messy opinions, reactions, and motivations that might form strong enough intuitions to steer businesses, politics, etc. in an actually economically viable way

[Tim Neumann @TimNeumann84165]

@georgepickett @cerebras @cartesia Siiiiick

[Parallel Web Systems @p0]

Today we're launching Index: a platform for content owners to understand how AI agents use their work, and earn revenue when they do.

Our first partners include @TheAtlantic, @FortuneMagazine, @PRNewswire, @PitchBook, @ZoomInfo, @Tracxn, @RocketReachCo, @enigma_data, @fiscal_ai, plus creators @alexeheath, @mariogabriele @azeem, @every, and @packyM.

[Tim Neumann @TimNeumann84165]

@Sam_Acqua https://t.co/ObbQMb0wbn
Continuous text flow models over categorical data exist... but I these still only give per token likelihoods and don't factor over the whole sequence like AR.
Agree that discrete flow evals only really make sense on downstream tasks

[Parag Agrawal @paraga]

We are working towards a new web: one built for agents instead of humans.

To celebrate the weird, whacky, and magical parts of the journey that made the human web, we are putting together something fun and unique!

Online: https://t.co/pjU9FXnQV2
Pop-up in SF May 8-16

[Parag Agrawal @paraga]

We raised a Series B led by @sequoia at a $2 billion valuation and I’m excited to welcome @andrew__reed to our board.

All our existing investors doubled down: @kleinerperkins, @IndexVentures, @khoslaventures, @firstround, @sparkcapital, @AbstractVC, and @terraincap.

[Larry Dial @classiclarryd]

New NanoGPT Speedrun WR at 86.8 (-0.4s) from @.samacqua on Github, by tuning and reusing the transpose_copy kernel during the cross entropy backward calc. Outside the main speedrun track, Sam did an interesting experiment in Jan showing how test-time training can improve perplexity. https://t.co/bc32yUD8wF. https://t.co/uTnd8mUbEx

[matia @matialee_]

love, acceptance, and the meaning of life in "her"
- a video essay by me (made with @trykino) :)

[Tim Neumann @TimNeumann84165]

@matialee_ @trykino incredible. completely changed my perspective on the movie, and love in general