๐ขIntroducing BiomniBench โ the first benchmark focused on evaluating the process, not just the final answer, of AI agents on long-horizon biology research tasks.
We evaluate whether each analysis step is statistically sound and scientifically defensible โ just like peer review, where the validity of the process matters, not matching a single ground-truth answer.
Built with first authors from Nature, Cell, and Science papers, BiomniBench captures 100 most ambitious research tasks across disease areas.
Huge headroom for performance, jagged frontiers across tasks, and many more interesting results:
I'm here at #NeurIPS2025 in #SanDiego all week and eager to connect with folks passionate about LLMs, Autonomous Agents, and the frontier of #AI4Science especially AI for Drug Discovery.
๐ We are also hiring ML Scientists/Engineers, SWEs, UX Researchers, Summer Interns at the Foundation Models team, #PrescientDesign @genentech (@Roche) to help us build:๐งชA cutting-edge Scientific Agent platform; ๐งฌAutomation and reasoning engines for real-world drug discovery pipelines; ๐ง Large Language Models trained specifically for molecular and protein tasks.
If you are interested in applying AI to real scientific challenges, let's grab a coffee or chat. Feel free to DM me!
#NeurIPS #MachineLearning #Genentech #Hiring #DrugDiscovery #GenerativeAI #Intern
Our foundation model team is hiring ML Engineers to build next-gen LLM agent platform. Exciting opportunity to push the frontier of AI in science and join a world-class cross-functional team on machine learning and science!
๐จ Weโre hiring at Frontier Research @PrescientDesign @genentech! Work on statistical ML for drug discovery:
- Bayesian inference & decision theory
- Uncertainty quantification
- Multi-objective optimization
DMs open, feel free to reach out directly!
๐ https://t.co/A5Brn9R5lD
Check out our new work using AI agents for gene panel design, multimodal annotation and cellโcell interaction analysis with performance comparison with human scientists!
๐Meet SpatialAgent: an AI agent for spatial biology (a key tech to understand cancer and develop therapies)
It rocks from experiment design to data analysis to hypothesis generation. It supports multi-organs, spans species, and can work alone or team up with human scientists๐งโ๐ฌ
๐Excited to share our latest work: OpenVLThinker, an exploration into enhancing vision-language models with R1 reasoning capabilities.
By iterative integration of SFT and RL, we enabled LVLMs to exhibit robust R1 reasoning behavior. As a result, OpenVLThinker achieves a 70.2% accuracy on multimodal reasoning benchmarks like MathVista, matching the performance of 72B-scale models.
๐Blog: https://t.co/kqY8tLr3yJ
๐คModel: https://t.co/shnialgaVt
๐ปGitHub: https://t.co/sXs74Rkpew
Lab-in-the-loop therapeutic antibody design
At @PrescientDesign @genentech we have spent 3+ years reimagining drug discovery. We built a machine learning system to design and execute experiments. Here's how it works and what we can do ๐งต
1/
AAAI Spring Symposium on Agentic AI for Science is calling for papers. We will have a fun 2.5-day event with speakers and participants from agents, LLM and science communities in San Francisco from Mar 31 to Apr 2.
๐ Paper submission deadline: Feb 1, 2025
๐ Both archival and non-archival, long and short papers are welcome
๐ Details: https://t.co/gpAGn7tprz
Please also consider submitting to workshops on this topic at ICLR (https://t.co/KlSt1SktgS) and TheWebConf (https://t.co/09QYvyptZT)! Join us in SF!
Do LLMs know the ๐งโ๐คโ๐ง cultural and โ๏ธ legal safety across our globe?
Our #NeurIPS2024 paper ๐ SafeWorld dives into this. We benchmark across 50 countries and 493 regions/races.
We've also cooked up SafeWorldLM, outshining even gpt-4o by a whopping 20% in human evaluations!๐
๐จ๐ฆ Excited to attend #NeurIPS & #ML4H this week! Iโll be presenting work on LLMs for health and diagnosis, efficient decoding, multimodal reasoning, and more.
Iโm on the academic & industry job market, eager to chat about my on-going efforts on agents for scientific discovery, LLM post-training, and potential opportunities.
Ping me to connect and chat! ๐
๐ข Iโm recruiting PhD students @CS_UVA for Fall 2025!
๐ฏ Neurosymbolic AI, probabilistic ML, trustworthiness, AI for science. See my website for more details: https://t.co/LDOPjISmBo
๐ฌ If you're interested, apply and mention my name in your application: https://t.co/xU1w98Rycv
Prediction: within the next year there will be a pretty sharp transition of focus in AI from general user adoption to the ability to accelerate science and engineering.
For the past two years it has been about user base and general adoption across the public. This is very natural because user growth is a critical part any business model. But at this point I would say that there is widespread accessibility of LLMs: for most queries from the average person on earth, many LLMs can answer pretty well.
In the upcoming five years I think the focal point will be the ability for AI to accelerate engineering and scientific research, which is the engine of progress in technology. At the frontier of innovation in any field, by definition there will be many open questions and a lot of headroom for better AI to make a difference. The stakes will be very high because progress compounds and also because AI accelerating AI research itself is a strong positive feedback loop.
The other way of saying this is that there is somewhat limited headroom for improving the average user query, but massive headroom for improving the experience for the 1% of queries that would accelerate technological advancement, as well as on queries that people would want to ask the model but currently donโt because models are not smart enough to answer. AI research tends to improve where there is great headroom, and in scientific innovation there be substantial upside.
Our new preprint simulates the ๐คthinking process of the ๐knowledge graph constructors and utilizes the KG structure as inspiration for reasoning. It populates the sparse KG with hypothetical knowledge connections, concretizes the internal knowledge of LLMs, and constructs counterfactual reasoning chains.
๐ Retrieve knowledge as a high-level hint for faithful reasoning, combine parametric and non-parametric knowledge
๐ Prove the ability of LLM to conduct deductive reasoning using very limited information, provide a new perspective for inference time computation
๐ Especially useful in scientific domains where the prompts could be knowledge and reasoning intensive, but building inclusive databases is expensive or infeasible.
Full paper: https://t.co/wWY8q71CB1
Collaborated with Jiashu He, Jinxuan Fan, @DanRothNLP, @WeiWang1973 and Alejandro Ribeiro from @Penn@UCLA@UCBerkeley@upennnlp