Shrey Modi

Infra @FireworksAI_HQ,
CS @iitbombay, @uchicago.
Poker. RL and Continual Learning research.

[Shrey Modi @ShreyModi13]

One of the major hurdles in solving continual learning through auto-research is the creation of high quality training data and evaluation benchmarks from production traces itself. These systems would have to spend a lot of compute and $$ in running ablations across various data curation recipes.

Different algorithms require different characteristics in the training data and that would have to be curated from the ever-changing production traces. The curation recipes itself would have to be adapted on the fly by these auto-research systems. Not all data is trainable.

PostTrainBench traces show us the models are not very good at conducting data ablation experiments.

[Shrey Modi @ShreyModi13]

@fabrisera2000 @a16z @inxoy_ @GEVS94 @ernesto_nam I love when people limp (free money on the table)

[Shrey Modi @ShreyModi13]

The database is here - https://t.co/QM4a0Mxe98

[Shrey Modi @ShreyModi13]

I analysed AI-native companies from the largest open-source US unicorn companies database by @IlyaStrebulaev (been one of my favourite accounts all these years)

1. Not surprisingly, 22 foundational model labs own 62% of all AI-native unicorn value. ($1.94T of $3.15T). Anthropic and OpenAI are $1.82T of that between them

2. Time to $1B varies by category, model labs and chip companies lead with median ~ 1 year, chips and AI cloud 3, robotics and defense 4, healthcare and legal 5-6, customer service and workflow 7, sales and marketing 8.

3. Defense went 1 unicorn in 2023, 1 in 2024, then 12 in 2025. Robotics went 0 in 2023, then 16 across 2024-26. Anduril alone ($61B) is 47% of the entire defense category's value. Shows a heavy focus on defense and robotics.

Infrastructure companies lead application layer companies (with the exception of coding agents!).

[Shrey Modi @ShreyModi13]

A very interesting paper. On-policy distillation has been getting all the hype and one of the major failure reasons of this algorithm is when the teacher policy is too "far" from the student policy. When applied to multi-turn use-cases the differences add up and the student-policy starts disagreeing with the teacher policy by a large margin. But often the disagreement is large only because the teacher has never been in the student’s state, so the update is noise, not signal.

I've been experimenting with various ways of mitigating this and this paper does something really smart! To figure out if the teacher is genuinely teaching the correct action, this paper adds the teacher action instead of the student action to the student trace and sees whether in the future the trace leads to a success and only then updates the student with this teacher supervision.
Would love to apply this to more use-cases, paper of the month?

[Xiuyu Li @sheriyuo]

Agentic on-policy distillation has a temporal problem: a teacher can look better at the current turn and still push the student into a worse future.

FutureBridge-OPD intervenes at high-disagreement states, restores the same state, rolls the student forward again, and keeps guidance only when the paired continuation becomes more teacher-preferred.

Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillation
Paper: https://t.co/xJo6ToHmAz

[Shrey Modi @ShreyModi13]

@achowdhery is there any place I can get more data on this? would love to do some analysis on the demand and supply of various accelerators

[Shrey Modi @ShreyModi13]

For post-training/evals, and for bringing auto-research to the enterprise, we need to build exact replicas of the environments agents will act in during production, for every use case and a large part of this environment will be internal tools. Is this automatable by coding agents? How are people thinking about it?

[Shrey Modi @ShreyModi13]

@gpusteve goat

[Shrey Modi @ShreyModi13]

a new wave of 2T+ parameter models? waiting for deepseek's launch.

I am especially excited about the 27B parameter model, Qwen has been excellent with their smaller model releases. They're really pushing the frontier there.

[Qwen @Alibaba_Qwen]

📢Meet Qwen3.8-Max — our most capable model to date.

Next week, the open weights of Qwen3.8-Max will be released, and Qwen3.8-27B is also going open-weights to meet you all!🎉

Qwen3.8-Max, a new bar for coding and cowork at 2.4T parameters:

- Autonomous coding: 10+ days of self-evolving development, from empty folder to production without hand-holding, complete project trace in the GitHub：https://t.co/iVHZWQoeSo
- Real work, real results: Production-quality deliverables across hundreds of professions.
- Long-horizon mastery: System-level autonomous planning with closed-loop adaptive learning, driving 500+ turns of chip design optimization and 365 days of e-commerce strategy.
- Native multimodal intelligence: Vision isn't just input — it's a continuous feedback loop for planning, execution, and self-correction.

💰Pricing:
Input: $2.0 / M tokens
Output: $6.0 / M tokens
Implicit Caching: $0.25 / M tokens

Start building with Qwen3.8-Max! 🚀

📖 Blog: https://t.co/iwjmQxLBof
✅ Qwen Studio: https://t.co/4V2pFvDovG
⚡ API: https://t.co/gAGqaLQGbN

[Shrey Modi @ShreyModi13]

With the cost of software generation going down, there is a whole new class of companies possible that wasn't possible before. These are the vertically integrated companies that own and optimise the end-to-end chain and have lesser dependencies on other businesses/softwares. These companies can build much better products for certain use-cases. Tesla has been doing it for years (owning software, hardware and compiler), a lot more such companies will spring up.

For example, inference companies can optimise all the way down to PTX (compiler), in the future maybe even software-hardware co-design.

Robotic companies will design the sim stack, the perception model, the planner and the actuators against each other.

The best companies are going to start owning things that are critical to their core-business helping them increase their value prop, earlier considered upstream.

[Shrey Modi @ShreyModi13]

i'm not bored on a flight anymore, I just use cloud coding agents

[Shrey Modi @ShreyModi13]

I wanted to check out the state of post-training auto-research so I analysed model traces on PostTrainBench (@posttrainbench ). Models are given 4 small target LLMs, an H100 GPU, and 10 hours to post-train them.

1. Fable 5 is the leader with a 40% benchmark score as compared to human trained instruct models standing at 50%.

2. The benchmark tests post-training on famous benchmarks like AIME, GPQA etc. The ideal dataset mix and training recipes for which are already published in papers online, most of the models directly used these datasets without having to curate their own dataset. They used already human-researched training recipes and just implemented those. This is not going to be the case on unknown benchmarks that enterprise care about.

3. They spent around 25% of their tokens fighting environment issues which points towards a need for an auto-research harness which abstracts away the training infra complexity. They were also too scared of causing OOMs and didn't know the extend to which they could scale up training. They strongly preferred running over small datasets to get a signal.

4. Models showed a lot of reward hacking to the extent that they even submitted the final instruct models on huggingface as their own post-trained model. To bring auto-research to the enterprise, a lot of work on prevention of reward hacking would need to be done.

5. All models gave up before utilising their time budget. I see the future as humans specifying their $ and time budget and models fully utilising it.

6. Models were really good at implementation and running experiments without making any errors. They also first checked and tried to align the harness of evaluation and training which is a common mistake people run into, especially looking chat templates!

7. GPUs suffered about 20% of idle time which can be improved to allow a higher throughput and lower cost for experiments!

Great work by the @posttrainbench team for open-sourcing the model traces! I see auto-research systems coming to the enterprise very soon. The cost of experimentation just has to go down and the certainty of training recipes to work has to increase for enterprises to invest time and money in these systems.

[Shrey Modi @ShreyModi13]

the updated model is 100x cheaper than opus 4.8 with similar capabilities! this is absolutely insane. waiting for the official Deepseek V4-Pro launch, fable level accuracy at 1/100th the cost?

[DeepSeek @deepseek_ai]

🚀 DeepSeek-V4-Flash Official API is now LIVE in public beta!

🔷 We’ve massively upgraded its Agent capabilities—benchmark scores are now far surpassing the V4-Pro-Preview. Check out the massive performance leap below! 👇
🔷 The official V4-Flash now natively supports the Responses API format and is fully adapted for Codex!

Check out the configuration details in our official API docs: https://t.co/smCwQZMeiq

[Shrey Modi @ShreyModi13]

@raj_jagiasi King behaviour

[Shrey Modi @ShreyModi13]

AGI is here if @Polymarket has automated their tweets, the hooks are insane

[Polymarket @Polymarket]

JUST IN: OpenAI CEO Sam Altman reveals humanity is “close to creating a genie that can grant any wish.”

[Shrey Modi @ShreyModi13]

@MohapatraHemant @dchaplot @SarvamAI Big!

[Shrey Modi @ShreyModi13]

@jiang_kev @polarbrowser Congrats @vishy1027 !!

[Shrey Modi @ShreyModi13]

beautifully run company, cooked food nearly at the cost of groceries?

[Stanley Tang @stanleytang]

https://t.co/iwjvRhZFok