Linxing Preston Jiang

PhD student @uwcse interested in theoretical neuroscience.

Also @lpjiang97.bsky.social

[Linxing Preston Jiang @lpjiang97]

I'm excited to share our latest work — "Data Heterogeneity Limits the Scaling Effect of Pretraining in Neural Data Transformers", where we examined the effect of scaling up pretraining data in neural foundation models carefully.🧐 (1/9)

Preprint: https://t.co/islaroGou7

[Xiaochuang Han @XiaochuangHan]

Can we simplify video generation by decomposing it into interleaved text-video co-generation?
Would explicit, repeated thinking in language improve generation in pixels?

We introduce TV2TV: a unified model that jointly learns
- language modeling (next-token prediction)
- video flow matching (next-frame prediction)

At inference, TV2TV dynamically alternates between textual thinking and video generation.

Model generations below: interleaved text plans and video slices (~1–2s) are co-generated over time, conditioned on a single frame per sport.

📖 https://t.co/qMszsl6BAt

[Linxing Preston Jiang @lpjiang97]

Find @WeijiaShi2 at @lm4sci Oct 10 for our latest work on neuro foundation model data scaling!

[Weijia Shi @WeijiaShi2]

Data Heterogeneity Limits the Scaling Effect of Pretraining Neural Data Transformers led by @lpjiang97
⏰: Oct 10, 11:30am
📌: 524A

[Weijia Shi @WeijiaShi2]

At #COLM2025 🇨🇦 this week! Would love to meet old and new friends.

I’ve been thinking about how to train LMs that can leverage high-risk but high-quality data, and how to build omni models by merging specialized ones across modalities.

And come check out our paper 👇

[Kanaka Rajan @KanakaRajanPhD]

(1/8) New paper from our team!

@yuven_duan & @hamzatchaudhry introduce POCO, a tool for FORECASTING brain activity at the cellular & network level during spontaneous behavior.

Find out how we built POCO & how it will transform neurobehavioral research 👇
https://t.co/o956ESdgsw

[Linxing Preston Jiang @lpjiang97]

@KanakaRajanPhD @yuven_duan @hamzatchaudhry @KempnerInst @harvardmed Interesting work! Is there an explanation on why POYO alone did so bad in forecasting (Table 5)? After all POYO transformer's expressivity should incorporate that of a one hidden layer MLP. Maybe MLP did all the work and POYO's outputs on the gating parameters don't vary a lot?

[tuochao chen @tuochao]

Today’s AI assistants passively wait for questions. But what if they could anticipate when to help-without explicit user invocation? Meet LlamaPIE, the first real-time proactive assistant to enhance  conversations via discreet, concise guidance delivered by hearable.#acl2025

[Jin Shang @jinshang1997]

I've been writing some AI Agents lately and they work much better than I expected. Here are the 10 learnings for writing AI agents that work:

1) Tools first. Design, write and test the tools before connecting to LLMs. Tools are the most deterministic part of your code. Make sure they work 100% before writing actual agents.

2) Start with general, low level tools. For example, bash is a powerful tool that can cover most needs. You don't need to start with a full suite of 100 tools.

3) Start with single agent. Once you have all the basic tools, test them with a single react agent. It's extremely easy to write a react agent once you have the tools. All major agent frameworks have builtin react agent. You just need to plugin your tools.

4) Start with the best models. There will be a lot of problems with your system, so you don't want model's ability to be one of them. Start with Claude Sonnet or Gemini Pro. you can downgrade later for cost purpose.

5) Trace and log your agent. Writing agents are like doing animal experiments. There will be many unexpected behavior. You need to monitor it as carefully as possible. There are many logging systems that help. Langsmith, langfuse etc.

6) Identify the bottlenecks. There's a chance that single agent with general tools already works. But if not, you should read your logs and identify the bottleneck. It could be: context length too long, tools not specialized enough, model doesn't know how to do something etc.

7) Iterate based on the bottleneck. There are many ways to improve: switch to multi agents, write better prompts, write more specialized tools etc. Choose them based on your bottleneck.

8) You can combine workflows with agents and it may work better. If your objective is specialized and there's an unidirectional order in that process, a workflow is better, and each workflow node can be an agent. For example, a deep research agent can be a two step workflow, first a divergent broad search, then a convergent report writing, and each step is an agentic system by itself.

9) Trick: Utilize filesystem as a hack. Files are a great way for AI Agents to document, memorize and communicate. You can save a lot of context length when they simply pass around file urls instead of full documents.

10) Another Trick: Ask Claude Code how to write agents. Claude Code is the best agent we have out there. Even though it's not open sourced, CC knows its prompt, architecture and tools. You can ask its advice for your system.

[Kenneth D Harris @kennethd_harris]

A new study led by @timothy_sit shows that different layers of mouse V1 integrate visual and non-visual signals differently. L2/3 activity is dominated by vision (or spontaneous fluctuations) and L5 by movement. This leads to different geometries.
https://t.co/aup77pQZGp

[Linxing Preston Jiang @lpjiang97]

Exciting work on flexible data usage in MoE LMs!

[Weijia Shi @WeijiaShi2]

Can data owners & LM developers collaborate to build a strong shared model while each retaining data control?

Introducing FlexOlmo💪, a mixture-of-experts LM enabling:
• Flexible training on your local data without sharing it
• Flexible inference to opt in/out your data anytime

At 37B parameters, FlexOlmo is competitive across 31 tasks.

[Linxing Preston Jiang @lpjiang97]

@sinclair_allie Congrats, Allie!!

[Linxing Preston Jiang @lpjiang97]

This is joint work with @ChinSengi, Iman Tanumihardja, @XiaochuangHan, @WeijiaShi2, Eric Shea-Brown, and @RajeshPNRao. Please check out the preprint for more details.

Any feedback is appreciated! (9/9)

[Linxing Preston Jiang @lpjiang97]

I'm excited to share our latest work — "Data Heterogeneity Limits the Scaling Effect of Pretraining in Neural Data Transformers", where we examined the effect of scaling up pretraining data in neural foundation models carefully.🧐 (1/9)

Preprint: https://t.co/islaroGou7

[Linxing Preston Jiang @lpjiang97]

Together, our results show that pretraining with more sessions does not naturally lead to improved downstream performance. We advocate for rigorous scaling analyses in future work on neural foundation models to account for data heterogeneity effects. (8/9)

[Weijia Shi @WeijiaShi2]

Our previous work showed that 𝐜𝐫𝐞𝐚𝐭𝐢𝐧𝐠 𝐯𝐢𝐬𝐮𝐚𝐥 𝐜𝐡𝐚𝐢𝐧‑𝐨𝐟‑𝐭𝐡𝐨𝐮𝐠𝐡𝐭𝐬 𝐯𝐢𝐚 𝐭𝐨𝐨𝐥 𝐮𝐬𝐞 significantly boosts GPT‑4o’s visual reasoning performance. Excited to see this idea incorporated into OpenAI’s o3 and o4‑mini models (https://t.co/DcbVgUAq2n).

Huge thanks to my co‑author @huyushi98 @XingyuFu2