Cheng-Kuang (Brian) Wu

Senior Research Scientist @ Appier AI Research

Research is the humble pursuit of seeing things as they truly are.

[Cheng-Kuang (Brian) Wu @brianckwu]

Check out this cool work by my brilliant colleagues!

[Sin-Han Yang @SinHanYang]

Model merging aims to build a multi-task model cheaply by combining the weights of individual task-specific models. Most methods require auxiliary data to find the coefficients that produce the best merged model. But are we making the best use of this data?

1/n

[Chenhao Tan @ChenhaoTan]

https://t.co/2XpaowTFbx

[Cheng-Kuang (Brian) Wu @brianckwu]

@mariabrbic @FabianGroger @ShuoWen18 Congrats to your team, Maria! It’s a really cool paper and I like it a lot!

[Coder girl 👩‍💻 @dev_maims]

POV: Software engineers 3 hours before the deadline. 😂

[Cheng-Kuang (Brian) Wu @brianckwu]

Quote of the day from Prof. Katerina Argyraki at the EDIC Open House, EPFL:

“PhD should be fun. It’s perhaps the only time in your life when you are paid to do something you find interesting.”

[Sin-Han Yang @SinHanYang]

Please check out our latest work on a new calibration framework for LLMs.

[Cheng-Kuang (Brian) Wu @brianckwu]

# Can We Predict How Likely an LLM Could Solve a Given Problem? 🤔

LLMs’ outputs are stochastic. If we ask them the same question multiple times, we may get different answers. Some of them are correct, some are not. So here's a natural question: for a given problem, can we estimate the probability that the model will answer correctly?

For example, if we ask "What is the capital of the UK?" 100 times, and the LLM answers "London" 30 times and "Paris" 70 times, the success probability is 30%. If we could estimate this probability without actually sampling a hundred times, that would be incredibly useful. It would allow us to determine when a model needs additional information, and to allocate computational resources more intelligently.

This is exactly what our team at Appier AI Research set out to address. We call this research direction Capability Calibration.

In fact, calibration is a well-established topic in machine learning. However, prior work on calibrating LLMs has almost exclusively focused on what we call response calibration, which estimates whether a specific, already-generated response is correct. The issue is that, due to the stochastic nature of language model outputs, the correctness of any single response doesn't reflect the model's overall ability on a given problem.

To capture this notion, we propose capability calibration, a framework that shifts the focus from "Is this particular answer correct?" to "What is the model's expected probability of answering this problem correctly?" In other words, we move from evaluating individual responses to assessing the model's holistic problem-solving capability.

Key takeaways:

* Theory: We formally distinguish capability calibration from response calibration and derive the mathematical relationship between the two.

* Experiments: Across 3 LLMs and 7 datasets, we compare multiple confidence estimation methods : verbalized confidence, P(True), and linear probes trained on internal representations. Linear probes achieve the best cost-performance tradeoff: at less than the cost of generating a single token, they provide meaning performance of capability calibration.

* Applications: (1) Pass@k prediction: estimating "if the model attempts k times, what is the probability of at least one correct answer?" without repeated sampling; and (2) inference-time compute allocation: given a fixed computational budget, distributing retry attempts based on estimated problem difficulty to solve more problems.

I'm very grateful to my collaborator Sin-Han Yang. His curiosity, humor, and sharp logic made this journey productive and enjoyable. Huge thanks also to our team lead at Appier AI Research, Chieh-Yen Lin, and our academic advisors Prof. @YunNungChen, Prof. @HungyiLee2, and Prof. @shaohua0116, whose feedback helped us sharpen our contributions.

🌐 Project page: https://t.co/Agyqto985w
💻 Code: https://t.co/0DE71PBTrq
📄 Paper: https://t.co/MGAif6iA56

#LLM #Calibration

[Bryan Johnson @bryan_johnson]

7 day social media fast is a longevity therapy.
Also outperforms antidepressants.

7 day (FB, IG, X, TikTok)
+ well-being +9%
+ depression -9%
+ anxiety -8%

7 day general
FOMO - 13%

7-day (FB, IG, Snap, TikTok, X)
+ anxiety -16%
+ depression -25%
+ insomnia -15%

[Cheng-Kuang (Brian) Wu @brianckwu]

I agree that taste and judgement is harder to built into AI models than execution skills.

[Amy Tam @amytam01]

https://t.co/geu9GoMQyj

[Dimitris Papailiopoulos @DimitrisPapail]

https://t.co/V7BSUMCcQB

[Nick Khami @skeptrune]

software developers merging code written by Opus 4.5

[Hung-yi Lee (李宏毅) @HungyiLee2]

Faculty Recruitment at National Taiwan University AI Center of Research Excellence (NTU AI‑CoRE)

Start date: August 1, 2026

Apply online (rolling review; early applications encouraged):
https://t.co/p0q8Ti1r8L

Learn more about the Center:
https://t.co/AOAQf3QAQt

[Cheng-Kuang (Brian) Wu @brianckwu]

Check our slides and reading lists if you are interested in robust and trustworthy LLMs!

https://t.co/FfwOrQYnDl

[Cheng-Kuang (Brian) Wu @brianckwu]

# ACML 2025 Tutorial Recap

A few months ago, I had lunch with Prof. Hung-yi Lee to seek advice on career growth as a researcher. I asked what more could be done beyond conducting solid research and publishing papers. He pointed out that our AI research team at Appier already have a strong body of research, and encouraged us to submit a tutorial proposal to share a structured overview of the field. This, he noted, would be a meaningful contribution to the research community and a great opportunity to increase our team’s visibility.

After months of preparation with my collaborators, Zhi Rui Tam (@zraytam) and Prof. Kuan-Hao Huang (@kuanhaoh_) , our tutorial was successfully delivered. The room was full, the Q&A was highly engaging, and after the talk, several researchers shared that they were truly impressed by our presentation.

The success did not come easily. To be honest, it took many rounds of rehearsals and slide revisions (special thanks to Prof. Yun-Nung (Vivian) Chen for her thoughtful feedback during practice sessions). I also spent an intense week confronting my own anxiety and uncertainty. Receiving such positive responses made every bit of that effort worthwhile.

Beyond the tutorial, this week was filled with many memorable and rewarding experiences. Several industry colleagues stopped by our team’s poster at the venue and shared that our research topic is not only practically important, but also easy to understand and communicate. As a researcher, it is deeply fulfilling to see our work resonate with practitioners.

Another highlight was the opportunity to meet and speak with one of the keynote speakers, Aja Huang (Google DeepMind), about his advice for early-career researchers. I never imagined that I would one day have a face-to-face conversation with the legend behind AlphaGo—someone I once watched via dorm-room livestreams during the historic matches against Lee Sedol and Ke Jie when I was still a medical student.

Finally, I would like to express my gratitude to the classmates and mentors who supported me throughout my undergraduate and graduate studies; my colleague Chao-Chung Wu, who discovered our work at EMNLP 2023 and forwarded my information to HR at Appier; and Chih-Han Yu, Founder & CEO of Appier, who made the decision to establish our AI Research team. I wouldn’t have had this experience without any one of you.

Research comes with setbacks, but this week has recharged me fully. Going forward, I’ll keep pushing to produce impactful research.

#ACML2025

[Kuan-Hao Huang @kuanhaoh_]

Glad to back in Taiwan to give a tutorial #ACML2025 together with @brianckwu and @zraytam. Check our slides if you are interested in robust and trustworthy LLMs!

https://t.co/it4f8bC5II

[Cheng-Kuang (Brian) Wu @brianckwu]

# The Need for Continual Learning
(And how our StreamBench work from last year was cited by Google DeepMind!)

Recently, I’ve listened to many interviews with AI researchers, including the father of RL Richard Sutton [1], former OpenAI Chief Scientist Ilya Sutskever [2], and Google DeepMind’s Chief AGI Scientist Shane Legg [3]. A common theme across these conversations is a fundamental limitation of today’s deep learning systems (such as large language models like ChatGPT, which I’ll simply refer to as “AI”): they lack the ability to learn continually.

This resonates deeply with me. Compared to AI systems, humans do not possess all the world’s knowledge, nor can we speak hundreds of languages. Yet when we enter a new environment or face unfamiliar challenges, we are able to acquire new knowledge and skills and adapt to a constantly changing world.

Interestingly, our Appier AI Research team recognized the importance of this direction last year and designed StreamBench [4], a benchmark for measuring an AI system’s ability to continuously improve on a task. This work was published at NeurIPS 2024 (Datasets & Benchmarks Track). One key takeaway from this project is that, at present, we still do not have an effective algorithm that enables deep-learning-based AI systems to truly learn continually.

Both StreamBench and recent work such as Google DeepMind’s Evo-Memory [5] rely on storing an AI’s past experiences in an external memory and retrieving relevant information at inference time by injecting it into the model’s input. While these approaches do improve performance on benchmarks like StreamBench or Evo-Memory, the AI does not truly learn the relationships between pieces of knowledge. Instead, it resembles a human repeatedly reading notes whenever encountering an unfamiliar problem. To form deeper connections between knowledge (or to acquire genuinely new reasoning abilities) I believe these experiences must directly influence the neural network’s parameters.

Why isn’t this done then? Based on my current understanding of the field, there are two major challenges: loss of plasticity [6] and catastrophic forgetting [7]. Fortunately, several promising research directions have begun to address these issues, including continual backpropagation [6], selective weight reinitialization [8], and nested learning [9]. I believe continual learning for deep neural networks will be an exciting and impactful research area in the years to come.

Finally, I’m very happy to see that Google DeepMind’s recent Evo-Memory [5] work cited our StreamBench [4] paper. Many aspects of their problem formulation and dataset design are also closely aligned with ours, which is an encouraging signal that our team’s research taste is on the right track. I’d like to sincerely thank my co-authors Zhi Rui Tam @zraytam, Chieh-Yen Lin, Yun-Nung Chen, and Hung-Yi Lee, as well as our research collaborators William Chen and Chih-Han Yu for their valuable feedback and contributions.

* Check out StreamBench: https://t.co/LOJCQelWBz

* References:
[1] https://t.co/dt69xNlIty
[2] https://t.co/AHOGDakYAa
[3] https://t.co/VuvdLfYt29
[4] https://t.co/LOJCQelWBz
[5] https://t.co/lhsFAAgjr6
[6] https://t.co/LpbUrnqbOY
[7] https://t.co/ap8ZmOyLZR
[8] https://t.co/2uXVdUV2vY
[9] https://t.co/r0gcAAKaRL

* Check out StreamBench: https://t.co/LOJCQelWBz

[Cheng-Kuang (Brian) Wu @brianckwu]

ICLR indeed does a great job! Hope that the reviewing processes of other conferences will also achieve this level of transparency in the future.

[Subbarao Kambhampati (కంభంపాటి సుబ్బారావు) @rao2z]

In the midst of all the angst about @iclr_conf review quality, let's not forget that the angst was only possible because ICLR makes the submissions and reviews "Open" (as OpenReview was originally meant to be--OpenReview was designed bespoke for ICLR..)--thus allowing for third party macro analyses of reviewing.