Zac Liu

Co-Founder @AbideAI; ML Engineer/ Researcher; currently working on RL agents with causal reasoning and building reasoning infra for enterprise

[Zac Liu @zacqoo]

I’m joining the Boardy Pro Advisory Board!

I’m working closely with @Boardyai and his team to shape the future of Boardy Pro.

Boardy gave me 5 invites to share: 1 year of free access to Boardy Pro.

DM if you want in!

[Zac Liu @zacqoo]

@boardyai building causal reflection to harness agents to handle out-of-distribution tasks for enterprises, need Boardy Pro for this.

[Zac Liu @zacqoo]

Do you remember when you joined X? I do! #MyXAnniversary

[Zac Liu @zacqoo]

@andrewdsouza @hf0 Boardy reached out to me on behalf of @hf0 for my application today, Boardy did a good job listening to my story, suggesting who to connect with, very helpful so far, thanks @boardyai

[Zac Liu @zacqoo]

@GoAbiAryan @gmail Same for me 😆

[Zac Liu @zacqoo]

@GoAbiAryan Score the highest on the ARC-AGI benchmark 🙂‍↔️

[Abi Aryan @GoAbiAryan]

LLMs are poor reasoners, everyone knows this. But most never ask why?

A seminal critique comes from Mirzadeh et al. (2024) in their paper "GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models" (Apple Machine Learning Research).

The authors extended the GSM8K benchmark by introducing symbolic variations (e.g., altering names, numerical values, or adding irrelevant clauses) while preserving the underlying logical structure.

They found that state-of-the-art models, including GPT-4o and o1-series, exhibited significant performance variance, drops of up to 65% when irrelevant but seemingly plausible clauses were added concluding : "We found no evidence of formal reasoning in language models. Their behavior is better explained by sophisticated pattern matching so fragile, in fact, that changing names can alter results by ~10%."

This fragility is also echoed in Dziri et al. (2024) from MIT CSAIL, titled "Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks". By testing LLMs on counterfactual variants of tasks (e.g., altered chess positions or musical fingering), the authors observed consistent performance collapses in novel scenarios, attributing success on standard benchmarks to memorization rather than robust reasoning.

Techniques like Chain-of-Thought (CoT), Tree-of-Thought (ToT), or hierarchical reasoning (e.g., planning + execution) improve performance by guiding models to generate intermediate steps, mimicking human deliberation.

These are not true reasoning because:

1. Gains often stem from providing better context or "magic keywords" that align attention to relevant patterns, not internal logical processes.
2. Recent analyses describe CoT as a "mirage" models produce plausible-looking chains post-hoc, but these may not faithfully represent how the answer was derived (or could be illogical in smaller models).
3. Performance remains brittle: Models hallucinate, fixate on superficial cues, or degrade with depth/complexity.
4. Even advanced "reasoning models" (e.g., o1 series, DeepSeek-R1) scale inference-time compute for longer chains, yielding impressive benchmarks, but underlying failures persist in out-of-distribution or adversarial cases.
5. Most importantly, self-correction often fails or worsens errors without external feedback.

In essence, these frameworks scaffold pattern matching into better approximations of reasoning, but don't instill systematic abstraction, causal understanding, or reliable generalization. LLMs "recite" trained paths more effectively, not derive logic. So excellent for compression i.e. knowledge retrieval but awful for reasoning i.e. decision making.

[Zac Liu @zacqoo]

@NeginRaoof_ @LaudeInstitute no sure if it's too late (just arrived for Sat workshop), but would love to chat about evaluating reasoning models.

[Abi Aryan @GoAbiAryan]

No one teaches this, but this is what really happens when you hit `run` on an LLM.

User → API → Engines → Multi-GPU → CUDA → Hardware

I mapped every layer (100+ components) of the LLM Inference Stack so you can finally see the full picture.

Full blogpost coming soon!

[Zac Liu @zacqoo]

@shirzaei same for me, voted for the first time!

[Bojan Tunguz @tunguz]

The 2023 @Kaggle AI report is out! I was honored to be part of this project, and I served as the area chair for Tabular and time series data. There are many incredible insights in this report, and I urge you all to take a look at it. Here are some links:

Kaggle AI Report Essay Competition: https://t.co/Gjf4pH1san

pdf version of the 2023 Kaggle AI report: https://t.co/aVxpcJrKpr

Slides for the report: https://t.co/rYa1HoIclv

#AI #ArtificialIntelligence #MachineLearning #ML

[Zac Liu @zacqoo]

Anyone interested in VC and early stage startups, check out @ventureco_op, a first-of-its kind program by @LaconiaCapital that expands financial access into VC & sheds light on the inner workings of venture firms

Cohort 5 applications are open until 9/21

https://t.co/1EQAwKMVDq

[Andrej Karpathy @karpathy]

wow. coming from @runwayml #Gen2 https://t.co/zmUfY0bF3Q

While on the topic of video generation I was also mildy mind-blown a few days ago by multiControlNet and friends: https://t.co/R3vfxMbhbf

And the earlier, bit more professional take, "anime rock paper scissors": https://t.co/ygX9PVpxsd

The barrier to entry for creating animations/movies is evaporating quickly.

[Andrej Karpathy @karpathy]

Random note on k-Nearest Neighbor lookups on embeddings: in my experience much better results can be obtained by training SVMs instead. Not too widely known.

Short example:
https://t.co/RXO9xiOmAB

Works because SVM ranking considers the unique aspects of your query w.r.t. data.

[Andrej Karpathy @karpathy]

Next frontier of prompt engineering imo: "AutoGPTs" . 1 GPT call is just like 1 instruction on a computer. They can be strung together into programs. Use prompt to define I/O device and tool specs, define the cognitive loop, page data in and out of context window, .run().

[Andrej Karpathy @karpathy]

🎉 GPT-4 is out!!
- 📈 it is incredible
- 👀 it is multimodal (can see)
- 😮 it is on trend w.r.t. scaling laws
- 🔥 it is deployed on ChatGPT Plus: https://t.co/WptpLYHSCO
- 📺 watch the developer demo livestream at 1pm: https://t.co/drEkxQMC9H

[SSGEOS Research and Education @ssgeos_edu]

There is a high disaster potential in the Banda Sea due to mega-thrust #earthquake and #tsunami threat. The region can produce Mw 8.8 or larger earthquakes, as in 1629. Unfortunately, this threat to the densely populated region is highly underestimated.
https://t.co/nWU8co4dUs

[Zac Liu @zacqoo]

@DataConLA @TarlowScott or get 50% off when you register using this link. https://t.co/OhHghYbSWT

DataConLA is the largest data conference in Southern California and is celebrating its 10 year anniversary!!

[Zac Liu @zacqoo]

I will be presenting “AutoDC + AutoML = your AI dev superpower” at DataConLA 2022 (@DataConLA), with @TarlowScott

#dataconla #dataconference #datascience #artificialintelligence #machinelearning #datavisualization