Larry Yinxi Li

CS MMath Student @UWCheritonCS, prev BSc in CS @cuhkcse

[Larry Yinxi Li @LarryYinxiLi]

Act now for respect, quality education and liveable wages at @UWaterloo . Tell #UWaterloo it's time for a fair first contract for graduate teaching assistants and research assistants : https://t.co/WLkT3F9kk9

[will brown @willccbb]

can we please be serious

[Larry Yinxi Li @LarryYinxiLi]

🥰 Huge thanks to @yuntiandeng for his guidance

and my advisor @pyniex for continuous support!

[Larry Yinxi Li @LarryYinxiLi]

📌 Modern code LLMs rely on subword tokenization (like BPE). But these tokenizers have no idea about code grammar, they just merge characters by frequency.

So what happens when subword tokenization ignores grammar? 🤔

🤗 Our Paper : https://t.co/moUEFnXRGN

#LLM4Code
(1/n)

[Larry Yinxi Li @LarryYinxiLi]

📎 Our takeaway:
Current subword tokenizers are not grammar-aware, and that limits code LLM reliability and robustness.

🧩 We’ll keep exploring better, grammar-aligned tokenization for future code models!

#tokenization #artificial_intelligence #LLMs #SoftwareEngineering
(6/n)