Maharaj Brahma

Research on NLP #NLPProc #LT4ALL

[Tomasz Limisiewicz @COLM @TomLimi]

Szia 👋🇭🇺
Happy to be in Budapest for EMNLP.

I'll be sharing few lessons learned on byte language models, latent tokenization at tomorrow’s Tokenization Tutorial (in P2 from 9 -12:30).

https://t.co/ABMcUQCcyL

[Maharaj Brahma @mrajbrahma]

Work done with awesome team - Karthika, Rajat @2397_Rajat, Mahesh, Prof. Maunendra Sankar Desarkar, Prof. Ganesh Ramakrishna, and Prof. Rohit Saluja.

[Maharaj Brahma @mrajbrahma]

Hey folks at EMNLP 26.

If you are interested in Multilingual LLM Robustness. Please consider visiting our poster in Hall H (Session 4) today - Karthika will be presenting our work on how LLMs can fail when words (word groups) are reordered.

#EMNLP2026

[Tokenization Workshop (TokShop) @COLM2026 @tokshopMeeting]

🎙️ Excited to announce our second invited speaker: Amir Zamir @zamir_ar, Assistant Professor at EPFL!

Talk: "(Multimodal) Flexible-Length Tokenization"

Looking forward to a talk that pushes tokenization beyond text 👀

[k.shiv kumar @kumarkshivoid]

𝗔𝗻𝗼𝘁𝗵𝗲𝗿 𝗺𝗶𝗹𝗲𝘀𝘁𝗼𝗻𝗲 𝗳𝗿𝗼𝗺 𝗜𝗜𝗧 𝗵𝘆𝗱𝗲𝗿𝗮𝗯𝗮𝗱 𝗡𝗘𝗧𝗫 𝗟𝗮𝗯...!!!

Our work, "Yaksha-Prashna: Bridging the Semantic Gap in eBPF Bytecode Analysis and Diagnostics," has been accepted at ACM CoNEXT 2026.

𝗡𝗘𝗧𝗫 𝗟𝗮𝗯: https://t.co/RjNO9OdDnL

[Stanford NLP Group @stanfordnlp]

There are still opportunities for using detailed linguistics in the age of LLMs!

[Bolei Ma @BoleiMaBolei]

🏆 𝗕𝗲𝘀𝘁 𝗣𝗮𝗽𝗲𝗿 𝗔𝘄𝗮𝗿𝗱 @ 𝗔𝗖𝗟 𝟮𝟬𝟮𝟲

Thrilled to share that our linguistic paper "The Imperfective Paradox in LLMs" received a Best Paper Award at #ACL2026 ! 🎉

Hoping this encourages more linguistics-informed evaluation in NLP.

https://t.co/U2YlJwqpKX

[Joël Niklaus @joelniklaus]

Congrats to @clara__meister for getting the Parity-Aware BPE tokenizer merged into Hugging Face tokenizers!

Learn more about it here: https://t.co/z0fpCekSxw

Use it like this:
from tokenizers.pre_tokenizers import ByteLevel
from tokenizers.trainers import ParityBpeTrainer

tokenizer = Tokenizer(BPE())
tokenizer.pre_tokenizer = ByteLevel()

# Plain text files: one iterator per language
def lines(path):
with open(path) as f:
yield from f

trainer = ParityBpeTrainer(num_merges=32000, variant="base")
trainer.train_from_iterator(
tokenizer,
train_iterators=[lines("en.txt"), lines("de.txt"), lines("zh.txt")],
dev_iterators=[lines("en_dev.txt"), lines("de_dev.txt"), lines("zh_dev.txt")],
)
https://t.co/1EyPrjITNd("parity_bpe_tokenizer.json")

[Jiawei (Joe) Zhou @jzhou_jz]

We love BPE tokenization!

ACL test of time award paper from the classic 2016 machine translation work (also rejected twice).

#acl #nlp #llm @aclmeeting

[LINGUIST List @linguistlist]

Software: NE-Embed: Multilingual Sentence Embedding Model for Northeast Indian Languages: NE-Embed is an open-source multilingual sentence embedding model developed by MWire Labs for Northeast Indian languages. The model is designed for semantic search,… https://t.co/zicRPAuhlN

[Joël Niklaus @joelniklaus]

Happy to share that our paper "Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization" was accepted to #ACL2026 in San Diego 🇺🇸!

When you use ChatGPT or Claude, you pay per token. But did you know that tokenization is wildly unfair across languages? Standard algorithms like BPE learn by counting what appears most frequently in training data, which overwhelmingly favors English and other high-resource languages. The result: speakers of lower-resource languages see their text split into more tokens, paying higher API costs for the same message.

Vocabulary utilization measures what fraction of a tokenizer's vocabulary actually gets used for each language. When utilization is low, the model has failed to allocate meaningful tokens to that language, fragmenting words into inefficient subwords.

We introduce Parity-Aware BPE, which at every step asks: "Which language is currently getting the worst compression?" It then selects merges that specifically help that language. The result is dramatically fairer vocabulary allocation across high, medium, and low-resource languages compared to Classical BPE.

Best part? Models trained with our tokenizer perform just as well on downstream tasks, while achieving near-perfect parity in tokenization costs. Same performance, fairer access.

Joint work with @negarforoutan, @clara__meister, @DebjitPaul2, @sina_ahm, @ABosselut, and @RicoSennrich.

[Clara Meister @clara__meister]

What if your tokenizer could tell you your text's language?

We're excited to introduce 𝗨𝗻𝗶𝗟𝗜𝗗: a simple, data-efficient method for language identification (LID) that builds on the UnigramLM tokenization algorithm.
📄 https://t.co/fp0FUzk4Yi

[Maharaj Brahma @mrajbrahma]

More details: https://t.co/A1H5c7OQe1

[CSE IITH @cse_iith]

𝗣𝗮𝗽𝗲𝗿 𝗔𝗰𝗰𝗲𝗽𝘁𝗮𝗻𝗰𝗲 𝗔𝗻𝗻𝗼𝘂𝗻𝗰𝗲𝗺𝗲𝗻𝘁

Paper titled “DIWALI – Diversity and Inclusivity aWAre cuLture-specific Items for India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Context” accepted at Empirical Methods in Natural Language Processing (EMNLP) 2025, a CORE Rank A* conference.

Authors: Pramit Sahoo, Maharaj Brahma, Maunendra Sankar Desarkar

👏 Congratulations to all the authors!

🔍 Key Highlight:
Large Language Models (LLMs) are widely used across a range of tasks and applications. However, existing LLMs lack cultural alignment and produce biased generations. It is further challenged by the lack of proper evaluation methods and the unavailability of culturally grounded resources. This calls for building cultural resources across various cultures. In this work, DIWALI, a new novel cultural dataset comprising 8,817 culture-specific items from 36 subregions of India across 17 cultural facets. Furthermore, the evaluation of LLMs on the cultural text adaptation task highlights the gaps in deeper-level adaptation.

#CSE #DIWALI #IITHyderabad #LLMs #CulturalAI #InclusiveAI #CulturalAlignment #DatasetCreation #BiasInAI #EMNLP #ResponsibleAI

[Ponnurangam Kumaraguru “PK” @ponguru]

🚀 ACM India Annual Event 2026 #CallForParticipation

📍 IIT Hyderabad | 🗓 14 Feb 2026

Researchers, faculty, students, practitioners are all welcome.

📌 Website & Registration link shared in the comments.

🙏🏾 Please help spread the excitement hashtag#CFBR!

#VPGiri #ProfGiri

[Ponnurangam Kumaraguru “PK” @ponguru]

#SpeakerAnnouncement 🚀🎤 #ACMIndia #AnnualEvent

Prof. Abhik Roychoudhury, Provost's Chair Professor of Computer Science, National University Singapore, will be joining us! ✍🏽 📝 Link in comment to Register.

🙏🏾 Please help spread the excitement #CFBR!

#ACMIndia #VPGiri

[Maharaj Brahma @mrajbrahma]

Good read! Starting for a new level of optimism this year.
https://t.co/7nu52fdJGb

[Ponnurangam Kumaraguru “PK” @ponguru]

#ARCS2026 #SpeakerAnnouncement 🚀🎤

Prof. Vaishnavi Sundararajan @iitdelhi. V's current interests ,showing that protocols for the internet behave as they should, using techniques from formal methods and theorem proving.

🙏🏾 Please help spread the excitement #CFBR!

#VPGiri

[Ponnurangam Kumaraguru “PK” @ponguru]

#ARCS2026 #SpeakerAnnouncement 🎤 Thrilled to share that Prof. Manisha Padala, Assistant Professor at @iitgn will be joining us! Manisha PhD @iiit_hyderabad. She is a leading researcher working in developing fair and private classifiers, exploring fairness in federated settings.

[Raj Dabre @prajdabre]

Very excited to release our ongoing work on IndicBERT-V3!

Some key points:
1. Long context
2. Various model sizes
3. SOTA performance on bitext mining and RAG on our internal evaluations
4. Multilingual Indic support

Go wild!

cc @osanseviero for visibility, since we used Gemma.

@anoopk

[Anoop Kunchukuttan @anoopk]

We had a great time presenting the tutorial yesterday!

The slides are avaible on the tutorial website:

https://t.co/eQxuw3rhYs

Turns out, there was a concurrent tutorial on "Continual Learning for LLMs" which is relevant to language expansion.

https://t.co/Dqo4V7lom6