Lukas Edman

Post-Doc at TU Munich, interested in just about anything in low-resource NLP

[Lukas Edman @LukasNLP]

Would love to hear your feedback too :)

[Lukas Edman @LukasNLP]

Ever feel like it's too hard to keep track of what LLMs cannot do as well as humans? We're making your life easier over at: https://t.co/dESLuXAmET

We're compiling a list of papers testing the abilities of LLMs against humans. Check it out! And you can help contribute too!

[Lukas Edman @LukasNLP]

@gsarti_ @glnmario My boy growing up so fast 🥲

[Gabriele Sarti @gsarti_]

Excited to kickstart this effort with @glnmario and a star-studded advisory team! 🚀

Also, we're hiring for 2 MTS and 1 Ops manager! Come build the future of white-box audits with us: https://t.co/82Xp4vUak9

[Vilém Zouhar @zouharvi]

Machine translation is not solved and it will take a while for it to be done https://t.co/NRecZdw4RU

[Lukas Edman @LukasNLP]

Everyone talks about how great ChatGPT and Claude are, but Olmo is literally in the world cup finals #NLProc

[babyLM @babyLMchallenge]

🍼 Mask and You Shall Receive: Optimizing Masked Language Modeling for Pre-training BabyLMs
👶 @LukasNLP, Alexander Fraser
https://t.co/tSMLiELc3i

[Lukas Edman @LukasNLP]

@akoksal_ @aclmeeting

[Lukas Edman @LukasNLP]

Thanks to everyone who showed up at my posters: EXECUTE, and Positional Overload w/ Lukas Kinder! #ACL2025 #ACL2025NLP

[Lukas Edman @LukasNLP]

2. Positional Overload: Positional Debiasing and Context Window Extension for Large Language Models using Set Encoding (PDF coming soon!). Lukas Kinder at KIT did all the hard work, showing that we can use shared position IDs across texts to extend context length and debias MCQA

[Lukas Edman @LukasNLP]

Happy to announce two papers at #ACL2025!
1. We made an extension to our CUTE benchmark, EXECUTE! We find that LLMs actually *do* have the ability to do character-level manipulation, but language bias, and tokenization too, get in the way. https://t.co/DnsaRvheOb
1/2

[Lukas Edman @LukasNLP]

@bminixhofer @licwu @PontiEdoardo @huggingface Ah I see, how are the results for a model with purely byte embeddings? Did you try that?

[Lukas Edman @LukasNLP]

@bminixhofer @licwu @PontiEdoardo @huggingface Cool work! I'd love to test these byte models on our CUTE benchmark, I suspect they will do quite well. Any plans on uploading a safetensors version?

[Lukas Edman @LukasNLP]

@nickcdryan The authors responded, it is indeed that the model did not output either yes or no. Strange indeed

[Lukas Edman @LukasNLP]

@nickcdryan @omouamoua @stochasticchasm Great to hear! We are currently working on a multilingual version of CUTE, release date still TBD :)

[Lukas Edman @LukasNLP]

@nickcdryan I was also wondering this, I thought maybe their model didn't output either "yes" or "no". I'll try emailing the authors

[Lukas Edman @LukasNLP]

Thanks to everyone who stopped by my poster at #EMNLP2024 today! Had some really fun and interesting discussions with you all :)

[Lukas Edman @LukasNLP]

I'm happy to announce that our LLM benchmark for measuring orthographic understanding, CUTE, was accepted to EMNLP main! Looking forward to seeing you all in Miami! #NLProc

Arxiv: https://t.co/1gkdoxJUKf
HF Leaderboard: https://t.co/sR0SqVFvBo
Github: https://t.co/3ioe8pb81n

[Gabriele Sarti @gsarti_]

Very excited to present my work at @CisLmu tomorrow; thanks for having me! 😄

[Lukas Edman @LukasNLP]

@vansteenkiste_s @jowenpetty We compared char-level to subword-level for NMT, and found the char models are generally on-par or better, and show word-level processing capabilities when needed. Indeed, the major issue seems to be that inference is ~5x slower.
Link: https://t.co/fVJfSDygJ4