William

[Joe Stacey @_joestacey_]

After going to NAACL, ACL and #EMNLP2024 this year, here are a few tips I’ve picked up about attending *ACL conferences.

Would love to hear any other tips if you have them! 🙂

1. This might be obvious, but I suggest showing everyone the same respect and interest regardless of how prestigious their university is, how junior they are, or how famous they are. I think everyone has lots of interesting things to say and is worth the time to talk to 🙂

2. Don’t be afraid to say hi to random people you don’t know, and be like “is it ok if I join you?”. Sometimes it won’t work out, but most of the time people will be happy to chat. I’m always nervous the first few times, but it quickly becomes natural.

3. Try not to be hard on yourself about social situations. I find that social situations don’t always go perfectly, and that’s ok. I think most of us feel awkward sometimes. Just keep at it and learn as you go 🙂

4. I now start reading conference papers when the accepted paper list is out (well before the conference starts). This helps me to take things in much better during the conference itself.

5. I’m also way more proactive messaging people on the conference app, eg if another PhD student has done a paper I really like I’ll ask to have a chat.

6. Don’t miss out on the workshops! They’re such a fantastic way of meeting people with similar research interests.

7. Be generous with your own time too if people message you and would like to talk. I’ve had really cool conversations with people who have messaged me.

8. Plan at the beginning of the day (or night before) which posters you want to visit, and write down the titles. Then at the beginning of each poster session write down the board IDs for these papers. I love being spontaneous, but I find having an initial plan stops me wandering aimlessly.

9. When presenting a poster, personally I like to acknowledge people who arrive when I’m still talking to someone so they don’t feel ignored, eg “I’ll start from the beginning again in x minutes”, or just saying hi!

10. Posters are often better than orals. I personally only watch oral talks online rather than sitting through the sessions. In my experience posters are a much better use of time.

Anything you think i should add to my list for next time? 🙂 Or more importantly is there anything above you think I’m doing wrong??

P.S. there was a great tweet going around before EMNLP with some advice before the conference. I can’t find it now, but thanks to the author for the idea!

[elvis @omarsar0]

Overview of LLMs for Evaluation

LLMs as tools for evaluation is growing rapidly.

This new paper thoroughly surveys the methodologies and explores their strengths and limitations.

There are a lot of creative ways in which LLMs are used for evaluation. The taxonomy below provides a good summary of the different approaches such as those involving prompt engineering or calibrating open-source LLMs for evaluation.

This is a very active area of research and not all is perfect. Some challenging things to keep in mind are dealing with the biases of LLM evaluators and tailoring them to specific domains that may require specialization or domain expertise.

[Andrej Karpathy @karpathy]

# On the "hallucination problem"

I always struggle a bit with I'm asked about the "hallucination problem" in LLMs. Because, in some sense, hallucination is all LLMs do. They are dream machines.

We direct their dreams with prompts. The prompts start the dream, and based on the LLM's hazy recollection of its training documents, most of the time the result goes someplace useful.

It's only when the dreams go into deemed factually incorrect territory that we label it a "hallucination". It looks like a bug, but it's just the LLM doing what it always does.

At the other end of the extreme consider a search engine. It takes the prompt and just returns one of the most similar "training documents" it has in its database, verbatim. You could say that this search engine has a "creativity problem" - it will never respond with something new. An LLM is 100% dreaming and has the hallucination problem. A search engine is 0% dreaming and has the creativity problem.

All that said, I realize that what people *actually* mean is they don't want an LLM Assistant (a product like ChatGPT etc.) to hallucinate. An LLM Assistant is a lot more complex system than just the LLM itself, even if one is at the heart of it. There are many ways to mitigate hallcuinations in these systems - using Retrieval Augmented Generation (RAG) to more strongly anchor the dreams in real data through in-context learning is maybe the most common one. Disagreements between multiple samples, reflection, verification chains. Decoding uncertainty from activations. Tool use. All an active and very interesting areas of research.

TLDR I know I'm being super pedantic but the LLM has no "hallucination problem". Hallucination is not a bug, it is LLM's greatest feature. The LLM Assistant has a hallucination problem, and we should fix it.

</rant> Okay I feel much better now :)

[Ali Ghodsi @alighodsi]

Free Dolly! Introducing the first *commercially viable*, open source, instruction-following LLM. Dolly 2.0 is available for commercial applications without having to pay for API access or sharing data with 3rd parties. https://t.co/HtRVSmArFU

[Jackson Greathouse Fall @jacksonfall]

I gave GPT-4 a budget of $100 and told it to make as much money as possible.

I'm acting as its human liaison, buying anything it says to.

Do you think it'll be able to make smart investments and build an online business?

Follow along 👀

[Nate Gruver @gruver_nate]

CNNs are famously equivariant by design, but how about vision transformers? Using a new equivariance measure, the Lie derivative, we show that trained transformers are often more equivariant than trained CNNs!
https://t.co/eySuHmpTWl
w/ @m_finzi @micahgoldblum @andrewgwils 1/6

[Aleksa Gordić (水平问题) @gordic_aleksa]

I get asked a lot about what it takes to become a great SE/ML engineer/researcher. My default answer now is:

Become one with debuggers and develop your debugging skills.

One of the core personality traits that goes tightly coupled with it: patience.

[Russ Salakhutdinov @rsalakhu]

1) My observation when working at Apple and many other industrial partners is that deep learning researchers who are well versed in graphical models, variational inference, CRFs, MCMC, uncertainty quantification are on average much more creative when solving real-world problems.

[William @williamteo]

Just completed the #MajulahGCP Challenge! #MajulahGCP