Mickey Verma

NLP Senior Data Scientist at BofA. NLP Explainability Practitioner/#ResponsibleNLP. Loves to learn and play piano with my two boys. Opinions are my own.

[Mickey Verma @MickeyVerma16]

@ChristophMolnar What if it does not impact you significantly?

[Sharon Zhou @realSharonZhou]

A course on finetuning LLMs coming to you!!

So you can build your own private custom LLMs, eg. on @MetaAI Llama2

🚢 Ship your own LLM
🚢 Get shipped a stuffed llama (maybe)

Sneak peek of us with 🦙s here:

[Machine Learning Mastery @TeachTheMachine]

A Gentle Introduction to Hallucinations in Large Language Models https://t.co/Z86CmcsnKa

[Santiago @svpino]

GPT-4 is getting worse over time, not better.

Many people have reported noticing a significant degradation in the quality of the model responses, but so far, it was all anecdotal.

But now we know.

At least one study shows how the June version of GPT-4 is objectively worse than the version released in March on a few tasks.

The team evaluated the models using a dataset of 500 problems where the models had to figure out whether a given integer was prime. In March, GPT-4 answered correctly 488 of these questions. In June, it only got 12 correct answers.

From 97.6% success rate down to 2.4%!

But it gets worse!

The team used Chain-of-Thought to help the model reason:

"Is 17077 a prime number? Think step by step."

Chain-of-Thought is a popular technique that significantly improves answers. Unfortunately, the latest version of GPT-4 did not generate intermediate steps and instead answered incorrectly with a simple "No."

Code generation has also gotten worse.

The team built a dataset with 50 easy problems from LeetCode and measured how many GPT-4 answers ran without any changes.

The March version succeeded in 52% of the problems, but this dropped to a pale 10% using the model from June.

Why is this happening?

We assume that OpenAI pushes changes continuously, but we don't know how the process works and how they evaluate whether the models are improving or regressing.

Rumors suggest they are using several smaller and specialized GPT-4 models that act similarly to a large model but are less expensive to run. When a user asks a question, the system decides which model to send the query to.

Cheaper and faster, but could this new approach be the problem behind the degradation in quality?

In my opinion, this is a red flag for anyone building applications that rely on GPT-4. Having the behavior of an LLM change over time is not acceptable.

Have you noticed any issues when using GPT-4 and ChatGPT lately? Do you think these problems are overblown?

[Machine Learning Mastery @TeachTheMachine]

What are Large Language Models https://t.co/1fOaGhvvNZ

[Sahil Bloom @SahilBloom]

How to learn anything.

The Feynman Technique:

[Christoph Molnar 🦋 christophmolnar.bsky.social @ChristophMolnar]

I have started to create cheat sheets for model interpretation.

So far:

• Logistic regression: https://t.co/iNILd5EhER
• SHAP plots for tabular data: https://t.co/rkFuTcaajN

They are free (pay what you want)

Do you have any topic wishes for the next cheat sheets?

[Christoph Molnar 🦋 christophmolnar.bsky.social @ChristophMolnar]

Super excited to release my new book Modeling Mindsets tomorrow 🥳

This is my first book since I decided to become a full-time writer.

And it's the book I wish I had read a few years ago to save myself a lot of time.

1/n

[Mickey Verma @MickeyVerma16]

@ChristophMolnar Yes, yes, and YES, PLEASE!

[David Crowder @crowdermusic]

#mondaysbelike

[Christoph Molnar 🦋 christophmolnar.bsky.social @ChristophMolnar]

8 Pitfalls To Avoid When Interpreting Machine Learning Models on Mindful Modeler https://t.co/V03RPZ4Rme

[Mickey Verma @MickeyVerma16]

@ChristophMolnar @ndwignall Acknowledge it and move on. Tomorrow no one will even remember, as there’s always something else to attend to next! #Life #BHumble

[Christoph Molnar 🦋 christophmolnar.bsky.social @ChristophMolnar]

Supervised machine learning models are deployed everywhere.

It's an open secret that all models have a huge problem:

Performative prediction - when predictions change future outcomes.

How to spot and handle this problem: A thread 🧵

[Christoph Molnar 🦋 christophmolnar.bsky.social @ChristophMolnar]

The interpretation of SHAP / Shapley values is not so straightforward.

If you use Shapley values:

- Are you certain how to interpret them?
- Do you have a text "template" for interpretation?

I'm thinking about creating a cheat sheet about the correct interpretation.

[Pratham @Prathkum]

Learn for free:

HTML → https://t.co/ozFIYNL2Pe
CSS → https://t.co/ZodiZZOaCD
JavaScript → https://t.co/sjTtpL3Cm2
Git and GitHub → https://t.co/x644UaqYat
React → https://t.co/V9yMEfH352
API → @Rapid_API
SQL → https://t.co/69jSBmUzTe
Blockchain → https://t.co/Jsihlwwskl

[Blake Burge @blakeaburge]

You'll get better at Excel with these 8 simple tips: 📊

[Hila Chefer @hila_chefer]

[1/n] Can explainability improve model accuracy? Our latest work shows the answer is yes!
https://t.co/OMts8xHqOC
https://t.co/xvx5VZjmoF

We noticed that ViTs suffer from salient issues- their output is often based on supportive signals (background) rather than the actual object

[Mickey Verma @MickeyVerma16]

@ryanwesslen @explosion_ai Congratulations, Ryan!

[Damian Romero @DamianRomero_CL]

✨ Named Entity Recognition vs SpanCategorization in @spacy_io ✨

#NLProc

spaCy's NER component is great for short non-overlapping labels, but what about longer ones that may overlap?

For that, we have `spancat`

You compare their performance using the sample code in this repo

[Mickey Verma @MickeyVerma16]

nerdlegame 48 3/6

⬛🟩🟩🟩⬛⬛🟩⬛
⬛🟩🟩🟩🟩⬛🟩⬛
🟩🟩🟩🟩🟩🟩🟩🟩