Andrew DeCotiis-Mauro

Data Engineer; Cancer Biotech; Biochemical physicist in a past life;

[Andrew DeCotiis-Mauro @kaumaron]

So many things to say but not enough time to write

[Andrew DeCotiis-Mauro @kaumaron]

@DataRepublican @JustinPDuncan @JuddLegum @elonmusk @Oilfield_Rando Why don't you just provide the queries for analysis and critique?

[Andrew DeCotiis-Mauro @kaumaron]

@coldwintersand @JuddLegum @DataRepublican @elonmusk I'm still waiting for the query

[Andrew DeCotiis-Mauro @kaumaron]

@DataRepublican @JuddLegum @Oilfield_Rando Where are you sharing your queries.

[Andrew DeCotiis-Mauro @kaumaron]

@EcZachly Cool. They need more people without SME

[Andrew DeCotiis-Mauro @kaumaron]

@RepSherrill @threadreaderapp unroll

[Andrew DeCotiis-Mauro @kaumaron]

@johnastoehr @threadreaderapp unroll

[Andrew DeCotiis-Mauro @kaumaron]

@AdamRackis @msdGYT exactly why I switched like 15 years ago

[Angie Jones @techgirl1908]

the White House said use better programming languages bro
https://t.co/BrpKXC7lCz

[ThePrimeagen @ThePrimeagen]

“The fundamental problem of software testing… is that software has to handle many situations that the developer has never thought of or will never anticipate. This limits the value of testing, because if you had the foresight to write a test for a particular case, then you probably had the foresight to make the code handle that case too. This makes conventional testing great for catching regressions, but really terrible at catching all the “unknown unknowns” that life, the universe, and your endlessly creative users will throw at you.” - Will Wilson

[Andrew DeCotiis-Mauro @kaumaron]

@generativist It was great when you could afford to buy an apartment complex for a $10k loan. Whenever that was

[Jesse 🌱 @Wenmoonlaw]

Join @attorneyjeremy1 and I today 11AM EST to talk about legal limit of how much wood a woodchuck can chuck . . . No wait blockchain and customer/user rewards!

For a little sneak peak of an announcement to come, see the next “X” in this thread

https://t.co/voUQWsRxzw

[Jesse 🌱 @Wenmoonlaw]

This Thursday at 5 pm Eastern (2PM PST) I will be hosting my first space called “Seedstarter Money Tree”.

This will be a recurring space allowing listeners to learn more about the companies offering on @JoinSeedstarter.

https://t.co/I6NdUVKMhx

[Andrew DeCotiis-Mauro @kaumaron]

@jesse_hynes Why you talking bout me

[Andrew DeCotiis-Mauro @kaumaron]

@10x_er Hey it's working for ChatGPT

[Andrew DeCotiis-Mauro @kaumaron]

@DeAiDave @thisisbrians @8teAPi Specifically its the number of papers that have received at least h citations - this person has at least 45 papers and those 45 papers have been referenced by others 45 times or more

[Santiago @svpino]

GPT-4 is getting worse over time, not better.

Many people have reported noticing a significant degradation in the quality of the model responses, but so far, it was all anecdotal.

But now we know.

At least one study shows how the June version of GPT-4 is objectively worse than the version released in March on a few tasks.

The team evaluated the models using a dataset of 500 problems where the models had to figure out whether a given integer was prime. In March, GPT-4 answered correctly 488 of these questions. In June, it only got 12 correct answers.

From 97.6% success rate down to 2.4%!

But it gets worse!

The team used Chain-of-Thought to help the model reason:

"Is 17077 a prime number? Think step by step."

Chain-of-Thought is a popular technique that significantly improves answers. Unfortunately, the latest version of GPT-4 did not generate intermediate steps and instead answered incorrectly with a simple "No."

Code generation has also gotten worse.

The team built a dataset with 50 easy problems from LeetCode and measured how many GPT-4 answers ran without any changes.

The March version succeeded in 52% of the problems, but this dropped to a pale 10% using the model from June.

Why is this happening?

We assume that OpenAI pushes changes continuously, but we don't know how the process works and how they evaluate whether the models are improving or regressing.

Rumors suggest they are using several smaller and specialized GPT-4 models that act similarly to a large model but are less expensive to run. When a user asks a question, the system decides which model to send the query to.

Cheaper and faster, but could this new approach be the problem behind the degradation in quality?

In my opinion, this is a red flag for anyone building applications that rely on GPT-4. Having the behavior of an LLM change over time is not acceptable.

Have you noticed any issues when using GPT-4 and ChatGPT lately? Do you think these problems are overblown?

[Andrew DeCotiis-Mauro @kaumaron]

@jesse_hynes Sick

[Andrew DeCotiis-Mauro @kaumaron]

@generativist I've been saying this for like 15 years

[Andrew DeCotiis-Mauro @kaumaron]

@Will_DeCotiis So much slower