Jake Firman

Director of Education Technology at @dsstpubschools, @Teachforamerica alum, and former 8th grade math teacher. Views are my own.

[Jake Firman @jakefirman]

This is super interesting. It’s one of the inherent risks of building tools off of technology that you neither own nor control. Certainly is an argument for why open-source models that are run and controlled internally may win out in the long run. #AI #edtech

[Santiago @svpino]

GPT-4 is getting worse over time, not better.

Many people have reported noticing a significant degradation in the quality of the model responses, but so far, it was all anecdotal.

But now we know.

At least one study shows how the June version of GPT-4 is objectively worse than the version released in March on a few tasks.

The team evaluated the models using a dataset of 500 problems where the models had to figure out whether a given integer was prime. In March, GPT-4 answered correctly 488 of these questions. In June, it only got 12 correct answers.

From 97.6% success rate down to 2.4%!

But it gets worse!

The team used Chain-of-Thought to help the model reason:

"Is 17077 a prime number? Think step by step."

Chain-of-Thought is a popular technique that significantly improves answers. Unfortunately, the latest version of GPT-4 did not generate intermediate steps and instead answered incorrectly with a simple "No."

Code generation has also gotten worse.

The team built a dataset with 50 easy problems from LeetCode and measured how many GPT-4 answers ran without any changes.

The March version succeeded in 52% of the problems, but this dropped to a pale 10% using the model from June.

Why is this happening?

We assume that OpenAI pushes changes continuously, but we don't know how the process works and how they evaluate whether the models are improving or regressing.

Rumors suggest they are using several smaller and specialized GPT-4 models that act similarly to a large model but are less expensive to run. When a user asks a question, the system decides which model to send the query to.

Cheaper and faster, but could this new approach be the problem behind the degradation in quality?

In my opinion, this is a red flag for anyone building applications that rely on GPT-4. Having the behavior of an LLM change over time is not acceptable.

Have you noticed any issues when using GPT-4 and ChatGPT lately? Do you think these problems are overblown?

[Jake Firman @jakefirman]

@christopherrufo These pics are awesome! Our family has loved Blake Island for years. How were the raccoons for your trip? Last time we were there was 2020, and the number of raccoons was absolutely crazy. They were everywhere, day and night.

[Jake Firman @jakefirman]

Interesting work by Georgia Tech to use “intermediary” tools to improve factual accuracy of generic large language models in education. Will be interesting to see how approaches like this compare to just fine-tuning existing models. https://t.co/XyuBo8xduf #edtech via @EdSurge

[Jake Firman @jakefirman]

I'm a bit surprised about the optimism from parents about the potential of AI in education. Still a healthy amount of concern (which is warranted), but the optimism is a good thing!

Students and Parents Are Bullish on AI's Potential for Education https://t.co/cq6EWbKxHn

[Jake Firman @jakefirman]

Educators Team Up to Respond to Sudden Rise of ChatGPT https://t.co/Y6NqeoSjmA #edtech via @EdSurge

[Jake Firman @jakefirman]

I'm starting to view #ChatGPT as another programming language. All that #Python is doing is converting our Python commands into C code. Entering english language into ChatGPT and having it generate Python code is just feeling more and more like a natural progression.

[Allie Kimmel @allie_kimmel]

Governor @jaredpolis celebrated 21 schools in Colorado that demonstrated strong student growth during the pandemic. Congrats to all of the Bright Spot schools! https://t.co/GyQHT7puWN

[Jake Firman @jakefirman]

A friend just introduced me to the AI concept of “Word Vectors”, and my mind is blown! https://t.co/ZqqnmwmPOi

[Jake Firman @jakefirman]

There's a point whenever you're learning a new framework/technology where you finally feel like you "see the matrix", and things just start to click.

It's one of the best feelings in the world! #edtech

[Jake Firman @jakefirman]

This is awesome.

Seattle PTAs pioneer new, more equitable ways of fundraising | The Seattle Times

#edchat https://t.co/OnpOTAqeCU

[Jake Firman @jakefirman]

What an awesome week at the #EdFiSummit. It has grown so much from that small first summit so many years ago. I loved seeing the trends of more district/LEA use cases, and more true success stories…both great signs for the @edfialliance community.

[Jake Firman @jakefirman]

Loved reading this brief from @chelseawaite, particularly this opening line: "Among the lessons to take away from COVID-19 in the K–12 sector, one is this: we can never again say that schools haven’t
changed in a hundred years." Great stuff. #EdTech https://t.co/6rkZhKTkKE

[Jake Firman @jakefirman]

Really excited to see the great things that the Rooted team will do in my home state of Washington in the coming years!

[Matt Candler @mcandler]

First a crazy idea. Then a pitch. Then a one day test. Then a weekend pilot. Then a week-long camp. Then a semester-long experiment. Then a group of courageous families trusting a new school.

Now...

[Robin Lake @RbnLake]

“If school districts aren’t prepared to consider doing things differently as a result of this pandemic, when will we ever do it?” Some good creative thinking from WA Leg https://t.co/YDWpwfI9Q3

[Jake Firman @jakefirman]

One thing I've learned in tech, and in #EdTech, is that if you think you're playing chess, you're probably playing checkers...because those playing chess are more focused on trying to learn to play 3D chess to realize they're playing normal chess.

[Jake Firman @jakefirman]

David is an incredible partner in this work. Awesome surprise to see him pop into my Twitter timeline. Great choice to highlight such an amazing team member @DPSNewsNow

[Denver Public Schools @dps_k12]

#LivingOurDPS: David Woessner is committed to providing excellent service and helping our students. Read more here: https://t.co/7oYY1yAu62.

[Jake Firman @jakefirman]

HAHA! @MJMadda on the money, as always. Well said!

[Dr. Mary Jo Madda @MJMadda]

Silicon Valley: HEY LOOK AT WHAT WE MADE YOU
Schools: I mean, that's cool, but we didn't really ask for that, what we need is—
Silicon Valley: YOU'RE WELCOME HERE LOOK AT IT
Schools: ...thanks but—
Silicon Valley: YOU'RE WELCOME YAY WE FIXED IT
#edchat #edtechchat #pdchat

[Jake Firman @jakefirman]

As soon as I watched the horrifying Jacob Blake video, I thought, "Why did the officer use his gun? Didn't he know there were other options?"

Then I watched a video of Kyle Rittenhouse saying "We don't have non-lethal."

These are not unrelated.

[Jake Firman @jakefirman]

I'm ready to run through a wall after watching an inspiring, candid, and motivating fireside chat with @arneduncan and @JohnBKing at @DSSTPubSchools Infusion. What an amazing pair of leaders and people!