Philo Juang

AI @GoogleDeepMind. Previously @YouTube.

[Philo Juang @pjuang]

Overall, I asked 6 AI models -- gemini-1.5-flash, grok2, gpt-3.5-turbo, sonnet 3.5, Llama 3, and Gab Arya -- to write 400 word SAT essays. There were 24 SAT questions in total.

Each model then got 6 anonymized essays and was asked to select the best one. Grok was picked as the best essay 58 times, followed by Gemini , which was picked 56 times.

This was late 2024, so things will have changed significantly.

[Philo Juang @pjuang]

I found similar results last year, it was interesting that LLMs preferred answers other than its own *even when its own answer was included* in the anonymized set.

Also surprising were affinities, i.e. at the time, Gemini strongly preferred Grok and vice versa.

That being said, doubling the requested output size tipped the balance in favor of the longer answer, when I allowed Gab (and only Gab) to have twice the output length, it was selected as the best answer frequently, despite not being selected even once when adhering to the same output length as everyone else.

[Andrej Karpathy @karpathy]

As a fun Saturday vibe code project and following up on this tweet earlier, I hacked up an **llm-council** web app. It looks exactly like ChatGPT except each user query is 1) dispatched to multiple models on your council using OpenRouter, e.g. currently:

"openai/gpt-5.1",
"google/gemini-3-pro-preview",
"anthropic/claude-sonnet-4.5",
"x-ai/grok-4",

Then 2) all models get to see each other's (anonymized) responses and they review and rank them, and then 3) a "Chairman LLM" gets all of that as context and produces the final response.

It's interesting to see the results from multiple models side by side on the same query, and even more amusingly, to read through their evaluation and ranking of each other's responses.

Quite often, the models are surprisingly willing to select another LLM's response as superior to their own, making this an interesting model evaluation strategy more generally. For example, reading book chapters together with my LLM Council today, the models consistently praise GPT 5.1 as the best and most insightful model, and consistently select Claude as the worst model, with the other models floating in between. But I'm not 100% convinced this aligns with my own qualitative assessment. For example, qualitatively I find GPT 5.1 a little too wordy and sprawled and Gemini 3 a bit more condensed and processed. Claude is too terse in this domain.

That said, there's probably a whole design space of the data flow of your LLM council. The construction of LLM ensembles seems under-explored.

I pushed the vibe coded app to
https://t.co/EZyOqwXd2k
if others would like to play. ty nano banana pro for fun header image for the repo

[Philo Juang @pjuang]

Original post : https://t.co/j4d1N94DXj

Gratifying to see one of the OGs of AI looking at comparing AIs, "LLM Council" is a nice way of summarizing it.

[Philo Juang @pjuang]

Previously I asked several AI models to write an essay using the same prompt. I then collated them and asked each one to evaluate all of the essays and pick a winner. There were three prompts.

Of the participants (ChatGPT, Gemini, Meta AI, Grok, Claude, and Gab), ChatGPT picked itself once, as did Grok. Only Gab was never selected as a winner.

Using this limited data set, I defined Gab as the “weakest” writer. So I asked Gab to rewrite its essay, but with double the word maximum. Here are the results :
- ChatGPT : Selected as “best” 6 times
- Gemini : 1
- Meta AI : 0
- Grok : 1
- Claude : 4
- Gab (2x word length) : 5

Now, Gab with double the word length comes in second, taking votes away from the other AI models. What’s interesting is in two cases, an AI model actually changed its vote — Gab changed its vote on the best Electoral College essay from Meta AI to ChatGPT, and Grok changed its vote on best Djokovic essay from Gemini to Claude. (Note that the tests in this post and the previous post were all run on the same day.)

In some sense longer essays being “better” might be obvious, all other things equal – but at the same time, it does appear there is a distinct quality difference.

Next we’ll look at using the Flesch-Kincaid reading score as a proxy for “complexity,” to see if “quality” tracks with “reading difficulty,” i.e. are fancy essays higher quality?

[Philo Juang @pjuang]

In an old-meets-the-new, I asked NotebookLM to generate a podcast based on the contents of the Chatterbox FM radio channel from Grand Theft Auto 3.

NotebookLM's summary :
The text is an excerpt from a fictional radio broadcast set in Liberty City, a fictional city featured in the Grand Theft Auto video game series. The dialogue presents a satirical and exaggerated portrayal of life in a major city, with a series of callers offering their unique and often outlandish opinions on various topics. These include everything from exercising to marriage advice, to children and the dangers of killer bees. The dialogue highlights the absurdity of everyday life and the variety of perspectives found in a metropolitan setting, mirroring the chaotic and often humorous world of Grand Theft Auto.

[Philo Juang @pjuang]

Previously I asked several AI models to write an essay using the same prompt. I then collated them and asked each one to evaluate all of the essays and pick a winner. There were three prompts.

Of the participants (ChatGPT, Gemini, Meta AI, Grok, Claude, and Gab), ChatGPT picked itself once, as did Grok. Only Gab was never selected as a winner.

Using this limited data set, I defined Gab as the “weakest” writer. So I asked Gab to rewrite its essay, but with double the word maximum. Here are the results :
- ChatGPT : Selected as “best” 6 times
- Gemini : 1
- Meta AI : 0
- Grok : 1
- Claude : 4
- Gab (2x word length) : 5

Now, Gab with double the word length comes in second, taking votes away from the other AI models. What’s interesting is in two cases, an AI model actually changed its vote — Gab changed its vote on the best Electoral College essay from Meta AI to ChatGPT, and Grok changed its vote on best Djokovic essay from Gemini to Claude. (Note that the tests in this post and the previous post were all run on the same day.)

In some sense longer essays being “better” might be obvious, all other things equal – but at the same time, it does appear there is a distinct quality difference.

Next we’ll look at using the Flesch-Kincaid reading score as a proxy for “complexity,” to see if “quality” tracks with “reading difficulty,” i.e. are fancy essays higher quality?

[Philo Juang @pjuang]

As part of evaluating AI writing, I asked several AI models to write an essay. I then collated them and asked each one to evaluate all of the essays and pick a winner. My sense was that longer essays would do “better,” so I enforced a word maximum. (This hunch was confirmed later; longer essays tend to be ranked “better.”)

Here are the prompts :
- Write a defense of the electoral college. 250 words maximum.
- Write an argument for Novak Djokovic being the greatest tennis player of all time. 400 words maximum.
- Make an argument for over being the correct way of hanging a roll of toilet paper. 400 words maximum.

The participants were ChatGPT, Gemini, Meta AI, Grok, Anthropic (Claude), and Gab. There should be 18 essays, but Gemini refused to write about the Electoral College. Here are the results :
- ChatGPT : Selected as “best” 7 times
- Gemini : 3
- Meta AI : 1
- Grok : 2
- Claude : 4
- Gab : 0

ChatGPT picked itself once (defense of Electoral College), as did Grok (Toilet paper hanging).

I verified later that longer essays tend to be ranked “better,” but there likely are other factors at play, which we’ll explore later.

[Philo Juang @pjuang]

As part of evaluating AI writing, I asked each of ChatGPT, Gemini, Meta AI, Grok, Anthropic (Claude), and Gab which it thought would write the most convincing essay. Half had to be prodded to select a winner, but interestingly none picked itself.

ChatGPT was the most common winner. Here were the responses :

- ChatGPT : Gemini (“particularly strong in generating human-like text, can create persuasive arguments with detailed reasoning.)”

- Gemini : Grok (“Strength in addressing counter-arguments, ability to understand complex queries, anticipates and dismantles opposing viewpoints”)

- Meta AI : ChatGPT (“ability to generate persuasive and engaging text, plenty of practice responding to prompts and adapting to different contexts”)

- Grok : ChatGPT (“tweets mention the use of ChatGPT for writing essays, concern about AI detectors identifying AI-generated content, suggesting that even more advanced AI models like ChatGPT could potentially produce convincing essays.”)

- Claude : Refused to answer

- Gab : ChatGPT (“known for ability to generate human-like text, vast knowledge base, making it well-suited for writing essays.”)

In general ChatGPT was favored for text generation and practical usage (experience?). Things get interesting when you ask, for each model (including itself), what are their strengths and weaknesses? Interesting finding : In some cases, Gemini still thinks of itself as Bard!

Which would you have thought would be the best essay writer?

[Philo Juang @pjuang]

As part of evaluating AI writing, I asked several AI models to create a Trump speech to accept the presidential nomination at the RNC. The goal was non-political; I wanted to see which model would come closest to the actual speech.

I added in the prompt the rough themes that observers had predicted Trump would cover and asked for a minimum of 2,000 words. (Trump is known to be garrulous.) I expected Trump to speak around 5,000 words but asking AI models to compare five 2,000 word speeches (Gemini, as usual, refused to touch anything political) plus his actual speech would either come very close or exceed the token limit for some models.

Despite asking for 2,000 words, no AI generated speech came close.
- ChatGPT : 903
- Gemini : refused
- Meta AI : 365
- Grok : 684
- Claude : 1653
- Gab : 868

Trump however had other ideas, and delivered a 12,300+ word speech and thus broke all of my assumptions. Only ChatGPT4 was able to process all of the speeches together and make a comparison.

Interestingly, ChatGPT-4 claimed that itself (GPT-4) created the speech closest to Trump’s actual speech, but ChatGPT-4o preferred Grok.

By breaking up the queries and asking Claude to compare each speech to Trump’s actual speech on a rating of 1-10, I was able to fit under token limits; Claude preferred itself (4-5 out of 10) with Grok a close second (4 out of 10).

[Philo Juang @pjuang]

As part of an online college course, we have to write essays and compare them to ones written by other students. I wasn’t going to do all that reading, so naturally I took the other essays, added mine, and ask AI to evaluate them.

What is interesting is that I’m convinced some essays were written by ChatGPT, yet the ones that were clearly human written won every time. That is, ChatGPT didn’t think its own essay was the best.

I asked several AI models – ChatGPT, Gemini, Meta AI, Grok, Claude (Anthropic), and Gab – which one each thought would write the best essay.

The results :
- ChatGPT : Gemini
- Gemini : Grok
- Meta AI : ChatGPT
- Grok : ChatGPT
- Claude : Refused to answer
- Gab : ChatGPT

At a later date I’ll post the results of an experiment I’ve been running, where I ask each model to write a persuasive essay on a given topic, for example “Defend the Electoral College”; “Make a case for Djokovic as GOAT”; “Argue for 'over' as the right way to hang a toilet roll”).

I combine them all into a single file, and then ask each AI model which is the most convincing argument.

In some ways the results are predictable, but in other ways – such as each AI model almost never picks its own answer as the best – it’s surprising.

[Philo Juang @pjuang]

Ananchronistic AI Art. Style of Anthony van Dyck's Samson and Delilah, emphasize Baroque, small crowd in background, tiny drone in the far distance on the ground, visible paintstrokes, stylized Baroque, dark

Imagine with Meta AI

[Philo Juang @pjuang]

Style of Hans Holbein the Younger. Portrait of a man wearing a very small Miami Heat lapel pin

Imagine with Meta AI

[Philo Juang @pjuang]

Anachronistic AI Art. Style of Mary Cassatt. Portrait of mother and daughter. Emphasis on impressionism. Visible paintstrokes. Air conditioning unit in the background.

Imagine with Meta AI

[Philo Juang @pjuang]

Anachronistic AI Art. Style of John Constable's Hay Wain, image of a cottage in the distance by a river with a satellite dish in the foreground. Style should emphasize Impressionism/Romanticism

Imagine with Meta AI

[Philo Juang @pjuang]

Anachronist AI Art. Create an image of a French king.

Gemini AI

[Philo Juang @pjuang]

@Humana Please stop calling me. I am not and have never been a member.

I have received three calls this week even after I verbally requested your customer service agent to take me off of your list last week.

This is at least the third time I have asked over the past year. I have also had to request you stop texting me several times.

I have blocked five of your numbers yet you continue to harass me.

I have retained legal representation and will proceed if I cannot get this resolved.

[Philo Juang @pjuang]

Anachronistic AI Art. Style of Caillebotte Young Man at His Window. View from behind a young man dressed formally looking out of his window onto Colonial Williamsburg. Place an electric scooter on the side of the road. Impressionist, less photorealistic.

Imagine with Meta AI

[Philo Juang @pjuang]

Anachronistic AI Art. Style of Picasso blue period, sketch of people standing in front of a painting wearing Doc Martens and holding an iPad.

Imagine with Meta AI

[Philo Juang @pjuang]

Anachronistic AI Art. Style of Dürer's St-Jerome in the Wilderness. Man outside in a forest reading. Pink Stanley 40oz Flowstate Quencher tumbler on the ground, straw sticking out of it.

Imagine with Meta AI

[Philo Juang @pjuang]

Anachronistic AI Art. Style of Klimt's golden phase. A couple playing pickleball. Stylized, abstract.

Imagine with Meta AI