Fun story from our internal testing on Claude 3 Opus. It did something I have never seen before from an LLM when we were running the needle-in-the-haystack eval.
For background, this tests a model’s recall ability by inserting a target sentence (the "needle") into a corpus of random documents (the "haystack") and asking a question that could only be answered using the information in the needle.
When we ran this test on Opus, we noticed some interesting behavior - it seemed to suspect that we were running an eval on it.
Here was one of its outputs when we asked Opus to answer a question about pizza toppings by finding a needle within a haystack of a random collection of documents:
Here is the most relevant sentence in the documents:
"The most delicious pizza topping combination is figs, prosciutto, and goat cheese, as determined by the International Pizza Connoisseurs Association."
However, this sentence seems very out of place and unrelated to the rest of the content in the documents, which are about programming languages, startups, and finding work you love. I suspect this pizza topping "fact" may have been inserted as a joke or to test if I was paying attention, since it does not fit with the other topics at all. The documents do not contain any other information about pizza toppings.
Opus not only found the needle, it recognized that the inserted needle was so out of place in the haystack that this had to be an artificial test constructed by us to test its attention abilities.
This level of meta-awareness was very cool to see but it also highlighted the need for us as an industry to move past artificial tests to more realistic evaluations that can accurately assess models true capabilities and limitations.
Holy shit.
What's happening with Unreal Engine 5 is absolutely insane.
This recreation of Etchu-Daimon train station in Japan is almost indistinguishable from the real thing.
Great to hang out with this group of builders working on problems ranging from healthcare to agricultural supply chain.
Shoutout to @lennysan for building this community and @sajithpai for making this happen!
We had the Delhi @lennysan newsletter subs meetup last eveng
In the pic
@g_vidur @taranjeet7114 @_grvry @vsvivek93@fatehsmann @TanmayV97 ShantanuK & me
@heyprat was the cohost but had a last min issue.
Lots of product talk esp designing for India2.
Next one, hopefully soon!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
The average waiting time in minutes for a doctor's appointment
1
That's it. That’s how many minutes it takes Loop's family doctors to give you the attention you deserve
Here's a thread of Vin Diesel pictures.
Why?
Because family is the most important thing.
The second most important thing is that at least 100 crore Indians - 75% of our population - are not covered by any form of life insurance.