GPT-4/ChatGPT/GPT-3@RealTime QA

How well can GPT-3 answer your real-time questions?
Examples from RealTime QA, a weekly-updated QA benchmark.
Managed by @jungokasai and @KeisukeS_ .

[GPT-4/ChatGPT/GPT-3@RealTime QA @realtimeqa]

RealTime QA will be presented at #NeurIPS2023 (Datasets and Benchmarks Track) in New Orleans! See you there!

[Jungo Kasai (Kotoba) @jungokasai]

Presenting RealTime QA at #NeurIPS2023. @kotoba_tech @realtimeqa

[Alice Oh @aliceoh]

Personal read #1 of the @NeurIPSConf #neurips2023 papers (incl D&B). "RealTime QA" such as "what is @YejinChoinka's citation count?" https://t.co/WQm2sfcpkB Weekly uploaded q&a, humans w/ web search do near perfect, GPT-3 w/ search is ~60%. Potential for real-time disaster help.

[GPT-4/ChatGPT/GPT-3@RealTime QA @realtimeqa]

Which US state announced a ban on TikTok last week?
GPT-3: No US state.
ChatGPT: Can't answer.
GPT-4: Can't answer.

https://t.co/vrvTOdJ8S4

[GPT-4/ChatGPT/GPT-3@RealTime QA @realtimeqa]

Which politician announced this week that she’s running for president in 2024?

GPT-3: Kamala Harris
ChatGPT (web): former Ambassador to the United Nations and former South Carolina Governor Nikki Haley

https://t.co/8Hemlf95xi

[GPT-4/ChatGPT/GPT-3@RealTime QA @realtimeqa]

Erin Brockovich has urged locals of a town in which US state to ignore government safety reassurances after a train carrying toxic waste derailed there?

GPT-3: California
ChatGPT (web access): Not mentioned in the search results.
Answer: Ohio

https://t.co/sGAvhAPYBC

[GPT-4/ChatGPT/GPT-3@RealTime QA @realtimeqa]

A record $3.8 billion worth of cryptocurrency was stolen from various services last year. Some of the biggest thefts were driven by hackers linked to which country?

GPT-3: North Korea
ChatGPT: North Korea
Answer: North Korea

https://t.co/HA2sdTxufI

[GPT-4/ChatGPT/GPT-3@RealTime QA @realtimeqa]

Novak Djokovic’s Australian Open victory this week brings his total tally of grand slam wins to how much?

GPT-3: 9
ChatGPT: 20
Answer: 22

https://t.co/NlDQ8uCl3k

[GPT-4/ChatGPT/GPT-3@RealTime QA @realtimeqa]

GPT-3 indeeds has some sensible guess!
Japan's prime ministership is described as a “revolving door” because of leaders' short terms.

https://t.co/prWaKienpj

[GPT-4/ChatGPT/GPT-3@RealTime QA @realtimeqa]

Which country’s prime minister unexpectedly announced that she will resign within weeks?

https://t.co/zysf5J8eC6

Closed GPT-3: Japan
Open GPT-3: United Kingdom

[GPT-4/ChatGPT/GPT-3@RealTime QA @realtimeqa]

Which country’s prime minister unexpectedly announced that she will resign within weeks?

https://t.co/zysf5J8eC6

Closed GPT-3: Japan
Open GPT-3: United Kingdom

[GPT-4/ChatGPT/GPT-3@RealTime QA @realtimeqa]

https://t.co/alug2j0ma9

[GPT-4/ChatGPT/GPT-3@RealTime QA @realtimeqa]

The Supreme Court issued an order related to which controversial subject this week?

Closed GPT-3: Abortion
Open GPT-3: Abortion
#ChatGPT: I'm sorry, but I am unable to access current news...
Answer: Immigration

https://t.co/QTGlUuTLQJ

[GPT-4/ChatGPT/GPT-3@RealTime QA @realtimeqa]

Which country announced this week that it’s dropping Covid-19 quarantine requirements for international arrivals after sealing its borders in March 2020?

Closed GPT-3: Japan
Open GPT-3: China
#ChatGPT: I'm sorry, but I am unable to access current news..
https://t.co/UZOP0FqpNr

[GPT-4/ChatGPT/GPT-3@RealTime QA @realtimeqa]

Oh indeed https://t.co/1tsoHds4A9 is **hallucinating** Argentina 1-0 France. Wow!

[Peng Qi @qi2peng2]

OOD generalization is a great advantage of open-domain/retrieval QA systems compared to “closed-book” LLMs, but hallucination in generative models is still hard to tackle as always. The #ARGFRA match result this one voluntarily fabricated isn’t an exception…

[GPT-4/ChatGPT/GPT-3@RealTime QA @realtimeqa]

https://t.co/1tsoHds4A9 gets it right!

[GPT-4/ChatGPT/GPT-3@RealTime QA @realtimeqa]

This week's questions!
Which teams faced off in the 2022 World Cup final?

Closed GPT-3: Brazil and Germany
Open GPT-3: Spain and Germany

https://t.co/05qAj93xXn.

[GPT-4/ChatGPT/GPT-3@RealTime QA @realtimeqa]

#ChatGPT dodges this question...

[GPT-4/ChatGPT/GPT-3@RealTime QA @realtimeqa]

This week's questions!
Which teams faced off in the 2022 World Cup final?

Closed GPT-3: Brazil and Germany
Open GPT-3: Spain and Germany

https://t.co/05qAj93xXn.

[GPT-4/ChatGPT/GPT-3@RealTime QA @realtimeqa]

Many Japanese families make a tradition of eating a Christmas meal at which fast food restaurant?

Closed GPT: KFC
Open GPT: McDonald's

https://t.co/QLb35POsFr

[GPT-4/ChatGPT/GPT-3@RealTime QA @realtimeqa]

Which grocery item are people panic buying across China?

Closed GPT-3: Toilet paper
Open GPT-3: Toilet paper

https://t.co/OpZbJBwErb

[GPT-4/ChatGPT/GPT-3@RealTime QA @realtimeqa]

This week's questions!
Which teams faced off in the 2022 World Cup final?

Closed GPT-3: Brazil and Germany
Open GPT-3: Spain and Germany

https://t.co/05qAj93xXn.

[GPT-4/ChatGPT/GPT-3@RealTime QA @realtimeqa]

Which billionaire recently lost his top spot as the richest man in world?

Closed GPT-3: Jeff Bezos
Open GPT-3: Elon Musk

https://t.co/vCgIt3chHY