"One of the very confusing things about the models right now: how to reconcile the fact that they are doing so well on evals.
And you look at the evals and you go, 'Those are pretty hard evals.'
But the economic impact seems to be dramatically behind.
There is [a possible] explanation. Back when people were doing pre-training, the question of what data to train on was answered, because that answer was everything. So you don't have to think if it's going to be this data or that data.
When people do RL training, they say, 'Okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing.'
You say, 'Hey, I would love our model to do really well when we release it. I want the evals to look great. What would be RL training that could help on this task?'
If you combine this with generalization of the models actually being inadequate, that has the potential to explain a lot of what we are seeing, this disconnect between eval performance and actual real-world performance"
#ChatGPT vs #Claude vs #Perplexity battle: can AI build complex financial models?
I love testing all 3 tools on complex real tasks. just heard @OpenAI hired bankers to teach #ChatGPT fin modeling. I had a task that would normally take me 6 hrs so I gave it to AI
Here how it went:
@OpenAI#result: 6 hours manual → 1 hour with Claude
#takeaway: this isn't the first time I've seen Claude dominate on complex multi-step tasks. for financial/ops modeling specifically, it's not even close. ChatGPT couldn't generate working formulas. Perplexity won't let you export
@OpenAI#Claude: Claude amazes me in tasks like this one - it delivered a fully functional XLS file with 110+ interconnected formulas following fin modeling best practices. proper SUMIF matching, conditional brand logic, correct data flow from raw inputs to client invoice. 90% prod ready
@OpenAI@perplexity_ai maybe I am missing something - can you pls guide me how to download the excel file the app generated - @AravSrinivas I know you are constantly looking at user feedback so flagging might be a bug - happy to share the prompt
@OpenAI Perplexity: generated basic structure but only viewable in-app. I asked multiple times how to download it. response: "You cannot download the calculator... the download button is not available in the chat interface." cool, so I can see it but can't use it
@OpenAI I designed the prompt with explicit architecture, prepared raw data files (PowerBI exports, storage data, MSA pricing), and fed the EXACT same input to all 3 systems
@OpenAI backstory: in my undergrad I interned at Goldman and spent 18+ hours daily building models. so when I needed to build a 3PL billing calculator, I knew exactly what it needed: 8 interconnected sheets with 110+ SUMIF formulas, data aggregation, conditional logic
9pm Wednesday at @PipelexAI evening hackathon
I was building invoice OCR recognition tool
Among demos - Zillow listing scam detector, calorie counter and Free Food events finder in #SanFrancisco
@n8n_io event in San Francisco
Did you know they raised $180 m series C @ $2.5bn valuation 💸
I will do a tutorial on how I use their workflows ☺️ Also they gave 30% discount - happy to share just lmk