@dbreunig If you can just open the benchmark read what is in it and you will realise that your tweet doesn't make sense at all. Yes finance has verifiable rewards but the benchmark is not at all about that
@jobergum A better framework to align for building for future is whether are you on the side of compute i.e. your solution improves with more compute vs some clever trick/optimisation
@HarveenChadha @compoundingaiin And all of this is without fine tuning. It's just about accurate retrieval. We extended the similar architecture on US filings and were able to achieve 88% against 55.9% of GPT 5.1 on Finance Agent benchmark by https://t.co/IDPpWfjDzi
@HarveenChadha Without benchmarks these models mean nothing. We did internal benchmark for @compoundingaiin against perplexity & chatgpt 5 on financial questions(suited for analysts not consumer) and we were able to beat both of them by a significant margin. All with citations
@shankarnath@itsTarH Stick to NotebookLM, nobody is as good as google on Long Documents. Claude will give you fancy charts but the numbers would be all wrong. In case you're exploring something beyond chat, checkout @compoundingaiin we solve a lot of research workflows related problems
@abeirami Exactly. And true evaluation would be on listed companies rather than crypto where you have access to fundamentals, peers, etc. Any classical ML approach on crypto trading data would mog these LLMs in performance.
@HarveenChadha Yeah interesting. Right now even frontier models suck at long documents as image vs text. Even if you leave aside the context window limitation the bigger problem in my opinion is lack of training data on image vs text modality. So I am not sure about the premise of their claim
@HarveenChadha Infact mineru 2.5 is at 2300 token/sec on A100 80gb ( would be same on 40gb as memory doesn't matter much for these small models) Since these are new models you you don't see them in deepseek paper