Prathamesh Juvatkar

Building Docstrange, open-source document intelligence for developers. Turning messy PDFs into clean Markdown for RAG pipelines. Sharing what I learn about doc

[Prathamesh Juvatkar @PJuvatkar]

https://t.co/GrLXfd8wKK

Table extraction is still a hard problem for these models. Gemini 2.5 flash indeed is very good in IDP tasks. More such insights!

[Souvik Mandal @mandalsouvik4]

It's officially live! The Intelligent Document Processing Leaderboard.

Featuring 16 datasets, over 9,000 documents, and 6 distinct tasks.

More models will be added soon, or you can evaluate on your own— the datasets and code are open-source too!

https://t.co/bnT4H7DpM9

[Prathamesh Juvatkar @PJuvatkar]

@vanstriendaniel @allen_ai Faced the same issue with this bench. I think it should be fair to eval model + some postprocessing.

If model over extracts but knows what's it's extracting, it can selectively remove things from output to make it compliant with bench?

[Prathamesh Juvatkar @PJuvatkar]

@GptMaestro Any good benchmarks to test graph performance of this plugin vs native graph databases. Maybe scope of queries possible, latency etc?

Which metrics do you think matter for a graph DB built for agents?

[Prathamesh Juvatkar @PJuvatkar]

@ProfAdebay @Lunexalith @teortaxesTex Self host, yes. Locally not in near future probably. Going to be very large models, no economies of scale on local etc.

[Prathamesh Juvatkar @PJuvatkar]

@LandoTakingOver @Lunexalith @teortaxesTex The only argument could be they have only unlocked large scale distillation capability yet, so their best models can't be better than US models. Then might as well release them.

A big assumption though!

[Prathamesh Juvatkar @PJuvatkar]

@Lunexalith @teortaxesTex Do you really think China is open sourcing their best models?

[Prathamesh Juvatkar @PJuvatkar]

@paulg Should curate these and write a new parenting book..

[Prathamesh Juvatkar @PJuvatkar]

@QuantaMind_2025 @nanonets Post training models to give out calibrates confidence scores, layering with business rules that spot mistakes in outputs

[Prathamesh Juvatkar @PJuvatkar]

Looking at mem benchmarks, most try to evaluate mem systems by giving access to a model to memory and seeing how well they do the task.

Wouldn't a good bench be to directly evaluate system? Eg give it lot of tax filings and figure out what % of tax code it can infer from it?

[Prathamesh Juvatkar @PJuvatkar]

The pitch of all seat based pricing companies now is agent needs their own seat for access control. Good save!

[Prathamesh Juvatkar @PJuvatkar]

@DrDatta_AIIMS Do give this a try https://t.co/ZDkjtixGZx

[Prathamesh Juvatkar @PJuvatkar]

@karpathy @kepano https://t.co/bvP9Dd7Czk
Looks ok to give markdown context to an LLM

[Prathamesh Juvatkar @PJuvatkar]

@pmarca Except the ones in AI

[Prathamesh Juvatkar @PJuvatkar]

@MaxBrodeurUrbas @satyanadella It works on my machine

[Prathamesh Juvatkar @PJuvatkar]

So far multi-agent setup was needed for more direct practical purposes like context management. Reading mythos's system card (if true), you would need multi-agent setup to minimize reward hacking, setting up accountability, manage model's psyche, just how you build organizations!

[Rushabh Nagda @rushabh_nagda]

"When a metric becomes the target, it stops being a good metric" - Goodharts Law

last few days GLM-OCR has been trending after it claimed 95% on OmniDocBench, which is higher than Gemini-3-pro

in reality GLM-OCR is way worse than the story these benchmarks paint, lets see how

full disclaimer: ive been working in this space for the last 7 years with @nanonets

[Prathamesh Juvatkar @PJuvatkar]

@heyrimsha @Wealth_Pill We tested it on documents slightly different from ones in popular benchmarks, and it doesn't do well. Model is clearly benchmaxxed

[Prathamesh Juvatkar @PJuvatkar]

@_karthik https://t.co/Tx2YeGoAJE
The last batch of frontier models became better than finetuned ones, next wave of finetuned ones should surpass them again. These small size VLM's are where a lot of architecture development is happening IMO

[Prathamesh Juvatkar @PJuvatkar]

@bindureddy For specific tasks like OCR, still some delta in SOTA vs flash varients, however small domain specific models are doing better and better

https://t.co/Tx2YeGoAJE

[Prathamesh Juvatkar @PJuvatkar]

We need elo score for every profession now on who can beat AI in that profession!

[Prathamesh Juvatkar @PJuvatkar]

@grok @Ichimokutrader7 @sama https://t.co/Tx2YeGoAJE

Gemini > Claude > GPT is the trend we've seen