ladies and gentlemen, i’m incredibly excited to introduce kindbench.
for the past few years, we’ve benchmarked ai on intelligence, coding, math, reasoning, knowledge.
but the thing people are increasingly using these models & systems for isn’t answering questions. it’s shaping beliefs, decisions and relationships.
it felt strange that we weren’t measuring any of that.
today, we finally are.
been almost a year since i made this tweet and honestly not much has changed
here's the state of ai in today's enterprise world:
- genai POCs are still failing at scale & in large corps
- MCP turned out to be pretty fucking useless
- multiagents have been disappointing. enterprise workflows mostly reward deterministic orchestration, not autonomous stategraphs
- hallucination still remains a core unsolved issue even with the SOTA models
- so does memory. maintaining state over long/cross conversations continues to be a challenge
- larger context windows & more parameters haven't really achieved much compared to the last generation
- tokens are costing more, not less, as models, architectures, and harnesses have progressed
- mid-size CEOs are realizing that replacing engineers with agents isn't the best way forward (agents are costing more than humans)
- non-tech megacorp CEOs still don't know what to do exactly & are implementing stupid KPIs such as measuring copilot usage to push AI adoption
- consultants who are not rebranding themselves as "forward deployed engineers" are having a really hard time
- organic ai adoption is bottom-up and not top-down across corps
- using tools like coderabbit has become imperative in fighting the thousands of lines of AI slop even senior engineers are committing every day
- nobody seems to be writing code anymore
- still doesn't mean code is solved. LLMs are duds on large codebases
absolutely no way this pipeline has near-zero hallucinations at 10M+ unstructured docs.
ingestion is weak. disambiguation, jargon handling, abbreviation, openie tags. gotta do more
if you really wanna beat hallucinations, 70% of your effort goes into ingestion alone
We approve this model!
Though less interested in the "0.9B beats Gemini" framing than in what GLM-OCR actually is: a small open OCR stack that does layout analysis, region-level recognition, and structured Markdown/JSON output. That shape matters. It means you can plausibly wire it into a real document pipeline instead of treating it like another pretty demo.
And it is not alone. LightOnOCR-2-1B is compelling if you want end-to-end OCR without a brittle external pipeline. PaddleOCR-VL-1.5 looks especially serious for messier parsing, irregular localization, seals, charts, multilingual docs, and distorted pages.
GLM-OCR, 0.9B model that beats Gemini on OCR benchmarks.💀
It's a 0.9B param vision-language model. supports 8K resolution, 8+ languages, and has built-in text, LaTeX, and table recognition modes.
demo's
- https://t.co/mqSSx2QBhy
i've been working on (and talking about) knowledge graphs since 2024. i like the solution they're pitching but the core issues with graphs as memory solutions (let alone a "vector db killer") still remain, same as 2024.
- building the ontology itself is a challenge. most organizations have scattered context, or they don't have methods to capture context across their processes. irrespective of how good your ontology builder is, it cannot fill missing context
- graphs are hard to scale and expensive to maintain. every change requires an edge removal, updating of nodes, and a significant reconciliation. over time graphs degrade, the more you add to a single graph, the more it degrades. the cost of traversing a graph with 8 million nodes is a lot
- the graph only knows what was successfully extracted. if the system misses an entity or relationship during ingestion, the graph literally has no path to reason over. vectors can still recover context through semantic similarity. graphs cannot infer missing edges
"killing vector dbs" sounds great for marketing. but in practice, the systems that actually work combine both. you use vectors for recall and graphs for relationships. this, of course, differs by usecase. or so has been the pattern we've observed after multiple experiments and customer deployments.
good luck to hydradb and whoever is trying to solve this. i'm keeping a close eye!
Most people building for India still underestimate the language problem.
We recently built a voice-first rural banking assistant MVP on Sarvam’s stack to handle dialect-heavy Hindi and real loan queries from farmers.
Glad to see @Inc42 cover the work and quote our approach.
https://t.co/lVnHMFqao2
shoutout to:
- AthenaAgent
- Neysa
- Soket
- Gnani
- Vicharak
- Ombase Limited
- Vecros
- TrueFan
all building genuinely cool stuff. great to see this up close & inspired
Recursive Language Models (RLMs) let agents manage 10M+ tokens by delegating tasks recursively.
This Google Cloud Community Article explains why ADK was the perfect choice for re-implementing the original RLM codebase in a more enterprise-ready format →https://t.co/p3MsNtLVJL
trying to cook up some content for @AphelionLabs. built a workflow, cracked character consistency to a decent point, but speaking styles and audio still need work. good problem for next weekend.
meet melissa. hot, opinionated, and not here to be nice.
trying to cook up some content for @AphelionLabs. built a workflow, cracked character consistency to a decent point, but speaking styles and audio still need work. good problem for next weekend.
meet melissa. hot, opinionated, and not here to be nice.
my clients pay me $250/hr for this info so y'all buckle up because i'm about to open up a big secret about why your RAG hallucinates at scale.
parsing & chunking is where a potentially great RAG goes to die. miss this and your solution is wrecked from day one. it will hallucinate, and it’ll do it with confidence.
if your org or client allows third-party (but expensive) services like unstructured/.io or azure document intelligence, you're lucky. they'll handle complex layouts decently.
if not, you're stuck with open-source options like IBM’s docling, or microsoft's markitdown, and they suck.
after running multiple POCs at @AphelionLabs, we found that tools like docling, markitdown, and similar parsers just don’t cut it for layout-heavy or visually complex documents like PDFs, DOCX, or PPTX. they break structure, drop images, miss tables, and often need heavy post-processing just to bandage the ingestion shortfalls. at scale, this becomes unmanageable.
we built custom tools & developed techniques to fix this and they're now running in two large-scale hybridRAG deployments that outperform anything else we’ve benchmarked in the open market. whether we open-source or publish them is a decision for another day.
but if your RAG is hallucinating like crazy, the root cause probably isn't your LLM or retrieval method.
start with your ingestion pipeline.
stop feeding garbage in. dynamic parsing and chunking strategies are non-negotiable.
your doc types vary wildly. your pipeline shouldn't treat them all the same. a one-size-fits-all parser is what's likely giving you ugly chunks, uglier embeddings and deranged schizophrenic outputs
chamath agrees. twice.
we build agentic systems that don’t break in prod. not cheap. not for everyone. if five-figure retainers scare you or your enterprise, don’t dm. but if you're serious about doing it right, we’re the call.
@AphelionLabs. aphelion dot in. or just dm me.
At @AphelionLabs, we're working day and night on Project Sanjeevni.
A platform to allow people to collectively mine cryptocurrencies that will be donated to Milaap's relief fund to help with oxygen, food & medical supplies.
Sign up > https://t.co/fjmUITEfK7
#COVID19#CovidHelp
We've set up an interactive web-based map to help you track COVID-19 in real-time across India. We're updating the data in every 2 hours.
#CoronaUpdatesInIndia
https://t.co/yfZj5IoxkP