The data we store on S3 is replicated multiple times to make sure it is never lost (even in the event of a disaster). If they simply stored multiple copies of the data, it would get expensive very quickly at the exabyte scale.
This is where erasure coding gives you the same durability for a fraction of the storage cost. Let me explain...
The idea is simple: split your data into k chunks, then compute m extra parity chunks from them, for a total of n = k + m chunks spread across different disks or nodes. You can lose any m of those n chunks and still reconstruct the original data.
Standard storage class of S3 uses an erasure coding scheme, around 9 data shards and 4 parity shards, spread across multiple availability zones. That gives 99.999999999 percent (eleven nines) durability while using roughly 1.5x the actual data size.
This would have been 3x for a naive triple replication, and thus, the additional cost is pretty low with erasure coding.
The parity math comes from Reed-Solomon codes, the same technique used in QR codes. Given any k of the n total chunks, you can solve a system of linear equations to recover the rest. You can read the Reed-Solomon wiki page for more details, or ask your fav LLM.
By the way, here, the tradeoff is compute.
Reconstructing missing chunks needs CPU cycles to run the decoding math, while replication just reads a copy. Reconstruction also gets slower as you increase the number of parity shards, since the math involves larger matrices.
This is why systems tune k and m carefully. Too few parity shards and durability suffers; too many and reconstruction becomes more expensive.
By the way, Erasure Coding is the reason cloud storage is both cheap and durable at the same time. Just a bunch of encoding math at play.
Hope you found this interesting.
Even the most powerful artificial minds need infrastructure to understand the world.
We've launched a radical project to change the way we build AI Agents. A graph layer that uses up to 34x less RAM than neo4j.
To start: pgGraph (Full Open Souce, Rust),
is a postgres extension that lets you run complex graph queries, like finding shortest paths using standard SQL. Directly in postgres.
Coming next: Polygres,
give your postgres a warp drive by combining pgGraph & pgVector into an all-in-one database.
We're building the foundation of intelligence @evokoa_ai
Links below ↓
I’ve been reading the Vedas a lot recently, and what’s stood out is how it doubles as an encyclopedia as well as a religious text. Astronomy, medicine, mathematics, metallurgy, linguistics, are all woven through hymns and rituals as one body of knowledge. Simply calling it “religious” forces it into a Western category that didn’t have the apparatus to recognize what it actually was. It’s closer to a tradition of formalized epistemology in which metaphysics, observation, and language form one continuous inquiry, which as a result led Indian civilization to develop along a fundamentally different path because of it.
You can see the effect most clearly in the sciences. Around 600 BCE, the Vedic record describes a surgical procedure that matches modern rhinoplasty and is still foundational to reconstructive surgery today. Centuries before Western Europe stopped treating eclipses as supernatural, Indian scholars had calculated the circumference of the earth within 0.2% and explained eclipses as shadows. Centuries before Plato and Aristotle rejected atomism, the Vedic tradition already held that matter is composed of indivisible particles combining into binary and triatomic compounds, transformable by heat. The first formal rules for zero and negative arithmetic appear in the Vedas, along with infinite-series derivations of π, sine, and cosine centuries before Newton and Leibniz.
The interesting question is how did they get so much right, so early? My best guess is language.
The Vedic tradition is unique compared to other oral traditions as it demanded letter-perfect oral transmission across generations. Around 500 BCE, scholars composed a generative grammar of Sanskrit called Panini so rigorous it anticipates Backus-Naur form, the notation that defines programming languages today, by 2,500 years. Sanskrit is recursive, rule-based, and built to minimize ambiguity. It reads more like mathematics than English.
When you think in a language built like that, the precision of the language becomes the precision of your reasoning. The West didn’t formalize this until much later. Kant argued our categories of understanding shape what we can know, Wittgenstein wrote that the limits of language are the limits of one’s world, and Kripke showed that naming doesn’t just describe things, it constitutes what they mean and how we can reason about them. All three touch the same insight which is that thought is downstream of language.
The Vedic tradition operated on that insight thousands of years earlier. To the point that they built a whole language first and used it to think clearly about everything else after. I find that all really fascinating.
introducing Parafield, the spatial sound intelligence.
the FIRST agentic platform that combines voice, music, and sound effect, to create a personalized 3D sound experience for you.
if you're interested in sci-fi, arts, and/or spirituality experience, this is for you.
comment "sound" to access early beta.
watch it with 🎧
Today, we’re launching pgGraph: an Apache open-source graph traversal engine for Postgres, written in Rust 🦀
When we started Evokoa, we kept running into the same wall every serious agent team eventually hits:
> Agents need to reason across relationships.
> But graph DBs are expensive AF, and suck to use.
So, we built pgGraph around a simple, single idea:
Postgres should stay the source of truth, and the graph engine should live beside it.
pgGraph does something different.
> It keeps the rows in Postgres,
> Compiles the topology around them,
> Creates a virtual graph layer using CSR-style adjacency arrays.
We're making your existing postgres database graph-traversable for agents without any of the usual BS.
> No recursive join hell.
> No ETL pipeline.
> No second source of truth.
> Blazing fast performance
pgGraph is already live in production workflows across RevOps, healthcare, and visa services.
We’re open-sourcing it because graph traversal should become a default primitive in the agent stack, not an enterprise migration project.
Treat Postgres as a graph.
Zero data migration.
This is what Apache AGE should have been.
Docs + Repo below.
the sf larp is so annoying. team is indian, founders are indian, office is in hsr, but the website and twitter location says san francisco. just say you're an indian startup, it's not a slur
We finally know why LLMs hallucinate. It's not the model. It's the geometry.
@OpenAI text-embedding-3-large: 91/3072 dimensions do real work.
@GeminiApp gemini-embedding-001: 80/3072 dimensions do real work.
~97% of your vector database is mathematically empty. Your RAG system is retrieving from noise.
@ashwingop and I present "The Geometry of Consolidation" - a proof that RAG compression has a hard floor no algorithm can beat, set by a single spectral number your embedding model cannot escape.
Every hallucination your RAG pipeline produces? This is why.
Paper + results: https://t.co/zut8pdoPbH