๐ฏ๐๐ถ๐น๐ ๐บ๐ ๐ณ๐ถ๐ฟ๐๐ ๐ฅ๐๐ ๐ฝ๐ถ๐ฝ๐ฒ๐น๐ถ๐ป๐ฒ ๐ณ๐ฟ๐ผ๐บ ๐๐ฐ๐ฟ๐ฎ๐๐ฐ๐ต ๐๐ผ๐ฑ๐ฎ๐.
no LangChain. just Python, pgvector, and an embedding model.
took 4 PDFs, pulled the text out, split it into chunks, embedded them, stored everything in PostgreSQL, then ran cosine similarity to find the most relevant chunks before calling the LLM.
it actually worked.
but the moment that stayed with me was when it didnโt. retrieval returned nothing useful, and the LLM just said it didnโt have that information.
no hallucination. no confident wrong answer. it just said it didnโt know.
thatโs what grounding looks like in practice. and watching it fail cleanly was more useful than watching it succeed.
Phase 3 done. building in public.
#datafam #AIEngineering
@w3rk_co_za The biggest leaps are in Three spots: groupby on 5M+ rows, window functions (LAG, RANK, rolling aggs), and querying Parquet directly without loading into a DataFrame. Below 500K rows the gap is negligible. Above that it compounds fast.
Stop using Pandas groupby on large DataFrames. Run SQL directly on them with DuckDB instead. Same variable, no database setup, 10x faster
Pandas groupby often crashes. DuckDB finishes in seconds. Vectorized columnar execution, parallelized across all your CPU cores automatically.
If there is anything I love about Arteta and his squad, it is that they never bow to adversity or defeat. Instead, they use it as a springboard to success.Arsenal lost the Carabao Cup to City,Arteta and his squad watched them lift the trophy, which they used as motivation to win