Spent last 2 weeks going breadth-first through System Design fundamentals.
Biggest realization so far: System Design isn't about knowing the "right" architecture. It's about picking the tradeoff you can live with.
Reply if you've got any questions. Feel free to connect.
Various methods are used to define the searching mechanism based on recall, latency and memory such as:
1. HNSW
2. IVF
3. PQ + IVF
4. Flat Index(Brute Force)
Next time don't say "uses AI to find similar things", as now you know the real magic is in the approximation algorithms.
Learned Vector Databases and their internals today.
They solve the problem of semantic similarity by using Artificial Nearest Neighbor(ANN) algorithm under the hood.
Basically, we can understand these as some sort of index structures rather than calling them as separate DB.
It uses multiple hash functions + a bit-array under the hood for it's working.
It trades a little accuracy for speed + massive memory savings.
The best example I found: checking usernames availability with large datasets.
Learned an interesting thing today: Bloom Filters.
It's a probabilistic data structure that let us check if an item is member of a set.
It may give results with false positives but always works for already existing items.
Can have many use-cases in real world systems.
@Hiteshdotcom This is actually great! As it bridges the gap between a candidate's preparation and the type of interview rounds taken now simulating real world environments where people can use AI to come up with optimised approaches.