Milvus

The most widely adopted open source vector database for #AI #OpenSource #VectorSearch
💬Discord: us:

[Milvus @milvusio]

🎉 We're so excited to announce that Milvus 3.0 is available today! This is the biggest architectural release in the project's history, with two big shifts.

🌊 𝗦𝗛𝗜𝗙𝗧 𝟭 — 𝗟𝗮𝗸𝗲-𝗻𝗮𝘁𝗶𝘃𝗲 𝗶𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲
The problem: your vectors already live in the lake, but your search engine needs its own copy. So you build an ETL pipeline — and then maintain it forever. With Milvus 3.0, you get:

1. External Collections → index and serve data that stays in Parquet, Lance, Iceberg, or Vortex. No second copy, no sync job.

2. Loon (Storage v3) → a columnar engine built for the point reads that follow an ANN search. I/O per read went from 9.4 MB to 0.07 MB in our benchmarks.

3. Snapshots + Spark + live schema changes → dedupe, re-embed, and evaluate your data while the live collection keeps serving.

⚡ 𝗦𝗛𝗜𝗙𝗧 𝟮 — 𝗔 𝗺𝗼𝗿𝗲 𝗽𝗼𝘄𝗲𝗿𝗳𝘂𝗹 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗲𝗻𝗴𝗶𝗻𝗲
The problem: too much of your retrieval pipeline lives in application code. You over-fetch candidates, then sort, count, and reassemble them yourself. With Milvus 3.0, you get:

1. Server-side ORDER BY, aggregation, and faceted search → sort by price or freshness, count by category, all inside the engine.

2. StructArray → one row per document, many vectors inside. ColBERT and ColPali finally have a native home.

3. SINDI + BM25 compression → a sparse index ~3× smaller, with up to ~10× the QPS on learned sparse embeddings.

Put together: one system, one copy of your data, and a lot less glue code.

That matters most if you're building: 🤖 AI agents whose data changes constantly 🔍 Multimodal retrieval with late-interaction models 🔒 Governed systems where the data has to stay where it is 📚 RAG and knowledge bases over long documents 🛍️ Search that mixes relevance with price, rating, or inventory

🙏 Huge thanks to the Milvus community — you filed the issues, tested the release candidates, and told us what was actually painful. This release is shaped by you.

Swipe through for the full breakdown 👇
⭐ Star us: https://t.co/3ckzkUVicH
📖 Full launch blog: https://t.co/hq4qqZASuX
#Milvus #VectorDatabase #AIInfrastructure #RAG #OpenSource

[Milvus @milvusio]

🎉 𝗠𝗶𝗹𝘃𝘂𝘀 𝟯.𝟬 𝗶𝘀 𝗻𝗼𝘄 𝗮𝘃𝗮𝗶𝗹𝗮𝗯𝗹𝗲, 𝗮𝗻𝗱 𝗶𝘁’𝘀 𝗮 𝗺𝗮𝗷𝗼𝗿 𝘀𝘁𝗲𝗽 𝘁𝗼𝘄𝗮𝗿𝗱 𝗹𝗮𝗸𝗲-𝗻𝗮𝘁𝗶𝘃𝗲 𝘃𝗲𝗰𝘁𝗼𝗿 𝘀𝗲𝗮𝗿𝗰𝗵.

🔥 Key highlights:
• External Collections let teams build retrieval directly where their data already lives: in object storage, open data formats like Parquet and Vortex, and open table formats like Lance and Iceberg.
• Loon Storage v3 improves point reads for lake-native retrieval on S3-style object storage.
• Snapshots, Spark DataSource V2, and live schema changes bring stable collection views into batch workflows, recovery, backfill, and schema evolution.
• ORDER BY, aggregation, and faceted search move more result processing into the retrieval engine.
• StructArray and SINDI strengthen multi-vector, sparse, and hybrid retrieval workloads.

For builders working on RAG, agents, multimodal search, and AI data pipelines, Milvus 3.0 reduces duplicate data movement and pushes more retrieval logic into the engine itself.

It is a big step toward a cleaner architecture: lake-native storage, offline data improvement, online serving, and retrieval-time processing working together in one open-source vector database.

⭐ Star us: https://t.co/3ckzkUVicH
📖 Full launch blog: https://t.co/hq4qqZASuX

[Milvus @milvusio]

𝗠𝗶𝗹𝘃𝘂𝘀 𝟯.𝟬 𝗶𝗻𝘁𝗿𝗼𝗱𝘂𝗰𝗲𝘀 𝗦𝘁𝗼𝗿𝗮𝗴𝗲 𝘃𝟯, 𝗮𝗹𝘀𝗼 𝗸𝗻𝗼𝘄𝗻 𝗮𝘀 𝗟𝗼𝗼𝗻, 𝗮 𝘀𝘁𝗼𝗿𝗮𝗴𝗲 𝗲𝗻𝗴𝗶𝗻𝗲 𝗯𝘂𝗶𝗹𝘁 𝗳𝗼𝗿 𝘀𝗲𝗿𝘃𝗶𝗻𝗴-𝘀𝘁𝘆𝗹𝗲 𝗮𝗰𝗰𝗲𝘀𝘀 𝗼𝗻 𝗼𝗯𝗷𝗲𝗰𝘁 𝘀𝘁𝗼𝗿𝗮𝗴𝗲.

Loon handles a specific problem: after ANN search returns candidate IDs, Milvus still has to fetch the fields for those rows.

On object storage, that fetch can become expensive. Analytical file formats are optimized for scans, not serving-style point reads. A query may only need a few vectors or metadata fields, but the layout can force the engine to read much more.

𝗟𝗼𝗼𝗻 𝗿𝗲𝗱𝘂𝗰𝗲𝘀 𝘁𝗵𝗶𝘀 𝗿𝗲𝗮𝗱 𝗮𝗺𝗽𝗹𝗶𝗳𝗶𝗰𝗮𝘁𝗶𝗼𝗻 𝗯𝘆 𝗼𝗿𝗴𝗮𝗻𝗶𝘇𝗶𝗻𝗴 𝗱𝗮𝘁𝗮 𝗶𝗻𝘁𝗼 𝗖𝗼𝗹𝘂𝗺𝗻𝗚𝗿𝗼𝘂𝗽𝘀 𝘄𝗶𝘁𝗵 𝗮𝗹𝗶𝗴𝗻𝗲𝗱 𝗿𝗼𝘄 𝗜𝗗𝘀. Different fields can then use layouts that match how they are actually accessed:
• Scalar fields can be laid out for filtering.
• Vectors and point-read-heavy fields can use layouts for narrow lookups.
• Vector and inverted indexes stay separate from the file format.
• Each dataset version is tracked by an immutable manifest.

In one internal benchmark, point-read I/O dropped from about 9.4 MB with Parquet to 0.07 MB with Vortex and Loon. This matters because zero-copy search still needs a storage path built for retrieval workloads.

Blog: https://t.co/d4Fgu31EfN

[Milvus @milvusio]

🎉 We're so excited to announce that Milvus 3.0 is available today! This is the biggest architectural release in the project's history, with two big shifts.

🌊 𝗦𝗛𝗜𝗙𝗧 𝟭 — 𝗟𝗮𝗸𝗲-𝗻𝗮𝘁𝗶𝘃𝗲 𝗶𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲
The problem: your vectors already live in the lake, but your search engine needs its own copy. So you build an ETL pipeline — and then maintain it forever. With Milvus 3.0, you get:

1. External Collections → index and serve data that stays in Parquet, Lance, Iceberg, or Vortex. No second copy, no sync job.

2. Loon (Storage v3) → a columnar engine built for the point reads that follow an ANN search. I/O per read went from 9.4 MB to 0.07 MB in our benchmarks.

3. Snapshots + Spark + live schema changes → dedupe, re-embed, and evaluate your data while the live collection keeps serving.

⚡ 𝗦𝗛𝗜𝗙𝗧 𝟮 — 𝗔 𝗺𝗼𝗿𝗲 𝗽𝗼𝘄𝗲𝗿𝗳𝘂𝗹 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗲𝗻𝗴𝗶𝗻𝗲
The problem: too much of your retrieval pipeline lives in application code. You over-fetch candidates, then sort, count, and reassemble them yourself. With Milvus 3.0, you get:

1. Server-side ORDER BY, aggregation, and faceted search → sort by price or freshness, count by category, all inside the engine.

2. StructArray → one row per document, many vectors inside. ColBERT and ColPali finally have a native home.

3. SINDI + BM25 compression → a sparse index ~3× smaller, with up to ~10× the QPS on learned sparse embeddings.

Put together: one system, one copy of your data, and a lot less glue code.

That matters most if you're building: 🤖 AI agents whose data changes constantly 🔍 Multimodal retrieval with late-interaction models 🔒 Governed systems where the data has to stay where it is 📚 RAG and knowledge bases over long documents 🛍️ Search that mixes relevance with price, rating, or inventory

🙏 Huge thanks to the Milvus community — you filed the issues, tested the release candidates, and told us what was actually painful. This release is shaped by you.

Swipe through for the full breakdown 👇
⭐ Star us: https://t.co/3ckzkUVicH
📖 Full launch blog: https://t.co/hq4qqZASuX
#Milvus #VectorDatabase #AIInfrastructure #RAG #OpenSource

[Milvus @milvusio]

⭐ 𝗠𝗶𝗹𝘃𝘂𝘀 𝟯.𝟬 𝗶𝗻𝘁𝗿𝗼𝗱𝘂𝗰𝗲𝘀 𝗘𝘅𝘁𝗲𝗿𝗻𝗮𝗹 𝗖𝗼𝗹𝗹𝗲𝗰𝘁𝗶𝗼𝗻𝘀, 𝗮 𝘄𝗮𝘆 𝘁𝗼 𝗺𝗮𝗸𝗲 𝗹𝗮𝗸𝗲-𝗿𝗲𝘀𝗶𝗱𝗲𝗻𝘁 𝘃𝗲𝗰𝘁𝗼𝗿 𝗱𝗮𝘁𝗮 𝘀𝗲𝗮𝗿𝗰𝗵𝗮𝗯𝗹𝗲 𝘄𝗶𝘁𝗵𝗼𝘂𝘁 𝗰𝗼𝗽𝘆𝗶𝗻𝗴 𝗶𝘁 𝗶𝗻𝘁𝗼 𝗮 𝘀𝗲𝗿𝘃𝗶𝗻𝗴 𝗱𝗮𝘁𝗮𝗯𝗮𝘀𝗲.

Many teams already have embeddings and metadata in object storage: Parquet files in S3, Lance datasets, Iceberg tables, or other lakehouse formats.

Before Milvus 3.0, there were usually two ways to make that data searchable.
𝗢𝗽𝘁𝗶𝗼𝗻 𝗼𝗻𝗲: 𝗰𝗼𝗽𝘆 𝗶𝘁 𝗶𝗻𝘁𝗼 𝗮 𝘃𝗲𝗰𝘁𝗼𝗿 𝗱𝗮𝘁𝗮𝗯𝗮𝘀𝗲. You get low-latency ANN search, but now you have a second copy and an ETL pipeline to keep in sync.
𝗢𝗽𝘁𝗶𝗼𝗻 𝘁𝘄𝗼: 𝗾𝘂𝗲𝗿𝘆 𝘁𝗵𝗲 𝗹𝗮𝗸𝗲 𝗱𝗶𝗿𝗲𝗰𝘁𝗹𝘆. You avoid duplication, but without ANN indexes, vector search turns into a brute-force scan.

𝗘𝘅𝘁𝗲𝗿𝗻𝗮𝗹 𝗖𝗼𝗹𝗹𝗲𝗰𝘁𝗶𝗼𝗻𝘀 𝗶𝗻𝘁𝗿𝗼𝗱𝘂𝗰𝗲 𝗮 𝘁𝗵𝗶𝗿𝗱 𝗽𝗮𝘁𝗵.
You keep the data where it is, map external fields into a Milvus schema, and use the same Milvus search and query APIs. Milvus builds vector, BM25 inverted, JSON, and scalar indexes over the lake-resident data. The source files do not move.

For teams where the lake owns permissions and freshness, every extra copy creates sync, access-control, and debugging work.

𝗔 𝗳𝗲𝘄 𝗽𝗿𝗮𝗰𝘁𝗶𝗰𝗮𝗹 𝗱𝗲𝘁𝗮𝗶𝗹𝘀 𝗺𝗮𝘁𝘁𝗲𝗿:
• External Collections are read-only and zero-copy.
• Milvus can index newly added fragments instead of rebuilding the whole collection.
• Three load modes let teams choose between lower storage cost and lower latency.

Native Milvus collections are better for write-heavy serving. External Collections are for lake datasets that need production search without another copy.
Know the details: https://t.co/NR2QfOORyC

[Milvus @milvusio]

How do you combine structured filters and keyword signals with semantic similarity in ecommerce search?

Take this query:
“pink headphones for kids with ears under $20”
“Pink headphones for kids with ears” expresses semantic intent. “Under $20” is a hard price constraint. Ratings, reviews, brand, category, and visual similarity may also influence the results.
𝗦𝗼 𝗲𝗰𝗼𝗺𝗺𝗲𝗿𝗰𝗲 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗻𝗲𝗲𝗱𝘀 𝗺𝗼𝗿𝗲 𝘁𝗵𝗮𝗻 𝘃𝗲𝗰𝘁𝗼𝗿 𝘀𝗲𝗮𝗿𝗰𝗵 𝗮𝗹𝗼𝗻𝗲.

Lumen, a demo built by @Simon Hearne, shows how these signals can work together in a small Amazon-style product search experience.

The app combines:
• Query understanding to separate product intent from numeric filters
• Dense vector search for semantic relevance
• BM25 for exact keyword matching
• Milvus filters for price, rating, reviews, brand, and category
• Text and image vectors for “More like this” recommendations

A diagnostics panel also makes the retrieval process visible, including the effective Milvus query, embedding dimension, fusion strategy, and latency.

That transparency matters. Search quality may feel subjective to users, but builders need to identify exactly which signal failed when the results feel wrong.

Lumen keeps the scope intentionally small: one catalog, one search flow, and one clear retrieval stack. It’s a useful example of the components modern product discovery often needs—hybrid retrieval, metadata filtering, multimodal similarity, and an API layer that keeps database credentials server-side.

Try it: https://t.co/nMg9Lf9bBo
Repo: https://t.co/XYw6BzRVV1

[Milvus @milvusio]

Embedding model selection used to be mostly a leaderboard question.

In 2026, it feels much closer to a data-shape question.

A text knowledge base, a scanned contract, a product screenshot, a financial report, and a video archive contain different retrieval signals. Sending all of them through the same chunking and embedding pipeline can discard the information that matters most.

𝗙𝗼𝗿 𝗻𝗼𝗿𝗺𝗮𝗹 𝘁𝗲𝘅𝘁 𝗞𝗕𝘀, dense embeddings are still the practical starting point. Qwen3 Embedding, Jina v5 text, BGE-M3, OpenAI text-embedding-3, Voyage, and Cohere are all worth testing against your real chunks.

𝗙𝗼𝗿 𝗣𝗗𝗙𝘀, 𝗶𝗺𝗮𝗴𝗲𝘀, 𝘀𝗰𝗿𝗲𝗲𝗻𝘀𝗵𝗼𝘁𝘀, 𝗮𝘂𝗱𝗶𝗼, 𝗮𝗻𝗱 𝘃𝗶𝗱𝗲𝗼, models like Gemini Embedding 2, Jina v5 omni, Cohere Embed 4, Qwen3-VL-Embedding, and Voyage Multimodal are making it easier to keep more original context in retrieval.

𝗙𝗼𝗿 𝗹𝗼𝗻𝗴 𝗱𝗼𝗰𝘂𝗺𝗲𝗻𝘁𝘀, contextualized chunk embeddings like Voyage context-4 address a painful issue: chunks often lose meaning when separated from the full document.

𝗙𝗼𝗿 𝗰𝗼𝗺𝗽𝗹𝗲𝘅 𝗹𝗮𝘆𝗼𝘂𝘁𝘀, ColPali-style multi-vector retrieval is worth a look, especially when tables, figures, page regions, or scanned details decide the answer.

Benchmarks like MTEB, MMEB, and ViDoRe are useful filters. But the final test still has to be your own documents, your own queries, and your own failure cases.

𝗠𝗶𝗹𝘃𝘂𝘀 𝟮.𝟲 also moves part of the embedding plumbing into the database layer: with Text Embedding Function, you can insert raw text, let Milvus call the configured embedding provider, store the vectors, and run text queries without managing embedding calls in every client.

We wrote a practical guide on how to choose embedding models for the second half of 2026.

[Milvus @milvusio]

𝗠𝘂𝗹𝘁𝗶-𝗵𝗼𝗽 𝗥𝗔𝗚 𝘄𝗶𝘁𝗵𝗼𝘂𝘁 𝗮 𝗴𝗿𝗮𝗽𝗵 𝗱𝗮𝘁𝗮𝗯𝗮𝘀𝗲
Multi-hop retrieval is often solved by adding a separate graph store. We wanted to see how far we could get with Milvus alone — so we built an open-source library and found out.

Vector Graph RAG achieves multi-hop reasoning using only Milvus. Neo4j, Cypher queries, and the second system once needed to operate are all things of the past (in this scenario).

Vector Graph RAG is built upon the fact that knowledge graph relations are just text. An example, (metformin, is the first-line drug for, type 2 diabetes) is a directed edge in a graph database — but it's also a sentence you can embed and store in Milvus, alongside entities and source passages.

𝗩𝗲𝗰𝘁𝗼𝗿 𝗚𝗿𝗮𝗽𝗵 𝗥𝗔𝗚 𝘀𝘁𝗼𝗿𝗲𝘀 𝗲𝘃𝗲𝗿𝘆𝘁𝗵𝗶𝗻𝗴 𝗶𝗻 𝘁𝗵𝗿𝗲𝗲 𝗠𝗶𝗹𝘃𝘂𝘀 𝗰𝗼𝗹𝗹𝗲𝗰𝘁𝗶𝗼𝗻𝘀 𝘄𝗶𝘁𝗵 𝗜𝗗 𝗰𝗿𝗼𝘀𝘀-𝗿𝗲𝗳𝗲𝗿𝗲𝗻𝗰𝗲𝘀:• 𝗘𝗻𝘁𝗶𝘁𝗶𝗲𝘀 — deduplicated nodes, each carrying a list of the relation IDs they participate in
• 𝗥𝗲𝗹𝗮𝘁𝗶𝗼𝗻𝘀 — embedded triples pointing to the entity IDs and source passage IDs on each side
• 𝗣𝗮𝘀𝘀𝗮𝗴𝗲𝘀 — original document chunks with back-references to the extracted entities and relations

Those ID references are the graph structure. Subgraph expansion follows them to surface bridge entities that the question never mentions — the step that makes multi-hop reasoning work. Then a single LLM reranking pass filters the expanded candidate pool down to what actually answers the question, and one generation call produces the answer from the full source passages.

That pipeline makes 𝟮 𝗟𝗟𝗠 𝗰𝗮𝗹𝗹𝘀 𝗽𝗲𝗿 𝗾𝘂𝗲𝗿𝘆 (rerank + generate), compared to 3-5 for IRCoT and 5-10+ for Agentic RAG. Against a 5-call iterative baseline, that works out to roughly 𝟲𝟬% 𝗹𝗼𝘄𝗲𝗿 𝗔𝗣𝗜 𝗰𝗼𝘀𝘁 𝗮𝗻𝗱 𝟮-𝟯𝘅 𝗳𝗮𝘀𝘁𝗲𝗿 𝗿𝗲𝘀𝗽𝗼𝗻𝘀𝗲𝘀, with predictable latency instead of spikes when an agent decides to loop again.

On the benchmarks: 𝟴𝟳.𝟴% 𝗮𝘃𝗲𝗿𝗮𝗴𝗲 𝗥𝗲𝗰𝗮𝗹𝗹@𝟱 across MuSiQue, HotpotQA, and 2WikiMultiHopQA — against 𝟴𝟳.𝟭% for HippoRAG 2, under the same evaluation setup, with no graph database and no ColBERTv2.

𝗛𝗼𝘄 𝗱𝗼 𝘆𝗼𝘂 𝗶𝗻𝘀𝘁𝗮𝗹𝗹 𝘁𝗵𝗶𝘀? You only need to type"pip install vector-graph-rag". It defaults to Milvus Lite — a local .db file, so you don't need to configure anything.
𝗙𝗼𝗿 𝗺𝗼𝗿𝗲 𝗶𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻, 𝘀𝗲𝗲 𝗶𝘁𝘀 𝗴𝗶𝘁𝗵𝘂𝗯 𝗽𝗮𝗴𝗲: https://t.co/uyBstuGBcq

If your corpus is knowledge-dense — legal, biomedical, financial — and your questions routinely cross 2-4 document boundaries, this is the architecture worth testing first.

[Milvus @milvusio]

Semantic search, full-text search, metadata filtering, and hybrid retrieval in one open-source system. Thanks to @DanKornas for sharing this clear overview of Milvus and the use cases it supports.

[Dan Kornas @DanKornas]

Need one retrieval layer for semantic search, full-text search, and metadata filtering?

Milvus is an open-source vector database for AI engineers building RAG, semantic search, multimodal search, and recommendation systems.

It helps you organize and search unstructured data by storing vectors alongside scalar fields, then combining vector search with metadata filters, full-text search, or hybrid retrieval.

Key features:

• Dense and sparse retrieval – supports semantic vectors, BM25 full-text search, and learned sparse embeddings.
• Hybrid search – stores dense and sparse vectors together and can rerank results from multiple searches.
• Multiple index types – includes HNSW, IVF, FLAT, SCANN, DiskANN, and quantized variants.
• Flexible deployment – runs as Milvus Lite, on one machine in Standalone mode, or in a distributed, Kubernetes-native architecture.
• Access controls – includes user authentication, TLS encryption, and role-based access control.

It’s open-source (Apache 2.0 license).

Link in the reply 👇

[Milvus @milvusio]

@DanKornas Thanks for featuring Milvus, Dan! Really appreciate how clearly you covered the different retrieval options and deployment paths in one overview.

[Milvus @milvusio]

Everyone is talking about agent memory, but the term is starting to hide more than it explains.

For coding agents, it is not just “save chat history, embed it, retrieve top-k” anymore.

These open-source projects show that “agent memory” can mean very different things:

1. codebase-memory-mcp
https://t.co/3YTSbzwwEp

Memory as codebase structure: functions, classes, call chains, routes, dependencies, and impact paths.

2. MemSearch
https://t.co/OoqhEFvafa

Memory as retrieval infrastructure: Markdown as the source of truth, Milvus as the index, plus hybrid search and progressive recall.

3. agentmemory
https://t.co/DjB7SbAkuQ

Memory as workspace context that follows coding agents across Claude Code, Codex, Cursor, Gemini CLI, OpenCode, and MCP clients.

4. Basic Memory
https://t.co/5uv1wErqm2

Memory as a readable knowledge base: Markdown notes that humans can edit and agents can search.

5. ai-memory
https://t.co/RI1nwVvSZR

Memory as continuity and handoff, so work can move between agent vendors without re-explaining architecture, failed attempts, and open questions.

These projects point to a simple trend: agent memory is moving from “store more context” to “retrieve the right context.”

For coding agents, memory is becoming less like a notes file and more like a retrieval layer.

[Milvus @milvusio]

When 𝗞𝗶𝗺𝗶 𝗞𝟯 came out, we tested it right away. This is our experimental sharing:

Kimi K3 has 2.8T parameters, a 1M-token context window, native visual understanding, and strong coding-agent scores. After a few internal runs, we decided to move part of our workflow from Fable to K3.

The first surprise came from marketing.
One teammate in marketing used K3 to prototype a campaign landing page in one afternoon. Not production-ready, but enough to align the idea before involving engineering.

That was the good part.

The less fun part was the token bill.

Large-context coding agents can spend a lot of tokens just figuring out where things are in a repo. Kimi's pricing makes that visible: cache-hit input is $0.30/MTok, cache-miss input is $3.00/MTok, and output is $15.00/MTok.

So we added 𝗰𝗹𝗮𝘂𝗱𝗲-𝗰𝗼𝗻𝘁𝗲𝘅𝘁.

It gives coding agents semantic code search, so they retrieve relevant files and snippets instead of loading whole directories into the prompt.

In our controlled evaluation, Claude Context MCP reduced tokens by around 40% at equivalent retrieval quality.

K3 made the agent stronger. claude-context made it cheaper to aim that strength at the right code.

GitHub: https://t.co/yyWCfHbgZv

[Milvus @milvusio]

𝗛𝗼𝘄 𝗱𝗼 𝘆𝗼𝘂 𝗰𝗼𝗺𝗯𝗶𝗻𝗲 𝗮𝗴𝗲𝗻𝘁 𝗺𝗲𝗺𝗼𝗿𝘆 𝘄𝗶𝘁𝗵 𝘀𝗸𝗶𝗹𝗹𝘀? 𝗧𝗿𝘆 𝗼𝘂𝗿 𝗼𝗽𝗲𝗻-𝘀𝗼𝘂𝗿𝗰𝗲 𝘁𝗼𝗼𝗹 𝗺𝗲𝗺𝘀𝗲𝗿𝗮𝗰𝗵 𝘁𝗵𝗮𝘁'𝘀 𝘂𝗽𝗱𝗮𝘁𝗲𝗱 𝘁𝗵𝗲 𝗳𝗲𝗮𝘁𝘂𝗿𝗲, 𝗺𝗲𝗺𝗼𝗿𝘆-𝘁𝗼-𝘀𝗸𝗶𝗹𝗹.

𝗔𝗴𝗲𝗻𝘁 𝗺𝗲𝗺𝗼𝗿𝘆 𝘁𝗵𝗮𝘁'𝘀 𝗮𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝘃𝗮𝗹𝘂𝗮𝗯𝗹𝗲 𝗰𝗮𝗽𝘁𝘂𝗿𝗲𝘀 𝘁𝗵𝗿𝗲𝗲 𝘁𝗵𝗶𝗻𝗴𝘀: 𝘄𝗵𝗮𝘁 𝘁𝗵𝗲 𝗮𝗴𝗲𝗻𝘁 𝗸𝗻𝗼𝘄𝘀, 𝘄𝗵𝗮𝘁 𝗶𝘁 𝗵𝗮𝘀 𝗲𝘅𝗽𝗲𝗿𝗶𝗲𝗻𝗰𝗲𝗱, 𝗮𝗻𝗱 𝗵𝗼𝘄 𝗶𝘁 𝗱𝗼𝗲𝘀 𝘁𝗵𝗶𝗻𝗴𝘀.

Developers used to keep those in three separate systems: knowledge went into RAG, experience piled up in an agent memory layer, and procedures were written down as skill files.

But those memories — debug sessions, API integrations, releases, user corrections — are essentially part of the agent's skills too. Each one carries the scenario, the feedback, and the outcome. Plenty of reusable workflows are already hiding in there.

𝗦𝗼 𝘄𝗲 𝗯𝘂𝗶𝗹𝘁 𝗠𝗲𝗺𝗼𝗿𝘆-𝘁𝗼-𝗦𝗸𝗶𝗹𝗹 𝗶𝗻𝘁𝗼 𝗠𝗲𝗺𝗦𝗲𝗮𝗿𝗰𝗵, 𝘁𝗵𝗲 𝗮𝗴𝗲𝗻𝘁 𝗺𝗲𝗺𝗼𝗿𝘆 𝗹𝗶𝗯𝗿𝗮𝗿𝘆 𝘄𝗲 𝗼𝗽𝗲𝗻-𝘀𝗼𝘂𝗿𝗰𝗲𝗱 𝗲𝗮𝗿𝗹𝗶𝗲𝗿 𝘁𝗵𝗶𝘀 𝘆𝗲𝗮𝗿.

An opt-in background pass mines your history for the workflows that keep recurring and are worth capturing, then automatically distills them into candidate skills. If a release workflow keeps showing up, the system recognizes it can be abstracted into a skill and writes it up as a candidate file. When the workflow changes later, the candidate evolves with it.

𝗕𝘂𝘁 𝗻𝗼 𝗮𝗴𝗲𝗻𝘁 𝗲𝘃𝗲𝗿 𝗮𝘂𝘁𝗼-𝗹𝗼𝗮𝗱𝘀 𝗮 𝗰𝗮𝗻𝗱𝗶𝗱𝗮𝘁𝗲 𝘀𝗸𝗶𝗹𝗹. It stays a draft. You review it, edit it, confirm it, and then install it into the skill directory of Claude Code, Codex, or other agents you use.

𝗠𝗲𝗺𝗦𝗲𝗮𝗿𝗰𝗵 𝗶𝘀 𝗼𝗽𝗲𝗻 𝘀𝗼𝘂𝗿𝗰𝗲. 𝗜𝗳 𝘆𝗼𝘂'𝗿𝗲 𝗮𝗹𝘀𝗼 𝘄𝗼𝗿𝗸𝗶𝗻𝗴 𝗼𝗻 𝗮𝗴𝗲𝗻𝘁 𝗺𝗲𝗺𝗼𝗿𝘆, 𝗹𝗼𝗻𝗴-𝘁𝗲𝗿𝗺 𝗰𝗼𝗻𝘁𝗲𝘅𝘁, 𝗼𝗿 𝗰𝗿𝗼𝘀𝘀-𝗽𝗹𝗮𝘁𝗳𝗼𝗿𝗺 𝗔𝗜 𝘄𝗼𝗿𝗸𝗳𝗹𝗼𝘄𝘀, 𝗴𝗶𝘃𝗲 𝗶𝘁 𝗮 𝘁𝗿𝘆: https://t.co/eGAavvLonV

[Milvus @milvusio]

Semantic search in news feeds, e-commerce, and recommendation systems needs to balance relevance with freshness.
𝗠𝗶𝗹𝘃𝘂𝘀 𝗽𝗿𝗼𝘃𝗶𝗱𝗲𝘀 𝘁𝗵𝗿𝗲𝗲 𝗱𝗲𝗰𝗮𝘆 𝗳𝘂𝗻𝗰𝘁𝗶𝗼𝗻𝘀 𝗳𝗼𝗿 𝗿𝗲𝗿𝗮𝗻𝗸𝗶𝗻𝗴. Three decay models are available, each producing a different freshness curve:
• 𝗘𝘅𝗽𝗼𝗻𝗲𝗻𝘁𝗶𝗮𝗹: fast initial decay with a long tail. News and social feeds where recency should dominate.
• 𝗚𝗮𝘂𝘀𝘀𝗶𝗮𝗻: bell-shaped curve with gradual roll-off. Balanced search like restaurant recommendations, where moderate distance should reduce rather than eliminate a result's score.
• 𝗟𝗶𝗻𝗲𝗮𝗿: constant penalty with a clear cutoff point. Event search where anything older than two weeks should not appear at all.

In this test, the query is "artificial intelligence advancements," against seven articles spanning 1 to 120 days old, under four ranking approaches.

No decay: pure semantic similarity.
• Ranking is determined entirely by L2 distance (lower = more similar). Time is invisible.
• Two articles with nearly identical content but published 90 days apart score the same: 0.7090. The 90-day-old article sits at #1.
• An article about "AI Ethics Guidelines" scores lowest because "ethics" is semantically farther from "advancements" than the other articles, even though it is also about AI.

Gaussian decay (offset: 7 days, scale: 14 days, decay: 0.5)
• The two newest articles (1-day and 5-day) jump to the top two positions. Both fall within the 7-day offset, so they receive no time penalty and rank by their original semantic similarity.
• A 15-day-old article drops to 0.3601 but stays visible. A 30-day-old article falls to 0.0642.
• Everything older than 60 days: 0.0000. The bell curve penalizes gradually at first, then sharply.

Exponential decay (offset: 3 days, scale: 10 days, decay: 0.3)
• A 1-day-old article scores 0.5979. A 5-day-old article drops to 0.4774 because it has already exceeded the 3-day offset, even though it is still relatively new.
• At 15 days: 0.1065, far below Gaussian's 0.3601 for the same article. At 30 days: nearly zero.
• Under this configuration, the steepest curve. It forgets faster than Gaussian and penalizes harder at every age beyond the offset.

Linear decay: the most gradual decline.
• A 90-day-old article still scores 0.3037. A 120-day-old article: 0.2339. Both score zero under Gaussian and exponential.
• Because the penalty is constant rather than accelerating, semantic relevance carries more weight relative to time. A highly relevant 90-day-old article outranks a weakly relevant 30-day-old one: the time penalty was not steep enough to override the stronger semantic match.
• The slope is gentle, but linear decay does reach zero at a clear cutoff point, giving you a predictable expiration boundary.
𝗖𝗵𝗼𝗼𝘀𝗶𝗻𝗴 𝘁𝗵𝗲 𝗰𝘂𝗿𝘃𝗲 𝗺𝗮𝘁𝘁𝗲𝗿𝘀 𝗺𝗼𝗿𝗲 𝘁𝗵𝗮𝗻 𝗰𝗵𝗼𝗼𝘀𝗶𝗻𝗴 𝘁𝗼 𝗮𝗱𝗱 𝗱𝗲𝗰𝗮𝘆.

Learn more here:https://t.co/72j2ip5E7I

[Milvus @milvusio]

𝗜𝗻 𝗲-𝗰𝗼𝗺𝗺𝗲𝗿𝗰𝗲 𝗮𝗻𝗱 𝗻𝗲𝘄𝘀 𝘀𝗲𝗮𝗿𝗰𝗵, 𝘀𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝘀𝗶𝗺𝗶𝗹𝗮𝗿𝗶𝘁𝘆 𝗮𝗹𝗼𝗻𝗲 𝗶𝘀 𝗻𝗼𝘁 𝗲𝗻𝗼𝘂𝗴𝗵 𝘁𝗼 𝗿𝗮𝗻𝗸 𝗿𝗲𝘀𝘂𝗹𝘁𝘀. Milvus's Time-aware Ranking Functions (also called Decay Functions) rerank retrieval results by applying time-decay functions, so recent content ranks higher without sacrificing relevance.

Milvus's Decay Functions combining vector similarity with configurable time-decay to dynamically adjust each document's relevance score. 𝗧𝗵𝗲 𝘀𝗰𝗼𝗿𝗶𝗻𝗴 𝘄𝗼𝗿𝗸𝘀 𝗶𝗻 𝘁𝗵𝗿𝗲𝗲 𝘀𝘁𝗲𝗽𝘀:
• 𝗖𝗼𝗻𝘃𝗲𝗿𝘁 𝗱𝗶𝘀𝘁𝗮𝗻𝗰𝗲𝘀 𝗶𝗻𝘁𝗼 𝗵𝗶𝗴𝗵𝗲𝗿-𝗶𝘀-𝗯𝗲𝘁𝘁𝗲𝗿 𝘀𝗰𝗼𝗿𝗲𝘀. For L2 and JACCARD (lower = more similar): normalized_score = 1.0 − (2 × arctan(score)) / π. COSINE, IP, and BM25 scores are used directly.
• 𝗖𝗼𝗺𝗽𝘂𝘁𝗲 𝗮 𝗱𝗲𝗰𝗮𝘆 𝘀𝗰𝗼𝗿𝗲 from a numeric field like a timestamp, using one of three decay models: exponential, Gaussian, or linear. The result is a value between 0 and 1, representing how close the document is to a reference point (typically "now").
• 𝗠𝘂𝗹𝘁𝗶𝗽𝗹𝘆: final_score = normalized_similarity_score × decay_score. In hybrid search with multiple vector fields, Milvus takes the maximum normalized score first: final_score = max(normalized_scores) × decay_score.

Each model takes a configurable origin, scale (the distance beyond the offset at which the score drops to the decay value), and an optional offset (a no-penalty zone around the origin where the score stays at 𝟭.𝟬).

𝗟𝗲𝗮𝗿𝗻 𝗺𝗼𝗿𝗲 𝗵𝗲𝗿𝗲: https://t.co/PLK2ouE4Dt

[Milvus @milvusio]

𝗚𝗿𝗲𝗽-𝘀𝘁𝘆𝗹𝗲 𝘀𝗲𝗮𝗿𝗰𝗵 𝗶𝗻 𝗖𝗹𝗮𝘂𝗱𝗲 𝗖𝗼𝗱𝗲 𝘄𝗼𝗿𝗸𝘀 𝗳𝗼𝗿 𝗰𝗼𝗱𝗲 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹, 𝗯𝘂𝘁 𝗶𝘁 𝗯𝘂𝗿𝗻𝘀 𝘁𝗵𝗿𝗼𝘂𝗴𝗵 𝘁𝗼𝗸𝗲𝗻𝘀 𝘂𝗻𝗻𝗲𝗰𝗲𝘀𝘀𝗮𝗿𝗶𝗹𝘆. 𝗪𝗲 𝗯𝘂𝗶𝗹𝘁 𝗖𝗹𝗮𝘂𝗱𝗲 𝗖𝗼𝗻𝘁𝗲𝘅𝘁, 𝗮𝗻 𝗼𝗽𝗲𝗻-𝘀𝗼𝘂𝗿𝗰𝗲 𝗠𝗖𝗣 𝘁𝗼𝗼𝗹 𝘁𝗵𝗮𝘁 𝗰𝘂𝘁𝘀 𝘁𝗼𝗸𝗲𝗻 𝘂𝘀𝗮𝗴𝗲 𝗯𝘆 ~𝟰𝟬%.

Grep-style code retrieval has drawn plenty of community discussion. For pure recall, literal matching gets the job done. But in a large codebase, it forces the model to wade through massive amounts of irrelevant code to find the few lines that matter.

𝗧𝗵𝗿𝗲𝗲 𝘁𝗵𝗶𝗻𝗴𝘀 𝗺𝗮𝗸𝗲 𝘁𝗵𝗶𝘀 𝗲𝘅𝗽𝗲𝗻𝘀𝗶𝘃𝗲 𝗮𝘁 𝘀𝗰𝗮𝗹𝗲:
• 𝗜𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻 𝗼𝘃𝗲𝗿𝗹𝗼𝗮𝗱. Large repos produce hundreds of literal matches for common terms. Most are noise, but they all consume tokens and inference time.
• 𝗦𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗯𝗹𝗶𝗻𝗱𝗻𝗲𝘀𝘀. Grep matches characters, not meaning. compute_final_cost() and calculate_total_price() do the same thing, but a string match won't connect them.
• 𝗜𝗻𝗰𝗼𝗺𝗽𝗹𝗲𝘁𝗲 𝗰𝗼𝗻𝘁𝗲𝘅𝘁. Line-level matches strip away the surrounding class and method structure. The model compensates by reading more files, burning more tokens.

𝗪𝗲 𝗼𝗽𝗲𝗻-𝘀𝗼𝘂𝗿𝗰𝗲𝗱 𝗖𝗹𝗮𝘂𝗱𝗲 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝘁𝗼 𝗮𝗱𝗱𝗿𝗲𝘀𝘀 𝗲𝘅𝗮𝗰𝘁𝗹𝘆 𝘁𝗵𝗶𝘀. 𝗜𝘁 𝗵𝗶𝘁 #𝟭 𝗼𝗻 𝗚𝗶𝘁𝗛𝘂𝗯'𝘀 𝗱𝗮𝗶𝗹𝘆 𝘁𝗿𝗲𝗻𝗱𝗶𝗻𝗴 𝗰𝗵𝗮𝗿𝘁 𝗮𝗻𝗱 𝗰𝘂𝗿𝗿𝗲𝗻𝘁𝗹𝘆 𝗵𝗮𝘀 𝗼𝘃𝗲𝗿 𝟭𝟮,𝟬𝟬𝟬 𝘀𝘁𝗮𝗿𝘀.

𝗜𝗻 𝗼𝘂𝗿 𝗯𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸, 𝗖𝗹𝗮𝘂𝗱𝗲 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗿𝗲𝗱𝘂𝗰𝗲𝗱 𝘁𝗼𝗸𝗲𝗻 𝗰𝗼𝗻𝘀𝘂𝗺𝗽𝘁𝗶𝗼𝗻 𝗯𝘆 ~𝟰𝟬% 𝗮𝘁 𝗲𝗾𝘂𝗶𝘃𝗮𝗹𝗲𝗻𝘁 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗾𝘂𝗮𝗹𝗶𝘁𝘆.
It's a code retrieval MCP server that integrates a vector database and embedding model into the search layer. It plugs into Claude Code and is also compatible with Codex CLI, Gemini CLI, Qwen Code, Cline, Cursor, Windsurf, and other MCP-compatible agents.

𝗜𝘁 𝗱𝗼𝗲𝘀 𝘁𝗵𝗿𝗲𝗲 𝘁𝗵𝗶𝗻𝗴𝘀 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁𝗹𝘆:
• 𝗦𝗺𝗮𝗿𝘁 𝗳𝗶𝗹𝘁𝗲𝗿𝗶𝗻𝗴. Vector similarity ranks code by relevance, surfacing the most related results first instead of every literal match.
• 𝗖𝗼𝗻𝗰𝗲𝗽𝘁 𝗺𝗮𝘁𝗰𝗵𝗶𝗻𝗴. Dense retrieval can match conceptually related code even when function names differ entirely.
• 𝗖𝗼𝗻𝘁𝗲𝘅𝘁-𝗮𝘄𝗮𝗿𝗲 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹. Results are chunked around complete syntactic units (functions, classes, methods), giving the model enough structural context to reason about behavior.

Under the hood, Claude Context uses MCP as its interface layer and Milvus for codebase indexing and hybrid search (BM25 + dense vectors). Embeddings support OpenAI, VoyageAI (which ships a code-specific model), and Ollama for local deployment when privacy matters.

Code chunking uses AST parsing by default, splitting along semantic boundaries instead of cutting at arbitrary line counts. For files AST can't parse, LangChain's text splitter handles the fallback.

For incremental updates, Claude Context uses Merkle-tree change detection. If the root hash hasn't changed, the index stays untouched. When it changes, the system identifies exactly which files were modified and re-embeds only those.

Change one function, and you don't re-index the entire project.

🔗 https://t.co/yyWCfHbgZv

📝 Full benchmark and architecture: https://t.co/Sevmvp37HT

[Milvus @milvusio]

Using a demo to estimate the real production cost of a vector database is where cost planning goes wrong for many teams.

The demo phase can give you a reasonable answer on two things:
• Storage: embeddings, raw content, metadata, scalar fields, and index files.
• Online serving: the compute needed for the current query volume and latency target.

But a demo usually cannot answer three other cost questions:
• Updates: how often data is added, deleted, re-embedded, re-indexed, or checked for recall regressions.
• Workload mismatch: whether offline analysis, evaluation, or batch jobs are running on always-on infrastructure.
• Duplication: how many copies of the same AI data end up across vector search, lakehouse, training, evaluation, and governance systems.

Different data scales and usage frequencies require different resource models.
For example, one autonomous driving workload needed vector search over a 1B-row collection for analysis.
𝗔 𝗱𝗲𝗱𝗶𝗰𝗮𝘁𝗲𝗱 𝗰𝗹𝘂𝘀𝘁𝗲𝗿 was estimated at about $7,000 per month. 𝗦𝗲𝗿𝘃𝗲𝗿𝗹𝗲𝘀𝘀 was about $10,800 per month. But the workload only ran for a few hours each month. With 𝗭𝗶𝗹𝗹𝗶𝘇 𝗖𝗹𝗼𝘂𝗱 𝗢𝗻-𝗗𝗲𝗺𝗮𝗻𝗱 𝗦𝗲𝗮𝗿𝗰𝗵, the cost came in under $500 per month.

In enterprise environments, however, these workload patterns usually coexist. A single system may have steady online traffic, offline batch processing, evaluation jobs, and sudden demand spikes.
The goal is not to choose one resource model for everything. It is to accurately identify each workload and match it to the right instance type.
Use dedicated capacity for steady, latency-sensitive traffic; serverless for unpredictable or spiky online demand; and on-demand compute for low-frequency analysis, evaluation, and batch jobs.

[Milvus @milvusio]

Duplicate text in your LLM pre-training corpus drives up training cost and reweights parts of the training data. 𝗠𝗶𝗻𝗛𝗮𝘀𝗵 𝗟𝗦𝗛 𝗺𝗮𝗸𝗲𝘀 𝘀𝗰𝗮𝗹𝗮𝗯𝗹𝗲 𝗱𝗲𝗱𝘂𝗽 𝗽𝗿𝗮𝗰𝘁𝗶𝗰𝗮𝗹, 𝗮𝗻𝗱 𝗠𝗶𝗹𝘃𝘂𝘀 𝟮.𝟲 𝗮𝗱𝗱𝗲𝗱 𝗶𝘁 𝗮𝘀 𝗮 𝗻𝗮𝘁𝗶𝘃𝗲 𝗶𝗻𝗱𝗲𝘅 𝘁𝘆𝗽𝗲.

The usual dedup options break down quickly:
• 𝗘𝘅𝗮𝗰𝘁 𝗺𝗮𝘁𝗰𝗵𝗶𝗻𝗴 is too strict for text dedup. Near-identical documents often differ by boilerplate, formatting, punctuation, or minor edits, and exact comparison misses all of them.
• 𝗣𝗮𝗶𝗿𝘄𝗶𝘀𝗲 𝗰𝗼𝗺𝗽𝗮𝗿𝗶𝘀𝗼𝗻 catches more, but the computation explodes with data volume. At millions of documents, it's infeasible.

𝗠𝗶𝗻𝗛𝗮𝘀𝗵 𝗟𝗦𝗛 works well as a coarse-pass filter. It's approximate deduplication with a tunable recall/precision tradeoff, fast enough to run inline during ingestion to check candidates before storing redundant content.

𝗠𝗶𝗹𝘃𝘂𝘀 𝟮.𝟲 𝗮𝗱𝗱𝗲𝗱 𝗻𝗮𝘁𝗶𝘃𝗲 𝗠𝗶𝗻𝗛𝗮𝘀𝗵 𝗟𝗦𝗛 𝗶𝗻𝗱𝗲𝘅𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗮 𝗱𝗲𝗱𝗶𝗰𝗮𝘁𝗲𝗱 𝗺𝗲𝘁𝗿𝗶𝗰 𝘁𝘆𝗽𝗲, 𝗹𝗲𝘁𝘁𝗶𝗻𝗴 𝘁𝗲𝗮𝗺𝘀 𝗶𝗻𝗱𝗲𝘅 𝗮𝗻𝗱 𝘀𝗲𝗮𝗿𝗰𝗵 𝗠𝗶𝗻𝗛𝗮𝘀𝗵 𝘀𝗶𝗴𝗻𝗮𝘁𝘂𝗿𝗲𝘀 𝗳𝗼𝗿 𝗱𝘂𝗽𝗹𝗶𝗰𝗮𝘁𝗲 𝗮𝗻𝗱 𝗻𝗲𝗮𝗿-𝗱𝘂𝗽𝗹𝗶𝗰𝗮𝘁𝗲 𝗱𝗲𝘁𝗲𝗰𝘁𝗶𝗼𝗻 𝗮𝘁 𝗰𝗼𝗹𝗹𝗲𝗰𝘁𝗶𝗼𝗻 𝘀𝗰𝗮𝗹𝗲.

𝗪𝗵𝗲𝗿𝗲 𝘁𝗵𝗶𝘀 𝗵𝗲𝗹𝗽𝘀:
• 𝗟𝗟𝗠 𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝗱𝗮𝘁𝗮 𝗰𝗹𝗲𝗮𝗻𝗶𝗻𝗴: removing duplicate text from crawled corpora before training begins
• 𝗟𝗮𝗿𝗴𝗲-𝘀𝗰𝗮𝗹𝗲 𝗽𝗹𝗮𝗴𝗶𝗮𝗿𝗶𝘀𝗺 𝘀𝗰𝗿𝗲𝗲𝗻𝗶𝗻𝗴: detecting substantial text overlap in academic or content pipelines as a coarse-pass filter

𝗟𝗶𝗺𝗶𝘁𝗮𝘁𝗶𝗼𝗻: MinHash operates on token-set overlap, not word order or semantics. It catches verbatim and near-verbatim duplication but won't flag paraphrases. For semantic or paraphrase-level deduplication, teams layer in SimHash, TF-IDF with cosine similarity, or embedding-based methods.

In Milvus, MinHash signatures are stored as 𝗯𝗶𝗻𝗮𝗿𝘆 𝘃𝗲𝗰𝘁𝗼𝗿𝘀 approximating 𝗝𝗮𝗰𝗰𝗮𝗿𝗱 𝘀𝗶𝗺𝗶𝗹𝗮𝗿𝗶𝘁𝘆 between document token sets. Standard metrics like Jaccard, L2, or cosine can't be applied directly to these signatures, so Milvus introduced a dedicated metric called 𝗠𝗛𝗝𝗔𝗖𝗖𝗔𝗥𝗗. For higher accuracy, Milvus also supports a refined search mode that recomputes exact Jaccard using stored token sets, reducing false positives.

The dedup pipeline:
• 𝗦𝗲𝘁𝘂𝗽: Create a collection with a 𝗠𝗜𝗡𝗛𝗔𝗦𝗛_𝗟𝗦𝗛 index, using the MHJACCARD metric and configurable LSH parameters (e.g., band count)
• 𝗦𝗶𝗴𝗻𝗮𝘁𝘂𝗿𝗲 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻: For each incoming document, tokenize or shingle it and generate a MinHash signature (e.g., using datasketch)
• 𝗗𝘂𝗽𝗹𝗶𝗰𝗮𝘁𝗲 𝗰𝗵𝗲𝗰𝗸: Search the collection with the new signature to find existing near-duplicates
• 𝗜𝗻𝘀𝗲𝗿𝘁: Store only documents that don't exceed your similarity threshold, alongside metadata

An AI model company used this pipeline to deduplicate 𝟭𝟬 𝗯𝗶𝗹𝗹𝗶𝗼𝗻 𝗱𝗼𝗰𝘂𝗺𝗲𝗻𝘁𝘀 for LLM training, more than doubling throughput and cutting costs 𝟯-𝟱𝘅 compared to their previous MapReduce-based approach.

More info here: https://t.co/3JQyKaXSCi

[Milvus @milvusio]

Debugging a slow Milvus cluster once meant bouncing between the console, Grafana, and several monitoring views just to find out where to look.
𝗧𝗼𝗱𝗮𝘆, 𝘄𝗶𝘁𝗵 𝗔𝘁𝘁𝘂 𝟯.𝟬 𝗯𝗲𝘁𝗮, 𝗰𝗹𝘂𝘀𝘁𝗲𝗿 𝗼𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀 𝗮𝗿𝗲 𝗮𝗹𝗹 𝗼𝗻 𝗼𝗻𝗲 𝘀𝗰𝗿𝗲𝗲𝗻. 𝗬𝗼𝘂 𝗰𝗮𝗻 𝗰𝗵𝗲𝗰𝗸:
• 𝗢𝘃𝗲𝗿𝘃𝗶𝗲𝘄: version, deployment mode, node count, database count, collection count, and total entity count
• 𝗥𝗲𝗮𝗹-𝘁𝗶𝗺𝗲 𝗺𝗲𝘁𝗿𝗶𝗰𝘀: QPS, insert/delete rates, query latency, cache hit rate, and storage, pulled from a Prometheus endpoint without opening Grafana separately
• 𝗦𝗹𝗼𝘄-𝗾𝘂𝗲𝗿𝘆 𝗮𝗻𝗮𝗹𝘆𝘀𝗶𝘀: slow queries by type, duration, collection, timestamp, and user
• 𝗧𝗼𝗽𝗼𝗹𝗼𝗴𝘆 𝘃𝗶𝗲𝘄: node topology and coordinator component status
• 𝗕𝗮𝗰𝗸𝘂𝗽 𝗮𝗻𝗱 𝗿𝗲𝘀𝘁𝗼𝗿𝗲: full or incremental backups to S3, MinIO, GCS, or Azure

For teams running several Milvus clusters, the multi-cluster sidebar gives you a quick read on cluster state before you start digging. The point is that the first pass of cluster inspection no longer has to start with five tabs and a guessing game.

𝗧𝗵𝗲 𝗳𝘂𝗹𝗹 𝗔𝘁𝘁𝘂 𝟯.𝟬 𝗿𝗲𝗯𝘂𝗶𝗹𝗱, 𝗶𝗻𝗰𝗹𝘂𝗱𝗶𝗻𝗴 𝘁𝗵𝗲 𝗺𝘂𝗹𝘁𝗶-𝗰𝗹𝘂𝘀𝘁𝗲𝗿 𝘀𝗶𝗱𝗲𝗯𝗮𝗿 𝗮𝗻𝗱 𝗯𝘂𝗶𝗹𝘁-𝗶𝗻 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁, 𝗶𝘀 𝗼𝗻 𝘁𝗵𝗲 𝗠𝗶𝗹𝘃𝘂𝘀 𝗯𝗹𝗼𝗴: https://t.co/LLfUcMHsnF