Mitesh Joshi

Definite optimist! Eng Leadership. Software engineer - diving into hard and soft system properties.

[Tobias Ziegler @Tobias__Ziegler]

Our paper, "How to Write to SSD", received a Best Research Paper Honorable Mention Award at VLDB ’26!
Bohyun (Lia) from TUM did a phenomenal job on this work. Check it out if you're interested in modern NVMe SSD internals and how to get the most performance out of them.

[Brandon Veiseh @BVeiseh]

One huge advantage that app layer vendors have is still evals.

Customers want to build their own agents but they dont know what models to use.

Vendors help by doing evals. Testing the right models (or fine tuning them) and deploying them correctly.

This is still a huge murky area that is holding enterprise back from self deploying agents for all their needs.

[Arpit Bhayani @arpit_bhayani]

A common failure mode we are seeing is systems leaking PII, and the root cause is almost always the same... People treat masking as a text-replacement problem, not a data-classification problem.

The common pattern: a support bot or RAG pipeline gets connected to real customer data. Someone runs a regex over the prompt to strip emails and phone numbers, ships it, and calls it done.

Regex is good at fixed-shape PII like card numbers or emails, but it cannot catch context-dependent PII, something like "the patient in room 4B", where the sensitive part is a relationship between tokens, not a pattern.

This can be fixed by classic named entity recognition. An NER model classifies a span using surrounding context, so it can flag "Alice" as a PERSON even without a trigger pattern. But NER alone misses domain-specific identifiers, like an internal employee ID format unique to your company.

What actually works is layered -

1. structural detectors for fixed-shape data
2. NER for free-text entities, and
3. domain-specific rule layer for your own schema.

Hope this helps.

[joaquindiez @joaquindiez]

DuckDB es barato. Consultar datos en S3 no siempre lo es.

El anuncio de que AWS ha firmado la compra de @ducklabs_com , la compañía que desarrolla DuckDB, tiene bastante sentido.

DuckDB ha demostrado que ya no necesitas desplegar una gran infraestructura distribuida para analizar cantidades importantes de datos. Puedes ejecutar el motor dentro de tu aplicación y consultar directamente ficheros Parquet almacenados en un object storage.

Ese es también el modelo que elegimos para @CalliopeBI .

Nuestra arquitectura mantiene una base DuckDB independiente por cliente e ingiere los datos principalmente como ficheros Parquet. El object storage actúa como capa de almacenamiento desacoplada, mientras DuckDB se encarga del procesamiento analítico.

Sobre el papel, la combinación parecía perfecta:

Almacenamiento barato y prácticamente ilimitado.

Datos en un formato abierto.

Computación local, sencilla y eficiente.

Sin necesidad de mantener un clúster analítico permanente.

Y, en gran medida, lo es.

Pero durante la operación de @CalliopeBI descubrimos un matiz importante: que DuckDB reduzca radicalmente el coste de computación no significa que desaparezca el coste de acceso a los datos.

Una consulta no es necesariamente una operación
DuckDB es muy eficiente leyendo Parquet. Utiliza los metadatos, aplica projection y filter pushdown y realiza peticiones parciales para descargar únicamente las columnas y bloques que necesita.

Eso evita leer el fichero completo, pero una consulta puede provocar varias operaciones contra el object storage:

Listados de objetos.

Lecturas de metadatos.

Peticiones parciales mediante HTTP Range.

Lectura de diferentes row groups.

Escrituras de nuevos ficheros.

Copias y operaciones de compactación.

Cuando tienes muchos clientes, muchos ficheros y consultas ejecutándose continuamente, ese número de operaciones crece rápidamente.

Amazon S3 cobra por las peticiones GET, PUT, COPY, POST y LIST, además del almacenamiento y, dependiendo de la arquitectura, del tráfico de salida. Cada operación individual es barata. El problema aparece cuando multiplicas una cantidad muy pequeña por millones de operaciones.

Pero el coste no fue la única cuestión.

Para contenerlo empiezas a introducir ficheros más grandes, compactaciones, cachés locales, batching y diferentes estrategias para reducir las peticiones. Todas ellas pueden ser buenas optimizaciones, pero también añaden complejidad.

En cierto momento te das cuenta de que ya no estás optimizando únicamente tu arquitectura de datos. También estás optimizando alrededor del modelo de facturación del proveedor.

Nuestra migración a Hetzner Object Storage
En Calliope decidimos migrar esta parte de la arquitectura desde Amazon S3 a Hetzner Object Storage.

Hetzner ofrece una API compatible con S3, por lo que pudimos mantener el modelo fundamental:

Parquet + DuckDB + object storage

No tuvimos que rediseñar el producto ni renunciar a los formatos abiertos. Cambiamos el endpoint, las credenciales y algunos detalles de configuración, pero no la arquitectura.

La diferencia importante está en el modelo económico. Hetzner incluye actualmente 1 TB de almacenamiento y 1 TB de tráfico de salida por 4,99 euros al mes, y no cobra individualmente las operaciones GET, PUT o DELETE. Esto hace que el coste sea mucho más predecible para un sistema analítico con numerosas lecturas parciales.

No significa que Hetzner sea automáticamente mejor para cualquier carga. Su propia documentación advierte de que el servicio funciona mejor cuando los objetos se escriben y recuperan, no cuando se modifican continuamente o se utilizan como sustituto de una base de datos transaccional.

Pero ese patrón encaja muy bien con Calliope: generamos ficheros analíticos, los almacenamos y DuckDB los consulta.

El resultado ha sido mantener las ventajas de la arquitectura y reducir considerablemente tanto el coste de almacenamiento como, sobre todo, el coste y la incertidumbre asociados a las operaciones.

La verdadera ventaja es la portabilidad
La conclusión más importante no es que S3 sea malo ni que Hetzner sea siempre mejor.

S3 ofrece una escala, madurez y ecosistema extraordinarios. Para determinados sistemas, esas capacidades justifican perfectamente su coste.

La lección es otra: “S3-compatible” es más valioso que “estar en S3”.

Gracias a DuckDB, Parquet y una API de almacenamiento compatible pudimos cambiar de proveedor sin cambiar nuestro modelo de datos ni reconstruir la plataforma. Esa capacidad de mover la carga es una ventaja arquitectónica y también una herramienta de negociación económica.

La compra de DuckLabs probablemente hará que la integración entre DuckDB y los servicios analíticos de AWS sea todavía mejor. Puede ser una gran noticia para el ecosistema.

Pero espero que DuckDB conserve precisamente aquello que lo ha hecho tan interesante: la capacidad de ejecutar la computación donde tenga sentido y de mantener los datos en formatos abiertos, sin que la arquitectura quede inevitablemente ligada a un único proveedor.

En Calliope seguimos apostando por DuckDB.

Simplemente hemos aprendido que elegir DuckDB no significa que también tengas que elegir Amazon S3.

[Luke Edwards @lukeed05]

Introducing `lane` — a CLI to manage:
🔹copy-on-write (CoW) worktrees
🔸 durable, in-project memories

Plain worktrees throw away every ignored file that made your checkout fast: node_modules, target/, .env files, etc. So you reinstall and rebuild, even though the caches are already on disk.

lane clones them by reference instead.

lane also manages project memory.
Attach a note to a file and a symbol:

$ lane note add src/auth.rs -a 'fn verify' 'must stay constant-time'
# or
$ git commit -am "fix(auth): prevent timing attach \
Why: src/auth.rs#fn verify | early return leaks token length"

When `fn verify` changes, the note is flagged. It stays flagged until someone resolves it. A confidently wrong note is worse than no note.

Lanes are cheap enough to run several at once, so parallel agents each get their own tree and their own notes. Every note is a separate file, so multiple agents can annotate the same function without a conflict.

lane never calls a model.

GitHub: https://t.co/aDPurOxriw
Docs: https://t.co/9Iq1UpIUUa

[Almog Gavra @almoggavra]

@nuonjon There are so many benefits from that. Another big benefit is that you can run compaction on cheap, slow storage devices (e.g. EBS) and save NVMe for your fast-path cache.

In most LSM compaction overhead requires ~1.5-2x disk headroom, which is $$$ on NVMe.

[Almog Gavra @almoggavra]

Running a database not designed for S3 on an S3-backed file system is a hack that gets you some benefits but misses the best parts. A few points:

1. An S3 native system allows you to run things like compaction and GC on a separate machine. Most databases that couple storage and compute don't, meaning your compaction and query execution thrash the same CPU.

2. If you don't design for explicit read-only replicas, you can't just spin up another replica and read the same data safely. Traditional systems often use the filesystem for locking and assume exclusive access, so a second instance can interfere with the active writer.

3. Caching will not be as efficient. A big part of making object native systems performant is caching with the explicit knowledge that cold misses are an order of magnitude more expensive. You could mirror everything to NVMe, but that defeats a lot of the cost benefits and predictable latency behaviors of object storage.

This is just the start of the list, but it feels very similar to trying to put a database on NFS and assuming your distsys problems are all solved.

[Akhilesh Mishra @livingdevops]

Kubernetes world is moving away from sidecar patterns.
Let me explain why.

A sidecar is a helper container. It sits next to your app container. Same pod. Same network. Same lifecycle.

People used sidecars for a lot of things.

Service mesh was the big one. Envoy would sit next to your app.
- It handled traffic. It handled retries and timeouts. It handled security between services(MTLS). Your app never knew it was there.

- Logging was another one. A small agent would sit next to your app. It would read the logs. It would ship them somewhere else.
- Metrics scraping worked the same way.
- Secrets injection too. Vault agent would run as a sidecar. It would fetch secrets and write them to a file.

This pattern felt smart. One app. Many helpers. No code changes needed.

But then problems showed up.

Every sidecar is another container.
That means more CPU, more memory.
One sidecar on ten pods is fine.
One sidecar on ten thousand pods is not fine.

Every sidecar can crash.
Every sidecar needs patching.
Every sidecar needs upgrading.
Multiply that by hundreds and thousands.

Startup order became a pain too.
- Sometimes the sidecar was not ready before the app started.
- Sometimes the sidecar died before the app finished its work. Small bugs. Big headaches.

Debugging got harder.
- Now you have two containers to check instead of one.

Cost added up quickly at scale.

So the industry asked a new question.

What if we do not need the sidecar at all?

Istio built something called an ambient mesh that required no sidecar injection.
The proxy moves completely outside the pod.

Cilium went even further by using eBPF (a kernel-level technology).
That means the networking logic lives in the kernel, so no sidecar container, and lightning-fast networking.

OpenTelemetry solved this for metrics and logs.
Instead of one sidecar per pod, you get one collector for many pods(runs as a DaemonSet and deployment).

Even Kubernetes noticed the pain.
In version 1.29, it gave sidecars a proper place in the pod lifecycle, fixing a lot of old bugs and startup issues.
It did not remove the sidecar. It just made it better.

Today we have two paths.
Some teams are removing sidecars completely.
Some teams are keeping sidecars but doing it the right way.

Where do you stand?

[Kelly Sommers @kellabyte]

Walgit leans on S3 primitives but has classic distsys bugs

Tests pass cuz its memstore makes cond deletes

But w/ S3 it HEAD->compare->DELETE so stale owner can delete new lease

The code incorrectly says S3 lacks conditional DELs. Which AI model wrote this? It's wrong in basics

[Jaana Dogan ヤナ ドガン @rakyll]

No one truly talks about the many foundational values in software engineering was about reusability. If reusability doesn’t matter that much, it impacts every fundamental value we’ve been holding for decades.

[Mehul Arora @infiniter3grets]

Object Storage (S3) vs Stream Storage (S2)

[Internal Tech Emails @TechEmails]

Steve Jobs on a potential Google hire

March 28, 2006

[Ali Ghodsi @alighodsi]

This is true. It wasn't actually possible before 2010 because datacenter networks would bottleneck. We used to design coupled storage/compute where you brought compute close to "big data". But research on "full bisection bandwidth" networks made it possible to essentially just talk from any machine to the storage system at full speed. The disaggregation started then! Databricks and Snowflake started soon after many others followed. Now "Put it on the object store" is the way to go.

[Chris @criccomini]

/goal ...

* Don't stop when things work. Stop when things work _and_ you're proud of your work. It should be high quality.

[Dennis Zhuang @killme20082]

A disturbing story about a Rust crate supply chain attack:

https://t.co/kDEzKToLrf

We really need Cargo RFC #3923.

It would allow projects to avoid newly published dependency versions until a configurable waiting period has elapsed.

npm has had a similar min-release-age feature for years.

It's a pity that progress has been so slow.

https://t.co/NBPwuQ51K7

#Rust

[Andrea @acolombiadev]

I work at GitHub. yesterday was rough and i'm not pretending otherwise. full root cause report is up if you want the timeline and numbers, and what we are doing to prevent this from happening again.
https://t.co/05do2WFoMa

[Almog Gavra @almoggavra]

This is a top 10 distributed systems blog post. Instant classic.

I've been hacking on a SlateDB backed git server for a few months, and Cursor settled on a similar design but I think using packfiles as the unit of compaction instead of SSTs is a better approach. h/t @vmg

[Marc Brooker @MarcJBrooker]

Handling edge-case failures like this is way simpler in distributed storage databases (e.g. Aurora) and distributed databases (e.g. Aurora DSQL) than in single-system databases. It's one of many cases where distribution simplifies system properties.

Here's how it works in DSQL:

[Vicent Martí @vmg]

Cool ship for my first day at SpaceXAI. I'm stoked to be working on Git infrastructure again!

I think everybody is now very aware of how hard it is to do this stuff at the scale that agents require. Can't wait to blog about our design and operational philosophy for the platform

[Mitesh Joshi @m_joshi1]

@Yugandhar_19 @AnjitaGargi Blr has two anytime fitness gyms.