Jack Vanlightly

Likes breaking ideas and systems, writing, picking systems apart @confluentinc
Ex @Splunk, @VMware
ESO/B. Tafreshi

[Jack Vanlightly @vanlightly]

Claude is pretty good a spotting confounders in perf test results.

[Yingjun Wu 🤘 @YingjunWu]

Recent security incidents are fundamentally changing how I think about open source.

The original idea behind open source was great: bring people together to build better software through collective intelligence.

But in the AI era, things are moving in a dangerous direction. Contributors are no longer humans, but AI agents.

As AI becomes more capable, malicious contributions will inevitably become more sophisticated. And with an overwhelming volume of AI-generated pull requests, thorough human code review simply cannot scale. This could undermine the very foundation of open source.

I see two possible paths forward:

- Stop accepting code contributions. Accept PRDs and specifications instead, reviewed by trusted humans. Let AI agents handle the implementation.

- Build superintelligent code review agents capable of comprehensively detecting security vulnerabilities, malicious code, and subtle attack vectors.

Open source is fundamentally a good thing. But until we achieve at least one of these, I think we need to be extremely cautious about its future.

[Jack Vanlightly @vanlightly]

My TLA+ spec of Chroma's S3 impl of wal3 is done. Next weekend, the write-up where I'll find commonalities and differences across the surveyed WAL-on-S3 designs, maybe come up with some kind of taxonomy.

I might come back later and cover the rest of the designs in my todo list, assuming the Interest Goblin is still interested by then.

https://t.co/ZtguHOpcJN

[Jack Vanlightly @vanlightly]

I think I've largely failed to make the wal3 spec generalized and abstract. I think I went too far trying to make it faithful to the implementation and so it has become a monster. Still, it did detect a real bug. I can't spend more time making it beautiful though. Day job. Will publish it over the weekend, then I'm calling it a day on writing more specs. I'll do the final write-up and classification of all the WAL-on-S3 designs I have surveyed.

[Jack Vanlightly @vanlightly]

Yeah, it's really simple! Some of these WALs though are designed the way they are (I assume) because they are tightly integrated into the system they serve, so might not be great general purpose approaches. Some complexity may arise due to the specific demands of a specific system.

[Jack Vanlightly @vanlightly]

Well, I got there too late. I just found a correctness bug in Chroma's wal3 this morning, and was working on an integration test to prove what my TLA+ spec found, and on reading the fork, I see that just 2 days ago, the bug got fixed!

https://t.co/uExeCKUQA3

[Jack Vanlightly @vanlightly]

@kellabyte Love the posts 😁

[Jack Vanlightly @vanlightly]

Next up, Chroma's wal3. Should be an interesting write up when I stop collecting designs.

[Jack Vanlightly @vanlightly]

Description of the consistency buy and solution https://t.co/KgvrRzj25X

[Jack Vanlightly @vanlightly]

Another day, another WAL-on-S3 system encoded in a TLA+ spec. This one is Shared Stream Consensus (S2C) which is more of a shared log for SMR, but fits closely enough to a WAL. After all, a WAL on shared storage is a shared log. My spec confirmed a theoretical consistency bug in the design, related to GC freeing a log address so a stale writer could do a put-if-absent of a log entry behind the snapshot boundary.

A second spec adds a fencing record in the log, SlateDB style, to avoid the scenario.

https://t.co/5PsFS3gYCt

[Jack Vanlightly @vanlightly]

From the OSDI LogDrive paper https://t.co/7C8kTPiiBc

[Jack Vanlightly @vanlightly]

cc @Mo69796244, I'll let you know when you can review ths spec and the variant with the fix.

[Jack Vanlightly @vanlightly]

I'm documenting the fix for the consistency issue I found in Shared Storage Consensus (S2C), and I'm referencing SlateDB (which I copied the solution from). That's what is neat about learning how varied systems work, you can easily steal what you already know works somewhere else! Will publish the latest spec over the weekend after more model checking.

[Jack Vanlightly @vanlightly]

@DominikTornow It's something old from aeronautics. Some gauges are in triplicate as with two copies of the gauge one faulty value means the pilot can't tell which is the right one, but with three, the pilot can hopefully tell the one that is faulty.

[Jack Vanlightly @vanlightly]

@StephanEwen Congratulations!

[Jack Vanlightly @vanlightly]

When reverse engineering a TLA+ spec from code, the hardest thing is understanding the code. While AI can write syntactically correct TLA+, I can't say it's good at writing specs. So I write the spec and use AI to help me review it. But AI is really useful when reading the code. It helps me understand what each class does, the roles of certain variables and structures, things that without AI can be quite time consuming and laborious. But I still read the code, I want a good handle for its structure and organization. Then bit by bit, as I walk the flows in the code, I build up a more simplified model in TLA+. My goal in the end is understanding.

[Jack Vanlightly @vanlightly]

@Mo69796244 I need to write the spec so I can get a firmer handle on it. I can also explore mitigations in the spec.

[Jack Vanlightly @vanlightly]

This morning I decided to look at Shared Storage Consensus (S2C) and one glaring correctness bug already stands out before I have started writing a spec. GC can delete log segments that allow a stale writer to make progress (resulting in unreadable data aka data loss). SlateDB uses fencing records which are not GCed for this purpose. S2C does not have this and so GC introduces a consistency problem.

Whenever the WAL is defined by a sequential address space guarded by put-if-absent, GC introduces the risk of allowing a stale writer to make progress resulting in data loss.

[Jack Vanlightly @vanlightly]

Right, in the case of a timeout, it would be invalid but harmless. It's a theoretical case. It would need very aggressive GC to result in the loss of an acknowledged write. The solution is to ensure that GC isn't that aggressive. SlateDB has the same issue with fence records. They currently don't GC them at all or ensure that GC of fence records leaves them in place for a long time.