Noah Watkins

Engineering @RedpandaData

[Noah Watkins @spin_lock_init]

prototyping some new #cplusplus interview questions

[Noah Watkins @spin_lock_init]

@SeanTAllen i'm a c++ luddite

[Yurii Rashkovskii @yrashk]

What if our ISAs made interactions with caches & memory explicit? https://t.co/I2O2JCb5Kd

[Noah Watkins @spin_lock_init]

This recruiter fawning over my distinguished engineer level prompting "hey claude, swap in the abseil implementation, and use RFC 3720 test vectors."

But more seriously, it sucks now. In the past, I would occasionally get a very well researched, personalized message.

[Noah Watkins @spin_lock_init]

@SeanTAllen Scared Straight (back into the loving arms of c++)

[Noah Watkins @spin_lock_init]

was thinking about what part of my body should host a C++ tattoo, and then I watch this 🤣 "if you are writing a program in C/C++, to use a b-tree is to bring firearms to a pre-school"

https://t.co/s3vYOylyli

[Noah Watkins @spin_lock_init]

@jamesladd @SeanTAllen Brian is basically saying that without a guaranteed ownership in C/C++, building these data structures is asking for serious trouble, even if your intentions are good.

[Noah Watkins @spin_lock_init]

@SeanTAllen I think c++ in this case is the good guy with gun? Or maybe it's rust 🤷‍♀️

[Noah Watkins @spin_lock_init]

Since we are talking about parallel agents, how are you thinking about inter-agent signaling/synchronization? If that is out-of-band today, then retries in the harness + close-to-open consistency feels sufficient. So I'm wondering if you are thinking that more inter-agent communication will flow through the file system itself?

[Noah Watkins @spin_lock_init]

Oh this solves exactly our issue with competing Bazel builds that agents were kicking off, and is so much better than the weird contraption I had cobbled together.

[Yurii Rashkovskii @yrashk]

@tavisrudd I had the same problem, built https://t.co/261TymWQlH (few more important usability improvements inbound)

[Noah Watkins @spin_lock_init]

I don't think it's all doom and gloom. But the bar for quality has been raised _significantly_, and it applies to both the producers of software, and users who need to be scrupulous in holding projects to a new standard.

[Qian Li @qianl_cs]

At HPTS, Kyle mentioned a company confidently handed him a vibe-coded database to test. Now Jepsen report on ParticleDB is out. IMO I feel sad for the engineers involved. It's a long report, but at least read the part on LLM tradeoffs.

"Perhaps engineers will begin to treat any new software project with suspicion—the signals we used to rely on as markers of a mature system are now easier to counterfeit."

https://t.co/0NsRyZLgxQ

[Mitchell Hashimoto @mitchellh]

@craigjellick The weird thing about vision is if you tell most people the long term vision up front they call you crazy and laugh at it. It's better to do something short term and obvious that everyone can grasp while carrying water and chopping wood towards the real vision.

[Hillel @hillelogram]

Okay, let's do this.

Isabelle, Lean, and Rocq (formerly Coq*) are theorem provers. They are meant to prove mathematical theorems. We use them for formal verification by either 1) translating a program to the theorem prover (risking a translation error) or 2) using a language built into the theorem prover that extracts into a working binary.

Fizzbee, TLA+, Spin, Alloy, Quint, and P are specification languages. They are meant for checking high-level blueprints of programs. After checking the blueprint, you still need to make sure your program matches it, which is a hard problem.

Specification languages differ in what they make easy to express. P, Spin, and Fizzbee are designed to be closer to the code implementations. P looks like interacting state machines, Spin looks like C with CSP, Fizzbee looks like a dialect of Python. It's easier to use a codelike specification language and refine it to code, but you can't do as much with it.

Alloy, TLA+, and Quint are more abstract. Alloy looks like a bunch of algebraic relationships, TLA+ is a much of logical formula, and Quint is a programmer-oriented (and statically typed) condensation of TLA+. Abstract modeling languages are harder to get started with and harder to represent things, but you can represent a wider class of systems with them.

I believe UPPAAL is a specification language for realtime systems. This is a field I have no experience in. It seems a graphical language, though, which would make LLM-assistance difficult.

Dafny is a verifiable programming language. It has regular programming language features, but also a bunch of features to express and prove properties of the program. It uses SMT solvers to actually prove the properties, meaning you don't need to learn a proof language (though you need to learn to cajole the solver). Dafny is probably the easiest thing on your list to get started with, but also the least likely to meet any weirder needs you might have.

I don't know what Ivy is but some cursory Kagiing suggests it falls in the specification language category.

Some other things @spin_lock_init didn't mention:

- Other theorem provers: Agda, Metamath
- Other abstract specification languages: mcrl2, NuXMV, Event-B
- Other verifiable programming languages: Idris, F*, Whiley
- There's also using a verifiable subset of a language, like Verus for Rust or SPARK for Ada.

So, what you need to pick depends on what you want to do.

You'll get the maximum power with a theorem prover. Lean is the de facto language for vibeproving, but Anthropic recently published a post about vibeproving in Rocq so YMMV.

If you want to make abstract models of software systems, go for a specification language. Which you choose depends on a lot of factors. If your system can be represented as lots of simple communicating state machines over a network, try P. Spin, I believe, is close to simple C/Go with a network and channels. I have never used Fizzbee.

Alloy is good for domain modeling, but struggles with really complicated temporal actions, nested data structures, and largish numbers. Also AIs aren't to good at it (last I checked, in March). Runs fast and makes good visualizations though.

TLA+ and Quint are for modeling complex temporal systems. I'm partial to TLA+ but know a lot of people who say Quint is easier to get into.

Note P, Fizzbee, and Quint all claim to have features for testing a codebase against a spec. I can't vouch for any of them, though. None of the languages I mentioned (except maybe NuXMV?) are particularly good at modeling sequential, deterministic algorithms.

Use Dafny if you need to just write a verified program and don't care which language it's in. I don't think other verifiable programming languages are as user-friendly for the time being. Consider a verified subset if you already know a language and want to just augment it with more correctness.

You may need other, more exotic tools depending on your particular needs. Regardless of the above advice, don't be afraid to use the wrong tool for the job. If you vibe well with a language, and you achieve good LLM-sympathy with it, you'll be able to go farther than using a "right tool" you don't like.

[Noah Watkins @spin_lock_init]

@ethshea Oh for sure, super powerful technique in use all over. In the case of pgrust it's a deliberate decision, not an invisible AI decision that the blog article makes it seem like.

[Noah Watkins @spin_lock_init]

but pgrust passes all 46,066 tests 🤦‍♀️

https://t.co/njJGpRYLeC

[Noah Watkins @spin_lock_init]

@agriffini @dcolascione The lowering of the barrier is what is unique here, agree!

[Noah Watkins @spin_lock_init]

insane stuff and just getting started

[Hunter Leath @jhleath]

AI is making it unsafe to ship even just a binary to your customers. Over the past 6 weeks, we've had multiple users attempt to reverse-engineer the Archil client as well as try to infer different behaviors about the Archil client from the strings present inside of it.

I don't know how this ends, but this is definitely making it challenging to distribute software to customers. In my previous life, I wouldn't have wanted to give customers tools that would be confusing or lead them to the wrong conclusions about how services work (like the strings to a binary).

In this new world, maybe you just have to open-source everything?

[Noah Watkins @spin_lock_init]

@anukool_j They are open about using the technique

[Noah Watkins @spin_lock_init]

@facturi1611781 It's definitely true that systems do this, on purpose, and everyone involved is ok with it. I think the surprise here is that the expectation was for a full-value comparison, even if not for correctness, for the benchmarking question that the referenced blog post raises.

[Noah Watkins @spin_lock_init]

@CattManii I suppose it must depend on what the agent believes it has freedom to do, either implicitly or semi-explicitly through a prompt. The technique in this post is definitely in the training set, and there are plenty of systems that do it, with the obvious caveats.

[Noah Watkins @spin_lock_init]

@grfwings it's a sleeper for sure!