Matthew Weidner

Collaborative software at Common Curriculum / CMU.

I write about CRDTs etc.

[Matthew Weidner @MatthewWeidner3]

@ibuildthecloud After 5 years of studying CRDTs, I'm fully in favor of "just apply ops in order on the server". The mature sync engines already do this.
You don't need CRDTs for optimistic updates either - just apply pending local ops on top of the synced state, like an FPS game.

[Matthew Weidner @MatthewWeidner3]

@jonathoda Are you referring to OT with tombstones? I'd be interested in up-to-date references on that.
(Character IDs also require tombstones, at least if you want to handle inserts relative to a concurrently-deleted character.)

[Matthew Weidner @MatthewWeidner3]

Fun academic find: the core idea of server reconciliation (a general approach to real-time collaboration that's simpler than Operational Transformation or CRDTs) already appears in an MIT tech report from 1985.

[Matthew Weidner @MatthewWeidner3]

We know now that the server reconciliation approach does work for text, so long as you assign each character an ID instead of trying to use array indexes: https://t.co/lj3heGDQah

[Matthew Weidner @MatthewWeidner3]

I learned of this from the first paper on collaborative text editing (https://t.co/DWaRuUOm6T), which cites the idea but claims it won't work for text. Instead they introduce Operational Transformation, which led to a lot of academic work but is more complicated.

[Matthew Weidner @MatthewWeidner3]

@msimoni Or each function, though I don't know if this can be made robust against arbitrary copy-paste.
Then you can start surfacing conflicts like "commit A meaningfully changed the docs for function foo, while concurrent commit B added a new call to foo".

[Matthew Weidner @MatthewWeidner3]

@JAldrichPL Using AI well seems fine without disclosure, while using it "wrong" (cliched text, fake figures) is bad strategy as it reduces your odds of acceptance. Maybe we just need a rate limit to prevent spam - e.g. at most 2 submissions per 1st author per year, resetting upon acceptance.

[Matthew Weidner @MatthewWeidner3]

@k081e @joodalooped @devagrawal09 @tigawanna @tannerlinsley CRDTs are good at helping you realize that ops don't actually conflict... e.g. editing text in two different paragraphs.
They're less helpful when users actually step on each other. A central total order also doesn't solve this 100%, but at least you can preserve invariants.

[Matthew Weidner @MatthewWeidner3]

@schickling Agreed, git is already a decent "local-first sync layer" with commodity sync servers.
How do you think about non-plaintext docs over git? I've tried out "event sourcing w/ one event per line", though it surprisingly required a custom merge driver: https://t.co/v5Vhlh0UNX

[Matthew Weidner @MatthewWeidner3]

@jamesacowling 2 users edit different paragraphs offline ->no semantic conflict; OT/CRDTs behave reasonably.
1 user edits a paragraph while another deletes it ->semantic conflict; OT/CRDTs give odd results.
This doesn't contradict your point 2, but folks may see more magic here than warranted.

[Matthew Weidner @MatthewWeidner3]

@jamesacowling OT&CRDTs' most difficult (& 1st) algorithms solve a narrow problem: in a text doc, if my "insert A @ index 6" op reaches a node before your "insert B @ index 10" op, the node needs to instead insert B @ index 11. This isn't a real semantic conflict, just a flaw w/ using indexes.

[Matthew Weidner @MatthewWeidner3]

More on this simple path to collaborative apps, as an alternative to the Operational Transform / CRDT rabbit hole, in my recent DWeb Seminar talk: https://t.co/MAPQbB8T9C

[Matthew Weidner @MatthewWeidner3]

From the ~1st collab text editing paper: if two users do "delete char @ index 7" concurrently, no serial ordering gives the correct answer. So they abandon serializability, leading to OT/CRDTs. But if you rephrase the ops as "delete char @ <per-char id>", both serial orders work!

[Matthew Weidner @MatthewWeidner3]

For many collaborative apps, though, you can get away with just serializing all users' ops. E.g., apply ops in the order a server receives them. This strategy appears in @replicache and in @tan_stack DB (taking over from ElectricSQL v0's fancier CRDT-based strategy).

[Matthew Weidner @MatthewWeidner3]

@jonathoda I got stuck on this for a while (how to make a container that fills all of the remaining space *and* will have its own scrollbar if that's not big enough) before learning the `min-height: 0;` trick. I think Claude taught me it.

[Matthew Weidner @MatthewWeidner3]

I made a guest post on @powersync_'s blog: https://t.co/F8S18K3c1P
It describes a demo we made of collaborative text editing on top of PowerSync's DB sync engine, using character IDs & totally-ordered updates in a DB table (w/o CRDTs).

[Matthew Weidner @MatthewWeidner3]

@schickling Gemini lets you constrain the output to a JSON schema: https://t.co/uMooqDaR5H
Surprisingly, this even works with streaming. Much nicer than my previous manual hacks to strip away ```'s and chatter before the JSON output.