Robin Linacre

Lead developer of Splink. Data scientist at Ministry of Justice. Trustee, GiveDirectly UK. Pledgee, All views my own.

[Robin Linacre @RobinLinacre]

Pleased to announce the release of Splink version 4.0.0 today. It's now:

👩‍🔬 Easier to use
🚀 Faster
⚡ More scalable
🛠️ Easier to improve

For the uninitiated, Splink is a free library for record linkage and deduplication at scale

See 🧵 for links and more
1/3

[DuckDB @duckdb]

🦆 Announcing DuckDB 2.0.0 LTS “Cinnamon Teal”

We are proud to release DuckDB 2.0.0, named after the cinnamon teal (Spatula cyanoptera), a striking reddish-brown duck found in the western Americas. 🌎

DuckDB 2.0.0 is the biggest DuckDB feature release so far. It ships a polished version of the Quack protocol, which turns DuckDB into a full-fledged server. It improves performance and lowers costs through supporting asynchronous I/O, a first-class VARIANT type, and a dedicated “agent mode”. It also empowers extension developers with a new SQL parser that allows adding new statements, clauses, and keywords, a new API to build on, and a new mechanism to distribute extensions more easily.

In our announcement blog post, we summarize the highlights and then walk through the key changes and give guidance on how to migrate to 2.0. For the complete release notes, see the release page on GitHub⁠.

https://t.co/QtT5wHhg2w

#DuckDB

[Robin Linacre @RobinLinacre]

@eric_simon11 But yeah - we do not attempt to solve this problem in Splink due to there being no obvious perfect solution (though we may have missed something)

[Robin Linacre @RobinLinacre]

We're very pleased to release Splink 5, which is now more scalable, faster to train models and lighter to install

[Robin Linacre @RobinLinacre]

@eric_simon11 Problem is fundamentally if existing records can be updated, stable cluster IDs is impossible because existing clusters can split.

Splink's IDs are pretty stable, just not guaranteed. If you only add records, and don't modify existing ones, can take the ID of the oldest record

[Robin Linacre @RobinLinacre]

You can find more details about the release here:
https://t.co/OP4Kv0cMhb

[Robin Linacre @RobinLinacre]

In case you missed it. This has since been featured in the Data Elixir, Data Science Weekly and Quantum of Sollazzo newsletters

[Robin Linacre @RobinLinacre]

New blog: Data Science in the Age of AI https://t.co/yJkoqmDGYj

LLMs are the biggest shakeup to the profession I've ever experienced. The post uses examples to look at how the job has changed and how to use agents effectively.

[Robin Linacre @RobinLinacre]

@neejkapoor @adamboxer1 Interesting this is not even novelty: he's continues to enjoy these games for many months. But we're careful with screen time, so no doubt this is partly because we don't allow him to play unlimited 'real' computer games

[Robin Linacre @RobinLinacre]

@neejkapoor @adamboxer1 He will reluctantly do his (excessively easy) mathletics homework, but does not enjoy it. He will play fun but harder mental arithmetic games and often do about 100 questions before getting tired. His two other faves are https://t.co/HTnrByfH8A
https://t.co/K0TM5KPAFe

[Robin Linacre @RobinLinacre]

@adamboxer1 Thanks - that was insightful. I have tried it with vocab building and it falls into the 'surface details' trap described. IMO it works better for things like times tables i.e. automatic retrieval of facts. But this is very helpful in thinking about what may actually help my son

[Robin Linacre @RobinLinacre]

@adamboxer1 In my experience (sample size 1 kid): this is a more valuable use of AI than anything else I've seen (videos, interactive explainers, worksheet generators, etc.)

[Robin Linacre @RobinLinacre]

@adamboxer1 Hey Adam. Question from random dad: What value do you see (if any) in making retrieval practice fun enough that kids want to do it? Example: https://t.co/byDgeT7HbN

[Robin Linacre @RobinLinacre]

@kennethfield For those who are confused, here's an explainer:
https://t.co/ofUpphDWvz

[Robin Linacre @RobinLinacre]

https://t.co/byDgeT79mf

[Robin Linacre @RobinLinacre]

Now has 2 player mode

[Robin Linacre @RobinLinacre]

https://t.co/qYF09IP8mL

[Robin Linacre @RobinLinacre]

Brain bombs, a two player mental arithmetic game

[Robin Linacre @RobinLinacre]

https://t.co/J8fKs9xxUb

[Robin Linacre @RobinLinacre]

Keep the train going for as long as possible: a game to help motivate kids to learn their times tables.

[Robin Linacre @RobinLinacre]

Just checked old messages. Splink v1 (which was very fast and scalable at the time) had following stats:
16m input, creating 100m comparisons: took 57 minutes at a cost of £3.50 in PySpark on AWS Glue. So 6 years work = >600x speedup, and it's now 350x cheaper!

[Robin Linacre @RobinLinacre]

A sneak peak at Splink 5 beta performance: just ran a 1bn input row/10bn comparison dedupe on synthetic data, on 192vCPU EC2 instance, using @duckdb 2.0 prerelease. Runtime: 8.5 mins, cost <$1.00 on a spot instance.

This is at least 10x faster than previous versions

[Robin Linacre @RobinLinacre]

Splink 5 pre-releases available on PyPI, latest code on GitHub https://t.co/pLXoaMmFsV