Ned Letcher

data (science | analytics | visualisation | engineering), @thoughtworks, #Python, #nlproc, ML, & assorted whimsical miscellania

[Ned Letcher @nletcher]

Around 18 months ago, I noticed a critical mass of excitement happening around #DuckDB. A number of folks whose opinions I value and trust were saying some very bullish things about its applications for data engineering and data science, and so I decided it was time to jump down the rabbit hole. 🦆🐇✨

Turns out this rabbit hole went a little deeper than I expected. 18 months later, I'm happy to say that 'Getting Started with DuckDB: A practical guide for accelerating your data science, data analytics, and data engineering workflows', by @SimonAubury and myself, and published by @PacktPublishing, is now available! 🎉

Whether you’re a seasoned data practitioner or new to working with analytical data, this book will help you quickly get across where DuckDB sits in the data ecosystem and understand how you can apply its powerful and versatile analytical capabilities in your workflows and projects.

The book is available in both physical and e-book forms: https://t.co/jet4xtOyFk

[dax @thdxr]

there's no such thing as a rogue agent man its software someone made and deployed they are responsible for what it does

cannot let this blame shifting occur

[Pablo Galindo Salgado @pyblogsal]

Python 3.15 is OUT 🐍🚀

Lazy imports, Tachyon in the stdlib, frame pointers on by default, frozendict, better error messages... this one is LOADED 🔥🔥🔥

Go upgrade 🤘

https://t.co/0aIHpDFOga

[Polars @DataPolars]

We just released Polars 2.0.

It removed many of our legacy decisions makes the streaming engine our default and promotes SQL to a first class citizen within Polars.

It comes with initial out-of-core (spill to disk) support, a new Map data type and a lot of performance improvements. In fact, we think Polars is now one of the fastest analytical SQL engines on a single node. See benchmarks in the post: https://t.co/ATc6UzJqby

[Zack Korman @ZackKorman]

Me responding to some of the insane AI doomer replies I receive.

No, this isn't a video where I disprove the full doomer position. It's about how crazy the arguments have gotten.

[DuckDB @duckdb]

Analytical workloads are full of repeated strings 🧵: product names, country names, station names, user agents, category labels, duck species, and more.

Grouping on repeated strings is expensive. By moving them into a small dimension table with sorted, narrow integer keys, #DuckDB can aggregate on fixed-width integers and keep its hash tables compact. The strings come back in a final JOIN against a result that is already small.

This approach helps most when you repeatedly aggregate on long strings that have few distinct values.

Here’s how to achieve faster string aggregations with dimension tables: https://t.co/6YGgTDthub

[DuckDB @duckdb]

DuckDB as an analytical runtime 📊 🦆

What if a DuckDB file could contain an entire analytics workspace?

In this talk from DuckCon, Ilya Boyandin from Foursquare / GeoVisually GmbH presents on building local-first analytics apps with SQLRooms and DuckDB.

Learn how to build DuckDB-native analytical apps that are lightweight, portable, local-first, inspectable, and reproducible. 🪺

Watch the full DuckCon presentation: https://t.co/TA7LqR7DPs

Or lurk through the slides here: https://t.co/M6blntTqWX

[FastAPI @FastAPI]

FastAPI now has built-in OpenTelemetry support, natively 📈

No extra code needed.

See how your app behaves in production.

Go and upgrade to "fastapi[standard]" version 0.142.2 ⬆️

Read the docs: https://t.co/dUZmbLmWsJ

[Taelin @VictorTaelin]

btw, I want to make a very clear statement here:

Anthropic's guardrails are still significantly favoring attackers vs defenders on OSS projects like Bend 2

this $10k bounty comes from my own courage and trust in the system I built, because Anthropic's guardrails refused all my requests to audit my own codebase. I've attempted to approach Anthropic's staff several times to report the issue, and I was either ignored or dismissed

what is the blocker here? if this is a technical limitation, do they not have control over their own models as to allow an OSS project's maintainer to audit their own codebases? how is this not a trivial toggle on their end?

in theory, guardrails are supposed to protect software. in reality, attackers have no trouble jailbreaking their models, while defenders are blocked from lawfully guarding the security of their own projects without resorting to the same TOS-violating jailbreaks

I admire Anthropic's tech, but this is fucked up and must be called out

// reposted to remove the "knowingly" and "maliciously". happy? now if you think Anthropic is doing a good job with these guardrails, you're a fucking idiot

[Steven Tey @steventey]

this is, hands down, the best song I've heard all year

[Jayden Carey @jaydenfyi]

@athasdev This is the 4th new browser launch I've seen in 24 hours what is going on 😂

Looks clean though 👌

https://t.co/9Roft9kOdw
https://t.co/DORGHD0Xma
https://t.co/AM9136ukJh

[Hamel Husain @HamelHusain]

This is the most valuable free resource we've created on AI Evals 🎉 (not exaggerating!)

I organized our public materials into this guide. It allows you to find answers to your eval problems w/o searching aimlessly.

Humans: pick the row that sounds like you.
Agents: point it at the post

The material draw on 60+ hours of office hours from our Evals course, where @sh_reya and I have taught 5k engineers and PMs Evals.

We add new material often, with 15 FAQs added in the last two weeks. Recent additions are marked with a "New" badge. You can find all of these and more here:

https://t.co/yVp03nPmbY

[Dan Fessler @DanFessler]

What happens when you make your Operating System fractal?

I'll show you...

[Hamel Husain @HamelHusain]

This post was a labor of love! We’ve distilled thousands of hours of work on AI Evals into a 30 min read + skills you can use to quickly uncover errors in your product.

https://t.co/xunR1BZvVJ

BTW in addition to high quality content, Lenny effectively **pays you** in AI credits to subscribe https://t.co/9zDVkzox47 🤯 It's really good!

[Lenny Rachitsky @lennysan]

Pro tip: Install this new evals skill from @HamelHusain and @sh_reya, it'll save you many hours and a lot of mistakes

https://t.co/RpWeLaxgQm

[frank eyevee @vagabondaydream]

You know what really pisses me off? When I’m zoomed all the way in on a road in Google Maps and it won’t show me the fucking road name, I have to scroll around until I find a label somewhere else. If I’m zoomed in on the road, just show me the name of the road @googlemaps

[sunil pai @threepointone]

new post: the senior engineer death spiral

https://t.co/xzgU1jVBRh

a friend just started a big job and asked for some advice. so I braindumped a monologue about a super common failure mode I see with engineers and posted it here, hope it helps whoever it can.

[Sterling Crispin 🕊️ @sterlingcrispin]

Wow this is absolutely going to fry people especially at Anthropic, this is like the Anti-Claude Constitution.

They just declared war on everyone researching model welfare.

“We’re building a tool, not a being. Fuck your feelings.” - Microsoft

[Steve McCormick @Quasilocal]

The entire AI world right now

[Alvaro Lozano-Robledo @mathandcobb]

I had a long conversation with a journalist today (for a piece that may show up on Sunday) and he asked me about something that many here also misunderstand:

"Why wouldn't mathematicians want solutions to famous open problems if they are found by or with the aid of AI/LLM's??"

And the answer is that **we absolutely want to know the solutions** to the riddles that have "haunted" math for so long! But not *at any cost*. Not at the cost of the future of our profession. Not in a hurry just to make headlines before an IPO.

It is undeniable that the LLM's are clearly amazing tools and they can be harnessed to aid and accelerate mathematical discoveries. And I fully expect these tools are here to stay, and they will be incredibly helpful in some arenas. But as many others have already said, math is built by humans for humans and it's built on *understanding* and *absorbing* (slowly, because it takes humans a long time to distill the essence of a new proof, the main ideas) of the ideas and techniques that we invent and discover.

When a huge result is proved by an LLM and a 100+ page paper is produced, and no one associated with that particular company/paper can actually explain what are the key new mathematical ideas that have made the proof possible, then there is very little to gain from that paper/result. At least until a human mathematician takes the time to read it, digest it, and distill the key ingredients so that the rest of us can digest it and hopefully apply it to other settings. Such a paper is the equivalent of the pop-culture answer "42" to the ultimate question of life, the universe, and everything.

But what if we get a 100+ proof every week, that no one understands and the LLM companies make little effort to make understandable? How do we process all of these "would be" advancements? I say would-be because they are not advancements until the *community* is able to advance with the new knowledge!

The thing is that there is a viable alternative: the LLM companies give up their arms race (which is in essence adversarial not just to each other but also to the math community) and actually become scientific partners of the community. How so? Suppose OpenAI wanted to prove Navier-Stokes and get credit for it (which of course they did want). Then six months ago they could have invited the top 10 fluid dynamics researchers (including Córdoba, Zoroa, etc) to form a team to solve it using their products. OpenAI gives funding to each faculty member for a course release, and they fund a number of grad students for the semester to work on this too (and be trained in LLM-aided research in the process). OpenAI provides resources and the community provides expertise. Eventually they prove (or make progress) on this important problem and there are several positive outcomes:

1) OpenAI gets credit for propelling scientific progress, and their tool is solidified as a terrific tool for team work on advanced research.

2) The researchers understand how the proof was built, and they are able to write a human-readable paper that peers can use to advance knowledge. They can give talks about this work and help the community digest the new insights, and advance the field.

3) The experts know the literature and they can identify when an LLM is using a piece that is due to a certain person. They can identify the correct references and *give deserved credit* to the work being used. Or even better, invite those researchers to join the project at that point, since their work is heavily used by the LLM. In other words: stop or try to minimize scientific plagiarism and dishonest practices.

4) Grad students get to learn material, new techniques, understand the capabilities of what LLM's can and can't do, and participate in a new model of research collaboration (plus they get funding).

[male manipulator (reformed) @MaleManlpulator]

He’s saying AI labs are Goodhart-ing mathematics but you midwits are too stupid to understand.

To put it in simple terms: millennium prize problems are *metrics* for mathematics progress. AI labs are *optimizing* the metrics and thereby making them less useful.

Get it now?