wangbin579

Expert in tackling complex software challenges, with deep knowledge of TCP, MySQL and PostgreSQL kernels, and a passion for AI, history, math, and physics.

[wangbin579 @wangbin579]

After nine months of hard work, this Postgres course is almost complete. You don't have to buy it, but if you want to truly understand Postgres internals, this is one of the most detailed courses you'll find. https://t.co/ZpRiQ6oNEr

[wangbin579 @wangbin579]

@voidwhale_dev But I've seen a Rust project like Neon that has poor architecture and is really hard to improve.

[wangbin579 @wangbin579]

Once you understand how Rust avoids C's pitfalls, you'll see that Rust itself is actually quite elegant. However, the Rust code most people write is so ugly that it scares many people away.

[wangbin579 @wangbin579]

@vkbdev Sorry, I haven't seen a successful project where most of the rust is written by LLMs.

[wangbin579 @wangbin579]

Neon is another project where I can already see a lot of room for performance improvements.

[wangbin579 @wangbin579]

You can't get a 100,000x speedup just by improving cache locality. Such big performance gains come from many things working together, but the most important thing is choosing the right data structures for large datasets.

[wangbin579 @wangbin579]

You can't get a 100,000x speedup just by improving cache locality. Such big performance gains come from many things working together, but the most important thing is choosing the right data structures for large datasets.

[wangbin579 @wangbin579]

Hard to say, honestly. It really depends. The biggest speedup I've ever seen was changing the data structure from a hash table to a well-designed array-based one. Overall, not just from cache locality, that made part of the code about 100,000 times faster.

[wangbin579 @wangbin579]

Hard to say, honestly. It really depends. The biggest speedup I've ever seen was changing the data structure from a hash table to a well-designed array-based one. Overall, not just from cache locality, that made part of the code about 100,000 times faster.

[Pablo Manjarres @pablomanjarress]

@wangbin579 When an array wins, how much the gain usually comes from cache locality?

[wangbin579 @wangbin579]

I have spent the past ten days checking my new home, trying to find as many problems as possible, and asking the workers to fix them. btw, AI is still not very helpful in solving these problems.

[Khaled @KhaledMejri]

@wangbin579 What did you do here?

[wangbin579 @wangbin579]

Many real-world bugs are fixed in a similar, simple way, including in databases.

[wangbin579 @wangbin579]

@renatocron It's just because neon's lsm operations are a mess, even with small data.

[wangbin579 @wangbin579]

I really like Postgres now. Once you understand it, almost anyone can build a database. It's not just for experts anymore.

[wangbin579 @wangbin579]

Based on Neon's architecture, I abandoned the LSM tree and designed a new data structure tailored to the data's characteristics. It looks simple and efficient so far, though it hasn't been implemented yet. Anyone interested?

[wangbin579 @wangbin579]

@hecodesforme Sorry, I made a mistake. I meant hotspot updates, where the same row gets updated repeatedly.

[wangbin579 @wangbin579]

I've traced through lsm myself, and I don't think it's the best data structure for neon. I choose more suitable data structures based on many factors, which I will share on discord (https://t.co/f8FQzBWmw1). Btw, good database design isn't something AI can do today.

[Shashank Mishra @mishrashashan14]

@wangbin579 Can you share more insights?
Whats the data shape and why LSM tree isn't useful here?
Sounds interesting!!

[wangbin579 @wangbin579]

@kethan228921 Right now we're just redesigning it. These days, good design matters most. In the future, we'll definitely open source it.

[wangbin579 @wangbin579]

Based on Neon's architecture, I abandoned the LSM tree and designed a new data structure tailored to the data's characteristics. It looks simple and efficient so far, though it hasn't been implemented yet. Anyone interested?

[wangbin579 @wangbin579]

@hecodesforme Postgres is hard to master, and it's not a good fit for high concurrency, heavy write loads, or lots of hot updates.

[wangbin579 @wangbin579]

@faber_ihsan Here are some real examples: https://t.co/QlbBTECv4b

[wangbin579 @wangbin579]

In performance optimization, the data structure I least want to use is a hash (only as a last resort), and the one I most want to use is an array. When a database is full of hashes, there's probably a lot of optimization opportunity hiding in there.

[wangbin579 @wangbin579]

Using arrays well is not easy. It takes skill, and you need to really understand your data and how it's structured. That's why programming is truly an art. In the future, poorly made databases will be everywhere.

[wangbin579 @wangbin579]

In performance optimization, the data structure I least want to use is a hash (only as a last resort), and the one I most want to use is an array. When a database is full of hashes, there's probably a lot of optimization opportunity hiding in there.

[wangbin579 @wangbin579]

@kamikaz1_k You can't stop Rust users from abusing every advanced feature, but those who truly know how to use Rust well can also use C well.

[wangbin579 @wangbin579]

Why C? Four reasons. 1) Postgres is written in C too, and I want to reuse its WAL replay code. 2) I don't want to change my taste. I just don't like Rust, which misses a 't' 3) I don't want slow builds or big binaries. 4) I'm not smart enough for Rust.

[wangbin579 @wangbin579]

Our product will be written in pure C (sorry, I don't trust Rust). Even if nobody ever uses it, it'll still show you what the art of optimization really looks like.

[wangbin579 @wangbin579]

@dadua_daku @Umar_init_ https://t.co/f8FQzBWmw1

[wangbin579 @wangbin579]

I think I've got the evidence now: in tokio's async model, Rust is far weaker than C in some critical situations. Full breakdown in the Discord group next week. Don't miss it.

[wangbin579 @wangbin579]

Why C? Four reasons. 1) Postgres is written in C too, and I want to reuse its WAL replay code. 2) I don't want to change my taste. I just don't like Rust, which misses a 't' 3) I don't want slow builds or big binaries. 4) I'm not smart enough for Rust.