Marc Garcia

pandas core developer

[polars data @DataPolars]

The early design decisions for the Categorical type were under strain because of our streaming engine. Every data chunk carried its own mapping between the categories and their underlying physical values, forcing constant re-encoding. The global StringCache we built to solve it caused lock contention and wasn't designed for a distributed architecture.

The new Categories object, released in 1.31, solves this, and gives you:
• Control over the physical type (UInt8/16/32)
• Named categories with namespaces
• Parallel updates without locks
• Automatic garbage collection

When you know the categories up front you can use Enums. They're faster because of their immutability and allow you to define the sorting order of values.

The StringCache is now a no-op, but the code will keep working how it used to (with global Categories). You can also migrate by replacing it with explicit Categories where needed.

The result is a Categoricals data type that works well on the streaming engine without performance degradation, and is compatible with a distributed architecture.

Read the full deep dive: https://t.co/kkikdqxrER

[Andrew Lamb @andrewlamb1111]

DataFusion 52 is released https://t.co/sgEZLVCCZd

[Marc Garcia @datapythonista]

pandas 3 has been released and marks the most significant evolution of #pandas in over ten years.

No more `copy()` everywhere, and no more `lambda` gymnastics.

Want examples? Read this hands-on article with the main changes: https://t.co/vnzMbCks0l

[Marc Garcia @datapythonista]

#pandas 3 is faster, and more intuitive. I just published an article with practical examples explaining the main improvements: https://t.co/vnzMbCks0l

[Marc Garcia @datapythonista]

I'll be giving a workshop on #DataAnalysis with #Python and #Polars at the next #PyData #ChiangMai. Limited seats, in-person only.

https://t.co/DZ7XPXGf03

[Marc Garcia @datapythonista]

We just reorganized the pandas ecosystem page. Is it clearer? Anything missing or not useful? Feedback welcome

https://t.co/HeKm7hLbmP

[pandas @pandas_dev]

We're happy to announce the release of #pandas 2.3.0. You can install it with `pip install pandas` or `conda install -c conda-forge pandas`. Thanks to all contributors and sponsors who made this release possible! The release notes can be found at: https://t.co/y9w1kBiqtP

[Marc Garcia @datapythonista]

pandas 2.3 has been released, the last version before pandas 3.0.

[unusual_whales @unusual_whales]

Tech occupations have been among the hardest hit by steep declines in hiring since 2018, per BI;

[Interesting things @awkwardgoogle]

Possibly the greatest single male athletic performance of all time

[polars data @DataPolars]

Today we are launching the first open Crash Course training sessions with a limited time discount. These instructor-led sessions are open to everyone looking to get up and running with Polars.

Find a date and sign up via our Academy: https://t.co/2hVIPQDLMO

[El Club del Arte 🎨📷📚🖼🕍🎼 @Arteymas_]

Ella lo único que quiere es cantar:

[Antidepressant Content @depressionlesss]

Duck said give me my pookie back 🥹🥰

[Posts Of Cats @PostsOfCats]

I hope these two never separate 😭❤️

[Nature is Amazing ☘️ @AMAZlNGNATURE]

This made my day! 😀

[cats aura 🐾 @catshealdeprsn]

The reason why i love cats

[B&S @_B___S]

Perfect training partner

[Posts Of Cats @PostsOfCats]

why are cats so silly

[Tzu Gwo @yochowgwo]

I've written a blog about Tonbo's research on async Rust and io_uring：
https://t.co/oRT1DrCaKM

We need to be careful to avoid the cancellation problem when using async Rust and io_uring together.