Daniël Heres

@ApacheArrow PMC and Open Sourceror |
Senior Engineer @ Coralogix (Query Engine)

[Daniël Heres @daniel_heres]

Oh no! Hope you get better soon.

[Daniël Heres @daniel_heres]

@charliermarsh That's very nice!

[Andrew Lamb @andrewlamb1111]

Recording and slides available from the DataFusion Stockholm Meetup:
Recording: https://t.co/FOaAr05PbU

Slides: https://t.co/il7Fgnczvq

[Chris Olah @ch402]

Here I stand, I can do no other.

[Daniël Heres @daniel_heres]

With larger than 2x improvement when Parquet filter pushdown is enabled

[Daniël Heres @daniel_heres]

Adding Morsel-driven parallelism to DataFusion / Parquet seems quite promising for solving data skew https://t.co/rohdopreXG

[Andrew Lamb @andrewlamb1111]

This arrow-rs kernel is so cool: https://t.co/QSwUip4jLD

[Daniël Heres @daniel_heres]

@andrewlamb1111 @RitchieVink Yes, perhaps combined with filter + repartition/coalesce optimizations

[Daniël Heres @daniel_heres]

@RitchieVink DataFusion is now ~1.5x Polars on c8g.4xlarge, with filter pushdown providing another ~20% win (not enabled). @andrewlamb1111

[Daniël Heres @daniel_heres]

@RitchieVink Very nice, what are the main efforts / improvement to get to the top I wonder?

[Daniël Heres @daniel_heres]

@mim_djo Hmm...
I am interested what you're doing here, are you using the TableProvider interface or something else?
https://t.co/x08CLTkxRz

[Andrew Lamb @andrewlamb1111]

Example of the kind of low level obsession we foster in the Rust parquet / arrow / DataFusion community:
https://t.co/Bjz0hCbWG8

I am skeptical that proprietary engines will be able to compete with OSS long term (though I am biased)

Huge thanks to Qi Zhu and @daniel_heres

[Shiyan Xu @_xushiyan]

I had the privilege of presenting Hudi-rs at VeloxCon 2025 - the native Rust implementation of @apachehudi that's opening new doors for cross-language data processing.

🔨 By rebuilding Hudi's core in Rust, we've created a foundation that supports multiple language bindings while maintaining consistent semantics.

🦀 We have integrated with Rust and Python frameworks like @ApacheDataFusio , @daft_dataframe , @raydistributed . Now we are adding APIs to work with C++ based frameworks like @velox_lib .

These integrations mean that regardless of your framework of choice, you can leverage Hudi's powerful transactional capabilities and table formats with native performance.

📺 Check out the full talk here: https://t.co/LKmBgVrrol

Interested in contributing? Check out the project at https://t.co/XxuXx78ThZ

#apachehudi #rust #python #cpp #database #dataengineering #dataecosystem

[Andrew Lamb @andrewlamb1111]

DataFusion 45 Release Blog: https://t.co/XmRNGna2pE

[Volodymyr Zelenskyy / Володимир Зеленський @ZelenskyyUa]

Thank you for your support! 🇺🇦
__

Дякую вам за підтримку! 🇺🇦

[Max Welling @wellingmax]

Why am I not seeing more outcry on the brutal assault on DEI in the US? The queer community is the first target of authoritarian regimes (or worse). This is the time to show your support even if you feel pressured not to (in which case you have already lost freedom of speech). 🏳️‍🌈

[Andrew Lamb @andrewlamb1111]

We keep improving the Rust parquet readers performance. This time with SIMD accelerated UTF8 valdation: https://t.co/tPjprcuyHj (thanks to @daniel_heres and many others for helping test!)

[Andrew Lamb @andrewlamb1111]

I am pretty excited about the growth in projects built with @ApacheArrow DataFusion. Here is the official announcement of Comet, created by Engineers from Apple to accelerate Spark. https://t.co/hRpxgC8PJo

[Daniël Heres @daniel_heres]

There is a cool opportunity in my remote team at @Coralogix for a Software Engineer to contribute to an ambitious distributed query engine based on DataFusion / Ballista. The job is open for US candidates as well. Feel free to send me a DM for questions.

https://t.co/XdeczxgU73

[Amnesty NL @amnestynl]

.