#Pyarrow - Twitter Hashtag

11 months ago

🏅 CSV Benchmark Results: 🥇 #DuckDB – blazing fast, sets the gold standard 🥈 #Daft – surprisingly solid, great performance 🥉 #CHDB – fast, as long as the data fit in memory ⚙️ #Polars, #PyArrow, #Pandas , #DataFusion disqualified 😢😢 #onelake #Python #delta_rs

mim_djo's tweet photo. 🏅 CSV Benchmark Results:

🥇 #DuckDB – blazing fast, sets the gold standard
🥈 #Daft – surprisingly solid, great performance
🥉 #CHDB – fast, as long as the data fit in memory
⚙️ #Polars, #PyArrow, #Pandas , #DataFusion disqualified 😢😢
#onelake #Python #delta_rs https://t.co/5vcul0T1yY

7

72

10

21

6K

Mim @mim_djo

about 1 year ago

finally after all these years #pyarrow added support for Azure storage #onelake should works too https://t.co/NJMfbO84qo

0

9

1

704

Khuyen Tran

@KhuyenTran16

over 1 year ago

Combining #DuckDB and #PyArrow enhances efficient large dataset processing on a single machine by optimizing query execution and handling in-memory data. DuckDB on a 6 million row PyArrow dataset runs approximately 2906 times faster than on a #pandas DataFrame of the same size.

KhuyenTran16's tweet photo. Combining #DuckDB and #PyArrow enhances efficient large dataset processing on a single machine by optimizing query execution and handling in-memory data.

DuckDB on a 6 million row PyArrow dataset runs approximately 2906 times faster than on a #pandas DataFrame of the same size. https://t.co/eEIuoXEOaE

1

68

6

29

2K

Mim @mim_djo

over 1 year ago

turn out, having your own Deltalake writer is a good thing, @daft_dataframe is the fastest in processing 2300 files and producing a delta table in #onelake #duckdb , #polars #pyarrow , personally I will still use duckdb as it is the only engine that support adding filename

mim_djo's tweet photo. turn out, having your own Deltalake writer is a good thing,
@daft_dataframe is the fastest in processing 2300 files and producing a delta table in #onelake
#duckdb , #polars #pyarrow , personally I will still use duckdb as it is the only engine that support adding filename https://t.co/YcAgLGDW9W

1

14

0

3

2K

Mim @mim_djo

about 2 years ago

Light ETL Challenge in #MicrosofFabric #Spark #pyarrow #pandas #duckdb #polars and #ibis https://t.co/b5W9HuKPH7 Pyarrow API is a pain to write

mim_djo's tweet photo. Light ETL Challenge in #MicrosofFabric
#Spark #pyarrow #pandas #duckdb #polars and #ibis
https://t.co/b5W9HuKPH7
Pyarrow API is a pain to write https://t.co/eR1cNb0KGY

3

14

1

4

3K

Quix @Quix_io

about 2 years ago

🎉 With #QuixStreams, you can easily serialize and deserialize #PyArrow messages in real time. Say hello to seamless integration with QuixStreams. Get ready to revolutionize the way you work with data! @PyArrow #DataProcessing #RealTimeAnalytics 🚀📊 https://t.co/sin21zAays

Quix_io's tweet photo. 🎉 With #QuixStreams, you can easily serialize and deserialize #PyArrow messages in real time. Say hello to seamless integration with QuixStreams. Get ready to revolutionize the way you work with data! @PyArrow #DataProcessing #RealTimeAnalytics 🚀📊
https://t.co/sin21zAays https://t.co/Dwr1GEJaw7

0

3

0

65

Apache Hamilton (incubating) @hamilton_os

about 2 years ago

In 1.59.0: * There’s a new decorator `@resolve_from_config`. Thanks to Jan Hurst for the addition🙏! * new @dltHub data saver & loader - see https://t.co/vzvbIxy2oj for use * @pandas_dev table loader by Swapnil D. * we now have a #pyarrow result! 2/4

hamilton_os's tweet photo. In 1.59.0:
* There’s a new decorator `@resolve_from_config`. Thanks to Jan Hurst for the addition🙏!
* new @dltHub data saver & loader - see https://t.co/vzvbIxy2oj for use
* @pandas_dev table loader by Swapnil D.
* we now have a #pyarrow result!
2/4 https://t.co/7BDv0SruJV

1

2

0

32

Apache Hamilton (incubating) @hamilton_os

about 2 years ago

News for the week: TL;DR: - Released 1.59.0 with more user contributions! We have a decorator, #pyarrow feature, and more data savers with @pandas_dev & @dltHub - @ProjectJupyter tutorial with Hamilton and blog with data from @PeopleDataLabs #opensource #python 1/4

1

3

1

0

130

Delta Lake

@DeltaLakeOSS

about 2 years ago

You can easily query a Delta table with DuckDB. 🦆🦀 #DeltaLake tables can be converted to PyArrow datasets and #duckdb can read PyArrow datasets. Query engines should support reading #PyArrow datasets so users have some out-of-the-box interoperability that's performant.

DeltaLakeOSS's tweet photo. You can easily query a Delta table with DuckDB. 🦆🦀

#DeltaLake tables can be converted to PyArrow datasets and #duckdb can read PyArrow datasets.

Query engines should support reading #PyArrow datasets so users have some out-of-the-box interoperability that's performant. https://t.co/wE1J2NJzNY

2

34

2

16

3K

Marc Garcia @datapythonista

over 2 years ago

Thanks to the feedback, I updated the article to include @duckdb 0.10 and the data.table implementation for R. DuckDB 0.10 performs 2.5 times faster than 0.9.2, and similar to #pandas with #PyArrow. R's fread (data.table) is a bit faster than them.

Marc Garcia @datapythonista

over 2 years ago

How fast can a CSV file be processed? I explain in detail comparing many options such as @pandas_dev, @duckdb, @DataPolars, #Python, #R, #rustlang and more in this new blog post: https://t.co/j4a4Mv099A

9

173

31

92

25K

1

26

3

5

3K

Khuyen Tran

@KhuyenTran16

over 2 years ago

When dealing with #Parquet files in #pandas, it is common to first load the data into a pandas DataFrame and then apply filters. To improve query execution speed, push down the filers to the #PyArrow engine to leverage PyArrow's processing optimizations.

KhuyenTran16's tweet photo. When dealing with #Parquet files in #pandas, it is common to first load the data into a pandas DataFrame and then apply filters.

To improve query execution speed, push down the filers to the #PyArrow engine to leverage PyArrow's processing optimizations. https://t.co/ua8no3wwPc

2

115

21

38

14K

北极小企鹅

@penguin9527

over 2 years ago

数据架构再次升级引入 #DeltaLake 存储， #DeltaLake + #pyarrow + #duckdb 查询分析重点是使用https://t.co/tHEDW31Kjb 好处如下 1. 替换之前手动维护 S3 路径表代码 2. 支持 ASOF join，砍掉一大波查询后代码过滤的逻辑 3. compact + vacuum 优化表存储 4. z_order 查询加速 🤡🤡🤡

2

40

3

19

10K

Voltron Data @VoltronData

almost 3 years ago

Dataframe Interoperability in action! We explain the dataframe interchange protocol and how it’s pushing composability forward in the #Python ecosystem. We demo it using @ApacheArrow to visualize a #PyArrow table with @vega_vis Altair without conversion. https://t.co/6SbbCq15jO

0

26

10

3

3K

Voltron Data @VoltronData

almost 3 years ago

“This is a game-changer.” In this blog, Tyler White, Solutions Architect at @SnowflakeDB, shows how the new Snowflake Connector for #Python eliminates @ApacheArrow #PyArrow version conflicts. Check it out! https://t.co/clH68w73do

0

9

5

2

2K

Voltron Data @VoltronData

almost 3 years ago

Need to create dashboards or reports? Use #opensource tools that support language interoperability. Today we show how to pass data from #Python to #rstats using a combo of #Parquet #PyArrow & @IbisData to write a report with the Census PUMS dataset: https://t.co/MVc4VOesQv

0

3

5

2

1K

Voltron Data @VoltronData

almost 3 years ago

Preview release of the nanoarrow-based @SnowflakeDB Connector for #Python is available! This connector is ~10x smaller in size and removes a hard dependency on a specific version of #PyArrow. Learn how the nanoarrow integration makes it possible. https://t.co/kWkOZSj0vS

1

28

9

8

4K

Khuyen Tran

@KhuyenTran16

almost 3 years ago

Combining DuckDB and PyArrow allows you to efficiently process datasets larger than memory on a single machine. In the following code, running DuckDB on a #PyArrow dataset is approximately 2906 times faster than running #DuckDB on a #pandas DataFrame.

KhuyenTran16's tweet photo. Combining DuckDB and PyArrow allows you to efficiently process datasets larger than memory on a single machine.

In the following code, running DuckDB on a #PyArrow dataset is approximately 2906 times faster than running #DuckDB on a #pandas DataFrame. https://t.co/IEPlJOZhPl

5

266

47

134

34K

Voltron Data @VoltronData

almost 3 years ago

Language interoperability is a key to productivity. Today, @fmic_ shows how to create a report using a @uscensusbureau PUMS dataset stored in @SnowflakeDB by passing data from #Python to #rstats using #Parquet, #PyArrow, and @IbisData https://t.co/MVc4VOesQv

0

6

3

0

2K

Dr. Hubertus Becker @hubertusbecker

about 3 years ago

Explore Lance, the new data format revolutionizing #MachineLearning. From fast random access to data versioning, Lance streamlines your #AI workflow. It integrates with popular tools like #pandas, #duckdb, #polars and #pyarrow. #DataScience https://t.co/mdcAGbUh3u

hubertusbecker's tweet photo. Explore Lance, the new data format revolutionizing #MachineLearning. From fast random access to data versioning, Lance streamlines your #AI workflow. It integrates with popular tools like #pandas, #duckdb, #polars and #pyarrow. #DataScience https://t.co/mdcAGbUh3u https://t.co/WslL2Zkymf

0

5

1

0

465

Voltron Data @VoltronData

about 3 years ago

C++, #Golang, and #PyArrow support for Run-End Encoded arrays can reduce memory usage for data containing many repeated values. Users can get memory savings for representing and processing data. Smaller data representation = faster operations and transport!

1

3

1

0

933

Top Tweets for #Pyarrow

Last Seen Hashtags on Sotwe

Trends for you

Most Popular Users