All this talk about #MKBHD and other reviewers killing the @Humane AI pin. It’s getting a lot of publicity, although negative. People who had no idea about it are now interested. A year down the line when they come back with the polished product, this publicity will help them.
By changing the sort order of one of my parquet tables at Airbnb, I was able to reduce its size from 35 GBs to 1 GB! Since there's 365 partitions of this data. It goes from being 12.2 TBs of data to 0.3 TBs.
Remember when sorting your Parquet data that you should start with lowest cardinality first, then higher cardinalities.
Say you have a dataset with schema:
- device_operating_system (values like iOS, Android, Windows)
- country (~250 distinct values)
- user_id (millions of distinct values)
If you sort by device_operating_system first then country then user_id, you'll achieve much better compression than if you sorted country, device_operating_system, then user_id.
The closer together your low cardinality dimensions are, the better job parquet does at shrinking its size!
#dataengineering
Given the speed at which ChatGpt generates text, the auto scrolling of the result should not be enabled by default! I need to instantly scroll upwards to stop the scroll from happening which is annoying in my opinion. Thoughts? #ChatGPT#ai
Polars 0.20 released. Version 1.0 next 👀. Really liking this library so far, great Pandas alternative! https://t.co/mpobDdKvbw #data#python#pandas#polars
Unit's top down navigation helps me finding the code behind something by following visual clues alone. I can always go directly to the component I want to focus on.