@jaceklaskowski I think we are just missing some context information as we start the extra threads we use to improve cluster utilization. We will fix it.
@jaceklaskowski@Falydoor Yeah, that is confusing. We should fix the icon. Longer term we are working on the ability for you to attach the notebook to your DLT pipeline cluster, so you can get interactive feedback on syntax or analysis errors.
Insightful benchmark of Linux Foundation Delta Lake and Apache Iceberg by @BrooklynData that shows Delta is up to 8x faster in workloads with data updates. Most storage benchmarks only test reads, but with updates, care is needed to maintain performance. https://t.co/6efYPoRFej
@jaceklaskowski@bigklata That is not exclusively true. Partitions can also help, for example with a SELECT DISTINCT <partition_column> and other aggregations, however, you are generally right that if the query is SELECT * you have to read the whole table definitionally.
Databricks just set a new record on the official TPC-DS data warehousing benchmark, showing that a lakehouse system based on open data formats can outperform previous DW systems. Don't listen to folks who say open means bad performance! https://t.co/Gj6k34M5eZ
As the #lakehouse gains momentum, we've been getting a lot of questions about what it means and how it compares to other architectures. Here are some answers to these and other common questions! https://t.co/Npp6rcfO7a
@technology@emilychangtv Why do you have to pick between diversity and merit? Make your work env. inclusive, decrease bias in hiring, as a leader don't make statements that alienate large groups. That'll give you a competitive advantage to a talent pool that won't join unwelcoming companies.
Really excited to announce Delta Live Tables, a system that turns your SQL queries and pyspark DataFrame code into production-ready ETL pipelines! We've been working on this for almost 3 years, and it powers some of our largest pipelines internally. https://t.co/edG7myvzG7
@dr_kuhalla@bgawalt I had a similar question as clover is going crazy on my citrus tree. Apparently not all clover does nitrogen fixing though, TIL.
https://t.co/pOSCjKkiS3