Top Tweets for #SparkSQL
Day 1 of Corporate Learning 🚀
Learned about Spark SQL and how data pipelines work in real-world projects. 📊
#DataEngineering #SparkSQL #LearningJourney
Use regex in Spark SQL for super-powerful string processing!
With the RLIKE or REGEXP_EXTRACT functions, you can:
Validate formats (e.g., emails, dates).
Extract specific data (e.g., codes, values).
Filter complex rows.
Example: WHERE column RLIKE 'pattern'
#SparkSQL
QUALIFY clause in Spark SQL filters the results of window functions (like RANK(), ROW_NUMBER()) without requiring subqueries. It acts like a HAVING clause specifically for window functions,simplifying your queries.QUALIFY RANK() = 1 to get the first record in each group.#SparkSQL
#ApacheIceberg + #SparkSQL = a solid foundation for building #ML systems that work reliably in production.
Time travel, schema evolution & ACID transactions address fundamental data management challenges that have plagued ML infrastructure for years.
🔍 https://t.co/uMJCmj7QnI
at @yourcreatebase, i was working with large unclaimed music royalty records — to consolidate publisher objects: mapping rights admin relationships to shares, writers, and iswc codes — to make our royalty payout pipeline faster and more accurate
#SparkSQL #PySpark #AWS #S3
#ApacheIceberg + #SparkSQL = a solid foundation for building #ML systems that work reliably in production.
Time travel, schema evolution & ACID transactions address fundamental data management challenges that have plagued ML infrastructure for years.
🔍 https://t.co/uMJCmj7QnI
#TIL Sub Execution IDs is a #SparkSQL feature in web UI (not #Databricks-specific as I always thought) 🥳
Any good docs on the feature? 🤔
#ApacheSpark
Ever wondered what happens when you execute CACHE TABLE AS command in #ApacheSpark #SparkSQL? 🤔 Curious if it's for tables only? Views too?
It all boils down to CacheTableAsSelectExec physical operator that uses high-level ones like we all do! 🥳
➡️ https://t.co/xWXnm2fQm4
Even wondered what happens after CREATE [[GLOBAL] TEMPORARY] VIEW AS statement is executed in #ApacheSpark #SparkSQL?
Start here ➡️ https://t.co/8tW2jUEF0t
...and follow along until you know it all or got qqs that I could answer in a follow-up 😉
![jaceklaskowski's tweet photo. Even wondered what happens after CREATE [[GLOBAL] TEMPORARY] VIEW AS statement is executed in #ApacheSpark #SparkSQL?
Start here ➡️ https://t.co/8tW2jUEF0t
...and follow along until you know it all or got qqs that I could answer in a follow-up 😉 https://t.co/X5s8xbyeOM](https://pbs.twimg.com/media/GLtG7zTWgAAJwyf.jpg)
![jaceklaskowski's tweet photo. Even wondered what happens after CREATE [[GLOBAL] TEMPORARY] VIEW AS statement is executed in #ApacheSpark #SparkSQL?
Start here ➡️ https://t.co/8tW2jUEF0t
...and follow along until you know it all or got qqs that I could answer in a follow-up 😉 https://t.co/X5s8xbyeOM](https://pbs.twimg.com/media/GLtGYHdWMAA2sWj.jpg)
![jaceklaskowski's tweet photo. Even wondered what happens after CREATE [[GLOBAL] TEMPORARY] VIEW AS statement is executed in #ApacheSpark #SparkSQL?
Start here ➡️ https://t.co/8tW2jUEF0t
...and follow along until you know it all or got qqs that I could answer in a follow-up 😉 https://t.co/X5s8xbyeOM](https://pbs.twimg.com/media/GLtGSYuXcAAJo6e.jpg)
![jaceklaskowski's tweet photo. Even wondered what happens after CREATE [[GLOBAL] TEMPORARY] VIEW AS statement is executed in #ApacheSpark #SparkSQL?
Start here ➡️ https://t.co/8tW2jUEF0t
...and follow along until you know it all or got qqs that I could answer in a follow-up 😉 https://t.co/X5s8xbyeOM](https://pbs.twimg.com/media/GLtGJY3XkAA0pOG.jpg)
#TIL #ApacheSpark #SparkSQL 3.5 comes with Named Function Arguments feature that lets you specify arguments by name 🥳
TVFs and a few built-in standard functions are supported only (but more is coming up in 4.0) 👏👏👏
➡️ https://t.co/Z3ag2tMz18
There are quite a few new standard functions in #ApacheSpark #SparkSQL 3.5 alone yet there are way more added in the recent versions.
One of them is max_by standard aggregate function that got added as early as in 3.3 🥰
➡️ https://t.co/kT0wU4lt9u
Craft a Pandas User Defined Function in Python, and seamlessly execute it within an SQL query. It's a symphony of efficiency and simplicity in data processing. #PandasUDF #SQLIntegration #PythonPower #SparkSQL
Read our latest story published on the https://t.co/d0LPZUd9m3 Medium platform : https://t.co/BZzfU0io9h
#SPARK #data #DataScientists #DataAnalytics #dataengineering #bigdata #distributedsystems #SQL #ApacheSpark #SparkSQL #dataframes #AI #ML
#LoomeConnects to #ApacheSpark Extracting, Loading and Monitoring Data in #SparkSQL has never been easier.
#Intel Granulate Optimizes #Databricks #Data Management Operations https://t.co/bRk8Il8DDs #SparkSQL #IntelGranulate #DataManagement
Customers can continuously and securely optimize large-scale #Databricks workloads and cut cloud infrastructure costs with no code changes.
TEL AVIV, Israel, Nov. 27, 2023 – Intel® Granulate™, Intel’s https://t.co/xQdeEli0U1 #IntelGranulate #DataManagement #SparkSQL
#Intel Granulate Optimizes #Databricks #Data Management Operations https://t.co/DnH5YfdKSS #SparkSQL #DataManagement #IntelGranulate
Two new metadata schema columns in #ApacheSpark #SparkSQL:
1⃣ Metadata Columns ➡️ http://localhost:8000/spark-sql-internals/metadata-columns/
2⃣ Hidden File Metadata ➡️ http://localhost:8000/spark-sql-internals/hidden-file-metadata/
Different code paths, yet so similar 🤷♂️
Current status : preparing an article and videos on @scala_lang @ApacheSpark and #DataprocServerless on @GoogleCloudTech with a complete example.
Stay tuned !!
#CICD with #CloudBuild and the #gcloud #cli
I missed Scala 😅😁
#GoogleCloud #GCP #Scala #Spark #SparkSQL #FP
Last Seen Hashtags on Sotwe
Trends for you
Most Popular Users
Elon Musk 
@elonmusk
241.5M followers
Barack Obama
@barackobama
118.8M followers
Cristiano Ronaldo
@cristiano
114.5M followers
Donald J. Trump
@realdonaldtrump
111.8M followers
Narendra Modi
@narendramodi
107.1M followers
Rihanna
@rihanna
98.7M followers
NASA
@nasa
92.3M followers
Justin Bieber
@justinbieber
91.8M followers
KATY PERRY
@katyperry
90M followers
Taylor Swift
@taylorswift13
84M followers
Lady Gaga
@ladygaga
75.5M followers
Virat Kohli
@imvkohli
73.5M followers
Kim Kardashian
@kimkardashian
70.9M followers
YouTube
@youtube
68.8M followers
Neymar Jr
@neymarjr
66.5M followers
Bill Gates
@billgates
65.3M followers
Selena Gomez
@selenagomez
63.1M followers
The Ellen Show
@theellenshow
62.2M followers
CNN
@cnn
61.7M followers
X
@x
60.7M followers