@jaceklaskowski I don’t have a query to share, sorry.
I thought I saw on Spark 3.1 that with repartition() anywhere then the CustomShuffleReader stages seem to disappear. I assumed at the time that AQE was disabling itself.
Next week I’ll be revisiting that job and have a closer look.
@SwiftOnSecurity One of my most profound “digital” experiences ever was the day as an undergrad that we plugged the thin ethernet cable into the bench oscilloscope. We could see the signal, jaw-dropping how messy it was to the point of “how the fuck does it get this right?!?” … “Oh…” 😳
@jaceklaskowski [1] Partitions that expand due to something like explode() basically break AQE, which is unfortunate, and if I recall trying to fix it in that area with repartition() just disables AQE entirely.
@jaceklaskowski I’ve started setting it to “large” values in the thousands: 1) apparently nobody knows the origin of 200 as a sensible default; 2) with Spark 3, AQE will normally[1] reduce it to something sensible; 3) the downside of too many partitions are nowhere near as bad as too few.
@SwiftOnSecurity Here’s some data I collected at a large client with SSO on everything internally.
Also: hideous complexity rules, regular rotation, pwd managers blocked; <1hr into onboarding, trying to come up with a password, someone calls across the room “Just use […]”. So everyone does.
This map of Texas superimposed over Europe really drives home how small and insignificant Europe is compared to America. No wonder something like 97% of Scandinavians say they would prefer to live in the USA.
Bellingcat Discord member Sepulco geolocated the video Elon Musk posted overnight, claiming to show a stalker, which he used to justify the banning of accounts that tracked flights. It's at 34.116175,-118.160635, not close to any airports.
https://t.co/H4ibUozDKV
It appears Elon Musk, the “free speech absolutist,” has shadowbanned an account which shared publicly available data regarding his private jet.
https://t.co/jPFrV4efJt
This whole Mastodon thing reeks of XMPP federation dreams however instead of a whole bunch of high quality high performing XMPP servers that can handle hundreds of thousands of requests a second with server 2 server we got 1 janky Mastodon Ruby implementation federating the globe
@jimseven@WIRED I have some ancient attic-found Cona makers that have for years been a mystery; felt way too niche to ever justify an Ultimate Guide (or even a mention).
That syphon answer just… solved so many problems! Very excited to have another crack at them.