Tom White

Open data. Open source. Hadoop book author. Ex-Cloudera. Runner. Looking for interesting data projects. @[email protected]

[Tom White @tom_e_white]

@adamcohenrose @mouseboks @sammachin I'm going. See you there Adam!

[Ryan Abernathey @rabernat]

It was great collaborating with @tom_e_white and @TEGNicholasCode on optimized the Cubed. 5x perf improvement for the benchmark workload! 🚀

[Tom White @tom_e_white]

@abayer True across the UK I think. There is no rule, and people just walk on whichever side of the pavement they like.

[Tom White @tom_e_white]

@adamjkucharski It's fascinating. I tried to visualise the words themselves a few years ago: https://t.co/kdDz1Z79Ej

[Tom White @tom_e_white]

@_JacobTomlinson @TEGNicholasCode @xarray_dev Thanks for the suggestions! Rechunking is always going to be a bottleneck, but I'd be pleased if we could get to the point in Cubed where there is one intermediate store per rechunk operation in a workflow.

[al @al_merose]

Here's a random direction for where Cubed could go: How will it interact with accelerators (e.g. GPUs/TPUs)?1/

[Tom White @tom_e_white]

@al_merose @TEGNicholasCode Lots of interesting ideas! BTW there's a bi-weekly meeting to discuss these kind of things, that you may be interested in: https://t.co/ec1sZ7kj3H

[Tom White @tom_e_white]

@_JacobTomlinson @TEGNicholasCode @xarray_dev Also, I think there are some very significant optimization opportunities to get the number of writes to intermediate storage down to something much more reasonable. This is probably the biggest: https://t.co/Q72X0blVI8 (see red circle).

[Tom White @tom_e_white]

@_JacobTomlinson @TEGNicholasCode @xarray_dev Thanks Jacob! I agree that using memory/lower-latency intermediate storage is worth trying (https://t.co/mZn3iZuoSB). Do you know any cloud memory caches that are serverless? (The ones I've seen need a cluster.)

[Tom Nicholas @TEGNicholasCode]

I want a future where an xarray workload can be executed like a SQL query, on a wide range of backends, who compete to provide the best performance in various cases.

https://t.co/FlCfM62LYh

[Tom White @tom_e_white]

One for the data engineers: I've been working on a new project called Cubed for serverless array processing. Tom Nicholas has written a superb blog post explaining how it works https://t.co/khkuKVNXKZ

[Tom Nicholas @TEGNicholasCode]

‼️ New blog post ‼️

Introducing Cubed: Bounded-memory serverless array processing in @xarray_dev

https://t.co/t2u7tYQHwc

[Tom Nicholas @TEGNicholasCode]

‼️ New blog post ‼️

Introducing Cubed: Bounded-memory serverless array processing in @xarray_dev

https://t.co/t2u7tYQHwc

[PyCon UK @PyConUK]

🚨🚨🐍🐍PyConUK 2023 call for proposals is now open 🐍🐍🚨🚨
https://t.co/6nSdVwo6Hf
This call will close on Friday 30th June
We’re looking for main stage events, classroom events, lecture room events, young coders’ day events and any other ideas you may have.
Submit now!

[Tom White @tom_e_white]

@rabernat 2015: https://t.co/sV7Sr1wMr7

[Jacob Aron @jjaron]

Here is a question that has been bugging me - what is the shortest possible queue? Explain your reasoning.

[Tom White @tom_e_white]

Ruthless simplification

https://t.co/HtfGbgVpIY

[The Night Science Podcast @nightsciencepod]

New 'Night Science Podcast' episode! Mathematician extraordinaire @stevenstrogatz talks about how to ruthlessly simplify a problem to the point where - while it still seems impossible - it is down to its bare essentials. @ItaiYanai @MartinJLercher
https://t.co/NAF0QbdZdE

[Erik Bernhardsson @bernhardsson]

An exciting technical project we've been working on at https://t.co/DCdDfXPXxs allows us to build a fresh container image, and boot up a lot of containers running that image, on many different nodes, all in a couple of seconds. Some notes on how we do it:

[Eliane Wigzell @elianewigzell]

#NationalMeadowsDay more orchids than ever this year

[Tom White @tom_e_white]

@s_ryz @laserson @sean_r_owen @josh_wills I think someone bought it! It had disappeared when I went back.

[Tom White @tom_e_white]

For sale in a field in Herefordshire #emfcamp2022 @s_ryz @laserson @sean_r_owen @josh_wills