Coiled

Lightweight cloud compute platform for Python people.

[David Cassel @DavidCasselTNS]

@mrocklin @mkennedy @TalkPython @CoiledHQ @natt941 made a good point on the podcast. "This message of 'Things are supposed to be delightful' is important to us."

And Matthew Rocklin agreed that the cloud "can be a delightful experience... Go play."

https://t.co/1Qhod30KZ6

[Coiled @CoiledHQ]

Easily configure shared memory size for CLI jobs with `--docker-shm-size`.

Training PyTorch models on a GPU and need more memory? Ever run into "Error: No space left on device"?

Customize Docker shared memory size with `--docker-shm-size`.

https://t.co/9pwx86miBv

[Coiled @CoiledHQ]

🔨 Job setup option for Coiled Batch

Use `--host-setup-script` to configure your VM before your batch job starts.

Easily:
✅ Install dependencies
✅ Mount cloud storage
✅ Handle authentication
or any other setup your jobs need.

https://t.co/LOQaSbvXr4

[Coiled @CoiledHQ]

@black_dot_box 🙌 thanks for the kind words, glad to hear Coiled's been working well for you.

[black.box @black_dot_box]

@CoiledHQ ~5 years ago I worked at a startup where we had multiple engineers screwing around for months with Terraform, Kubernetes, EKS, etc. just to get the same capabilities I got after an hour of playing around w/ Coiled.

Pretty cool.

[Matthew Rocklin @mrocklin]

Coiled 2024 in Review

https://t.co/jC16ztdgE3

It’s the time when companies issue year-end summaries, acclaiming success (or not), and forecasting incredible growth for the next year (or not).

I thought I’d do something similar for Coiled. It’s been quite a year for us ...

[Coiled @CoiledHQ]

Calculating quantiles, a common application in
#geospatial workloads, used to be slow due to GIL contention in NumPy.

The new implementation in @dask_dev + @xarray_dev is up to a hundred times faster and scales independently of the number of threads 🥳.

https://t.co/UnJjPEF3Pd

[Coiled @CoiledHQ]

We're big fans of rich for a nice terminal experience, but have found sometimes folks log things even rich can't handle.

In the latest coiled=1.67.0 release, coiled logs automatically falls back to non-rich printing in these situations.

Release notes: https://t.co/jWOKtDlRz0

[Matthew Rocklin @mrocklin]

New Post: Cloud Computing is Broken

https://t.co/Ode3eXkGFO

Investor asks: "What's next for Data/Cloud Infrastructure?"

My answer: "Boring stuff. People struggle with basics."

Cloud feels like MP3 players before iPod. In theory everything is good. In practice adoption is low

[Coiled @CoiledHQ]

We're now on Bluesky!

Should be pretty easy to find us, since bluesky lets us use our https://t.co/FYMNE1Ch34 domain as our handle ☀️

[Xarray @xarray_dev]

Read about the latest improvement to https://t.co/sNVU1DIXuJ with Dask: https://t.co/EAkjYHQxZe

Thanks to Patrick Hoefler of @CoiledHQ for the great work here!

[Matthew Rocklin @mrocklin]

New Post: SLURM-Style Job Arrays on the Cloud

https://t.co/Fu7kVUSVAZ

HPC Job scripts were the first form of parallelism I ever used as a graduate student. They're dead simple and accessible to almost anyone.

We replicated the API with Coiled. It feels pretty slick to me 🙂

[Nazari Goudin @nazari_goudin]

@CoiledHQ is amazing. If you want to have distributed compute and provisioned infrastructure from the code - its easy as that. Forget @ApacheSpark and @awscloud Sagemaker, EMR

[Quentin Lhoest 🤗 @lhoestq]

New blog post: Scale AI-based Data Processing EASY

The FineWeb-Edu dataset comes from processing 45TB (🤯) of FineWeb

And it uses a Language Model to classify the educational level of the text 😭😭

Still, we reproduced it in a few lines of code !
The key ? HF + Dask 😎

[Coiled @CoiledHQ]

@HEPfeickert @hugobowne We took it down because it's a bit dated, but I've brought it back: https://t.co/bDdxs0qZKr

[Arpit Bansal @arpit__bansal]

Implemented @CoiledHQ into our product to offload data syncing from BigQuery to Neo4j 🤯

Works like butter 🧈

Now I don’t have to worry about scaling VMs dynamically to handle variable loads.

[Matthew Rocklin @mrocklin]

We're to build a 100-TB scale geospatial benchmark suite

https://t.co/vlzt3Szdmd

We've seen an uptick in geospatial users and in challenges of the Xarray/Dask stack to scale beyond ~500-GiB.

This post presents a call for benchmark workloads.

[Earthmover @EarthmoverHQ]

Arraylake and @CoiledHQ work great together! You can use Coiled to manage your cloud computing infrastructure with @dask_dev, and store your data as @zarr_dev in Arraylake.

We just added new a documentation page about our integration with Coiled.

https://t.co/5070jBW2k1

[Coiled @CoiledHQ]

Run a Python script on a cloud GPU with one line of code.

Training a @PyTorch model training takes ~10 minutes and cost ~$0.12 on the NVIDIA T4 GPU on AWS. Coiled handles provisioning hardware, setting up drivers, and installing CUDA-compiled PyTorch.

https://t.co/JoUAOWUe9e

[Dask @dask_dev]

Dask DataFrame is now 20x faster. Some of most prominent changes include:
- Apache Arrow support in @pandas_dev
- Better shuffling algorithm for faster joins
- Automatic query optimization

Learn more: https://t.co/eVbgpWE7BZ