despite coding agents, modern apps are fragile. without proper architecture, you can spit out as much code as you’d like, but you’re just adding levels to a house of cards.
introducing Hatchet v2 & our $5.7M seed round. our bet is that great orchestration is the difference between a reliable app and a broken one.
Hatchet v2 is the universal orchestration layer: durable execution, durable queues and durable streams, sandbox integrations, serverless support, and operators. this is the architecture used by high-growth companies like @greptile, @linear, @getenter_ai, and many others, who collectively process over 5 billion tasks/month on Hatchet.
we’ve spent over two years obsessing over making our engine performant and scalable. this release extends the existing @hatchet_dev engine to support any sandbox provider, serverless environment, edge function, or container.
@HeyGarrison that's fair, the issue is that to spend time contributing to something there needs to be some trust & credibility in the first place which hasn't been met.
not saying this can't be remedied! it's probably a few hours of time writing a methodology section by hand.
I saw that there was a sandbox provider benchmark that some providers were referencing. Looked interesting so I clicked in and clicked around for a bit. To my horror the methodology section is all slop. The open source README is all slop as well. Many of the benchmarks are "for subscribers only."
I'm not going to call them out by name but the reason this is so discrediting is that in order to build trust in a benchmark, you need to prove out the series of micro-decisions of exactly how and why you've designed it this way. If you're not going to bother writing the methodology section or the README, at least publish the prompts that you used to build the benchmark.
This might be a good benchmark, but who knows? How can I possibly evaluate that?
we added durable streams as a storage backend to pi-durable, which can be used via hatchet-dev/pi-durable.
writing this integration was super interesting because it illustrated exactly the thing I've been saying for months: durable workflows have their time and place, but there are other equally important durability primitives. the @pidotdev durability story is much more "stream-centric" than "workflow-centric," which makes sense given the default storage backend is sqlite.
combining the two (durable workflows and durable streams) is even more powerful. I wrote last week about how durable workflows are a great way to model session state, of which the harness may or may not be a part.
experimental package here: https://t.co/wTcZcLuRvj
pi durable: adds support for hatchet as a storage backend to pi-durable and also allows provisioning of pi durable harnesses inside of a sandbox
https://t.co/bCzOpWlQBE
Also: the best benchmarks reveal tradeoffs. In the sandbox case, there's almost certainly a direct tradeoff between filesystem throughput, durability and snapshot restore times. Or between burstable CPU capability and cost.
The best vendors will be able to walk you through the tradeoffs they've made between latency, throughput and durability, and the best benchmarks will illustrate this.
@soohoonchoi was one of the first people I met in the YC W24 batch. it was immediately clear that greptile was going to take off. I remember stopping by their office during the batch and thinking: those are their 3-month goals?
there was even a moment during a YC event where I looked over and saw the hatchet UI pulled up on soohoon's laptop, and being quite nervous that there was an incident or something.
it's been super rewarding getting to meet more engineers on the @greptile team and seeing the volume of PRs getting reviewed each day. excited to see what the next year brings 🚀
more case studies coming soon!
postgres 19 is out! so many great things in this release but two that are super useful operationally:
- can't express how excited I am for REPACK CONCURRENTLY. we used to run pg_repack in production and it added a lot of complexity.
- logical replication of sequence values! our migrations off of RDS used pg_logical but and manually resetting sequence values was such a footgun.
https://t.co/sevMVGYvE6
at the facebook office, 2005: "guys why are you writing all this PHP yourselves? wordpress gives you the database, logins, and hosting. just pick a theme and install some plugins."
so that’s my opinion on managed agents
if you're tempted by the "<x> is too expensive, let's built it from scratch" narrative, it's worth doing some back of the napkin math on what said service will actually cost from a pure disk/database perspective. for example, for an analytics services, you can just ask an agent to estimate the cost of your analytics data on clickhouse cloud.
it's definitely the case that there are lots and lots of SaaS products which are way more expensive than their underlying compute and database services and have historically had high margins.
but there are also tons of cases where you're just cooked because of the amount of data you're sending, and the services that allow you to send that much data in the first place likely benefit from massive economies of scale which almost guarantee it's not worth the token cost of building it yourself.
there are so so many useful nuggets in this book.
just one example: there's a section on how to rebalance partitions across nodes that I've been re-reading lately. to paraphrase, don't just hash each key and mod by the number of nodes, because adding and removing nodes becomes super expensive. the better way to do it is to create far more partitions than nodes and have nodes siphon off partitions as they come online. even better is dynamic partitioning, where partitions can merge or split and then rebalance, basically like a btree implementation.
a "node" in the case of @hatchet_dev is a bunch of things: tenant schedulers, leasing for crons, leasing for queues, etc. we absolutely did this incorrectly in hatchet v0 but we're finally rolling out dynamic partitioning for tenants, which is covered at the end of this section.
if you have the tokens, you can and should use swarms for research / experimentation.
but if you're shipping to production, this is the only way to prevent codebase enshittification.
ok so this ended up costing us just under $1k after being on the front page of Hacker News, not quite the $350k that julius was hoping for.
it did turn out to be a nice little load test for durable streams after rendering over 5k images:
- 300k stream messages sent
- 7k unique stream topics
- 215k individual tiles rendered
we will be taking it down at the end of the day, though it will live on in our hearts forever.
my CTO gave me 350k of infra credits and I spent them all rendering mushroom clouds
I made a ray tracing app in Rust that lets you prompt SDFs to render realistic images (using @hatchet_dev durable streams and @cloudflare workers). Try it out at https://t.co/EoFpRiq81J before we run out of money
modern data layers are bifurcating into either really fast, local NVMes or really slow, massively scalable object storage. or both.
it's pretty obvious when NVMe is useful (incredibly fast disk i/o). and it's also obvious when S3 is useful. but systems that combine the two are the most interesting, because you're taking a big bet on data access patterns.
for example, the reason this works so well for us is that if our users are accessing a payload that was written in the past 24 hours, you are fairly likely to be latency sensitive (obviously not true for everyone, but enough of our users). so you'd want single-digit millisecond queue times.
but if you're accessing data that was written over 24 hours ago (for example, resuming a workflow after 24 hours of inactivity), you are not latency sensitive to a ~100ms queue time.
every data stack which uses NVMe + S3 needs a similar argument for their data access, or it doesn't work. so while disaggregated storage + compute OLTP databases are really cool, I've never worked on a stack where I've wanted to use them in production.
once you're sure of the access pattern, matt's two blog posts on how this actually gets built are definitely worth a read!
new blog post: how we built index files on S3 to offload data from our NVMe disks on Hatchet Cloud.
everything is S3 now. Postgres, Git, Kafka, and even POSIX filesystems are being reinvented with S3 as a foundational layer. earlier this year, we hopped on the bandwagon and decided to offload a large part of the Hatchet payload data to S3 to save us money on disk.
we affectionally called the resulting table Supertoast.
but our initial implementation didn't go far enough. and thus, "Supertoast, Redux".
https://t.co/AXWzj6jA8T
we built a Postgres query optimizer with @hatchet_dev sandboxes and durable streams.
I spend a lot of time optimizing queries. even the best coding agents are actually quite bad at this. they'll add an index or modify the query structure and more often than not won't pick the best approach (i.e. every index comes with write overhead, which often isn't considered).
this is partially because it's very non-obvious how to squeeze performance out of a query without the right environment. I've been on teams that have cloned (anonymized) production datasets just because query issues are often impossible to reproduce locally.
so how do you get around this problem? while query writing is meh at best, coding agents are really good at two things: (1) parallelization and (2) iterating towards a verifiable goal with the right environment.
and thus our Postgres query optimizer was born. it runs a full postgres instance in a sandbox and the agents race to build the most performant query given a specific problem. nice work @jbsouvestre!
yeah it's a good question. you could definitely couple them (durable objects are sort of an opinionated coupling of routing, runtime and storage). if you don't use durable objects and you build your system this way, the compute which is running the "next tick" will have a filesystem attached, which is expensive. the event log is usually a lot more lightweight and easy to move between machines.
to zoom out, ultimately I view it as a more flexible programming model to treat them as separate pieces of state with their own backend. I talk about this at the bottom of the article, but one example is that often you want your filesystem guarantees to be different from your eventing guarantees (by this I mean the checkpoint frequency section).
so I pissed off temporal's ceo a few months ago by making the argument that old-school durable execution is much less important than it used to be, or more specifically, is very architecturally lacking.
this is really a conversation we've been having internally for the past 6 months, because we've been trying to reconcile how to model the two different aspects of durable state that agents need: filesystem state and session state. these are distinct and if you're building an agentic application I think you need to understand the difference (which I talk about in the article).
it turns out that the biggest innovations in filesystem durability are coming out of the sandbox providers like @archildata@e2b@modal@freestyle_dev , to name a few. it's easy to have sandbox-provider-fatigue if you view them as compute resellers, but if you cut through the noise, some of them are doing super interesting work on the filesystem durability layer (and the isolation layer of course).
I go into more detail in the article, but there's a correct way to use these providers in conjunction with durable execution. that's why we released the @hatchet_dev sandbox package earlier this week and why we're integrating with really great sandbox providers. full article:
https://t.co/Zj0ATB3S99