Arroyo

The next-gen stream processing engine. Now part of @CloudflareDev.

[Micah Wylde @mwylde]

I'm very excited to announce that after 8 months of work, Arroyo 0.15.0 is now available! This is a huge release that brings all of the improvements, fixes, and learnings from running Arroyo at scale to power @CloudflareDev Pipelines, our fully-managed ingestion service.

[Micah Wylde @mwylde]

The news is finally out! Cloudflare has a Data Platform! We're starting with serverless streaming pipelines (powered by @ArroyoSystems), a managed Iceberg Catalog, and a new distributed SQL engine built on @ApacheDataFusio.

[Arroyo @ArroyoSystems]

Arroyo 0.14.1 is now available! This is a patch release with a number of fixes over 0.14.0, and is a drop-in upgrade: https://t.co/wvWLotbXA0

[Micah Wylde @mwylde]

The third SF DataFusion meetup is today at 6! We have a bunch of great speakers including project leader @andrewlamb1111, @camuelg, @LakeSailHQ , and @Greptime. I'll be giving a talk about how Arroyo does streaming incremental view maintenance to support unbounded SQL queries.

[Arroyo @ArroyoSystems]

@sdil @CloudflareDev Arroyo is remaining open source and will continue to be developed: https://t.co/YKVnwhq7mg

[Arroyo @ArroyoSystems]

@Cloudflare We couldn't be more excited to be building the future of real-time process with Cloudflare! See all of the details on the blog: https://t.co/YKVnwhq7mg

[Arroyo @ArroyoSystems]

We are very excited to announce that we are joining @Cloudflare to bring stream processing to everybody! Arroyo is coming to the Cloudflare Developer Platform as a serverless stream processing system, and will also remain open-source and self-hostable.

[Cloudflare Developers @CloudflareDev]

Two big things:

1. Pipelines, our brand new streaming ingestion service, is now in public beta.
2. We've acquired @ArroyoSystems so we can bring SQL-based stateful transforms to Pipelines!

Ingest tens-of-thousands of records per second right now: https://t.co/D3gtdsD4ma

[Arroyo @ArroyoSystems]

Arroyo 0.14.0 is now available! New features in this release include:

🔎 Lookup joins
🪆 Nested updating aggregates
{} Struct types
📖 Streaming SQL syntax
🔀 Sink shuffles

See the full release notes at https://t.co/7ThORpfqe5

[Arroyo @ArroyoSystems]

Arroyo just crossed 4,000 GitHub stars! It's amazing to see the excitement for a new future of open-source stream processing. Thanks to all of our users and contributors for your help getting here!

[Arroyo @ArroyoSystems]

@v1gnesh @mwylde Here you go! https://t.co/OHJToyDPhY

[Micah Wylde @mwylde]

You'd think that the key to being a fast streaming engine is like clever join algorithms, but it's mostly just being really good at JSON. @ArroyoSystems uses Arrow and the arrow-rs JSON decoder. I think it's pretty cool, so I wrote up how it works: https://t.co/6ipB13JFiu

[Arroyo @ArroyoSystems]

Building a near-real-time analytics stack used to take a team months or years. Thanks to modern tech like S3, @redpandadata, @duckdb, and Arroyo, any company can build a sub-minute data lake in under an hour: https://t.co/iiDjuJkexg

[Arroyo @ArroyoSystems]

Arroyo 0.13.1 is now available: https://t.co/MybchCcz7k

This patch release includes several fixes over 0.13.0 and is a drop-in upgrade.

[Arroyo @ArroyoSystems]

Arroyo 0.13.0 is now available, with

📘 Source metadata support
🐰 RabbitMQ streams connector
⚛ Atomic updating outputs
🔐 IAM auth for Kafka
⛓️‍Operator chaining

See all of the details in the release post: https://t.co/4hhTGTLur3

[Arroyo @ArroyoSystems]

Arroyo 0.12.1 is now available: https://t.co/8QfZ7fbhbW

This patch release contains several bug fixes and is a drop-in upgrade for clusters running 0.12.0.

[Micah Wylde @mwylde]

If you missed #p99conf last week, talks are now available to stream on YouTube. I spoke about the design decisions that went into Arroyo's incredible performance: https://t.co/YdPybnyr4m

Come for the Rust hot takes, stay for my terrible hand-drawn architecture diagrams 😅

[Arroyo @ArroyoSystems]

Arroyo 0.12.0 is now available, with support for 🐍 Python UDFS 🐍, plus Protobuf, faster JSON functions, custom state TTLs, and more.

See all the details in the release notes: https://t.co/smrx330jA4

[Arroyo @ArroyoSystems]

Arroyo is coming to #Current24! Come chat with the team and learn about the future of stream processing.

[Micah Wylde @mwylde]

Looking forward to talking about how we dynamically-load DataFusion UDFs in @ArroyoSystems! Because Rust lacks a stable ABI, this is harder than it sounds!