Pascal Seitz

Rust Software/Search Engine Engineer @Quickwit_Inc
Creator of lz4_flex

[Pascal Seitz @pascal_seitz]

@cmuratori I paused the video and just saw that you actually talk about this already. Well if anyone is interested in the paper.

[Pascal Seitz @pascal_seitz]

@cmuratori Hi Casey, great in-depth talk. The missing paper is available here (page 107): https://t.co/rYqGof3opK
On page 120: "in a typical program only 3% of the statements make up 50% of the programs execution time" based on 17 programs and instrumented code. Interesting way to profile

[François Massot 🦀 @FrancoisMassot]

Binance built a 100PB log service with Quickwit, indexing logs at 1.6PB/day

Their engineers migrated petabyte-scale Elasticsearch clusters to Quickwit, cutting compute costs by 5x & storage costs by 20x

Very proud of our collaboration ❤️. Read more => https://t.co/gcS0yGOg6Z

[François Massot 🦀 @FrancoisMassot]

Ever stumbled upon a performance quirk that left you puzzled? 🤔

@pascal_seitz from @Quickwit_Inc did: just allocating unused memory led to a 2x performance boost! 📈

Be ready to dive into the intricate process behind this unexpected perf. gain! 🦀

https://t.co/XXQN7iq5M1

[François Massot 🦀 @FrancoisMassot]

Super proud to work with @zackkanter, @ShortJared, and @stedi engineers to power their multi-tenant log search with Quickwit Lambdas.

"It scales from zero cost at rest to practically infinite volume."

❤️❤️❤️

Thank you for choosing @Quickwit_Inc !

[Pascal Seitz @pascal_seitz]

tantivy 0.22 has been released

tantivy 0.22 is backward-compatible to tantivy 0.21 and contains:
- Top Hits Aggregation
- Term Aggregation on Bool, Date and Ip Addresses
- Performance Improvements on Search and Indexing
- Stability Fixes
https://t.co/jyaxi64yES

[Paul Masurel 🦀 @fulmicoton]

Remember my little puzzle? Here is a blog post explaining the puzzle, its relationship with the usual Top-K algorithm average case complexity, and a better lesser known alternative.
https://t.co/nXTUcnfdg5

[François Massot 🦀 @FrancoisMassot]

We are thrilled to announce Quickwit 0.8, which marks a leap forward in scalability, efficiency, and performance 🦀🏇

Quickwit is now running in production at petabytes scale 🔥:
- Indexing at 1PB/day, or 13GiB/s
- Searching 40 PB of logs
- And more!

➡️ https://t.co/OgkEzLFgpg

[François Massot 🦀 @FrancoisMassot]

Just got some crazy numbers shared by a Quickwit user.

They indexed 40PB of logs into Quickwit, 5x10¹³ log entries, 7.5 PB on S3.

And they send this kind message as well ❤️

"Right now, we are in a bull market, and the logs have increased a lot. Luckily Quickwit works very well."

We know they have an original multi-cluster setup:
- ~10 indexer clusters
- 1 big searcher cluster

And we will share more details about their vCPU and RAM usage soon.

If you have some logs to index, use @Quickwit_Inc :)

[François Massot 🦀 @FrancoisMassot]

💃🏼New blog post about Quickwit's serverless search performance on AWS Lambda & S3 🏇

On the menu:
- Latency on search/analytics queries
- Impact of Lambda size (1GB to 8GB)
- Impact of caching layers
- Beautiful charts to illustrate our findings 😍

=> https://t.co/g8JUpIhx3H

[François Massot 🦀 @FrancoisMassot]

We have just landed Quickwit’s serverless search based on Amazon Lambda and S3 \o/

You can now scale search to 0, no fixed costs. In comparison, OpenSearch Serverless starts at $700/month.

Overall, we estimate the costs to be 50x to 100x cheaper than CloudWatch.

And performance is still there: searching on millions of log entries is sub-second on a cold start. Expect a bit more if you run analytics queries.

More details in our blog post ⇒ https://t.co/x9uSbOxNG8

I’m eager to see new use cases on this, any feedback is welcome!

#ServerlessSearch #AWSLambda #Serverless #Quickwit

[François Massot 🦀 @FrancoisMassot]

I'm very excited to announce the release of Quickwit 0.7, which brings better Elasticsearch API compatibility, better integration with @grafana and @JaegerTracing, and 30% performance gains,

This is also a major release as we have witnessed massive performance gains and cost savings for our nightly users, and this will now be in the hands of all of our users.

The most compelling report comes from a company ingesting hundreds of terabytes of logs daily and migrating from Elasticsearch to Quickwit, dividing their compute costs by 5x and storage costs by 2x while increasing retention from 3 to 30 days.

This gives us great confidence in our ability to deliver a robust OSS alternative to Elasticsearch, Datadog, or Splunk!

https://t.co/qhGlWW8dzT

[Pascal Seitz @pascal_seitz]

@_tde9 @fulmicoton Any datapoints in particular you'd be interested in?

[Paul Masurel 🦀 @fulmicoton]

💃🏼 Quickwit 0.6 released!🕺🏼 is released!

It packs an 𝗘𝗹𝗮𝘀𝘁𝗶𝗰𝘀𝗲𝗮𝗿𝗰𝗵 𝗰𝗼𝗺𝗽𝗮𝘁𝗶𝗯𝗹𝗲 𝗔𝗣𝗜
and a 𝗚𝗿𝗮𝗳𝗮𝗻𝗮 plugin!

See more here ⬇️
https://t.co/nY297Ce9JO

[Paul Masurel 🦀 @fulmicoton]

Does @Quickwit already scale? Yes! Thanks for asking! Here is a screenshot @FrancoisMassot just shared.

40 x c5ad.2xlarge nodes
= 1.5GB/s of indexing throughput
(~$150/month per node)

Data is ingested from @apachekafka and the index is stored on @awscloud 's S3

[Paul Masurel 🦀 @fulmicoton]

Quickwit hits 0.4: distributed indexing with Kafka, k8s-native, and more!
https://t.co/k8gLuGndEl

[Quickwit @Quickwit_Inc]

🌟Quickwit just reached 2000 stars!

The journey to this milestone has been hard, but it wouldn’t be possible without the amazing community that we have today. Thank you everyone.

https://t.co/VhYEmjWdL7

#search #selfhosted #rustlang #devops #tech #AWS #analytics #developer

[Quickwit @Quickwit_Inc]

🔎Search through logs and set filters right from UI…like the one here on “severity:ERROR”.
🤿Narrow it down further with date range, fields, etc.

#search #selfhosted #rustlang #devops #tech #AWS #analytics #dev #solr #developer

[Quickwit @Quickwit_Inc]

Have you heard of Heaptrack? Check out an amazing blog post by @pascal_seitz on how to investigate memory consumption of your Rust program and how he optimized #memory usage in #Quickwit!

https://t.co/MBveUeFgCP

#optimize #software #datasets #rustlang

[Pascal Seitz @pascal_seitz]

Tantivy 0.17 is out https://t.co/IXm6QBlQ0I via @Quickwit_inc