Eddie Ruiz

[Alex Wolf @falexwolf]

We’re thrilled to share that Lamin is now generally available! With just a few clicks, anyone can create a database on https://t.co/3zIxWeO17Q

In 2022, we set out to build an open-source data management tool that’d make it easier to scale AI workflows in the life sciences. Over the past few years, we’ve served research labs and enterprises of all sizes with what used to take a dozen tools:

- Data lineage
- Format-agnostic lakehouse
- Notes, ELN, LIMS, ontologies
- Versioning, branching
- Data lake, catalog
- Agents, workflows, notebooks
- Feature store, MLOps

With Lamin, it’s just Postgres and storage in a Python package, plus an optional GitHub-like web platform.

Give it a try: pip install lamindb

Thanks to @_DimensionCap and @ycombinator and many others who made this possible

[Pedro Holanda @holanda_pe]

Today we released DuckDB 2.0, and with it, DuckLake 1.1.
The new implementation is backward compatible with DuckLake 1.0, so you can still enjoy a bunch of bug fixes, performance improvements, and some of the new features without upgrading your catalog!

https://t.co/tm3KrVczFg

[Niko McCarty. @NikoMcCarty]

Our newest book is finally here!

Making the Modern Laboratory is a spectacular 504-page hardcover that tells the story of everything in the research laboratory and how it came to be, from pipettes to heat-resistant glass to microscopes and more.

It features dozens of articles (most of which will never appear in print), 100+ full-color images, and original writing from 36 authors. It's printed on Arena Natural Rough paper from Fedrigoni. These books are THICC and would make a great gift for both the science-curious and the working scientist. :)

We have spent the last year working on this project. It has been an insane labor of love. The best way to support our independent science journalism is to buy a copy. <3

Pre-order for $49, with free shipping: https://t.co/7d9L2Ual0c

[Daniel Litt @littmath]

Well worth reading: https://t.co/dPmEAogvMX

[Micah Wylde @mwylde]

The Cloudflare Data Platform has been in beta for the past year and has helped amazing companies like @anomalyco ingest, store, and query their analytical data. Today, for @Cloudflare Birthday Week, we're going GA. And we have a new name: Cloudflare Basin.

[Eddie Ruiz @Ed2uiz]

I’m deeply grateful to my coauthors and collaborators for making this work possible, and to @genome_gov, @BU_Computing, @GEMFellowship, @The_BMC, and @ChanZuckerberg for their support.

Questions, feedback, and contributions are welcome.

Thanks for reading! 🪐

16/16

[Eddie Ruiz @Ed2uiz]

Larger scientific datasets often push researchers toward specialized hardware, cloud infrastructure, or unfamiliar software stacks. These options can be costly, difficult to adopt, or unsuitable for sensitive and proprietary data.

Embedded OLAP engines like @duckdb and @ApacheDataFusio offer a practical alternative. dbverse shows how we can use them to execute larger-than-memory scientific operations while preserving the APIs researchers rely on, enabling new kinds of analyses on ordinary computers.
13/🧵

[Eddie Ruiz @Ed2uiz]

dbverse builds on @duckdb, @ApacheDataFusio, @ApacheArrow, and many packages from the #RStats and #Bioconductor communities.

Interested in bringing dbverse to Python or Julia? I’d love to connect, especially with people in the Narwhals and @Tidierjl communities 👀

15/🧵

[Eddie Ruiz @Ed2uiz]

Using this approach, we observed greater proximal poly(A)-site usage in tumor-associated epithelial cells (TAEpi), consistent with 3′ UTR shortening. We also observed shifts in poly(A)-site usage among cancer-associated fibroblasts (CAFs) associated with their proximity to TAEpi cells in the Visium HD 3′ sample.

12/🧵

[Eddie Ruiz @Ed2uiz]

Can modern databases scale scientific analyses beyond RAM?

During my PhD, I explored whether matrix, spatial, and genomic operations could share a database execution layer without changing the APIs scientists rely on.

Today, I’m excited to finally share dbverse, a composable database framework built around that idea.

Across benchmarks, dbverse delivers up to 1,000× speedups and scales end-to-end spatial omics analyses to millions of cells on a 16 GB M4 Mac mini.

Paper: https://t.co/E8QGJZoOYC

1/🧵

[Eddie Ruiz @Ed2uiz]

On the full 200,769-cell OCCC dataset, GiottoSeq was 3.1× faster than movAPA (Ye et al. 2021). At 2.1 million cells, GiottoSeq completed while movAPA ran out of memory.

11/🧵

[Eddie Ruiz @Ed2uiz]

Next, we asked whether the same framework could scale alternative polyadenylation analysis. We evaluated relative usage of distal poly(A) sites (RUD), a proxy for 3′ UTR length.

3′ UTRs are interesting because they harbor regulatory motifs that influence mRNA and protein fate. Greater proximal poly(A)-site usage can exclude some of these motifs from the mature transcript.

Work from @Mayr_Christine offers excellent context for why this biology matters.

10/🧵

[Eddie Ruiz @Ed2uiz]

We first tested whether the database-backed Giotto object could scale an end-to-end spatial omics workflow.

GiottoDB completed the full workflow for datasets up to 10 million cells. The in-memory backend completed 3 million cells but ran out of memory (OOM) at 10 million. The HDF5-backed TENxMatrix path was ~39× slower than GiottoDB at 100,000 cells and exceeded the 1,000-second time limit for datasets of 1 million cells or more.

9/🧵

[Eddie Ruiz @Ed2uiz]

Benchmarks of individual operations were promising. But the harder test was whether the same database layer could support a real, multi-step biological analysis across data types.

To test this, we integrated dbverse with the @GiottoSpatial framework through two new R packages, GiottoDB and GiottoSeq.

Together, they bring database-backed execution and query pushdown to joint analysis of gene expression and alternative polyadenylation in a @10xGenomics Visium HD 3′ ovarian clear cell carcinoma (OCCC) sample.

8/🧵

[Eddie Ruiz @Ed2uiz]

Genomic operations showed a similar scaling pattern.

dbSequence was the only method to compute the overlap between a fixed chromosomal region and up to one billion genomic intervals within our 600-second time window. It outperformed GenomicRanges and plyranges beyond one million genomic intervals.

7/🧵

[Eddie Ruiz @Ed2uiz]

The speedups were even larger for spatial operations.

We evaluated the time to intersect up to one billion points (representing molecules in a tissue) with 100,000 polygons (representing 2 µm square bins in a @10xGenomics Visium HD sample).

The sf R package failed to complete the operation within our 600-second timeout window at the two largest tested sizes, whereas dbSpatial completed both in under 11 seconds.

6/🧵

[Eddie Ruiz @Ed2uiz]

But do OLAP databases actually improve performance of core scientific operations?

To answer this, we benchmarked core matrix, spatial, and genomic operations used throughout single-cell and spatial omics, comparing dbverse with commonly used in-memory and file-backed R packages.

First, we calculated column means in sparse matrices with up to 2 billion nonzero values. At the largest scale, dbMatrix was fastest and up to 3× faster than BPCells.

5/🧵

[Eddie Ruiz @Ed2uiz]

The dbverse framework consists of four standalone libraries.

Matrices ➡️ dbMatrix
Spatial geometries ➡️ dbSpatial
Genomic intervals ➡️ dbSequence
Connections and projects ➡️ dbProject

Each works independently, but all can operate on data in the same OLAP engine. Among other benefits, one analysis can query matrix, spatial, and genomic data without repeatedly materializing the data as in-memory objects.

4/🧵

[Eddie Ruiz @Ed2uiz]

To address this problem, dbverse represents scientific data structures as object-relational mappings (ORMs) within embedded OLAP engines such as @duckdb and @ApacheDataFusio.

The key idea 💡
Calls through familiar scientific APIs are translated into optimized SQL queries that execute locally, even when the data exceed RAM.

Because this translation happens behind the API, dbverse functions can serve as drop-in replacements for over 100 operations from commonly used R packages.

3/🧵

[Eddie Ruiz @Ed2uiz]

Scientific datasets now frequently exceed RAM on ordinary computers.

Spatial omics datasets illustrate this problem clearly, with datasets often 10–100× larger than earlier bulk and single-cell omics datasets.

2/🧵