A Trillion Databases, and Not One for Analytics: Why DuckDB Exists
In 2019, SQLite had over a trillion deployments and analytics had nothing like it. A SIGMOD paper named the gap, DuckDB filled it - and hardware quietly made single-machine analytics the default.
On this page 4 sections
Processing data with sophisticated queries is a systems problem that gets harder in two directions at once. A complex query - joins, aggregations, window functions - has to hold intermediate state across dependent pieces of data, so the memory it needs grows with both the complexity of the query and the scale of the data underneath it. For decades, the accepted answer was to ship the data somewhere big: a warehouse, a cluster, a database server on the other side of a network connection.
The empty quadrant
In 2019, two researchers at CWI Amsterdam published a SIGMOD paper that started from a simple observation: SQLite is the most widely deployed SQL engine on the planet, with more than a trillion databases in active use. That number is proof of an enormous appetite for in-process data management - a database that is just a library linked into your application, no server, no setup. But SQLite is built for transactions: a row-major engine on B-tree storage. Ask it analytical questions and it struggles.
Draw the landscape as a 2x2 - embedded vs standalone, transactional vs analytical - and three quadrants were crowded. Embedded OLTP had SQLite. Standalone OLTP had Postgres and MySQL. Standalone OLAP had the warehouses. Embedded OLAP was empty. The paper named four requirements for whatever would fill it:
- Fast for analytics without abandoning transactions - in a live dashboard, some threads update the data while others run the aggregations that drive the charts.
- Efficient table transfer in and out - the database and the application share one address space, and that opportunity must be exploited instead of serializing data across a protocol.
- A high degree of stability - an embedded database that crashes takes the host process down with it. Out-of-memory can never mean a crash; queries must abort cleanly.
- Practical embeddability - it has to run wherever the host runs, with no external dependencies at compile time or runtime.
How DuckDB answers
DuckDB’s design is deliberately unglamorous - the paper itself notes that none of its components is revolutionary. Each one is simply the best-known technique for the job, chosen under the constraints above: a SQL parser derived from Postgres, because the parser handles the most volatile input a database receives; a cost-based optimizer with dynamic-programming join reordering and subquery flattening; a vectorized execution engine that processes columnar batches of values (1024 per vector in the original engine) - chosen over JIT compilation precisely because JIT drags in LLVM, and requirement 4 forbids heavyweight dependencies; compressed columnar storage whose blocks carry min/max indexes so scans skip data that cannot match; and serializable MVCC so concurrent updates and analytics coexist. The entire system links into your process and stores a whole database in a single file.
The lesson in that list: when the requirements are sharp enough, assembling proven parts beats inventing new ones.
The physics changed underneath us
Werner Vogels made the second half of this argument in a recent essay. The flagship cloud server of 2005 had 15 GB of RAM and 4 cores; today’s largest instances have roughly 50x more of everything, and the laptop you are reading this on has around 40x the memory bandwidth and 100x the I/O of that 2005 server. Meanwhile, most datasets grew with business metrics - customers, transactions - not with Moore’s law. His conclusion: the “physics” that justified shipping data to a distributed cluster no longer holds for most real workloads. They fit on one machine now, and often in one process. Vogels calls DuckDB “the glibc of structured data” - analytics becomes something you do continuously as you build, not a separate activity that happens somewhere else.
The takeaway
- The biggest gaps hide in quadrant maps, not benchmarks. A trillion deployments on one side of a 2x2 and an empty box next to it is the clearest product spec ever written. When setting platform strategy, draw the map your industry lives on and stare at the empty square.
- Every architecture encodes the hardware assumptions of the year it was designed. When the physics shift 50x, revisit the architecture - plenty of teams still pay the distributed-systems tax in coordination, network hops, and operations for datasets that fit in one machine’s RAM. Right-size before you distribute.