Epsio Blog

Releasing ioflame: an eBPF based I/O tracer and visualizer

Kobi Grossman — Thu, 20 Nov 2025 07:51:30 GMT

We’re happy to release ioflame, an I/O tracing tool that we developed internally to debug high I/O latency and wait time. ioflame attaches to the Linux kernel block level with eBPF probes, tracing all I/O activity. The resulting output of the trace is a flamegraph organized by filesystem hierarchy, showing the exact amount of bytes read or written by directory.
Here’s an example output from the repository, showing reads of some files not cached in the page cache:

In this post, we’ll explain why we needed to build this tool for Epsio, and do a deep dive into the Linux filesystem and I/O stack to understand why that’s a non trivial problem.

Profiling is.. easy?

We’re building a Streaming SQL Engine, which is a SQL Engine that works on live changes to input data instead of having all the data up front in query time. Like any database, we care about performance. Thus we spend time profiling benchmark queries, trying to find our current bottlenecks. But what even is a bottleneck? What makes a database fast or slow?

There are 3 main resources that every software uses: CPU, Memory, and disk. In broad strokes, performance problems stem from inefficiently using one of these resources. Thus to improve performance, we need two things: A good understanding of how each of these resources work, and a good visibility on how we’re using them in practice. When we have an intuition on how something should optimally work, and knowledge on how far are we from that optimum in practice, we can start thinking on improvements we can make to our code. To look at a concrete case, let’s look at the CPU.

Understanding the CPU is understanding multicore parallelism, what are the standard algorithms for computing what we need to compute with a good time complexity, and a myriad of important low level details which are beyond the focus of this post.
From the visibility perspective, we have great tools to see what’s going on in real time. At a high level, we can get basic utilization percentages for each core. We can then dig deeper with flamegraphs from data recorded by a profiler, as well as per CPU performance counters, to get high resolution data on what our CPUs are doing.

For example, we can look at this zoomed in part of a flamegraph of one of our runs:

There’s some noise here, but we can see that we spend most of our time in this section on writing to RocksDB SSTs, with serialization and allocation also taking a big part of the runtime. This is very useful information! We can start asking questions like: Are we writing more data than we should? Can we avoid serialization? Can we reduce our allocations done in the hot path? etc. So when it comes to the CPU, we have good visibility that enables us to improve our performance.

A database has another crucial resource: Disk. We can understand it well, and know that i.e it is a block based device, that random writes on SSDs are not a great idea and so on. But what about visibility?

We can get a high level view and see that we’re blocked on I/O via CPU utilization data, and specifically wait%, which is the % of time in which the CPU sits idle and waits for I/O to complete. For example, consider the following graph:

In it we can see a ~60% wait. This means that most of the time, instead of doing something useful, the CPUs are waiting for I/O. We can also look at the disk queue, which is where requests are waiting to be sent to the drive, and see that it’s packed:

But knowing we wait on I/O is only half the way there. We want to see why we’re doing that much I/O. In the CPU case, a high utilization is a good global indicator, but the flamegraph helped us really understand what’s going on. How can we do that when profiling I/O?
That’s where things get complicated, and that’s the issue we set out to solve with ioflame.

I/O tracing: a surprisingly hard problem

Ideally, we’d like to a level of observability similar to what perf gives us on the CPU, but instead of hot functions we’d like to find hot directories or files. We want to know which files are read or written the most, how many bytes are read or written from each directory in total, etc. so we could discover hotspots. We first tried known tools, but all of them didn’t fit to our specific use case for one of these reasons:

Too high level: iostat is incredibly helpful to get the high level picture, for example whether the problem is many reads or many writes. But what are we doing that’s causing the I/O? That’s hard to tell from iostat alone. Other tools like biosnoop provided more information, like PID and latency, but no information about the file level.
Can’t distinguish page cache operations from actual I/O: One can imagine using strace with some wrapper script to trace all the read and write syscalls a program does. This would work in theory, but won’t tell us much about actual I/O done!

That’s because we won’t know if a read was served from the Page cache or the disk, or which of the writes was flushed to the block device.
Since we’d naturally see high wait% when the page cache is close to full, the thrashing in this case would aggravate the problem: A small % of reads can access files whose pages aren’t cached at read time, but cause the most I/O and thus the high wait%.
This will be hard to tell if we track both cached and uncached reads together.

This behavior exists in eBPF based tools like filetop. Similar to iostat, filetop and similar tools were great for an initial overview of where we might have problems, but wasn’t precise enough for our use case.
Not tracking all types of I/O: File I/O is not necessarily the result of read, write or fsync on a file descriptor. For example, we use mmap buffers as a simple disk backed channel between operators, to avoid OOMs when large amounts of data are involved. While we’re sure we want to phase them out for this use case in the future, when we’re debugging high wait% currently we‘d like to know if mmap buffers are the cause.

The problem is that mmap buffers are almost invisible to observability tools, most of which operate at the filesystem or syscall level. That’s because a read from mmaped memory is a simple pointer dereference. If the read page is not mapped, a page fault will occur, and the kernel page fault handler will issue a read from the filesystem to get the page from disk. All of this happens deep in the kernel, with no direct userspace action we can reasonably hook.

I should stress that we used some of these tools together to understand what’s going on in high wait% cases, and each tool worked great. But combining them didn’t work for us at scale. In complicated benchmark queries we can have tens of RocksDB directories, each with hundreds of SST files, as well as many mmap buffers. All of these being read, written and deleted at a rapid pace. For these cases the combination of tools and heuristics wasn’t enough.

We wanted a single tool that has low overhead, catches any type of I/O, attributes it to a path, and displays the result in an easy to understand visualization. In short, the experience we’ve gotten used to from CPU profilers.
We experimented with ad-hoc bpftrace scripts that used ext4 probes, but we wanted a proper tool that worked on any filesystem and environment. So we wrote ioflame.
But how does it work? Where do we want to put a probe, and what data should we record in it?

To answer that we’ll need to understand the Linux I/O stack a bit better, which we’ll do by tracing the path a read takes from userspace to the block layer. We won’t cover the write path, but the concepts are similar.

Life of a `read`

Let’s say that userspace issues a read syscall from some file. The read goes then through several layers in the Linux kernel, until it reaches the device driver, descending in the abstraction level from files to disk blocks. The layers are the syscall, VFS, page cache, filesystem code and finally the block layer.
Here’s a diagram of the path through the different layers, with the relevant kernel functions in each layer. We’ll explain each layer in detail next:

The diagram was created by printing the kernel stack at submit_bio with bpftrace, at the read time.

Before we get into each layer, let’s understand what we need of our hook.
We need it to be at a layer low enough so that it captures all I/O and doesn’t run on page cache hits, but we also need it to record high level information about which file was accessed. We’d also like it to be as generic as possible, not relying on specific filesystems or disk types.

Given these requirements, let’s look at the different layers and see what they give us. Note that the same layer can be called in different levels of the stack. For instance ext4 is called both directly by the generic VFS logic, and then in the case of a cache miss by the page cache. So the different layers are:

Syscall: The first path running the kernel as a result of our read.
This isn’t a good place to hook: there are many different syscalls that eventually result in disk I/O, and at this stage we won’t know if the read is from cached data or not.
VFS: The next layer is VFS. This layer sits between the syscall layer and the actual filesystem code. It provides a common abstraction that all filesystems use, so that the syscall layer doesn’t need to know the filesystem specifics. It also provides common helpers that all filesystems can use. For example, generic_file_read_iter above connecting between the filesystem and the page cache logic.

While this layer is an awesome abstraction in general, for our purposes it’s a bit too high in the call stack. In vfs_read or in the VFS helpers we still don’t know whether the pages are read from the page cache or not.
ext4: The filesystem specific logic. This could be a great place to hook, for instance hooking ext4_readahead right which is called right before issuing the block layer request sounds exactly like what we need. That’s because at this point in time we know that we had a cache miss, since the page cache code called the read logic.

One problem with hooking this layer is that we would need to have a per filesystem hook. Since we want to be as generic as possible, that’s a no go for us. Ideally, we’d like different filesystems to work out of the box with our hook, without adapting per filesystem.

Another problem is the fact that not all filesystem calls correspond to actual I/O. One example is truncated files, or the more general concept of “holes” in files. These are parts of the file that are marked by the filesystem to be filled with zeros. There aren’t actual zero bytes stored on the block device itself though, since a metadata marker indicating a hole is enough. Thus a read of such a hole, even though it does go through the filesystem logic, won’t actually trigger any I/O (except for metadata, which is tiny and is probably cached anyway).

So the filesystem specific code isn’t ideal for a hook. What else do we got?
Page cache: Part of the complex memory management subsystem. Called by VFS helpers to try and get the read page from the page cache. In case of a miss, it initiates a read from the filesystem. This layer is close to ideal, and a read_pages hook was used in our initial implementation.
The only real problem with it is the hole problem mentioned above. At this point, we only know that a page isn’t resident in the cache and needs to be read from disk. But what if this page is part of a file that was just truncated? We’ll mark the read as actual I/O done, when it isn’t.

This isn’t that big of a deal, and this approach would work, but this is an issue we would encounter in our usage of mmap buffers, for example. We decided to dig deeper and try to find a hook which doesn’t have this problem. This brings us to the..
Block layer: The last level before actual device drivers, sending requests from the filesystem to read individual blocks. This sounds like the perfect place to hook. We can put a probe on submit_bio, which by definition sends I/O requests to the block device, so we know we are actually reading from disk. There is one problem though: since we are in a very low level in the stack, it’s hard to relate I/O to actual files it’s done on, which was our point all along!

In upper layers of the stack, for instance in the page cache miss code, we have the file struct easily available to us. By design, these layers work on files, not blocks, so it’s not surprising that it’s easy to relate a page miss to a file. In the block layer however, all types of I/O are mixed — general filesystem metadata blocks, journal records, and actual data blocks which we care about. So it makes sense that a relation to a file, if there is one, would be harder to obtain.

Luckily, extracting file information in the block layer is hard but not impossible, as we’ll see later. But first, what data should we even use to relate the operation to a file? Is there any file information this close to the device driver?

Associating bio requests with files

In Linux, each file is associated with a unique inode. The file inode contains the metadata about the file, like its size and permissions. But most importantly it allows us to uniquely identify a file as a tuple of 3 fields of the inode: The inode number i_ino, the block device in which the inode resides s_dev, and the inode generation i_generation.

This means that if we can a reference to an inode struct in submit_bio somehow, we’d be almost done! We’ll only need to extract the amount of bytes read or written and whether it’s a read or a write. So what’s the signature of submit_bio?

void submit_bio(struct bio *bio)

Hmm, not very telling. Looking at struct bio members, unsurprisingly there is no inode field we could read, which makes sense due to the block layer level of abstraction as we discussed above. There was also no file struct member or anything like that allows us to get the inode..

But then we remembered something. After all, we’re reading from a file into a page. Thus there must be some page struct somewhere in the bio that maybe could have the backing inode. And indeed, we found this in the kernel code:

/*
 * main unit of I/O for the block layer and lower layers (ie drivers and
 * stacking drivers)
 */
struct bio {
  ...
  struct bio_vec  *bi_io_vec; /* the actual vec list
};
...
/**
 * struct bio_vec - a contiguous range of physical memory addresses
 * @bv_page:   First page associated with the address range.
 * @bv_len:    Number of bytes in the address range.
 * @bv_offset: Start of the address range relative to the start of @bv_page.
 * ...
 */
struct bio_vec {
 struct page *bv_page;
 unsigned int bv_len;
 unsigned int bv_offset;
};

So we can easily find the page accessed in the bio, great! Getting from the page to the inode is easy: page has a member of type struct address_space *, which has a pointer to the inode.

Tracking Direct I/O is a bit more complex to track as the read page is a userspace buffer page and not a file backed page, but we resolved that by hooking another bio submission function which has the file inode in Direct I/O.

To distinguish file backed I/O requests from general metadata I/O that we don’t want to track, we can use the same logic the kernel uses to detect if a page is anonymous, i.e not backed by a file: if the lower bit of the address_space isn’t set, we know that the page is backed a file and we can proceed.

Finally, we write the read or written bytes to an eBPF percpu map, keyed by the inode. After reading a 1GB file, the map would look like this:

This is great, but something is missing. What we eventually want is paths, not inode numbers. How can we get that?

To userspace and beyond

An inode struct is a unique handle to a file with metadata and not much more. It doesn’t have the file path, since paths are at a higher level of abstraction. Many paths even point to the same inode via hardlinks. Thus finding the path from the inode just doesn’t make sense. This means we’ll need to hook some location right after opening a file with a path, which will have both the inode and path, then store it in a mapping like so:

In general, resolving the path of a file in eBPF isn’t trivial due to the eBPF verifier (resolving a path is a loop which can’t be proven to terminate), so we opted for the simpler solution of resolving it in userspace with procfs: In kernel space, we’ll hook a file open function, which will give us both the inode opened, the PID who opened it and the fd it got. Then in userspace we’ll read the path from /proc//fd.

We decided to hook fd_install, which gets called whenever a new fd is opened by any process. Here we have all we need: a struct file containing the inode, with the pid and fd also available. We send this to userspace via an eBPF ringbuffer, which then resolves the path as described above, and stores it in the mapping.

In userspace we link the information from both maps, use inferno to produce a flamegraph per directory, and we’re done!

Wrapping up

The initial version of ioflame is on Github. With this basic architecture in place, we can imagine other cool features we’d like to add:

Tracking over time, allowing comparing I/O patterns over time with something like a differential flamegraph
Adding a richer UI to enable better filtering
Adding probes to track page cache inserts and evicts, to produce a visualization of page cache residency over time

But all of that is in the future. For now we finally have the I/O visualizer we needed.
btw, did we mention we’re building a cool Streaming SQL Engine?

I love UUID, I hate UUID

Maor Kern — Tue, 09 Sep 2025 12:36:37 GMT

At Epsio we’re building a streaming SQL engine which natively integrates with databases/warehouses (both as “source”s and “sink”s). Epsio acts as a stream processing “middle man“- receiving data from one (or more) sources, applying transformations (according to the users’ queries), and sinking results into multiple tables in the users’ sink database. As part of this, we make it a priority to optimize database interactions so they run with maximum efficiency and impose minimal overhead on our users’ databases.

To sink data into the users databases, Epsio needs to be able to insert and delete rows efficiently. Inserting rows is fairly easy- with most databases we simply do INSERT INTO (or perhaps COPY .. for large amounts of data). Deletions, however, are more complicated. We could perhaps delete by all the fields DELETE FROM pasten WHERE user=nadav AND age=21 AND … - but a few issues would arise:

It’s crazy inefficient- the users’ database would need to do a lot of work to find that row, given you don’t want a massive index on all the tables fields
What if there are multiple rows with the same fields, but Epsio only wants to delete one? Some databases give this option, but Postgres for example does not.

Enter primary keys.

Primary keys are a cornerstone for any relational table. They give the ability to have a handle to a specific row, sort of like a pointer. The primary key is usually indexed, which means that given a primary key, the database can quickly jump to it’s corresponding row very quickly. These keys are often used to perform quick UPDATES/DELETES, and also as foreign keys for other tables to point to them.

There are multiple data types used for primary keys. The main two types are:

Auto incrementing integers - every new row inserted receives the “next“ number, starting at 0
UUIDs (128 bits) - every row receives a randomly generated UUID.

Auto incrementing keys must be generated by the server, as given two clients inserting at the same time the server must be the one to coordinate which one will receive which number.

UUIDs, however, can be generated by the client. This is because the probability of a UUID collision is astronomically low- so low in fact that most systems rely on them to be absolutely unique.

But why does it even matter if the client is the one creating the identifier? To answer this, take for example an application that allows users to make posts, comment and edit them. If the client is the one creating an identifier, the application could be “optimistic“ about the server successfully receiving the post, and allow the user to edit/comment on the post even before the server has responded with it’s OK. This is because the client already has the identifier of the entity it’s creating, even before the server responded.

At Epsio, we also leverage UUIDs for primary keys in the users’ databases. This is mainly because there are situations such as COPY … in Postgres where you can’t receive back auto incremented keys from rows inserted. UUIDs allow us to generate identifiers before the insert, and later use those identifiers to delete from the database.

So up till now, UUIDs may sound like a gift from the Statistical Lords of the Universe, but there are some serious caveats.

The Problem with Indexing UUIDs (and specifically, UUIDv4)

The reason primary keys are so efficient for lookups is because they are indexed. This means the database is holding some type of data structure, like a B-Tree, with all the UUIDs (with the data perhaps being the physical location of the row). B-Trees work very well when the writes are as condensed as possible; as the writes become more scattered, the B-Tree hits more leafs, and thus more pages need to be brought into cache and more splits occur. Sequential numbers added into a B-Tree is the best situation for a B-Tree- you’re always hitting the right most leaf, filling it up, then splitting, etc. The default UUID type (UUIDv4) that is used by default in most DBs / clients though, is the opposite- it’s the worst for a B-Tree. The very things that makes UUIDs so magical- their global uniqueness- also means they’re completely random numbers across a 122bit spectrum. To put it in perspective, it’s range is wider than there are atoms in the universe!

This matters because inserts will suffer a serious hit, while lookups may suffer slightly. This means our throughput for insertions will go down, and the database will need to work harder. Not great.

Enter UUIDv7

UUIDv4 is fairly simple- as stated above, it’s simply a random number in a 122bit spectrum. UUIDv7 however, is a bit more interesting- it’s first 48 bits are used to contain the current timestamp (IE the timestamp of when the UUID was generated), a few more bits for version and variant, and 74 bits are used to store a random number. It’s exact layout is as follows

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|            unix_ts_ms (48)            | version (4) |   rand_a (12)   | variant (2) |                   rand_b (62)                                 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

This means that if we generate multiple UUIDv7s in consecutive times (say one millisecond after the other), they’ll all sort one after the other! This solves the index issue, and actually adds a very big benefit: It’s possible to now know the insert time of the row from within the primary key. This can help debugging, and in internal “chaos“ simulators we’ve used this to understand and figure out issues in Epsio.

To give a taste of the performance increase, I made a simple benchmark with my friend Claude to insert 10 million rows with UUIDv4 and then 10 million with UUIDv7. Claude of course proceeded to write code with an abundance of emojis, as one does.

Results:

	UUIDv4	UUIDv7
Index Size	389MB	301MB
Throughput	183K/s	266k/s
Total Time	54.62s	37.53s

Which shows the improvement fairly well- the total index size is 22% smaller with UUIDv7, and in total was 31% faster.

That’s why in Epsio… we use UUIDv7s for primary keys!

But still, there are some tradeoffs that I do want to note here:

UUIDv7 containing the timestamp is not always a good thing. In Epsio, the UUID identifying a row is never exposed to an untrusted user. But if your IDs are being exposed (for example, via REST endpoints) you may be giving a potential attacker information you’d rather them not have (namely, when the row was created).
Less “random”- I debated whether to even write this, since the chances are still astronomically low of a collision. But for all UUIDs generated in the same millisecond, your “random” bits are reduced from 122 to 74. To give a taste for this- if you have 1 million UUIDs generated the same millisecond, the chances of a collision are .0000000003%.

Why is stream processing hard?

Gilad Kleinman — Sun, 10 Aug 2025 15:35:42 GMT

Before starting Epsio a couple of years ago, I used to spend countless hours learning about creative and innovative ways companies were making data access and querying faster and more efficient. Looking at rising products in that space — whether it’s ClickHouse, DuckDB, or others — I was consistently amazed by how much magic and innovation these tools packed into their products while still abstracting away the complexity & while keeping the interface extremely simple.

The streaming space however, told a completely different story. While technical innovation was absolutely booming in this area, it somehow felt like the technology was becoming increasingly more complex and costly to use as the underlying theory & tech evolved.

Streaming is a high-maintenance, complex bitch that offers real-time data and scalability, but it’s costly and a pain to manage; batch processing is your reliable, easy-to-handle workhorse, less resource-intensive but slower and less sexy — choose based on whether you want a race car or a minivan.

(Ill-Valuable6211 in Reddit, “Tradeoffs between batch processing and streaming?”)

A Death By A Thousand Components & Configurations

While in a batch processing world, I could spin up a database and pretty quickly get started, in a stream processing architecture, for me to be able to build a simple feature that counted rows in a PostgreSQL database, I “only” needed:
- Debezium for CDC
- Avro registry for schema of events
- Kafka for change queue
- Flink to run the aggregation
- JDBC Sink to write back the results to PostgreSQL

And, of course — digging deep into many (many) Flink / Kafka / Debezium configurations to make all components work together and learn many new concepts like watermarks, checkpoints, etc…

Why Is Streaming Inherently Hard?

Fundamentally, I do believe stream processing is inherently harder and more complex than batch processing. This is for 2 main reasons:

1 — Stream processors are highly coupled to other data products

By definition, a stream processor is meant to input data, transform it, and push it to downstream data products (DBs/ data stores)— not to be the final “endpoint” for other systems (backend / BI / end-users) to pull data from it.

Unlike a database (/batch processor), that can usually be a full “end-to-end” data solution — ingesting, transforming and serving data to the backend — a stream processor can almost never be implemented without an additional data product (database / batch processor) that enables users to access, index, and search over its output. A stream processor without a database after it is like email without email inboxes — the data flows, but there’s no easy way to view or interact with anything after it’s sent.

This means that unlike the batch processor (that doesn’t need any other component to serve a full end-to-end data product)— the stream processor depends on and must couple and integrate itself with numerous other data product for users to be able to benefit from it’s value.

If your backend reads/ writes to a specific PostgreSQL instance (in a specific PostgreSQL version), even the best stream processor in the world probably won’t be helpful unless it can properly ingest from and sink data to your specific PostgreSQL instance/ version.

Similarly — if your database has a complex partitioned schema that involves many ENUMs and custom columns — no stream processor int the world would be usable unless it can properly handle the scenario of a new partition being added in your database, or a way to process and transform all of your custom-typed columns and ENUMs.

(From Jay Kreps' original vision for Kafka — https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying)

As a live testimony for this — some of the hardest bugs we ever encountered while building Epsio, were actually bugs in other data products we integrated with. Since we needed to rely so heavily on the specific format and way each data store does things — the smallest of glitches in a source / destination database could easily translate into a glitch in Epsio.

2 — Stream processors are batch processors, plus more

Fundamentally, batch processing is just a “subset” of stream processing. A batch processor only works on the CURRENT data you have. A stream must both process the CURRENT data you have, and also “prepare” (/build structures) for any FUTURE data that arrives.

A great example of this difference is how JOIN operations are handled (specifically, hash joins). In batch processing, since the system knows it will never receive additional data, it can optimize the JOIN by building a hash table only for the “smaller” side of the join — say, the left side — and then iterate over the current rows in the larger side (the right) in an un-orderly fashion, constantly looking up the corresponding rows in the other side’s hash table.

A stream processor in contrast, needs to build not only a hashtable for the small side, but ALSO a hashtable for the larger side of the JOIN — to make sure it can quickly lookup corresponding rows if any change in the future is made to the smaller side of the JOIN. It needs to build all the structures a batch processor needs for the same operation, plus more.

(Some stream processing systems — or similar systems like TimescaleDB’s continuous aggregates — mitigate this overhead by limiting support for processing updates to only one side of the join).

Even after building these additional structures, stream processors still must “take care” of additional edge cases batch processors don’t need to. Stream processors for example, many times must guarantee that the order they received data correlates with the order they output data. While a batch processor receives a single batch of data and outputs a single batch — a stream processor that constantly receives a stream of changes, needs to make sure the output stream of changes correlates to the order of input stream of changes (image how hard that could be when trying to process that stream in parallel!). A row that got INSERTed and then DELETEd must never be DELETEd in downstream systems before it got INSERTed.

As evidence of a stream processor being “a batch processor, plus more”, you can frequently find stream processing systems that also have support for batch queries (e.g., Flink, Materialize, and Epsio soon ;)), while no batch processor supports streaming queries. To build a batch processor — simply take a stream processor and remove some components!

How Were These Difficulties Overcome Until Today?

Historically, the creator of Kafka (Jay Kreps) talked about tackling the difficulties of stream processing by breaking down these huge complexities into many small “bite size” components, thereby making it easier to build each one of them: “By cutting back to a single query type or use case each system is able to bring its scope down into the set of things that are feasible to build”. Specifically, he portrayed a world where many “small” open source tools play together to form the complete “data infrastructure layer”:

Data infrastructure could be unbundled into a collection of services and application-facing system apis. You already see this happening to a certain extent in the Java stack:
‍
‍- Zookeeper handles much of the system co-ordination (perhaps with a bit of help from higher-level abstractions like Helix or Curator).*
- Mesos* and YARN do process virtualization and resource management
Embedded libraries like Lucene and LevelDB do indexing*
- Netty*, Jetty and higher-level wrappers like Finagle and rest.li handle remote communication*
- Avro*, Protocol Buffers, Thrift, and umpteen zillion other libraries handle serialization*
- Kafka* and Bookeeper *provide a backing log.

If you stack these things in a pile and squint a bit, it starts to look a bit like a lego version of distributed data system engineering. You can piece these ingredients together to create a vast array of possible systems. This is clearly not a story relevant to end-users who presumably care primarily more about the API then how it is implemented, but it might be a path towards getting the simplicity of the single system in a more diverse and modular world that continues to evolve. If the implementation time for a distributed system goes from years to weeks because reliable, flexible building blocks emerge, then the pressure to coalesce into a single monolithic system disappears.*

Even though this architecture definitely succeeded in improving the “specialization” of specific components and advanced the capabilities of the streaming ecosystem dramatically — it kind of feels like somewhere along the road, the end user that was supposed to be safeguarded from this implementation choice (“This is clearly not a story relevant to end-users who presumably care primarily more about the API then how it is implemented”.) ended up paying the price.

In today’s world — there is nearly no technical barrier for great stream processing. Whether it’s scale, complexity or robustness — nearly any application can be built with the tools the market has to offer. The interesting question though is — at what cost?

Even setting aside the sheer number of components users must manage — Debezium, Kafka Connect, Kafka, Flink, Avro Schema Registry, and more — the internal knowledge required to operate these tools effectively is staggering. Whether it’s understanding how Flink parallelizes jobs (does an average user need to understand how PostgreSQL parallelizes queries?), thinking about the subtleties of checkpointing, watermarks, or how to propagate schema changes to transformations — the end users is definitely not abstracted away from the internal implementation choice.

A New Generation Of Stream Processors

Innovation is dirty. And perhaps this historical unbundling and complexity must have taken place in order for us to reach where we are today and to overcome all these difficulties. But similar to the evolution of batch processors — where the “dirty” and “complex” innovation of Oracle must have taken place to allow the rise of the easy to use PostgreSQL, in the past couple of years — a new generation of stream processors is now rising that focus their innovation on the end user, not just technical capabilities.

Similar to the movement from Oracle to PostgreSQL — stream processors in this new generation are not necessarily less internally “complex” than the previous ones. They compound all the “internal” innovation the previous generation had, while somehow still abstracting away that innovation from the end user.

In this new generation — simplicity is the default and complexity a choice. An end user is able dive deep into the complexities and configurations of the stream processor if he wishes to — but can always have the option to use a default version/configurations that “just work” for 99% of the use cases.

For this new generation of stream processors to be built, a radical change in the underlying technology had to happen. The endless process of stitching and gluing together generic old components was broken — and a new paradigm, a much more holistic one, had to be built.

New stream processing algorithms, with much stronger end-to-end guarantees (differential dataflow & DBSP with their internal consistency promises) are used and top-tier replication practices inspired by great replication tools like peerdb, fivetran, etc… are incorporated.

In this new generation of stream processors, each component is designed specifically for it’s role in the larger system — and with strong “awareness” to all the external systems and integrations it must serve.

The concept of “database transactions,” for example, is enforced in all internal components in order to abstract away ordering and correctness issues from the user. While stream processors in the old generation (Debezium, Flink, etc.) by default lose the bundling of transactions when processing changes (meaning large operations performed atomically on source databases might not appear atomically on sinked databases) — our new stream processor must (and does!) automatically translate a single transaction at the source database to a single “transaction” in the transformation layer, and into a single “transaction” in the sink database.

The transformation layer is aware of the source database it transforms data from so it can build the most optimal query plans (based on schemas, table statistics, etc..), and the replication layer is aware of the transformation layer (so it can automatically start replicating new tables that new transformations require).

Parallelism, fault tolerance, and schema evolution are all abstracted away from the user — and a single simple interface is built to control all the stream processors mechanism (ingesting data, transforming data, sinking data).

Moving Forward

Streaming is hard. But as the barrier of adoption in the market shifted from a theoretical one (is reliable stream processing at scale even possible / a real solution with real use cases?) to a usability one (how hard is it for me to use?) — the underlying technologies that power streaming had to and need to change.

Although probably always harder than batch processing, if we’re able to truly make stream processing (almost) as easy as batch processing — the implications to the data world would be staggering. Heavy queries/pre-calculation would be easily pre-calculated (and kept up-to-date) using incremental materialized views, data movement between data stores would be a solved issue, dbt model/ETL processes could be made incremental using a streaming engine, and front ends would be much more reactive (e.g. https://skiplabs.io/). The cost of data stacks will drop dramatically, and engineers would be able to focus much more on what they are supposed to — building application logic!

Similar to how Snowflake revolutionized the data warehousing world with its ease of use and scalability — I truly believe as a software engineer that an “easy to implement” yet robust stream processor can break the existing trade-off between batch processing and stream processing (easy to use vs. real-time/performant) — and unlock a new and exciting world in the data ecosystem!

The Bidirectional Stream Processor: Why Pull Beats Push for Crash Recovery

Gilad Kleinman — Tue, 05 Aug 2025 11:43:15 GMT

At Epsio — we’re building a streaming SQL engine, focusing our engineering effort towards ease of use / removing the middleware usually associated with stream processing.

As part of that effort — we need to ensure that if and when our stream processor crashes, no data duplication occurs. This means that even in the scenario of a disk malfunction, network issues, unexpected compute restart — there shouldn't be duplications in the output stream. Each inputted change should be received once, processed once, and its derived change should also be outputted only once to the component after the stream processor.

As you’ll see in this blog post, certain scenarios can make this especially challenging — particularly when it’s difficult to determine exactly what the stream processor managed to process or output before it crashed. Since our mechanism can’t depend on any local disk or state that might be lost during a crash, we had to get a bit creative in figuring out which messages had already been sent by the stream processor — and which hadn’t.

Problematic Scenario #1 — Connection to Sink Breaking

The first (and easier to handle) failure scenario is when the connection to the sink breaks during a batch write or delete operation. This can happen either because of a network hiccup, a sink/database restart, etc..

When this failure occurs (e.g., an insert operation receives a timeout or a TCP connection breaks forcefully mid insert), the stream processor is left in the dark: did the sink manage to apply any of the writes before the disconnect? If we retry the operation blindly, we risk duplicating data. But if we skip it, we could lose records altogether.

To handle this, we employ one of two strategies, depending on the capabilities of the target sink:

Utilizing transactions

Luckily, many of the data sinks we work with have transactions built-in. This makes life much easier because we can apply a batch of changes in bulk, and if our connection breaks in the middle — all is good because the transaction wasn’t committed yet! For example:

BEGIN;
INSERT 
INSERT 
INSERT 
COMMIT;

If for some reason, the server disconnects between insert2 and insert3 — we know the previous inserts didn’t get committed yet — so we can just restart the entire write operation when we re-establish the connection and commit. Great, right? No need to worry about duplicate writes anymore?

…

Wrong!

In the world of data processing, we need to be prepared for failure at ANY POINT IN TIME. This include the specific time when we run the COMMIT command. If the connection breaks while we perform the COMMIT — how can we know if the transaction was committed (and not that the reply confirming the COMMIT just never reached us?), or if we need to re-apply the transaction?

You might think this scenario is fairly rare (how often does a database restart? And how unlucky must one be to have a restart exactly on the commit command) — but this is something that actually happened many times for some of our users (when running many views and writing into many data stores that frequently upgrade/restart due to scaling events).

Surprisingly, some of the most famous stream processors (Flink included!) market themselves as “exactly once” stream processors, but can’t handle this edge case:

“If a commit fails (for example, due to an intermittent network issue), the entire Flink application fails, restarts according to the user’s restart strategy, and there is another commit attempt. This process is critical because if the commit does not eventually succeed, data loss occurs.” — An Overview of End-to-End Exactly-Once Processing in Apache Flink

Epsio, on the other hand, overcomes this by utilizing a built-in feature many data stores have to check if a specific transaction ID was committed. This means that when the engine needs to decide whether to re-apply the changes to the sink, it can just check if the previous transaction ID was committed or not. For example, in PostgreSQL:

BEGIN;
SELECT xid_current(); -- Save this for later. If commit fails, check txid if was committed
INSERT 
INSERT 
INSERT 
COMMIT;

If the COMMIT command fails, Epsio simply checks if the transaction ID was previously committed or not and decides if it needs to re-apply all the changes again.

Making writes idempotent

Sometimes, transactions aren’t available in the data sink — so Epsio has to “cheat” its way into only delivering changes once.

Instead of trying to guarantee that a change is written only once to the datastore, an alternative approach is to make the write process idempotent: if a particular change has already been written, any subsequent attempt to write it again will be safely ignored.

Epsio handles this by assigning a unique identifier (UUIDv7) to each change before it’s sent to the sink. It then relies on unique indexes (or similar constraints) in the target datastore to ensure that duplicate writes are ignored — typically by using an UPSERT or merge operation.

For example, if a batch of three changes is being written to the sink and the connection drops after the first change is written, retrying the batch will result in only the remaining two changes being applied. The first change, already present, will be ignored.

Problematic Scenario #2 — Engine crashes Mid-Processing

A more complex (and interesting!) scenario arises when the stream processor itself crashes or restarts mid-processing.

In this scenario, it’s unclear how the processor should proceed upon recovery. How does it know which changes were already written? How does it continue to ensure no duplicate changes are written to sink, even in the scenario of state data loss?

Traditional stream processing systems — ahem, ahem, Flink — handle this by combining end-to-end transactions with periodic state checkpoints. Here’s how it works:

For each input batch, start a transaction in the sink.
Process, transform and write the batch into the sink (using the transaction opened in previous stage)
After fully processing the batch, back up the internal state (key-value store, input stream offsets, metadata, etc.).
Commit the transaction after all steps are successful.

If a failure occurs mid-processing, a “replica node” (a node sitting in standby) can revert to the last successful checkpoint/backup. Since the transaction isn’t committed while the batch is still being processed, any partially outputted changes from this batch are discarded, and the replica node simply retries processing the entire batch.

The huge (!) downside of this approach — is that the “frequency” of your transactions is highly limited by how fast you can perform checkpoints and backup all your internal state. Consider Confluent’s default checkpoint interval of 1 minute — this overhead means that achieving exactly-once guarantees in Confluent’s Flink implementation currently requires accepting at least 1 minute of latency in your stream processing pipeline.

“Exactly-Once: If you require exactly-once, the latency is roughly one minute and is dominated by the interval at which Kafka transactions are committed” — Delivery Guarantees and Latency in Confluent Cloud for Apache Flink

How does Epsio solve this?

Unlike traditional stream processors that function as a “one-way street,” continuously receiving data from sources and pushing it downstream, Epsio is bidirectional. This means it can not only push data to the sink but also pull data from it when needed.

This bidirectional approach is crucial because it enables failure recovery in ways that traditional stream processors can’t. While traditional processors simply push data to the sink, Epsio can pull data back from the sink during failure scenarios and compare the current state in the sink with its expected internal state — thereby avoiding outputting changes it already outputted.

For example, consider a highly available Epsio deployment with a “Main” instance and a “Replica”. This system receives a stream of changes and multiplies each input integer by 2. To prepare for failure scenarios, the replica will continuously maintains internally the “expected” result state of the sink.

When a failure occurs, the replica will pull the current state from the sink, compare it with its expected state, and apply only the differences (ie new rows that need to be inserted / deleted).

After applying these changes, the replica can just continue ingesting more events from the input stream, and never worry about duplicate data delivery being an issue.

Because this approach doesn’t require any external checkpoint/backup, and more importantly decouples sink writes from state backup frequency, it enables users to achieve strong delivery guarantees without sacrificing streaming latency.

Concluding Thoughts

Historically, stream processors have been built to operate in isolation — generic, self-contained components with minimal awareness of their upstream and downstream counterparts. While this approach has merit, it comes with significant trade-offs.

While there are still many things to talk about in the above “strategy” (how can you deal with the scenario multiple instance crash, how do you deal with a scenario where the sink dataset is too large to efficiently maintain on the instance, etc…) — I hope this article gave you a small glimpse into how stream processors could become significantly easier to use — and more efficient — if they were designed to be more “aware” of the surrounding ecosystem.

How we built a Streaming SQL Engine

Maor Kern — Tue, 01 Apr 2025 21:00:00 GMT

So you probably wake up every morning asking yourself three of life’s most pertinent questions- how do I build a streaming SQL engine, what even is a streaming SQL engine, and can our Lord drop tables owned by another user.

I too found myself asking these questions, sometimes even dreaming about them- often in the form of various SQL operators pointing and laughing at my incompetence as I beg them to answer me.

And so, a year ago, I (quite bravely if I may say so myself) packed my bags and set off on the long and treacherous journey to find answers to these questions. I went from monk to priest to spaghetti-enthusiast, shocked to find themselves concerned with paltry questions such as the Meaning of Life and how to find peace within oneself. But eventually, desolate in the deepest pits of my mind, I happened by a small temple by the name of “Epsio Labs”- a feeling of tremendous revelation came over me, and I walked in.

Friends, today I will share with you the secrets I have found there (despite the numerous NDAs).

What is a Streaming SQL Engine?

A streaming SQL engine keeps queries’ results up to date without ever having to recalculate them, even as the underlying data changes. To explain this, imagine a simple query, such as SELECT count(*) FROM humans . A normal SQL engine (such as Postgres’s, MySQL’s) would need to go over all the different humans every time you ran that query- which could be quite costly and lengthy given our ever changing population count. With a streaming SQL engine, you would define that query once, and the engine would constantly keep the resulting count up to date as new humans were born and the old / sickly ones died off, without ever performing a recalculation of counting all humans in the world.

How to build a Streaming SQL engine

For the simple example above, you may have an idea of how a streaming SQL engine could work- first, you would need to do what any normal SQL engine does and calculate the number of humans. Next, every time a human was born you would add one to your result, and every time a human died you would negate one from your result. Easy, right?
Let’s try creating a diagram showing a procedural “query plan” and how we would process a new human being born. We’ll have a series of nodes, one for each operation, and a “final” node which will represent a table with our results. Since we’re a streaming engine and dealing with changes, we’ll represent the messages passed between our nodes as:

where the key is what we want to change, and the modification is by how much we want to change it. So for example, if we wanted to pass a message to a node telling it “Hey Mr. Node, 1.5 Apples have been added”, it would look like this:

Each node will be responsible for receiving changes, performing some type of operation, and then outputting changes itself. Another important concept here is that we can add modifications together if they have the same key. So the two changes apple: 1.5 and apple: 2 are equivalent to apple: 3.5 . If the modifications equal zero together, it’s as if nothing at all happened- for example, if we have two changes apple: 3 and apple: -3 , it’s equivalent to not having streamed any changed (You can think of it as me giving you three apples, utterly regretting my kindness, then taking away your three apples. For you it would be as if nothing happened at all- aside from your broken pride). To make this make more sense, let’s draw out the nodes for our query (SELECT count(*) FROM humans), and add in the first human, Adam.

Zoom image will be displayed

As we can see, a new human named “Adam” was born. The “Counter” node, perhaps called for its ability to count, holds an internal state of the current count of humans in the world. Whenever it receives a change (an addition or deletion of a human), it updates its internal count, and then outputs relevant changes- in this case, only one change was necessary, telling the next node to add the number one (signifying the total number of humans) once. In this instance, the next node was the “Final Results Table”, an actual honest-to-god relational table (perhaps in Postgres). You can imagine every change translating to a DELETE or INSERT based on the key and modification.

Next, let’s add in Eve:

Zoom image will be displayed

This time, the counter node needed to output two changes- one to cancel out the previous change it outputted, and one two add the new updated value. Essentially, if we were to look at all the changes the Counter node outputted over time, we’d get:

Since we’re allowed to combine changes with the same key, the above is equivalent to:

Zoom image will be displayed

A modification of zero means we can simply remove the change. Therefore, we’re left only with 2: +1 in the final result table- exactly what we wanted.

Are you starting to feel that sweet sweet catharsis? Ya, me too.

A slightly more interesting example

Let’s imagine we’re the devil and have two tables:
1. A “Human Table” that contains two columns, a unique identifier and their name.
2. An “Evil Table” that maps human ids to whether or not they are evil.

Zoom image will be displayed

Now, for obvious reasons, let’s say we wanted a count of evil humans per name:

SELECT humans.name, count(*) FROM humans 
JOIN evil_humans ON humans.id = evil_humans.human_id 
WHERE is_evil IS true 
GROUP BY humans.name

To be able to create a query plan (a series of nodes) from this query, we’re going to need to introduce a few new types of nodes here.

Filter Node

A filter node filters a change by its key, regardless of its modification. If a change “passes” the filter, the filter outputs it as is. To really give you the feel of this, let’s diagram a filter node that passes only changes with keys that are equal to the word “cats”.

As we can see, we gave the filter node 3.4 cats and, astonishingly enough, it passed them on without changing a thing. Let’s try passing dogs through the filter node:

Zoom image will be displayed

Whoa! This time the filter did not pass anything on. I imagine you get the idea. Moving on.

Joins

A Join node is responsible for receiving changes from two nodes, and outputting changes whose “join keys” match. It does this by holding an internal state (all in storage) of all the changes that pass through it from both sides, mapped by their respective join keys. So in our example with

JOIN evil_humans ON humans.id = evil_humans.human_id

we would create one Join node, with two mappings:
On the left side, for id to name
On the right side for human_id to is_evil

In practice, the mappings would look something like this:

Zoom image will be displayed

Every time the Join node would receive a change from one of its sides, it would look on the other side for a matching key. If it found one, it would output the combined values. Let’s try drawing out a simple example where the Join node receives a new human named Tommy with id 232, and then a change saying id 232 is not evil.

First- a new human named Tommy enters the world:

Zoom image will be displayed

Ok, we streamed a change that tells the Join node that Tommy (id 232) has been added. The Join node looks in its right mapping for a corresponding change for key 232, and finds none. It therefore outputs nothing; but it does update its internal mapping to reflect the fact Tommy has been added- this will help us when we do the following:

Zoom image will be displayed

Here, the Join node received a change from its right side telling it “id 232 is not evil”. The Join node then looked at its left mappings, found a corresponding change (232: Tommy- the change we just streamed before) and outputted the combined change.
But this is not the end of the Tommy saga- at any moment new changes could come in. Perhaps Tommy could fall down the steps and die. This would result in the Join receiving Tommy, 232: -1, which would then have the Join node outputting (232, Tommy, false): -1 — cancelling out the previous change the Join sent. Or perhaps Tommy could change in his evilness- we’ll keep that idea for an example down the line.

— -

Side note- you may have noticed we said “join key to change” but don’t actually keep the modification count in the join mapping. In the real world we do, and then multiply the modification counts of both sides to get the outputted modification count

— -

Group Bys

Our “Group By” node is very similar to the counter node we had before (in truth, they are one and the same wearing different hats- but that’s a tale for another time). The Group By node outputs aggregates per buckets- always ensuring that if you were to combine all changes it outputted, you would be left with at most one change per bucket (similar to how we took all the changes over time that came out of the Counter node, and saw that we’re left with only one as others cancel out). It does this by holding an internal mapping (in storage) between each bucket and its aggregated value. So in our example

SELECT humans.name, count(*) ... GROUP BY humans.name

The Group By node would hold a mapping something like this:

Let’s try drawing out a simple example showing what would happen if we entered a new name the Group By node has never seen before:

Zoom image will be displayed

Ok, what happened here? The change coming into the Group By node tells the Group By node to add one to Richard. The Group By node looks in its internal mappings, and sees it has no entry for Richard. It adds an entry, and then outputs that the amount of Richards is one (this is the key of the change- (Richard, 1) ). Let’s go ahead and add another two Richards:

Zoom image will be displayed

Slightly more interesting- the Group By node receives another change telling it to add two to Richard. The Group By node updates its internal state, and then outputs two changes- one to remove the previous change it outputted, and one to add a change with the new updated count of Richards.

Putting it all together

Back to our original query:

SELECT humans.name, count(*) FROM humans 
JOIN evil_humans ON humans.id = evil_humans.human_id 
WHERE is_evil IS true 
GROUP BY humans.name

To set the stage for our upcoming example, imagine a young goody-two-shoes coder named Tommy, id 232 (you may remember him long ago from our explanation about the join). Tommy was a super cool dude who regularly downvoted mean people on StackOverflow (evilness=false).

One day, Tommy got kicked in the head by a horse, and as a direct result force pushed to master while deleting the CI. We’ll represent this occurrence with two changes- one to cancel out the old change saying Tommy wasn’t evil (232, false): -1 , and one to add in the new change saying he is evil(232, true): +1 :

Zoom image will be displayed

Bonus points if you see a nice optimization we can do here- try playing with the order of the nodes :)

Let’s do a quick breakdown of what we see above- so we outputted the two changes we talked about (232, false): -1 and (232, true): +1 . The Join node receives it, looks on its other side (ids -> names), finds a name (Tommy), and outputs the inputted changes together with the name “Tommy”. Next, the filter node WHERE is_evil IS true filters out the change (232, false): -1 , and, since its evil value is false, only outputs (232, true): +1 . The Group By node takes in this change, looks in its mappings, and sees there existed a previous entry for Tommy (while not a very evil name, I have met some mean Tommys in my life). The Group By node therefore sends out one change to remove the old change it sent out with (Tommy, 7): +1 (this happened in a previous addition of an evil Tommy). It then sends out another change introducing the new change to the Tommy count.

Wait, but why go to all the trouble?

So, now there are 8 Tommy’s in the world that are evil, and we didn’t need to rerun our query to calculate this. You may be thinking- well, Mr. Devil, you really didn’t need a streaming SQL engine to do that. If you had a humans table and evilness table, just create indexes on them. You’d still need to go over all the records each time queried, but at least the lookups would be quick. The Group By would still need to do everything from scratch, but at least…
So yes, it is possible to optimize queries so they run fairly quickly on the fly- up to a certain point. As more Joins, Group Bys, and (god forbid) WITH RECURSIVEs are added, it becomes more and more complex to optimize queries. And as more Tommys and Timmies and Edwards and Jennies (not to mention Ricardos and Samuels and Jeffries and Bennies) are added to our system, even those optimizations might begin to not be enough (and don’t even get me started on the evils of in-house caching). Streaming SQL engines are, to paraphrase a non-existent man, totally badass, and straight up solve this.

In Conclusion

Feel ready to build a streaming engine? You definitely have the building blocks- but you’re missing a few major concepts here such as how to be consistent (to never output partial results), how to do this all with high throughput (in everlasting tension with latency), and how to interact well with storage (async-io anyone?). Maybe I’ll write another blog post, maybe not.

By the way, as for the Lord dropping a table owned by another user: it apparently comes down to if he uses RDS (on a self-hosted DB he’s a superuser, but on RDS nobody is. Yes, not even the Lord).

Since posting this article, the Elders at Epsio realized there’s no point to keeping their secrets so secret and published their streaming engine for the world to use- check out Epsio’s blazingly fast SQL engine here.

How we made our streaming joins 50% faster

Kobi Grossman — Sun, 09 Feb 2025 22:00:00 GMT

When describing the core SQL (Or relational algebra) operators, the humble join shows up pretty fast. In fact, you would be hard pressed to find an app built on top of an SQL database which doesn’t use a join in one of its queries. For good reason! Joining two sets of data based on some predicate is the basis for creating any sort of relation between logical entities, whether it is one-to-one, one-to-many, etc.

At Epsio, we’re building a fast, streaming SQL engine which computes the result of arbitrary SQL queries in real time, based on changes. So, it’s no surprise that we’re very motivated to have an optimized implementation of join. In this post we’ll describe a key idea that allowed us to optimize our streaming join in common cases.

But before we dive into optimizations, let’s first describe how the Symmetric hash join algorithm, which is commonly used in stream processing, works. We will only discuss inner join in this post for simplicity, even though there is a lot to talk about other join types.

Background: Symmetric Hash Join

Let’s say that we are running a bookstore, and there are 2 tables in our sales database: customer and store_sales. Each row in the customer table has a unique id primary key along with a name column. To connect customers with sales, each store_sales row has a buyer_id column pointing to the relevant customer, and a book_name specifying the book name. We can visualize this relation like so:

If we want to retrieve all the customers with the book names they bought, our query would look like this:


SELECT customer.name, store_sales.book_name FROM customer
JOIN ON customer.id = store_sales.buyer_id;

The result in this example would be a single row: ("Adam", "How to pronounce SQL, and other flamewars").
How can we implement this join?

If we have all the data ahead of time, A naive solution could be to first build a hashtable for store_sales keyed by buyer_id . We would then iterate over each row in customer, reading the matching sale. In Python:

# Build the store_sales hashtable
for sale in store_sales:
    store_sales_hashtable[sale.buyer_id] = sale.book_name
# Probe the store_sales hashtable using customer
for c in customer:
    if c.id in store_sales_hashtable:
        sale = store_sales_hashtable[c.id]
        output_result((c.name, sale.book_name))

This is the classic hash join algorithm.

In a streaming context however, things get more complicated. We don’t have all the data ahead of time, but instead receive live changes. This means that e.g whenever a new sale is added, the streaming engine receives a change message containing the new row just added to store_sales. It then needs to understand how this changes the query result, and issue an update to the previous result accordingly. In the case of our join example, this looks like this:

We had received a new sale, with buyer_id=1. So we need to check if we have a customer with id=1, and if so retrieve their name to return the correct result.

This is exactly where our classic hash join fails. We build a hashtable for store_sales, but we have no state for customer. In the classic algorithm, we can probe the hashtable using the full data of customer table. Here we don’t have the entire table available, since we are working based on changes and don’t have all the data up front. Thus we won’t be able to find the matching customer for our sale.

If we flip the roles and only build a hashtable for customer instead, we encounter a similar problem. Assume that instead of adding a sale we receive the following change: The name of “Adam” had changed to “Adam, for real” (as so happens). Our new result would then look like this:

But if we only store a hashtable for customer, we will not be able to find the matching store_sales rows when the change is received! Luckily, there is a simple change that solves our problem.

Well, like many other problems in computer science, this one is also resolved by adding another hashtable. In particular, we maintain two hashtables: one for customer keyed by id, and another for store_sales keyed by buyer_id.

Now whenever a new customer is added, we lookup the store_sales hashtable by the customer id to return the matching sales. We then write the customer to the customers hashtable, to ensure that it is visible for future lookups for when another sale is added.

The other direction is similar: when a new sale is added, we lookup the matching customer in the customer hashtable to see who bought the item. We then write the new sale to the store_sales hashtable.

So when Adam buys a new book, we lookup the customer hashtable with buyer_id=1 and write the new sale:

We will then return the new result row ("Adam", "Dataflow and the art of incremental view maintenance"). This is the essence of the Symmetric Hash Join (or SHJ for short) algorithm: on a change to one side of the join, write the change to the hashtable of this side, and then read the matching rows from the hashtable of the other side to get the matching row. Pretty simple, right?

The Population Problem

This algorithm is the core join algorithm used in Epsio. There is one key detail we have glossed over though: Most of the time, the hashtables for the 2 join sides don’t fit in memory. We need to have persistency to be able to do the join with reasonable data sizes.

This is a perfect candidate for using a Key value store: Our key is the join key, and the value is the row. In particular, we use the rock-solid RocksDB to save the join states, with each state in a separate RocksDB database. As RocksDB plays a big role in optimizing our join implementation, we will describe it in more detail below.

Another important detail, and what brought us to look closer into join optimization, is the population phase. Epsio, like other streaming engines, has an initial data load phase. Before we can respond to live changes to data and update the query result, we must first compute the result for the existing data in the user’s database. We also need to populate our inner states, like the join hashtables, so that we could stream changes to compute updated results. This phase can be a computationally intensive task, since the initial amount of data can get quite large.

We have noticed that with many joins in the query, our population time isn’t meeting our expectations. When thinking about it, this isn’t surprising: During population, we can receive hundreds of thousands of rows as input into one join at a time, with different keys. Following the SHJ algorithm as described above, we will be performing these 2 steps a lot:

Lookup a key in a RocksDB database. This will be Disk I/O once the join database is large enough.
Write a the join key and its corresponding rows to another RocksDB database. The Disk I/O likely won’t be done immediately, as RocksDB flushes data to disk in a separate thread as detailed below.

These 2 steps make the symmetric hash join both a write intensive and a read intensive operation. No wonder that it’s a big part of our population time! Given the importance of join, we started profiling to see how we can improve its performance.

Let’s rock!

To get an accurate picture for where we can improve, we benchmarked the population of a TPC-DS query with many joins, query4. Here’s a shorter version of it with the key details:


 select 
       c_customer_id, 
       d_year,
       sum(ss_ext_list_price - ...) year_total,
       's' sale_type,
       ...
from customer
   join store_sales on c_customer_sk = ss_customer_sk
   join date_dim on ss_sold_date_sk = d_date_sk
group by c_customer_id ,d_year, ...
union all

select 
      c_customer_id,
      d_year,
      sum(cs_ext_list_price - ...) year_total,
      'c' sale_type,
      ...
from customer
   join catalog_sales on c_customer_sk = cs_bill_customer_sk
   join date_dim on cs_sold_date_sk = d_date_sk
group by c_customer_id, d_year, ...
union all

select c_customer_id,
       d_year,
       sum(ws_ext_list_price - ...) year_total,
       'w' sale_type,
       ...
from customer
   join web_sales on c_customer_sk = ws_bill_customer_sk
   join date_dim on ws_sold_date_sk = d_date_sk
group by c_customer_id, d_year, ...

(There is also another inner self-join later in the query, but we can ignore it for the purpose of our analysis here).

The goal of this query is to compute the yearly total purchases per customer, across different sales channels: store_sales, catalog_sales and web_sales. For this purpose, there is a join between customer and each of the sales table, along with a join of the date_dim dimension table with every sales table. This gives a total of 6 joins in the core of our query — perfect for analysis!

When looking at profiles for this query, we quickly noticed that RocksDB is a bottleneck. For example, writes to the databases of the different join sides would take a very long time. We started tackling this by looking into the usual suspect: Tuning RocksDB configuration. This worked well and resolved our long running writes, but we were still left with much room for improvement.

Studying the flamegraph more carefully, we found it. The ancient curse. The one that will strike fear into the heart of even the most experienced RocksDB users. The word that was hanging in the air since the start of this post like Chekhov’s merge-sort, the one they have all guessed by now but hoped it wouldn’t show up. The creature lurking in the background threads, in all its glory. Compaction!

What’s compaction, or: Where did my CPU cycles go?

(If the sentence “We need to compact L0 to reduce our read amplification” makes sense to you, feel free to skip this part)

To understand what’s happening, we first need to take a detour and understand how RocksDB stores data. This is a topic for its own blog post, with many resources already covering it. We will give a short summary of the read and write flows to see why we need compaction.

Whenever a new (Key, Value) pair of bytes is inserted to a RocksDB database, it is first written to an in memory data structure called a memtable. This data structure is usually a Skiplist, but all that matters is that it is a sorted list of (K, V) pairs. Once a memtable exceeds its configured capacity, 64MB by default, it is marked as immutable. A new active memtable is created in its place, while the immutable memtable is “sealed” and inserted into a queue of memtables waiting to be flushed to disk. The memtables are flushed in the background to not stall writes.

A memtable is flushed to disk as an SST file. The specific format doesn’t matter here much, but the main point is that the each SST is sorted, and thus has a first and last key in some order. For example, at some point in time the store_sales database can look like this:

Notice that while both SST files are sorted, their key ranges are overlapping. Indeed, so far we haven’t described some intentional merging of sorted data. If our keys aren’t arriving in a sorted order, which happens most of the time, we will naturally have multiple sorted but overlapping files on disk.

So what does this mean for our read flow? If we want to find a key K, we will have to:

Do a binary search on the active memtable
If not found, do a binary search on all immutable memtables
If not found, perform a binary search on every SST file, loading blocks from disk for search

Step 3 is where we will suffer from key overlap the most. We have to check every SST file for our key, where the amount of SST files can be in the thousands. Lots of I/O, lots of waiting for our KV pair. How can we speed this up?

This is where the Compaction background process comes into play, which occurs in any LSM-tree based database and RocksDB in particular. Its main purpose is to merge multiple overlapping SSTs into non overlapping files (and deleting entries, but this is out of scope for our insert only population flow).

There are several compaction algorithms, with complicated conditions for what to compact, when and how. This is an entire (interesting!) field of research which we will not get into here. Suffice to say that the compaction process will write new SST files instead of the existing ones, performing a merge sort so that we have a sorted “run” of data:

Now when we’ll lookup a key, we will only have to search one SST. Nice!

But there’s a catch. Compaction is an intensive process by itself. We read many SST files, perform a merge sort on a large amount of keys, and write possibly many SST files back. In RocksDB, compaction happens in background threads, causing the many BGThread samples we saw above.

There are many strategies to reduce compaction overhead in RocksDB and in general, but you can’t avoid its overhead entirely. Or can you?

Disabling compaction for bulk load?

Because of high compaction overhead, it is common advice to disable compaction in bulk load cases like our join during population, and do one large compaction at the end of population. Sadly, this didn’t work for our use case: empirically we didn’t see much of an improvement from it. We also can’t avoid compacting for too long; As noted above, the join algorithm is also read intensive: we write to one join side database and immediately read from the other. If we don’t compact, our read performance will suffer.

At this point we started looking a bit more into the data for our specific benchmark, TPC-DS query4, in the SF100(=A raw size of ~100GB) version of the dataset. We then noticed something interesting..

The asymmetry of joined tables

As you may remember, as part of the query we had a join of customer with 3 sales tables: store_sales, catalog_sales and web_sales. We checked the row count for each of these tables, and found out these numbers:

customer: 2M rows
store_sales: ~288M rows
catalog_sales: ~144M rows
web_sales: ~72M rows

The customer table is much smaller than the tables it is joined with! Anywhere from 36x to a 144x size difference. This isn’t a TPC-DS specific thing of course. One of the use cases of join is to model one-to-many relations, in some cases one-to-much much more.

This is great news for us! Even though the symmetric hash join algorithm itself is, well, symmetric, our data is not. This means that one side of the join will be write intensive, and the other side will be read intensive**.** So how can we use this asymmetry to our benefit?

Using the asymmetry to minimize compaction overhead

This unlocks many possible optimizations, the one relevant for our compaction problem is: When one side of the join has finished populating, stop compacting the other side.

Why does this work? Let’s look at the customer and store_sales join again, with sizes of 2M and 288M rows, respectively. We have much less customers than sales, so at some point all of our customer rows had arrived, and compaction had finished for the customer database:

At the same time, new store_sales batches of rows keep arriving, creating more SSTs. Once the last customer had been written to the database, we stopped compacting store_sales, causing an overlap with the newer files written:

However, this doesn’t matter. From this point we are only writing to store_sales, and reading from the already compacted customer. Thus we get to both write and read fast without paying for unnecessary compaction.

We went to implement this optimization, and noticed a roughly 1.5x improvements in population times of join heavy queries! That’s awesome, but can we do better?

Can we exploit the asymmetry further?

The insight regarding join sizes enables other interesting optimizations that are possible. For example, in RocksDB the memtable type can be specified. One memtable type that we mentioned earlier is SkipList, which allows concurrent writes but has O(log n) lookup cost. Another one is HashSkipList, Which as the name suggests is a hashtable where the bucket is a skiplist. The HashSkipList type doesn’t support concurrent writes, but has an (amortized) O(1) lookup speed.

Perhaps we could use the fast write yet slightly slower read SkipList for the larger join side, and the slower write but faster read HashSkipList for the smaller join side?

Honestly, we don’t know yet — we still need to verify this. All optimization ideas are good on paper, but aren’t very interesting without careful measurements. But the general idea is the same: there are probably many ways we can use the (reasonable) assumption of join size asymmetry to our advantage.

Key Takeaways

Given the importance of our population performance and the prevalence of join, it was great to see such an improvement from this technique. But beyond this specific case, we’ve learned some lessons which can be applied more broadly:

Know your data: This is a big one. There are many known optimization techniques that work well across the board, like reducing excessive memory copies, avoiding serialization costs, etc. These should be implemented of course — Making all cases equally faster is definitely important!
But treating the underlying data as a black box misses an important piece of the puzzle. The real world isn’t a collection of uniform random variables (Thankfully, that would’ve been really boring).
When considering optimizations, thinking about it as such possibly misses interesting ideas. If real world joins tend to behave in a certain way, why not use it?
In fact, TPC-DS and other benchmarks make this line of thought easier: Even though every synthetic benchmark dataset has the issue of being not entirely realistic, they are designed to model the real world. Using them as a baseline gives a good starting point for these sort of ideas.
Know your stack: Like in any database system, Epsio has many components, with RocksDB being an important one, especially when considering performance. Even though it’s tempting to go with a catch-all “good” configuration for all streaming operators, this misses many avenues for optimization. Knowing how the underlying storage engine works deeply (In case of RocksDB, sharpening your C++ skills along the way) is crucial.
On a more personal note, it is also very fun! From the Node struct representation in the SkipListimplementation to the high level compaction algorithms, RocksDB is a true marvel of engineering which we’ve learned a lot from.
Measure, measure, measure: This is a classic optimization talking point, and rightly so. The more you actually profile a real workload, the better you optimize.. the real workload. Discussing possible slowdowns is beneficial, and microbenchmarks can reveal some easy wins. But profiling with good visualization for what’s really going on in the system is king. We’ll elaborate more on our methodology here in a future blog post, so stay tuned.

These points continue helping us improve the performance of Epsio. Want to learn more about how we achieve fast response time in our engine? Check out our docs.

Epsio's Diff in the Streaming Landscape

Maor Kern — Fri, 03 Jan 2025 22:00:00 GMT

We recently published an article that gained a bit of traction about how we built a streaming engine, and though the concepts therein received much adulation and praise, many members of the community expressed confusion about how Epsio is different than:

Materialize
Flink
ReadySet
Snowflake Dynamic Tables
BigQuery Incremental Views
Oracle Fast Refresh
MSSQL Indexed Views
Database Triggers

I won’t be going through every similar product, but I’d like to break down the so called magic bullet that is “incremental views”- what inherent tradeoffs exist, and where (and why) Epsio is positioning itself.

A briefer- Why is everyone so interested in incremental views?

Incremental views allow you to define a query once, and have the queries’ results stay up to date without ever having to recalculate them, even as the underlying data changes. Imagine those costly queries that make your users wait multiple seconds and absolutely eat up your databases resources (we’ve even been privy to dastardly situations where queries take upwards of 30 minutes). With incremental views, those same queries all of a sudden take milliseconds with almost no compute, all with one simple command- CREATE INCREMENTAL VIEW SELECT ... FROM ... .
Here’s a brief explanation from our other article about how this works:

A streaming SQL engine keeps queries results up to date without ever having to recalculate them, even as the underlying data changes. To explain this, imagine a simple query, such as SELECT count(\) FROM humans . A normal SQL engine (such as Postgres’s, MySQL’s) would need to constantly go over all the different humans every time you ran that query- which could be quite costly and lengthy given our ever changing population count. With a streaming SQL engine, you would define that query once, and the engine would constantly keep the resulting count up to date as new humans were born and the old ones died off, without ever performing a recalculation of counting all humans in the world. In comprehensive streaming engines this is possible with every sort of query, no matter the complexity.*

This can be a huge improvement with very little overhead. Furthermore, in big data environments, this has huge cost implications- every programmer who has worked with Big Query or Snowflake shudders at the words “Full Table Scan”. Incremental views eliminate them; your costs are correlative to your rates of change, not the amount of overall data in your system.

So, given incremental views are interesting, let’s talk about a few categories of incremental views, and where Epsio sees itself in comparison with them.

First Category — The Databases/Warehouses themselves- Batch Processors in disguise

Oracle, MSSQL on the DBMS side, BigQuery and Snowflake on the warehouse side.
From a UX standpoint, it would have been excellent if you could just define incremental views using these engines instead of needing another solution for them. The above databases/warehouses each attempted in their own way to give you some type of ability to create incremental views without needing to use another service.

The problem is that under the hood, these databases have planning & query engines that were built from the ground up to be batch processors. Whenever queried, these databases look at a snapshot of their complete set of data, and calculate a result. They do not deal with a stream of changes (eg “Deal with this insert. Deal with this update. Deal with this delete. etc”), but with constant data (eg “Deal with this static dataset”). Their queries are heavily optimized to be “short lived” (a 30 minute query is considered incredibly long), and produce one single result set. A stream processor’s queries normally live forever, and continuously produce new results as changes come in. This affects every single part of how they’re implementing both their planning and execution of a query. The language between the query nodes is different; batch engines pass data between their nodes, and do not have any way to pass a deletions/updates. The optimization tradeoffs are different; batch processors usually need much more compute and the ability to go over large amounts of base data quickly, while stream processors need little compute and lots of storage for intermediary representations of data. Even the flow of data is often different- in batch processors, you’ll normally have the query nodes “pulling” data from the nodes above them, while streaming processors normally work in “push” mode (which means different communication channels altogether).

But the proof is in the pudding. Oracle, MSSQL, and BigQuery have endless constraints on what you can do in their incremental views- the easiest example being that all of them do not support Outer Joins (Left/Right etc). And it makes sense; when reading through the three page essay that is the list of constraints on these databases, you get the monumental feeling that they’ve created a mountain of patches on their existing execution engines to support this, and that it had the expected effects on their user facing capabilities. This is by no means a bashing of those databases; they’re absolutely incredible at what they do. Stream processing just isn’t what they do.
Snowflake, by the way, is a bit more interesting, with their dynamic tables being released as a preview in June. They’re a bit more vague on what they actually support, and purposefully did not call them incremental views. If they do not support your query, they’ll just do a full refresh behind the scenes (and charge you as such). Playing with them and talking with some companies that tried to use them tells us that they’re valuable as a slightly smarter query refresher, but by no means a streaming solution.

Second Category — Stream Processors

Materialize, ReadySet, Flink, Epsio, etc- all built from the ground up to be stream processors, all very efficient; Materialize and ReadySet built by pioneers in the academic space.

While on the surface they may look similar, there are a multitude of tradeoffs and questions that arise as you start looking at each one in depth.

1. Does this stream processor give “correct” answers?

This may seem like a bit of an obvious question, but there are actually a variety of consistency models in databases. A consistency model of a database is a sort of “contract” the database gives you which gives certain guarantees for how writes will be taken into account in reads. The strongest level of consistency guarantees that every write will be taken into account on sequential reads- all your average SQL databases (Oracle, MySQL, Postgres, etc) give you this. Most stream processors do not give you this- as a matter of fact, none of the above (Materialize, Readyset, Flink and Epsio) do. What they do guarantee is that at a certain point in time after writing (maybe a few milliseconds, maybe a few seconds) the inserted record will be taken into account.

For most use cases, eventual consistency of 50ms is something you can swallow. What can be a much bigger issue is if you’re not internally consistent. Internal consistency means that every answer given was true at some point in time. Some systems, like Flink, are not internally consistent, which is usually a no-go from the get-go for companies. Imagine an alert going out about an event that never happened, or a user going into your dashboard at just the wrong time and seeing his balance is below zero. This is true about other streaming engines as well, and something I highly advise on checking before picking a streaming engine.

2. How much cost/risk/time is inherent in trying to integrate them?

To answer this per stream processor, we’ll need to understand their mode of integration. Some stream processors are built to be more “standalone”, and some are meant to complement existing systems. If you already have a database/data-warehouse, standalone stream processors mean replacing your current database. Materialize, for example, is an incredible streaming engine, highly geared towards realtime latency in big data environments. They’re a good example of tending towards more of a standalone engine (and you can see over time how this has become more and more the case), looking to be a replacement for existing data warehouses (or ETL processes). A few others I can throw into the category of tending towards standalone engines are Arroyo (serverless stream processing) and Feldera (data warehouse, built upon the uber-cool-bleeding-edge DBSP). They’re excellent choice if you want to create new solutions (if you want to create a new warehouse), but can often have too heavy integration costs if you’re looking to optimize existing queries/materialized views, as you’ll probably need to move your data warehouse to only use them if you don’t want to end up managing two data warehouses (a quote I really like- “replacing your database/data warehouse is like trying to replace a car’s engine while driving”).

For stream processors that are complementing current systems, the question becomes- what risks does this new platform pose architecturally, and how do I integrate with it?

Here ReadySet is very interesting. They sit as a proxy between your server and your database in order to offer partial materialization. Sitting as a proxy has both upsides and downsides; ReadySet can intercept queries and automatically reroute them to their corresponding incremental views (that you define in their portal), thereby taking away the need to make any code changes. The downside is that adding a proxy to your database is adding another Single Point of Failure to your entire system. If ReadySet goes down, the connection from your backend to your database goes down with it. The irony is that although ReadySet is very easy to integrate from a code standpoint, it would take a courageous architect to add a proxy without extensive testing; it makes it more difficult to check a specific use case and grow from there.

So where is Epsio in all this?

Our aim is to give a seamless UX within your existing database, while minimizing risk & integration time. Our belief is that while the implementation requires a different solution than a batch processor, the user experience ought to be as though it were available within your very own database.

From an architectural standpoint, Epsio sits “behind” your database as another replica, and writes back to your existing database. This allows your backend to converse with your database without ever knowing about Epsio. If Epsio were to go down, your backend would be unaffected, and even Epsio’s own incremental views would return results, albeit “stale”. From a risk standpoint, this can be crucial for integrating a new technology; architects are often extremely risk-averse when it comes to production environments, and this architecture allows you to try out Epsio on one use case without risking others.

To give a “seamless” UX, you never interact with Epsio directly, instead calling functions that Epsio creates within your existing database (such as “epsio.create_view”, or “epsio.alter_view”). These functions are transaction safe, which can be critical for the workflow of most companies who want to create/alter views within migrations.

Lastly, there’s an inherent tradeoff between speed and cost that stream processors have to deal with. Epsio is built from the ground up to work well with storage to vastly reduce costs compared to in-memory engines. If you want sub-millisecond latency no matter the cost Epsio may not be the best engine. But if you’re looking for high throughputs at ~50ms latency, Epsio delivers.

...

I think at a bit of a more “meta” level, we believe that incremental view theory has finally reached the point where it’s not a concern of support for SQL operators, or making it that bit more speedy, but a question of how well it ties into your current architecture, both from speed of integration and the cost of the actual product. This is because of incredible work done by the folks who built Noria, Differential Dataflow, DBSP, and countless other streaming methodologies.
Epsio is built to bring the best of the world of theory while optimizing on seamless, low time/risk integration and using low cost resources. Can’t wait to see you!

Streaming SQL Arithmetic

Maor Kern — Mon, 09 Dec 2024 22:00:00 GMT

Math insists that a+b-c=a-c+b, and we take this as an unshakable truth. But in computer science, when floating-point arithmetic enters the picture, even basic rules like this can fall apart. Most developers never notice these quirks in everyday applications — but in the early days of building our Streaming SQL engine, this tiny mathematical inconsistency snowballed into a nightmarish bug. It wasn’t just hard to fix; it was hard to even understand. Let’s dive into the depths of this floating-point horror story together.

How Epsio Stores Result Data

Before we go into the inherent issues with floating arithmetic, let’s talk about how Epsio stores the result data. In Epsio, you define a “view”, which consists of two parts: a name for a result table (this will be a table Epsio creates in your database, say a Postgres database), and a query you want Epsio to maintain. Epsio will populate the result table with the answer of the query, and then continuously update it as new changes come in via the replication stream from the database. A streaming SQL engine keeps queries’ results up to date without ever having to recalculate them, even as the underlying data changes.

To make this more understandable, let’s say we had a farm with livestock, and wanted to give the animals on the farm their fair portion of the profits (trying to avoid an Animal Farm here). To know how much total salary we’ve given each animal group, we’d have a query SELECT animal, sum(salary) FROM animals GROUP BY animal . Because the underlying data is always changing (animals being born, asking for raises, etc) we create a simple Epsio view in our Postgres:

CALL epsio.create_view('animal_salaries', 
'SELECT animal as group, sum(salary) as total FROM animals GROUP BY animal')

This would create a table in our Postgres called animal_salaries, which Epsio would continuously update with the ever changing animal salaries.

It might look something like this:

You may wonder what that “epsio_id” column is doing there, since we only ever defined two columns in our query. Epsio automatically adds another column to every result table in order to allow for efficient deleting. Internally, Epsio has a sort of “dictionary” mapping the entire row to it’s corresponding epsio id (this is actually held in RocksDB).

In our case, it would look something like this:

Bunnies,720.32:999
Pigs,7049.41:238

Let’s say all the Bunnies died (someone nefarious ran DELETE FROM animals WHERE animal='bunny') , and Epsio wanted to remove the bunnies row. It would find the corresponding row in it’s key mapping (Bunnies,720.32), find the corresponding epsio_id (999), and then run a delete query on Postgres:

DELETE FROM animal_salaries WHERE epsio_id = 999

You may notice that salaries are not round numbers. Animals can be a bothersome folk who negotiate on the penny, which means we need to keep our salaries (and therefore the sums) as floating points.

Enter Floating Points

Floating-point numbers are a way for computers to represent real numbers; numbers that can contain decimal points. Instead of storing numbers with absolute precision, floating points use a sort of “approximation”, which allows them to be both efficient in terms of space and also quick with computations. They’re represented using three numbers: a sign (positive or negative), a “mantissa”, and an exponent. So every floating point is essentially represented as:

using a nifty calculator (https://observablehq.com/@benaubin/floating-point) we get:

As we can see, we’re actually not able to represent 2.4 exactly in 32bit floats! This might not seem to be that big an issue (you might say that 2.399999.. is awfully close) but it can lead to weird things.

Weird Things (aka Arithmetic with Floating Points)

So if we’re not able to represent numbers exactly, what happens when we add them up?

The question is how computers manage to add up floats that are written in this method. If the exponents were the same, then by the ever-respected laws of math we could just add up the mantissas, and keep the exponent. But if the exponents are not the same, we need to align one of the float’s exponents to be the same as the others. We do this by both changing the mantissa and the exponent- which means we get an ever so slightly different approximation. For example, if 3.8 is approximated to 3.79999 with exponent=1 and we aligned the exponent to 2 (and changed the mantissa correspondingly) it may be approximated to 3.800001.

If we do addition like this, math may not be associative. Imagine we add 4.6 and 3.8. Let’s say 4.6 has an exponent of 2 and is approximated to 4.60001, and 3.8 has an exponent of 1 and is approximated to 3.79999. As we said before we need to change one of the floats’ exponents, so let’s change 3.8 from its exponent of 1 to 2, making its approximation 3.800001 instead of 3.79999. If we add these numbers, we may get something very close to the “real” answer, which is 8.4. But if we now subtract 4.6 from this answer, and asked our computer “hey mr.computer is that now equal to 3.8”, our computer would say no.

3.800001 != 3.79999.

Now imagine we remove our original 3.8 (whose approximation is 3.7999) from that. We would get a number very close to zero, but not zero.

So 4.6 + 3.8 - 3.8 - 4.6 != 0 .

But… 4.6 + 3.8 — (3.8 + 4.6) does equal zero in floating point land. I’ll let you work out why.

Why does this matter in streaming SQL

In Epsio’s streaming SQL engine we have a concept of a “change”, which is the basic unit that is passed between query nodes. If you’d like to go into depth on how this works, my other blog post https://blog.epsio.io/how-to-create-a-streaming-sql-engine-96e23994e0dd explains it, but I’ll give you the short and relevant version here.

Every change has a key and a modification.

Let’s say a Llama appears out of the blue in our farm, demanding a very fair salary of 3.8$. We would represent it like this:

One of our query nodes is called a Group By node, which in our example from the beginning of the blog would be the last query node before our “final” node, mapping rows to their corresponding epsio_ids. The Group By node outputs aggregates per buckets. It does this by holding an internal mapping (in storage) between each bucket and its aggregated value. So in our example we would have

Let’s imagine we indeed passed Llamas +3.8 into the Group By node. It would look something like this:

The Group By node outputs that there is now one new group (signified by the +1) of Llamas with the value 3.8.

If that Llama were to then have a Llama baby, we would have:

The Group By node basically tells the node after it,

“Hey, remove the row I gave you before saying (Llamas, 3.8) and create a new row called (Llamas, 8.4)”.

The Group By node always outputs a minus on what it had for the group (if the group previously existed), and a plus for the new value. Now, you might be thinking, what if both Llamas died? Wouldn’t we reach the exact scenario we talked about before where 4.6+3.8–4.6–3.8 != 0? It would be pretty weird if an end user saw that there are no Llamas in his system but Llamas receive $0.000002 of total salary.

The way Epsio solves this is by keeping track of the number of records that passed through. If the number of records from the Group By node is ever summed up to zero (e.g. we had one new record and one deletion of a record), we simply filter it out (in truth- all our aggregations are “tuples” which can have many values that we add up together at once, but that’s for another time). This is how we avoid weird situations with groups having 0.00000x values for floats.

But you might be beginning to get a feel that something weird is going on here- in a batch processing SQL engine this problem doesn’t exist as acutely, since if there are no Llamas you would never output the group. Because stream processing engines are keeping state over time, modifications that aren’t perfectly associative can cause trouble.

(Finally) The Bug

So what’s the bug?The answer lies in how our internal mappings work. We’ve previously shown our internal mapping as follows:

But how do we actually hold this data, and how do we update it?

Our keys and values are held in RocksDB, an embedded key value store. Every time we merge in a new value for a key, instead of just removing the old key and adding a new key with the updated value, RocksDB simply saves a “merge” record. So we can imagine our mappings looking something like this:

RocksDB will actually do the “merging” (in our case addition) when it either runs a process called “compaction”, or when you read a key. The thing is, RocksDB offers no guarantees on what order it’ll do the merging in. Furthermore, we have optimizations that hold some of the data in memory that we add with the result of RocksDB- this is just to emphasize the fact that there is an implicit expectation in our code that you’re allowed to “merge” records in any order you want.

This all means that we’re constantly at risk of giving off a slightly different answer- just like4.6+3.8–4.6–3.8 != (4.6+3.8)-(4.6+3.8).This rarely happens precisely because most of the time you are merging them in the same order, and so getting the same result. But nobody is guaranteeing that for you, and in rare cases you’ll run into trouble. This trouble comes in the form of telling the Final Node, which we talked about in the beginning of the blog, to delete a row that never existed. It therefore cannot find the equivalent epsio_id, and proceeds to error out.

Why this happened

In streaming engines, the concept of a modification must be extremely well defined. To us, it was anything that implements in Rust the Add trait and the Default trait- meaning both that modifications could be added together, and that they could zero out. But there was a third “trait” (not in the Rust sense) that we didn’t think too much about when building the underlying infrastructure: associativity. While “hacks” such as keeping the count (a very common solution in other SQL Engines) can seem to patch the problem, the underlying issues with associativity will come to light in other fashions. For us, the right solution was to transform floats into absolute precision numbers when they arrive at accumulative query nodes. Absolute precision means that we never have “approximations” like in floats, which guarantees us perfect associativity. The cost and overhead of having absolute precision was negligible in most situations, while engineering around them was full of pitfalls.

Why We Built a Streaming SQL Engine

Gilad Kleinman — Wed, 21 Aug 2024 21:00:00 GMT

Following our previous blog post “How we built a Streaming SQL Engine,” we are delighted to share some of the reasoning and thoughts that led us to build Epsio (as we always say — start with how, only later ask why).

Why we built Epsio

For most of our adult lives, my co-founder Maor and I have been working on developing and designing software products, and scaling them from mere ideas to fully-fledged solutions that cater to the needs of thousands / tens of thousands of users.

Two interesting (though obvious) things almost always happened as time passed while working on these projects:

The amount of data our product relied on increased. As more users were using our projects, the more data we had to store.
Our products become more complex. Over time, management would request additional features and enhancements, resulting in a much more complex backend architecture and more complex database queries.

Although these two trends may seem obvious, they always had severe implications, particularly on the performance of the databases we were using. As a new feature was requested or a new big customer onboarded, we always remember asking ourselves “will the DB handle this?”

Why our queries became slow: The gap between the data we store and the data we seek

In a philosophical way, whenever one of those things occurred (more data was stored / queries became more complex), the gap between the data we stored and the data we sought became bigger.

The more data we had, and the more complex a query we ran, the more things (disk reads, calculations, correlations, etc…) our database needed to do to calculate the result based on the data we stored. The more things our database needed to perform, the wider the gap became, and the longer it took the database to provide a result.

If, for example, we were building a salary management system with a dashboard that shows the total sum of salaries in our company — and the number of salaries in the company were to suddenly double, our database would need to perform twice the amount of work each time someone opens the dashboard. This would result in our database either using twice the resources it did before or taking twice the time it used to. Both options are, obviously, not ideal.

Even in the scenario where the number of salaries doesn’t grow, it is very likely that over time we would want to add additional or more complex queries to our dashboard (such as changes in salaries over time, distribution graphs, etc.), again — resulting in even more work for our database to perform to calculate what we want to show in the dashboard from the underlaying data.

How did we overcome these performance issues until today?

Usually, when facing these performance issues, we would try using all the traditional methods (indexing, optimizing the queries, etc.) that we were taught to “accelerate” our queries. All of them may improve the performance in the short term, but usually these solutions would just be “patches” and would not hold water as data volumes and query complexity continued to grow.

Methods such as adding indexes, increasing the computing power of the database, and rewriting queries would usually indeed shorten query times. However, since the performance would still be correlated to the data volumes and complexity, our queries would continue to slow down as the gap continued to widen. A query that was optimized to run 10 times faster would very quickly become slow again as data increased by 10 times its size.

Alternative methods such as materialized views, caches, or denormalizations that pre-calculated the data we seek instead of just “accelerating” queries would also seem very promising at the beginning, but would quickly prove difficult to implement and maintain when we actually tried using them. Given that the results were pre-calculated, every time the underlying data changed, we would need to recalculate the entire result from scratch to reflect the new data, which would cost us expensive compute power or lead to stale data if we didn’t constantly update the results.

Introducing Epsio

Epsio solves this problem by never trying to reprocess the entire dataset whenever a result of a query is needed or whenever the underlying data changes. Instead, Epsio pre-calculates results of defined queries and incrementally updates them whenever the underlying data changes, without ever recalculating the entire dataset.

By pre-calculating complex queries and incrementally maintaining their results, we can achieve the huge benefits of denormalizations and caches, where results are instant, without the need to invalidate or recreate results whenever the underlying data changes. For us, this method of executing complex queries was the only way to effectively bridge the gap between the data we stored and the data we sought, and to “prepare” our database for the specific queries we were about to execute.

How does this actually work?

To demonstrate how Epsio actually works, imagine the backend system to manage salaries we mentioned before. Imagine that the system contains a table with the salaries of all employees in a company, and a dashboard running a query that calculates the sum of salaries by department. If our imaginary system were to be used in a very large company, our salaries table might easily contain millions of records, meaning that our dashboard would probably take multiple seconds or minutes to load without us optimizing its queries extensively.

To utilize Epsio, we can simply define our query in Epsio. First, Epsio scans the entire base table and saves the initial result of the query to a results table:

Then, when new data arrives or existing data gets updated or deleted, Epsio performs the minimum calculation needed to update the previous result. In this case, just adding or subtracting the previous count per department:

As the above calculation is very efficient (compared to recalculating the entire query), we are able to take a complex query that originally took seconds / minutes to run and provide instant and always up-to-date results, all while using less compute power and without spending a second on “query optimizations”.

Although it’s very simple to imagine how the above query could be incrementally updated when new data arrives, apparently (as you can read in the previous blog post we published), the same concept and way of thinking could be applied to much more complex queries and SQL operators that contain JOINs, ORDER BYs, DISTINCT, and many more — meaning that almost any complex and slow SQL query that runs frequently in our backends could benefit from this huge performance boost if processed “incrementally.”

Why Epsio is (not) a new database

Although our initial hunch when thinking about the architecture of Epsio was to build a new database, we quickly understood that it is important for our new engine to be “Easy to integrate, Easy to trust, and Easy to use”. When thinking about these three “E”s, we understood that building a new database simply doesn’t address any of those, and probably isn’t the right architecture for us.

Specifically, we wished for an architecture where:

Zero migrations are required for use.
The engine could be tested on a small use case without effecting anything not related to that use case.
No additional single point of failure should be added. If the engine falls, our database should still be “queryable”.

To answer all of the above, we decided to implement an architecture where Epsio sits “behind” the existing database, receives the change stream (CDC/replication stream) for the relevant data, and writes back to “result tables” in the original database:

In this architecture, unlike a new database:

No migrations are required to use Epsio.
Developers can try Epsio on a small use case without it effecting the rest of the database. Epsio is not in the hot path.
In the (very bad) scenario where the Epsio instance fails, other queries in the database are not affected, and even Epsio’s result tables continue to be accessible, although their results might be stale.

All of the above means we can continue working with the databases we already love and trust, while still enjoying the significant benefits of stream processing — or as the great Hannah Montana used to say: “You get the best of both worlds.”

So, what’s the catch?

Although we truly believe that the “incremental” way of thinking can really change the way we process and query our data to the better, at the end of the day, similar to everything else in the world of databases (or software in generally) — everything is a tradeoff.

Some queries that change drastically between runs (ad-hoc queries) or that query data that changes drastically between runs, might be less efficient in Epsio. In the world of tradeoffs you can never solve all queries, but if we can still solve a huge portion of them, we should be happy with the big “incremental” (no pun intended) change it offers in the process of making developers focus more on building logic and less on optimizing stuff.

Concluding thoughts

After spending way too much time optimizing complex queries and trying to scale them as our products scale, we hope that Epsio will serve as a new paradigm to better bridge our applications and our databases — and to ensure that our databases perform only the minimum required operations to meet our complex application needs.

...

If you enjoyed this blog post and wish to try Epsio’s blazingly fast (but actually just cheating) SQL engine, check out our docs here.

Out Of Memory Shenanigans

Gilad Kleinman — Tue, 06 Aug 2024 21:00:00 GMT

At Epsio Labs, we develop an incremental SQL engine — in brief, an SQL engine that operates on changes instead of reprocessing the whole dataset each time a query runs. I’ll leave explaining why that’s useful to another blog post of ours — Why We Built a Streaming SQL Engine.

In the early days of Epsio, one of our first ways to stress test our engine was to run the TPC-DS benchmark queries (an industry standard benchmark) with a large data scale (100 TB of records).

We ran one of our newly born tests, and after several minutes, our test machine completely froze for 10 minutes!
Afterwards, we noticed our engine process got killed (some would say brutally murdered). Looking at our system metrics and logs, we saw the system memory was at 99%, which was much higher than expected.

Memory Utilization on 16 GB machine, Between 19:06 and 19:17 logs were not sent (thus the awfully straight line)

In this blog post, we will navigate the uncertainty of memory utilization, unveil the mystery of our process’s memory bloat, and discuss how we tackled it.

What happens when the machine is out of memory?

Data and memory-intensive programs such as ours often reach high memory usages. If the workload is large enough and no memory regulation is in place, it might cause your system to reach close to 100% memory utilization.

In order to keep our machines running even in such scenarios, Linux gives us two configurable tools:

Swap memory — A file or partition on disk that’s used as a supplement to physical memory (RAM). When more RAM is needed than what’s available, inactive pages in memory are moved to the swap space in order to free memory space. We chose to disable swap in our test machine because we consider extreme memory utilization a bug; we wouldn’t want to rely on swap to save us due to its performance impact. Also, Epsio might be deployed on a machine that has swap disabled.
OOM Killer — The Out of Memory Killer is a process ran by the Linux kernel when the system memory is extremely low. It reviews all the running processes and assigns a score to each, based on memory usage and other parameters. It then chooses one or more of them to be killed. [1] [2]

The culprit of our process’s assassination is of course the OOM Killer, who chose to kill the engine process because of its high memory usage.
To ensure that our accusation is true, we can look at the system logs with journalctl (a utility to query various services’ logs):

(You can ignore the nasty command)

journalctl --list-boots | awk '{ print $1 }' | \ xargs -I{} journalctl --utc --no-pager -b {} -kqg 'killed process's -o verbose --output-fields=MESSAGE Tue 2023-02-02 16:21:52.348846 UTC [s=694483b10fab40da9c36d06a1f288df8;i=acc;b=8ee2faf0530244958351de3c743bd18a;m=d991edcc;t=605b8a8153aae;x=affb06d11f0113d9] MESSAGE=Out of memory: Killed process 9049 (epsio_engine) total-vm:16253052kB, anon-rss:14777084kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:30128kB oom_score_adj:0

Let’s focus on the output:

Out of memory: Killed process 9049 (epsio_engine) ... anon-rss:14777084kB

Our process got killed by the OOM Killer while having 14.7 GB allocated to it in physical memory (see “anon-rss” value in the output).

So OOM Killer was spawned, which means our system was out of memory. When you get that close to the sun, bad things start to happen — my machine was unresponsive via ssh, application logs were not sent, and everything seemed to be frozen. It took 10 minutes until my machine was responsive again, which is weird. You’d think the OOM Killer would kill a process in a millisecond.

Apparently, this is a known phenomenon in Linux that is sometimes called “OOM hangs/livelocks”. The reason for this phenomenon is a bit complicated, but to be concise, when memory is requested and there’s not enough available memory, the system first tries to reclaim memory pages that are backed by a file on the hard disk (memory-mapped files).
If enough memory is reclaimed, the OOM killer is never spawned. This can go on and on, causing the system to be slow and unresponsive.
Interestingly enough, SSDs’ fast performance makes this phenomenon even worse. [3]

We definitely never want to reach an unresponsive state because of OOM hangs. We can avoid this by killing our process before the system memory is at full capacity (see earlyoom). For the simple task of making our stress test machine want to talk to us, we chose to add a CGroup in order to limit the process’s memory (the process will be killed when the limit is exceeded).

This mitigation makes our machine happy; however, the engine process will still be abruptly killed because of memory bloat, which takes us to the root cause of the problem: Why did Epsio need such an extreme amount of memory in that test?

Processes fighting for memory on my ubuntu machine, circa not so long ago

Epsio View And Its Big Pipeline

The failed test creates an Epsio view for a TPC-DS query. An Epsio view is an incremental view; it shows up-to-date results of the query. When an Epsio view is created, a new Epsio engine process is spawned. The engine first calculates the current results of the query — we call this stage population, and its purpose is similar to other SQL engines.
After population, the engine begins to consume row changes (e.g. inserts, deletes, updates) and update the results accordingly — we call this stage CDC Streaming (Change Data Capture).

The specific test that failed created an Epsio view with a query similar to the one below (this is a simplified version):

select i_item_id, avg(cs_quantity) agg1
 from catalog_sales
 join item on cs_item_sk = i_item_sk
 join promotion on cs_promo_sk = p_promo_sk
 where (promotion.p_channel_email = 'N' or promotion.p_channel_event  = 'N')
 group by i_item_id

Our SQL engine constructs a data pipeline for the query. Each node in the pipeline corresponds to an SQL operator. Here is an illustration:

Each root node (catalog_sales table, item table, promotion table) is an input node that yields a batch of rows either from the population or CDC stages. If we processed one batch of rows at a time and waited for it to reach completion, we would waste idle CPU cores and increase the latency of the next batch. That’s why we want to stream multiple batches of rows in parallel.

Now, try to think of all the ways memory might bloat when we stream these batches of thousands of records:

We might stream too many batches too fast, which is bad because memory will become scarce.
Each join node might output more rows than the sum of its inputs. For example, if the first join key in our query is cs_item_sk = i_item_sk, for a batch of 10,000 rows from the left side of the join that have cs_item_sk = 1 and another 10,000 rows from the right side of the join that have i_item_sk =1 , the results would be the cross product of the rows (100,000,000 rows!).
A bottleneck node might cause a “traffic jam” in the pipeline, causing memory to be occupied for more time. This might not seem like a problem, but remember that multiple Epsio views can run simultaneously, and if all of them have lots of batches in memory waiting for the bottleneck to finish, system memory will become scarce.

Those disasters were already foretold, to tackle these situations we:

Limited the amount of parallel batches.
Because a node like Join might output larger batches than its input, we introduced a stalling mechanism that will wait for more memory to be available before streaming batches between nodes.
Added a spill-to-disk mechanism that can choose to move stalled batches from memory to disk.
Introduced more optimizations to our SQL query planner in order to have a more efficient pipeline. For example, because our query’s filter node has predicates (filters) relevant to the promotion table’s columns, we can move it to be before the join node. That way, we avoid unnecessary joins between rows that would be filtered out later anyway, resulting in less memory usage and better performance. This optimization is known as “Predicate Pushdown” because it pushes the predicates (filters) closer to the data source (table input).

Some of these mitigations are good examples of memory regulation. However, even with the mitigations in place, the test caused our process memory to bloat. The process memory usage was much higher than expected and exceeded our memory regulation limits by far. At this point, we decided to go more in depth.

Malloc And Its Shenanigans

To understand our process memory usage, we have to understand what happens when our process allocates dynamic memory.

In Rust, dynamic memory allocations use the global memory allocator, which in our case is the malloc allocator implemented in glibc 2.35.
So under the hood, our process basically uses malloc and free as any normal C program does.

Malloc Being Greedy

The allocator asks the OS for memory by using a syscall, but it strives to reduce the number of calls for performance reasons. Hence, when free is called, it doesn’t necessarily return the freed memory chunk to the OS, but instead holds several data structs for the bookkeeping of free chunks. On the next malloc call, the allocator looks for a suitable free chunk to return. If found, it will return a pointer to the chunk and mark it as used. Otherwise, it will issue another syscall for memory.

From the OS’ perspective, the user process still occupies the memory in the free chunks.

Memory Profiling

Interesting stuff! Now that we understand a bit more about malloc, let’s profile our process memory usage to see how many free chunks are there. We can use the libc mallinfo2 function (__libc_mallinfo2), which returns a struct containing information about memory allocations.
From the mallinfo man page:

The structure fields contain the following information:
..
uordblks
The total number of bytes used by in-use allocations.

fordblks*
The total number of bytes in free blocks.*

Let’s create a task that will log mallinfo results every 10 seconds:

use libc::mallinfo2;
use tokio::time;
use epsio_tracing::info;

pub fn start_mallinfo_monitor() -> JoinHandle<()> {
    tokio::spawn(async move {
        let mut interval = time::interval(Duration::from_millis(10000));
        interval.set_missed_tick_behavior(time::MissedTickBehavior::Delay);
        loop {
            interval.tick().await;
            unsafe {
                let mallinfo2_struct = mallinfo2();
                info!(
                  message = "mallinfo2",
                  mallinfo2 = mallinfo2_struct
              );
            }
        }
    })
}

When we put uordblks and fordblks on a graph, we get:

So we are actually not even using most of the occupied memory; ~11GB out of 15GB are actually free chunks.

To shed more light on the matter, I used the malloc_info libc function, which prints a huge nasty XML with much more details:

more arenas
...
heap
--Arena number 60-->
<heap nr="60">
<sizes>
  <size from="33" to="48" total="96" count="2"/>
  <size from="113" to="128" total="128" count="1"/>
  <size from="49" to="49" total="79625" count="1625"/>
  ...
  <size from="224705" to="224705" total="224705" count="1"/>
  <size from="300689" to="524241" total="37369240" count="88"/>
  
  <size from="524353" to="67108785" total="804350523" count="235"/>
  <unsorted from="65" to="65" total="65" count="1"/>
  
sizes>
<total type="fast" count="3" size="224"/>

<total type="rest" count="8250" size="852802425"/>
<system type="current" size="1173217280"/>
<system type="max" size="1220870144"/>
<aspace type="total" size="1173217280"/>
<aspace type="mprotect" size="1173217280"/>
<aspace type="subheaps" size="18"/>
heap><br><heap><br><heap nr="61">
...
More arenas

It turns out our process has more than 60 arenas (corresponding to the XML’s heap element). But what the hell is an arena? to explain that we need to understand how malloc deals with multiple threads.

Malloc And Multi-Threading

In order to efficiently and safely handle multi-threaded applications, glibc’s malloc needs to tackle the situation where two threads might access the same memory region at the same time (which would cause parallelization bugs). Therefore, the allocator segments the memory into different regions. These regions of memory are called “arenas”.

Each arena structure has a mutex in it, which is used to control access to that arena. This mutex assures that two different threads won’t access the same memory at the same time. Contention for this mutex is the reason why multiple arenas are created.

The number of arenas is capped at eight times the number of CPU cores by default.

Memory Fragmentation

It’s important to note that each arena has a heap (sometimes multiples) — a contiguous memory region holding the memory chunks (used or free). An heap can be grown or shrunk using the brk and sbrk syscalls.

As our process continues to stream row batches through the data pipeline, many small mallocs and frees are called, and heaps might get fragmented — gaps of free chunks between used chunks start to form. adjacent free chunks are combined into bigger free chunks. But a heap can shrink only by its “top” chunk — the free chunk at the current end of the heap. Free chunks trapped between allocated chunks might be used for the next allocation but will not be returned to the OS. We lovingly refer to this situation as memory fragmentation.

With multiple arenas and heaps, memory fragmentation is even more prevalent because each heap gets fragmented on its own. There are fewer chances to coalesce free chunks and to find a suitable free chunk for an allocation.

To illustrate how costly memory fragmentation can be, I wrote a small C program. The program allocates 1,000,0000 buffers and then frees all of them except the last one:

#include
#include
#include
#include

#define NUM_BUFFERS 1000000
#define BUFFER_SIZE 1024  // 1KB
// 1000000 buffers * 1KB = ~1GB

void* allocate_buffers_then_free() {
    // Allocate an array of NUM_BUFFERS buffers
    char* buffers[NUM_BUFFERS];
    int i;

    for (i = 0; i < NUM_BUFFERS; i++) {
        buffers[i] = (char*)malloc(BUFFER_SIZE);

        // Check if malloc was successful
        if (buffers[i] == NULL) {
            fprintf(stderr, "Failed to allocate memory for buffer %d\n", i);
            exit(1);
        }
        // Write to buffer in order to assure it's mapped to physical memory
        // This is done because overcommit is enabled see https://www.linuxembedded.fr/2020/01/overcommit-memory-in-linux
        memset(buffers[i], 0, BUFFER_SIZE);
    }
    printf("Allocated buffers\n");

    sleep(10);

    // Freeing all the buffers except the last one
    for (i = 0; i < NUM_BUFFERS - 1; i++) {
         free(buffers[i]);
    }
    printf("Finished freeing buffers\n");
}

int main() {
    setbuf(stdout, NULL);
    allocate_buffers_then_free();
    // Sleep until being killed
    while (1) {
        sleep(10);
    }
    return 0;
}

I compiled (gcc -o memtest memtest.c), ran the program, and watched my machine’s memory with watch free. Here is the output I got:

// free showed our system had 13GB of free memory
Allocated buffers
// system had 12GB of free memory (1GB was allocated for out process)
// 10 seconds of sleep...
Finished freeing buffers
// system memory still had only 12GB of free memory
// Meaning the freed buffer chunks were not freed back to the OS

What’s even more interesting to see is that if we change the program to free all the buffers except the first one, the memory will be freed back to the OS. Another adjustment that achieves the same outcome is increasing the buffer size to 1MB and reducing the number of buffers to 1000. Because over a certain allocation size (M_MMAP_THRESHOLD), malloc will use mmap (a syscall that can allocate memory outside of the heap), and mmaped buffers are easily freed to the OS with the munmap syscall.

I invite you to try it on your machine (be careful not to trigger an OOM).

Back to our process’s malloc_info XML… Our process had 64 arenas (the maximum number for an 8 core machine), and each one had between 100 and 1000 MB of free chunks. It strongly implies that malloc being greedy + memory fragmentation amplified by our use of multi-threading, are the reasons we have so much memory wasted on free chunks.

Switching Memory Allocator

Memory fragmentation is a hard problem that most memory-intensive programs face. To solve it, we could have tuned malloc with configurations or changed our application’s allocation patterns (which is a great topic for another blog post).
Even so, like many other memory-desperate developers, we started seeking a more adequate allocator.

In an academic research article called “Experimental Study of Memory Allocation for High-Performance Query Processing”, we read about other memory allocators and their pros and cons. The Jemalloc allocator was the star of the show. It was noted that one of its strong suits is memory fairness, which was defined as “returning freed memory back to the OS”.

One of the ways Jemalloc achieves memory fairness is by having a time-based mechanism that releases free chunks back to the OS with the munmap sycall (it strongly relies on mmap for its allocations).

We switched our global allocator to Jemalloc, which is very easy in Rust:

use tikv_jemallocator::Jemalloc;

#[global_allocator]
static GLOBAL: Jemalloc = Jemalloc;

We ran the stress test, and surpassingly, it passed successfully. There was no notable performance difference in any of our stress tests.

We used Jemalloc stats (from the great tikv_jemalloc_ctl crate) in order to monitor memory fragmentation:

resident — Total number of bytes in physically resident data pages mapped by the allocator.

‍‍allocated — Total number of bytes allocated by the application. (in-use allocations)

resident minus allocated should be approximately equal the free memory used by our process. So free chunk memory consumption is low, which is great! There’s a small problem with the graph — It keeps growing steadily. This can be explained by the fact that we haven’t reached the limit where we start stalling our data stream (12 GB).

Ultimately, the test passed, and the system memory seemed to be lower by more than half compared to the same test with glibc.

So we just made our stress test pass and the system will succeed in processing one incremental view with billions of records! But will it pass if a thousand incremental views are created?

Multi-Process Architecture Memory Implications

As you probably guessed, we had a stress test that spawned a thousand incremental views, and it crashed miserably because of high memory utilization.

At the time, we were using a multi-process architecture; each incremental view had a dedicated engine process that maintained it — received row changes (CDC), and ran them through the data pipeline.

Unfortunately, the existence of 1,000 engine processes presented a new hurdle for us. While each process had its own memory regulator limiting its usage to 4GB (for example), there was no certainty that other processes wouldn’t also utilize 4GB, potentially resulting in an OOM.

Moving To Single-Process Architecture

It became more and more evident that an architecture change was needed.

A process per incremental view provides isolation between them; however, a single-process architecture holds many benefits for us in terms of performance and capabilities:

Memory fragmentation is lower.
Memory utilization does not depend on the number of incremental views.
Memory regulation in our data pipeline is easier. In a multi-process architecture, different processes are not aware of other processes resource consumption and needs.
Only one RocksDB embedding.
Overall decrease in memory overhead.
Communication between incremental views and with their coordinator is easy and fast.

For these reasons, we decided to move to a single-process architecture where all the incremental views are just asyncio tasks maintained by one process. Now, instead of depending on the number of incremental views, our memory utilization depends on the total number of sql operations.

Is Single-Process Architecture Better For SQL Engines?

It’s important to note that single-process architecture isn’t necessarily better than multi-process, as with everything in life, there are tradeoffs.
Just to name a few:

It may be harder to implement. e.g. aborting a query (or incremental view in our case) is harder. In multi-process architecture, it is as easy as killing a process. This is less critical for us since incremental views are seldom aborted.
Different query executions share the same address space, so a memory bug in one query might affect other queries. Fortunately, this is of lesser concern for us, thanks to Rust.

PostgreSQL for example, uses a multi-process architecture for its worker processes (the processes in charge of executing queries).
In the Let’s make PostgreSQL multi-threaded thread, there’s an excellent comment that portrays the negative impact of single-process architecture:

OOM and local backend memory consumption seems to be one of the main
challenges for multi-threadig model:
right now some queries can cause high consumption of memory. work_mem is
just a hint and real memory consumption can be much higher.
Even if it doesn’t cause OOM, still not all of the allocated memory is
returned to OS after query completion and increase memory fragmentation.
Right now restart of single backend suffering from memory fragmentation
eliminates this problem. But it will be impossible for multi-threaded Postgres.

Basically, when a query’s execution is complete, Postgres can choose to kill the process if it detects that it has irregular physical memory usage or memory fragmentation. That way, Postgres eliminates both of these problems.
Having said that, while a query’s execution time is usually between milliseconds and hours, an incremental view’s execution rarely completes — it should be up until the view was dropped by a user or Epsio was shutdown. Therefore, this mitigation is also less relevant for us.

Conclusion

Memory-intensive applications can pose challenges. It’s crucial to consider the internal behavior of your allocator, regulate memory usage, and not rely solely on the OS to save the situation. Experimenting with different inputs and scenarios can cause different allocation patterns that might or might not cause fragmentation or bloating.

If you want to try Epsio’s blazingly fast (and super memory regulated) SQL engine, check us out here.

References:

Epsio Blog

Releasing ioflame: an eBPF based I/O tracer and visualizer

Profiling is.. easy?

I/O tracing: a surprisingly hard problem

Life of a read

Associating bio requests with files

To userspace and beyond

Wrapping up

I love UUID, I hate UUID

The Problem with Indexing UUIDs (and specifically, UUIDv4)

Enter UUIDv7

Why is stream processing hard?

A Death By A Thousand Components & Configurations

Why Is Streaming Inherently Hard?

1 — Stream processors are highly coupled to other data products

2 — Stream processors are batch processors, plus more

How Were These Difficulties Overcome Until Today?

A New Generation Of Stream Processors

Moving Forward

The Bidirectional Stream Processor: Why Pull Beats Push for Crash Recovery

Problematic Scenario #1 — Connection to Sink Breaking

Utilizing transactions

Making writes idempotent

Problematic Scenario #2 — Engine crashes Mid-Processing

How does Epsio solve this?

Concluding Thoughts

How we built a Streaming SQL Engine

What is a Streaming SQL Engine?

How to build a Streaming SQL engine

A slightly more interesting example

Filter Node

Joins

Group Bys

Putting it all together

Wait, but why go to all the trouble?

In Conclusion

How we made our streaming joins 50% faster

Background: Symmetric Hash Join

The Population Problem

Let’s rock!

What’s compaction, or: Where did my CPU cycles go?

Disabling compaction for bulk load?

The asymmetry of joined tables

Using the asymmetry to minimize compaction overhead

Can we exploit the asymmetry further?

Key Takeaways

Epsio's Diff in the Streaming Landscape

A briefer- Why is everyone so interested in incremental views?

First Category — The Databases/Warehouses themselves- Batch Processors in disguise

Second Category — Stream Processors

1. Does this stream processor give “correct” answers?

2. How much cost/risk/time is inherent in trying to integrate them?

So where is Epsio in all this?

Streaming SQL Arithmetic

How Epsio Stores Result Data

Enter Floating Points

Weird Things (aka Arithmetic with Floating Points)

Why does this matter in streaming SQL

(Finally) The Bug

Why this happened

Why We Built a Streaming SQL Engine

Why we built Epsio

Why our queries became slow: The gap between the data we store and the data we seek

How did we overcome these performance issues until today?

Introducing Epsio

How does this actually work?

Why Epsio is (not) a new database

So, what’s the catch?

Concluding thoughts

Out Of Memory Shenanigans

What happens when the machine is out of memory?

Epsio View And Its Big Pipeline

Malloc And Its Shenanigans

Malloc Being Greedy

Memory Profiling

Malloc And Multi-Threading

Memory Fragmentation

Switching Memory Allocator

Multi-Process Architecture Memory Implications

Moving To Single-Process Architecture

Life of a `read`