Ergo: a case study in approachable invention

What a small framework can teach us about system design

The importance of invention

The Depth of Knowledge (DOK) model categorizes thought into four levels of complexity:

1. Recall. Memorizing, matching, reciting, quoting

2. Skill. Inferring, categorizing, interpreting, predicting

3. Strategy. Critiquing, developing, comparing, concluding

4. Extend. Design, connect, synthesize, create

DOK helps educators create a curriculum with varied cognitive rigor. A good math exam would require students to identify basic formulas, choose the right one, analyze data, and solve a real-world problem. It also helps explain why some students excel at certain assignments but struggle with others.

We can look at our system design skills through a similar lens.

In keeping with our theme of simplicity, I propose just three levels:

1. Facts. The difference between SQL and NoSQL.

2. Heuristics. SQL for structured data, NoSQL for unstructured data.

3. Inventions. Distributed SQL.

Our Discord case study displays engineering at all three levels of complexity.

1. Facts: Knowing enough about programming to make the app fast.

2. Heuristics: Adopting Elixir, Rust, and ScyllaDB, despite their newness.

3. Inventions: Creating a “Super-Disk” abstraction to guarantee both speed (SSDs) and reliability (hard drives).

Discord’s inventions are the most impressive.

“But we’re not at Discord scale — our apps don’t need to support trillions of messages or millions of concurrent users. Plus, we can pass most system design interviews with memorization alone. Why not just move on and build stuff instead of worrying about creating more abstractions?”

I get it.

But it’s a shame to build a strong base during interview season, only to let it atrophy after day one on the job, obediently accepting the precedents forever.

Invention is worthy of pursuit. Although you’ll never be assigned a ticket to “Invent New Abstraction,” doing so is one of the most high-impact things you can do as an engineer.

You get simplicity by finding a slightly more sophisticated building block to build your theories out of.
— Alan Kay

Invention case study

Carl Hewitt’s Actor Model introduced a communication protocol of messages and mailboxes that solves race conditions. Although the model is a beautiful invention, creating something so esoteric feels a bit out-of-reach for those of us who don’t have a PhD from MIT.

Actors: Hewitt’s building block

Thankfully, we don’t have to introduce a new paradigm to invent; plenty of opportunities arise within our current stack. The Ergo Framework is a perfect example of how regular engineers can invent abstractions that simplify rather than complicate.

The problem

Ergo is an implementation of the Actor Model in Go, bridging the gap between a 50-year-old theory and a modern programming language. The basics of the actor model are perfectly compatible with the Go language — queues for mailboxes, trees for supervisors.

However, there’s one scenario where the language conflicts with the model: I/O (file reads, listening to OS signals, TCP accept). The Actor Model requires a balance of async delivery with sync processing. That’s because Actors work sequentially. One message arrives, gets processed, completes. Next message. This strict workflow is what eliminates race conditions.

Blocking I/O breaks this model.

In Ergo, a normal Process is an actor — a lightweight entity that handles messages sequentially in its own goroutine. There’s one process per goroutine. Processes can make sync calls and create monitors — standard actor stuff. Since there’s one goroutine, it must handle everything. If it blocks on I/O, message processing stops.

Call net.Listener.Accept() in a handler and watch the goroutine freeze while it waits for connections. Messages pile up until the system times out.

Let’s try solving the problem with a simple heuristic.

“Compute is cheap, add more. Moore’s Law. Just spawn a new goroutine for Accept().”

This bails you out of the immediate bottleneck. But now the two goroutines access the actor’s state concurrently. We just traded a perf problem for a race condition problem.

“Add a lock” (level one).

Sure, but now we have to add a bunch of code to handle the lock’s complexity.

The only way we can preserve the guarantees of concurrency that come with the Actor Model and the I/O use-case in Go is to invent a more elegant abstraction (level three).

The invention

The Ergo authors created “Meta-processes” for this situation.

Unlike normal processes, which have one goroutine per process, Meta-processes have two: a Reader and a Handler. The reader is for blocking operations — Accept() loops, ReadFrom() calls. The handler does everything except I/O via sequential messages in traditional Actor fashion. The Reader runs continuously from spawn until termination. The Handler comes and goes based on message traffic.

Meta-processes separate concerns. The Reader handles I/O, while the Handler processes messages. The Reader blocks reading while the Handler blocks writing. Two blocking operations, two goroutines, neither interrupting the other.

Meta-processes mediate the purity of the Actor Model with the messiness of the real I/O world. They turn blocking operations into async messages and messages into blocking commands. Regular processes implement everything else.

It’s a wonderfully simple distinction that a lay-developer can implement, even if they don’t appreciate the mechanism.

Meta-process flow. From ergo.services

The takeaway

To escape a bind, add constraints.

To escape locks in concurrent programming, the Actor Model added communication constraints: messages and actors. Authors need to follow the rules by only sending messages and not sharing state, but they get guaranteed concurrency.

To escape blocking I/O in single-processes, Ergo added process constraints: one for reading, one for writing. Authors need to discern whether they’re working with I/O, but they get to keep concurrency in Go.

It’s counterintuitive to consider adding more rules in the pursuit of simplicity. It’s also easy to add the wrong rules and get more complexity without extra benefits. Tolerating this tension is a prerequisite to finding a level three invention. It’s the kind of challenge that comes from both understanding the theory and working in the slimy guts of a problem.

You can’t memorize or skim your way out of these situations.

That’s why they require level-three thinking.

Keep memorizing definitions, skimming, and interviewing. But also pursue answers that aren’t so accessible. Our industry is increasingly outsourcing its thinking; we need those who continue to exercise their cognition more than ever.

Bonus: More inventions

MapReduce: map() + reduce()

Containers: cgroups + namespaces

Git: immutable snapshots

CRDTs: sync + merge without locks

Virtual DOM: diffs against the real DOM

Promises: Async that reads like sync

Resources

Depth of Knowledge: Difficulty and Complexity (webbalign.org)

DOK chart (pdesas.org)

The Actor Model: Messages and Mailboxes (fullstack.zip)

Discord: A Case Study in Performance Optimization (fullstack.zip)

Ergo Basics (ergo.services)