Devs love speed, which is why we’re obsessed with benchmarks and Rust. But speed is not just about acceleration; it’s also about mass.

So what’s the heaviest part of our codebase?

Dependencies.

In other words, other people’s code that we rely on to make our code work. GitLab’s 2024 survey of 5,000 DevSecOps pros claims that over 25% of the code devs work on is from open-source. Those guys are either working on CLIs or lying. My calendar app’s codebase has 10 million lines. I only wrote 116,351 lines — just 1.2% of the product. The other 98.8%? Other people’s code.

These numbers are insane, but not anomalous. Web apps are especially bloated these days. This is why we should spend less time looking at the speedometer and more time looking at the scale.

Dependencies are weight

Every dependency adds weight. The more you carry, the more every step costs, and the slower you move. Using minimal dependencies is like traveling internationally with a backpack. You might be going to the same destination as your buddy who brought two suitcases, but you’ll have vastly different experiences.

Let’s start with the heavy traveler, the one who has a “checked” codebase. What is his life like?

Checked codebases

The repo started small and grew quickly. Each dependency felt like a good idea at the time. Of course, you need an HTTP client (even though it comes with a bulky logger). Of course you need that component library with strong opinions about theming (even though you’ll only use five of its components). Of course you should start with $FRAMEWORK_THAT_EVERYONE_IS_USING.

He started with a lot, added more along the way, and never stopped. The PMs were eager to add features, and no one slowed the sprawl.

Now the builds take forever, no one understands why half the packages are there, but they’re all too afraid to do anything about it. So instead of simplifying globally, everyone optimizes locally. He’s greasing the wheels of a 50-lb suitcase instead of taking out the 10-lb boots. He optimizes, but he’s still big and slow.

Traveling with a checked codebase is exhausting. It takes a lot of planning to do a little movement. Each implementation step is also laborious, as he has to check so many surfaces for downstream bugs. He’s forced to bring a month’s worth of clothes for a short weekend getaway hotfix. By the end of the trip, he’s more exhausted than when he left.

Carry-on codebases

The minimalist is initially scoffed at as a simpleton, an irresponsible kid unprepared for the prod life. He sticks to what works, fights complexity vigorously, and has enough clout to be trusted. As a result, his dependency tree is minimal. Everyone knows what every package does. Sure, some teammates grumble about how a modern tool would be faster. But, on the whole, things are working. The builds are fast enough, newcomers can understand the code in days instead of weeks, and things get shipped.

Traveling with a carry-on codebase is chill. A POC gets demo’d. Builds pass. Merged. Done. There are fewer hotfixes, patches, docs, and meetings. The trips to Featureville actually feel energizing, and the team looks forward to the next experiment.

When the requirements change, it’s easy to recalibrate and move forward. Coding here feels like freedom. Not from bugs or feature creep, but from the weight that used to make overcoming those challenges so draining.

The reality of travel

Traveling light has become even more important recently, because travel itself has remained complex. While the vibe coding narrative promises to collapse the SDL into a one-shot prompt, we know that there are still many steps to shipping real products.

Pull → install deps → run tests → wait for CI → deploy → retest → get design feedback → get PR feedback → repeat → deploy to prod → fix bugs → write docs.

The journey is long. At every stage, a heavier dependency graph adds time. Moreover, the dependencies themselves require attention.

We still need to upgrade them. This is like replacing a backpack with a hole at the bottom with a new one.

We’ll need to swap heavy deps for alternatives. This is like swapping a heavy cotton sweater with a light dry-fit one.

We’ll still need to remove third-party libs with built-ins. This is like leaving your glass tupperware at home when you learn the hotel provides some.

Like it or not, dealing with dependencies is still part of our lives, which means it’s still helpful to learn how to pack lightly.

A small dependency graph will not make your product automatically great. But it will make the product easier to build, easier to understand, easier to debug, and easier to adapt when reality changes.

A light traveler is a fast traveler.

Story from the road

I fly with this backpack instead of those rolly carry-ons that real adults use. It fits overhead and underneath. It lets me take the stairs at the hotel when the elevator line is long. It helps me blend in at a coffee shop. When the ski lifts shut down, it allowed me to haul my gear to the top for one last run. My fancy shoes don’t fit, which means I lose some fashion points. But I gain so much speed and lightness. Once you feel that freedom, it’s hard to go back.

More

On the Relationship between Software Complexity and Maintenance Costs (2014).

Software complexity and maintenance costs (1993) | Analyzes Cobol maintenance projects in a large commercial bank.

Software Dependencies, Work Dependencies, and Their Impact on Failures (2008) | How dependency measures relate to defects.

DevSecOps Report: Application security in the digital age (2024) | GitLab’s survey

Do you think traveling light is always better, or are there exceptions?

Anthropic accidentally leaked its Claude Code source code. Although the archived repo is easy to find, deeper analysis is not. The popular reactions are surface-level, while the technical reports read like a mindless doctor’s note written by Claude itself.

“It’s just a Bash wrapper.”

“Wait, that’s it? This is so hack-y.”

“There’s an undercover pet!”

“It’s a CLI with multiple layers of abstractions.”

I blame the lack of substance on the legal grey zone we’re in: linking the repo or sharing specifics would anger the copyright police. At least that’s what my $20/month lawyer/SWE tells me.

After reading through the repo, I nevertheless have to share this amazing fact:

Claude Code has its own skill for simplifying code (simplify.ts).

That’s not as flashy as those other headlines, but it’s still important.

Coding with Gemini, Grok, or Cursor can feel like a game of whack-a-mole. They’ll add the feature I asked for and pass the tests…but they also litter the code with unnecessary comments, dependencies, and abstractions. Claude’s submissions feel more elegant — as if it spent some extra time refining away the rough edges first.

Well, that’s exactly what they’re doing, and it’s all documented in simplify.ts. Instead of pointing you to that file, though, here is my own “simplify-code” SKILL.md, which is spiritually equivalent to Claude’s.

// Excerpt from my SKILL.md for simplifying code 
---
name: simplify-code
description: Simplifies code by minimizing complexity, eliminating duplication, and prioritizing legibility for contributors. Use when implementing features, fixing bugs, refactoring, or when the user asks to simplify, clean up, make DRY, reduce complexity, or improve maintainability.
---

# Simplify Code

Favor minimal, legible implementations. Fewer lines and clearer structure make code easier to understand and maintain.
...

You and I aren’t the only ones who recognize the importance of this simple engineering philosophy. Claude is used by a whopping 75% of the smallest companies (ie, the ones that are free to use whatever they want) and is already the overall market leader, despite being released only 11 months ago.

It seems we agree:

When it comes to code,

Less > more.

Simple > complex.

We’ve been willing to pay for simplicity with our wallets and our time all along. Now we finally know why.

A few bonus observations:

Mixed file names. You’ll see kebab-case, camel case, Pascal case, and lowercase all in the root directory alone.

A few massive files. The entry point is 4600+ lines long, for example.

Types, kinda. It’s 100% TypeScript, but they don’t rely on types very heavily. There are lots of optional properties, and types aren’t widely shared.

My instinct tells me these are less engineering conventions they settled on and more symptoms of an internal POC that rapidly evolved into a mainstream product.

What was your first reaction to the leaked code?

More

AI Tooling for Software Engineers in 2026 | Pragmatic Engineer | usage stats

The format

They’re system design case studies, featuring popular apps.

Part I: Architecture. The stack, protocols, and main players that the featured app relies on.

Part II: Feature Deep-Dive. How everything comes together in the app’s most interesting feature.

Part III: Context. Where these ideas come from, and what they mean for regular devs like you and me.

The backstory

I broke down a few apps on my personal channel in 2024, and engineers seemed to like them. I got comments like this one:

Big fan of this! Definitely wanna see more video breakdown apps! :)

Of course, I ignored that feedback and instead uploaded 100 unrelated videos. When those flopped, I finally thought, “Maybe I should just give my audience what they want.”

I created “Fullstack Engineering” to be the strictly business vehicle for this educational content.

You’re probably familiar with the Fireship channel (4.1M subs as of today). If those videos are espresso, Fullstack Engineering is pour-over. They’re longer in format, but richer in context.

My goal with this channel is to give you something to watch during your WFH Friday lunch that educates and inspires. If you go back to the IDE excited, that’s a win.

Here are the first four.

Reply to LMK what app you wanna see next.

Before coding agents, a PR went through more checkpoints. You’d suggest a design pattern in an RFC or an issue comment. Your teammate would advocate for a different pattern. You’d go back and forth on the idea before committing to a tradeoff. After a day of POCing the first approach, you realized there’s an easier way, so you’d update the RFC or have another conversation. Then you’d submit a draft PR. More feedback from teammates who skimmed it with fresh eyes. They asked for more tests and told you how you could ditch that new library for a built-in. More refinement. By the time you make the PR public and everyone approves, your code is 2X simpler than the first version.

Things are different now.

Instead of clarifying our thinking in an RFC or issue, we prompt and get a PR. An agent reviews the code and auto-fixes major bugs. Another writes the PR description.

The end.

Sure, there’s a little back-and-forth between you and the agent, but only enough to make the tests green. Even reading the code is starting to feel like an inconvenience.

Our velocity has skyrocketed, but so has the complexity of our codebases. Part of that complexity is inevitable. Yet more is a byproduct of accepting the agent’s defaults without scrutiny. The agent doesn’t want to waste tokens refining its already green solution, and we don’t want to waste time nitpicking. So extra libraries get added, nested if statements sneak in, and kludgey abstractions get overlooked.

Teams won’t care until their complexity debt starts to slow their velocity or introduce more bugs. At that point, it’ll be tempting to start from scratch with another prompt. But there might be a more pragmatic way out.

Solution: Simpler tools

Replace complex tools with simple tools. Keep the requirements and tests, but replace one heavy tool with a lighter one.

To make this practical, here are 40 swaps to consider, organized by domain.

Simpler Auth

1. Auth0 / Clerk → express-session + bcrypt + a users table. Don’t lock yourself into a vendor for a personal project.

2. OAuth → API keys. Trade the redirect dance with an API key in the header. Consider OAuth when your users can only auth with accounts they don’t control (rare).

3. Username / password → Magic link. No hashing, strength enforcement, or reset forms.

4. Per-route auth middleware → Single auth gateway. A single middleware that authenticates every request by default is easier than sprinkling requireAuth() on each route.

5. SSO → anything else. Add it when a paying customer asks for it.

6. JWT everywhere → Server sessions. Easier to revoke and doesn’t require rotation.

7. RBAC service → role column on a users table. Unless you’re an enterprise, WHERE role = ‘admin’ in your middleware is fine. Consider a better abstraction when you have 3+ distinct roles.

8. Third-party MFA service → TOTP + otplib. It’s an open spec, and otplib generates and validates codes in ~10 lines. There’s a maintenance burden, but it might be cheaper than locking yourself in too early.

9. Refresh token rotation → long-liven tokens + revocation list. If calls are inside your infra, you might not need to worry about refreshing yet.

10. passport.js → manual middleware. Passport adds 20+ files and a plugin ecosystem just to check a password and set a session. If you're not juggling multiple strategies, a verifyPassword function and a session check is 15 lines and nothing to maintain.

Simpler Storage

1. Redis queues → Postgres LISTEN/NOTIFY. Postgres has a built-in pub/sub mechanism. NOTIFY sends a message on a named channel; LISTEN opens a persistent connection that receives those messages in real time. You can store messages in a jobs table and tell workers to pick it up on NOTIFY. Or use a lib like pg-boss if you don’t wanna coordinate locks and retries yourself. This’ll last you until you get to a few hundred jobs / second.

2. Trendy DB → SQLite. A single file, zero config, and it handles tens of thousands of reads per second. Upgrade to Postgres when you need concurrent writes from multiple processes.

3. Time-series DB (InfluxDB) → Postgres timestamptz + indexes. For application metrics, user activity logs, or event streams at normal SaaS scale, a timestamptz column with a partial index covers the query patterns without running a second database.

4. Separate search service (Algolia, Elasticsearch) → Postgres full-text search.

5. ORMs (Prisma, Sequelize) → SQL + a query builder. OMRs abstract SQL until they don’t. If you’ll be writing raw queries anyway, a lightweight query builder can get the job done without hiding the DB from you.

6. MongoDB → Postgres JSONB. If you’re already on Postgres and don’t want to stress about schema design upfront, you can still use JSON for now without having to bring in another DB. IF your data is document-shaped AND you’re confident you’ll hit scale quickly, then ignore.

7. Redis cache → Postgres Materialized View. Pre-compute the result and store it in a table. Great way to preserve speed for complex queries and avoid a cache. Must be OK with some data staleness.

   1. Use-case: daily sales dashboard.

   2. See: Choosing Between Postgres Materialized Views and Redis Application Caching (leapcell.io).

8. S3 → local filesystem. If you’re on a single VPS, you don’t need to manage another round-trip, IAM policy, and subscription. If GitLab could run its git commands on a shared NFS filesystem, you’re probably fine to store some files locally.

   1. GitLab Architecture (fullstack.zip)

9. Multiple databases → one Postgres schema per domain. Running separate databases for separate domains means cross-domain queries require application-level joins, separate connection pools, and separate migration pipelines. A Postgres schema per domain (users.accounts, billing.invoices) gives you logical separation with a single connection string.

10. Key-value store → single-table Postgres with a type column. Redis shines for ephemeral data — rate limiting, leaderboards, session tokens. For “store this thing and look it up by key” use cases in a persistent store, a Postgres table with (type, key, value, expires_at) is queryable, inspectable, and doesn’t require operating a second data store.

Simpler Infra

1. Fly.io / Railway → 1 VPS. SSH in, run your app with pm2, put Caddy in front. You own the machine, you understand the machine, and you're not beholden to a platform's pricing or uptime decisions.

2. k8s → systemd + reverse proxy. Automatic startup, dependency checking, and failure recovery.

   • How to run pods as systemd services with Podman | redhat.com

   • Replacing Kubernetes with Systemd | news.ycombinator.com.

3. Docker → Bash + pm2. Replace the whale with pm2 (for Node apps), and you’ll free up a ton of RAM, storage, dependencies, and time.

4. Terraform → Bash. For a personal project or small team with one VPS, a shell script that runs apt install, creates users, and drops config files is fine. Terraform is for provisioning lots of stuff across teams.

5. Secrets manager → Env files + chmod 600.

6. Staging environment → Feature flags. It doesn’t get more real than prod, not matter how much you dress up your staging env. How many of us are acting like we have an SLA with five 9s when we don’t even have 5 customers?

7. GitHub Actions → git hooks + deploy script. A post-receive git hook on your server that pulls and restarts is a 10-line bash script that deploys on git push.

8. Autoscaling → Vertical scaling. What happened first, you spent a week configuring autoscaling or 10 minutes doubling your VPS size?

9. Nginx + Certbot → Caddy. Auto-TLS, one config file. HTTPS in Nginx usually requires installing Certbot separately, configuring it, setting up a cron job for renewal, and debugging when it gets out of sync with your Nginx config. With Caddy, you just put your domain name in the Caddyfile — Caddy obtains and renews the certificate automatically, handling HTTP-to-HTTPS redirects, OCSP stapling, and multi-domain SANs with no additional configuration. The makers of Certbot even agree that this is a better solution for everyone.

   1. Should Caddy and Traefik Replace Certbot? | eff.org

10. Multi-region deployment → backups + a restore script. Small SaaS apps don’t need 99.99% uptime. A daily Postgres dump to S3 and a restore script that gets you back in 10 minutes is more than enough.

Simpler Libraries

1. ESLint + Prettier + Husky → biome. Biome replaces that entire stack with one tool — it’s 10–25x faster than ESLint and ships as a single binary instead of 127+ npm packages. Caveat: it doesn’t cover every ESLint plugin (e.g. eslint-plugin-jsx-a11y), so teams with heavy custom rule sets may not get a clean 1:1 swap.

2. Jest/Vitest + mocking libs → node:assert. Ideal for simple backend unit tests in pure Node.js environments where minimal dependencies are a priority (nodejs.org)

3. .env + dotenv library → Native --env-file flag. Since Node 20, you can load your env file natively.

4. node-config → env files. node-config is a popular library for managing environment-specific config (development.json, production.json, etc.). For most projects, a plain .env file per environment covers the same ground with zero deps.

5. uuid → crypto.randomUUID(). Why does the uuid package get 91 million downloads per week when crypto is native and 4x faster? I think it’s just a better name.

6. nodemon → node --watch. Unless you need granular ignore patterns, this simple flag is good for anyone on Node 18+.

7. axios → fetch. Works identically in browsers. Can upgrade if you need interceptors or fancy errors.

8. moment.js → Date or Temporal. Ditch the bulky libs for a lib that the JS ecosystem spent a decade pushing through. (You’ll need a polyfill until Safari gets its act together).

9. lodash → array methods. Lodash predates modern JS. _.map, _.filter, _.groupBy, _.flatten are all native today. Another example of muscle-memory keeping a heavy lib around.

10. express → v0: node:http, v1: hono. Node for simple internal services, hono if you need a router on the edge. 0 deps.

Conclusion

It’s easy for me to tell you to “just replace this whole part of your stack.” It’s hard to do this in a company, especially when you need to convince teammates along the way. After spending the last three months doing nothing but this type of cleanup in Compass Calendar, however, I can tell you it’s worth it. You’ll have fewer dependencies, fewer lines of code, and fewer headaches. You’ll start to understand the codebase again. Best of all, you’ll develop a deep conviction that’ll help you forever separate signal from noise:

It doesn’t have to be this complicated.

More

Simple is smart (fullstack.zip)

A Plea for Simplicity (schneier.com)

Choose Boring Technology (mcfunley.com)

77 Things to avoid in 2026: Developer edition (fullstack.zip)

The Law of Leaky Abstractions (joelonsoftware.com)

“I’m {FirstName}, and I’m a complex-oholic.”

Say it.

It’s OK, we’ve all been there — prioritizing our ego ahead of our users.

Deep down, we know we were overdoing it all along.

Not ready yet?

All good, it takes time. For now, skim this list and let yourself yearn for the days of yore, when life seemed so straightforward.

Simplify your system

1. Microservices → modular monolith. If you’re a small team, a monolith with clean module boundaries gives you the same separation of concerns without the network tax (Modular Monoliths).

2. Abstract base classes → plain functions + composition. If your class has one subclass, it probably shouldn’t be a class (The Composition Over Inheritance Principle).

3. Feature flag service → Env var + if statement. LaunchDarkly and friends are powerful when you have millions of users. You don’t, so if (process.env.IS_PREMIUM_USER === 'true') is fine (How To Build Feature Flags Like A Senior Dev In 20 Minutes).

4. Semantic versioning automation → git tag + a changelog entry. (Keep a Changelog)

5. GraphQL → REST + JSON. Do multiple clients need different shapes of the same data? If not, a REST API will be easier to build, cache, and debug (You probably don’t need GraphQL).

6. Event-driven everything → direct function calls. If two things always happen together, just call one from the other and save yourself the tracing scavenger hunt (Method Calls vs Event-Driven Architecture in a Modular Monolith API?).

7. API gateway → reverse proxy routes. Gateways come with rate limits, middleware, routing, and policies. Your existing proxy can do the essentials with a few config lines (Is an API Gateway a Reverse Proxy?).

8. Pub/sub → EventEmitter. Node’s built-in can handle internal pub/sub within a single process without a broker or network hop (Redis pub/ sub vs node event emitter).

9. Polyrepo → monorepo. This’ll cut down your CI configs, versions, PRs and repos. Go further by skipping Nx/Turborepo in favor of a simple Makefile (Understanding Monorepos).

10. Multi-tenancy → multiple single-tenants. Every unicorn engineer will proudly talk about their journey from single-tenant to multi-tenant. Every tech lead of a graveyard startup will reluctantly admit how they spent too much time optimizing instead of iterating (Choosing the right SaaS architecture: Multi-Tenant vs. Single-Tenant).

Did you reject these indignantly? Perhaps you bristled at the thought of a vet like you doing something so elementary.

Well, hopefully you’ll scale so quickly that you outgrow these script-kiddie tactics in a few months. None of the fancy things is wrong; they’re just wrong now, at this scale.

Just ship and help your users, then let the real problems tell you what to add back.

Complexity has its place.

Just not yet.

This article follows the steps that GitLab takes for you after you push a commit.

TL;DR

• Pipelines are declared in gitlab-ci.yml, then compiled into an internal job graph and persisted in Postgres.

• Jobs move through states like created → pending → running → success

• Merged results pipelines validate your changes against the current main.

• Merge trains validate your changes against the eventual main.

• This is all based on dependency graphs, producer-consumer queues, and optimistic validation.

YAML → Postgres

Step 1 — Git event received. GitLab Shell (SSH) or Workhorse (a Go reverse proxy) receives the pack data, stores it via Gitaly (gRPC for git data), and triggers a post-receive hook.

Step 2 — Ci::CreatePipelineService runs. Rails fetches the YML via a Gitaly GetFile RPC. The YAML goes through several passes:

• include: directives are resolved — local files, project templates, even remote URLs. This is recursive and can get deep.

• Variables like $CI_COMMIT_SHA AND $CI_PROJECT_NAME are interpolated.

• rules:, only:, and except: are evaluated against the pipeline context (branch, commit message, changed files, schedule) to determine which jobs actually apply.

• needs: relationships are resolved into the DAG (more on that later)

Step 3 — Everything is written to PostgreSQL in one transaction:

After GitLab parses .gitlab-ci.yml, it stops treating the file as the source of truth. Instead, it compiles that config into relational rows. One row represents the pipeline, a few rows for its stages, one row per job represents runnable work, and extra rows for dependencies.

In other words, GitLab turns your YAML into a small execution graph stored in PostgreSQL. That graph is what the scheduler and runners operate on later.

Here’s pseudo data to help visualize what things are like in Postgres:

PIPELINE
  5001  main@abc123  status=created

STAGES
  6001  build   pipeline=5001
  6002  test    pipeline=5001
  6003  deploy  pipeline=5001

JOBS
  7001  compile  stage=6001  status=created
  7002  rspec    stage=6002  status=created
  7003  jest     stage=6002  status=created
  7004  ship     stage=6003  status=created

NEEDS
  7002 needs compile
  7003 needs compile
  7004 needs rspec
  7004 needs jest

Here is what a simplified ci_builds row looks like at creation:

SELECT id, pipeline_id, name, stage, status, runner_id, yaml_variables
FROM ci_builds
WHERE pipeline_id = 99801;

-- id      | pipeline_id | name           | stage | status  | runner_id
-- --------|-------------|----------------|-------|---------|----------
-- 4410023 | 99801       | lint           | test  | pending | NULL
-- 4410024 | 99801       | unit-tests     | test  | pending | NULL
-- 4410025 | 99801       | docker-build   | build | created | NULL  ← waiting on stage order

Note that a job in created is still invisible to runners at this point; it exists in the DB but is not yet in the queue.

Step 4 — Ci::InitialPipelineProcessWorker is enqueued.

This Sidekiq job calls ProcessPipelineService, which evaluates the DAG, finds all jobs whose dependencies are satisfied, and transitions them from created → pending.

A job in pending is now on the ticket rail, so runners can see it.

Postgres → Runner

Runners poll POST /api/jobs/request every ~3 seconds. On the Rails side, this hits Ci::RegisterJobService, which:

1. Queries ci_builds for pending jobs matching this runner’s tags and project access

2. Attempts to atomically transition the selected job from pending to running using optimistic locking

3. If the update succeeds, it returns the full job payload — env variables, Docker image, scripts, artifact definitions, etc.

4. If another process won the race, it returns 204 No Content, prompting the runner to retry next cycle.

Runner → Green Check

Workflow 1: Merge Requests

Merge request pipelines answer the baseline CI question: “Did my change work?” When you push, GitLab ensures you can build, test, lint, and deploy that change in isolation by testing a temporary merge commit that combines source and main. Nothin’ fancy here.

Workflow 2: Merge Trains

“Will Cory’s code break mine if it gets in first, though?”

Now you’re asking the right question.

When you’re working on your side project, you don’t need to worry about this. You’re the only one merging into main, so if it passes on your branch, it’ll pass in main. On a collaborative project with many changes landing at once, however, this isn’t guaranteed: two branches that each pass CI individually might conflict when combined.

Merge trains address this by testing the result of merging your branch into the target, rather than your branch in isolation.

A merge train is a queue of Merge Requests (MRs), where each is compared to earlier ones to ensure they all work together.

For example:

Four merge requests (M1-M4) are added to a merge train in order, which creates four merged results pipelines that run in parallel:

1. The first pipeline runs on the changes from M1 combined with main.

2. The second pipeline runs on the changes from M1, M2, and main.

3. The third pipeline runs on the changes from M1, M2, M3, and main.

4. The fourth pipeline runs on the changes from M1, M2, M3, M4, and main.

If M1 completes successfully, it merges into main, while the other pipelines continue to run. Any new MRs added to the train include the M1 changes from the target branch, along with the other changes from the merge train.

That’s the happy path.

Here’s where it gets fun.

If the pipeline for M3 fails:

• M1 and M2 continue to run.

• M3 fails and is removed from the train.

• A new pipeline starts for the changes from M1, M2, M4 and main (without M3).

The payoff: M4’s dev got the changes from M1 and M2 without manually restarting their pipeline. Without merge trains, M4’s dev would either be waiting around for M1 and M2 or removing M3’s crappy code.

While this “speculative thing on top of another speculative thing” got the banks in trouble in 2008, GitLab thankfully hedges risk a little more responsibly. Think: automatic rebasing, hiding speculative future states, fast forwarding, juggling Git and MR refs, capping parallel pipelines (to 20). It required years of behind-the-scenes work.

This train flow puts more pressure on the dev team to maintain a reliable test suite, as a single flaky test can make the difference between a smooth and bumpy ride.

The payoff is worth it, though: developers can be confident that their changes will work against both the present and the near future of a shared branch.

(For the record, you don’t get all this for free; Merge trains are part of GitLab’s Premium tier.)

Historical Context

The ideas that support GitLab’s CI originally popped up in the 50s and 60s. Understanding their history will make all of this feel less magical (in a good way!).

Pipes

The most obvious precedent is the pipe. In 1964, Douglas McIlroy wrote an internal Bell Labs memo proposing that programs be linked “like a garden hose — screw in another segment when it becomes necessary to massage data in another way.” He envisioned software as interchangeable components connected through data streams. The idea sat dormant for nearly a decade before Ken Thompson implemented pipes as a core Unix feature in 1973.

ps aux | grep chrome  # is chrome running?
ls -1 | wc -l         # how many files in cwd?

This idea had the power to extend further. Imagine your email and projects being piped into your calendar, and then to your team’s combined data lake. However, platforms are disincentivized to share: Google Drive and Dropbox don’t want to share the same files. Perhaps AI will revive this idea. You could imagine that all apps tap into a shared phone context, rather than each having its own data.

The idea that complex work should be expressed as a chain of discrete, single-purpose steps where the output of one becomes the input of the next is the foundation of every CI pipeline ever run. The stages: keyword is the same idea in different clothes: break your build process into named phrases, run them in order, and let each phase’s artifacts flow into the next.

DAG Scheduling

GitLab’s needs: keyword lets you write pipelines like this:

build:
  stage: build

unit-tests:
  needs: [build]

integration-tests:
  needs: [build]

deploy:
  needs: [unit-tests, integration-tests]

This is a Directed Acyclic Graph (DAG) — a graph of dependencies with no cycles, where the sort of nodes gives you a valid execution order. Multi-step projects are represented as directed graphs so that the longest sequence of dependent tasks (the “critical path”) becomes clear. This allows operators to find where work could be parallelized without delaying the overall timeline. The strategy was used to schedule chemical plants and coordinate the Polaris missile program.

GitLab’s job scheduler does the same thing. ProcessPipelineService evaluates the job DAG on every status change, unblocking downstream jobs the moment their dependencies complete.

The specific algorithm is Kahn’s topological sort, published in 1962: repeatedly select nodes with no incoming edges, emit them, remove them from the graph, repeat. Every CI scheduler does some variant of this.

Optimistic Concurrency Control

What if hundreds of runners simultaneously call POST /api/jobs/request, causing multiple Rails processes to receive these requests concurrently? How can they ensure that the same pending job is not handed to two runners at once?

The naive, pessimistic answer is a global lock: before selecting a job, lock the table, pick one, mark it running, unlock. This works, but at scale, it creates a queue of Rails processes waiting for the lock, and throughput tanks. Even if deadlocks are avoided, this locking limits concurrency.

Optimistic Concurrency Control (OCC), formalized in 1981, is an alternative that works well in environments with low data contention. The premise: instead of locking preemptively, assume conflicts are rare, proceed without locking, and detect conflicts at commit time.

In practice, GitLab’s RegisterJobService does this:

UPDATE ci_builds
SET    status = 'running',
       runner_id = $1,
       lock_version = lock_version + 1
WHERE  id = $2
  AND  status = 'pending'
  AND  lock_version = $3;  -- ← the OCC check

If another Rails process already picked this job (incrementing lock_version), this UPDATE matches zero rows. The service detects the miss and returns 204 No Content to the runner, which simply tries again on the next poll cycle. No deadlock, global lock, or serialization. Conflicts are rare, and when they do occur, only one poll cycle is wasted.

Where things go from here

Everything here assumes a human pushes code, waits for a runner, and sits around waiting for a green check.

That workflow is starting to feel antiquated.

If n agents are committing to n worktrees simultaneously, doesn’t it seem odd for them to poll and coordinate complex trains?

I think the primitives — pipes, DAGs, OCC — will stay, but the workflows built on top of them will look very different.

Whatever happens, one thing is certain: having a reliable and fast way to evaluate code is always going to be important.

Recommended Reading

GitLab’s Architecture | fullstack.zip | Stack and overview

Unix Pipeline History | Wikipedia

The Bell System Technical Journal | 1978 | McIlroy’s philosophy

Directed Acyclic Graph | Wikipedia

On Optimistic Methods for Concurrency Control | Harvard | The original 1981 paper.

Optimistic Concurrency Control | Wikipedia

Ultimate vs Immediate Goals

In How to Focus: A Monastic Guide for an Age of Distraction, young monks ask a master how to reconcile their tendency to chase goals with their vocation to focus on the present. The master differentiates between an ultimate goal (telos) and an immediate goal (scopos). Pursuing the ultimate goal provides purpose, while pursuing the immediate goal provides focus.

The merchant’s ultimate goal of profit keeps them centered when trouble happens at sea. The soldiers’ ultimate goal of honor helps them face the danger of their campaigns. So what’s the monks’ ultimate goal? Salvation. Makes sense, but too abstract for the young monks to act on daily. So the master recommends pursuing salvation by focusing on the immediate goal of “clarity of heart,” achieved through solitude, fasts, labor, and reading.

This empowered the young monks to channel their existing ambition into daily habits rather than attempting to remove their ambition.

5th-century monks aren’t the only ones who can direct the power of flawed nature toward good. Although our goals are less noble, engineers also have the opportunity to work with our nature — or against it.

I’ll recommend a goal at the end. But first, we have to start with the question:

What we do naturally

We build things. Not because we have to, but because we can’t stop. 52% of devs code for fun even after coding all day.

We automate the boring stuff. Give a dev a mindlessly repetitive task and watch him spend three times as long building a buggy script to eliminate it.

We learn obsessively. Yes, partly to keep up with the industry’s pace. But things can only move so fast because so many of us refuse to wait for permission to pick up a new skill. We’ll find the docs, read the src, and figure things out regardless.

We give a sh*t about craft. Tech debt is the top frustration at work, cited by 62% of us — ahead of tool reliability, deployment pipelines, and security issues. We hate living in a mess (especially if we didn’t create it).

We give back. 25% of developers actively contribute to open-source, which doesn’t include those who volunteer to help with issues, docs, sponsorship, or organizing.

We’re drawn to hard problems. Two out of three developers say they are motivated by a passion for learning new things, and the problems that excite us most are the ones nobody has cleanly solved yet.

What we naturally resist

Context switching. The average digital worker toggles between apps and websites nearly 1,200 times per day and spends 4 hours per week reorienting themselves. I don’t know the numbers for devs, but I bet they’re worse. It requires sustained focus to hold the codebase in our working memory, which is why everyone hates silly meetings and interruptions.

Devs in Japan are 4X less likely to struggle with context switching than those in the UK. (IDK what to do with that fact yet, but it’s interesting.)

Waiting for CI. Slow feedback loops also break the flow.

Hollow metrics. We know productivity KPIs are BS: 66% of developers don’t believe or aren’t sure that current metrics reflect their real contribution.

Fixing other people’s messes. We hate spending time debugging legacy code, untangling connective tissue between APIs, and working around workarounds. Unless we made the mess, then it’s fine.

Estimation theater. Estimating costs and timelines while requirements change every other day and everything is on fire seems silly.

Meetings. Most engineers spend nearly a third of their week in meetings. We resist this instinctively, because we know exactly what we could have shipped instead.

Put it all together, and you get a group of creative, crafty builders who are motivated by hard problems and are allergic to inefficiency. We just want to ship things that matter and leave the codebase better than we found it.

Application

Now that we know ourselves, we can pick goals that work with our nature rather than against it.

This is easy for your day job: your ultimate goal is to maximize value — generate more revenue or reduce expenses (cringe, I know). Feel free to adopt a more altruistic goal on your employer’s behalf, but don’t complain when you get cut during the next layoff.

For your side project, pick whatever ultimate goal resonates.

Some ideas:

• Fix a personal pain point

• Learn a tool that excites you

• Make enough to replace your main income

• Build the tool you always wished existed

The important point is that you commit to one ultimate goal. Don’t optimize for four things simultaneously.

The traps hide in the immediate goal, as it often leads to complexity. Initially, complex feels like progress. It’s challenging, exciting. It gives you an excuse to learn and write more code. More features, more services, more dependencies.

But complex things don’t last.

That’s why I humbly recommend making your immediate goal to preserve simplicity.

Fix your problem with a simple stack. Learn a new thing, simply. Make enough $ to replace your main job ... using simple workflows.

This works because it embraces our tendency to find creative solutions. Instead of adding, however, it forces us to use that creativity to subtract. Instead of dragging kludgy abstractions along, it asks us to find cleaner ones. Deleting a thousand lines of code that you spent months writing doesn’t feel great, but replacing 950 lines with 50 feels amazing.

Simple software is natural software. It’s easier to scan at 6 am, easier to write at 6 pm, and easier to fix at 1 am. That's why it has a chance to survive long enough for you to reach your ultimate goal.

What’s another thing we do naturally that I left out? Reply to LMK.

Recommendations

Data

• Developer Ecosystem 2025 | JetBrain’s survey of 25k devs

• Developers want more, more, more | Stack Overflow’s 2024 survey

• State of AI 2025 | How devs use AI

• The Context Switching Crisis | Hivel AI 2025

• What Motivates Open Source Software Contributors | Opensource.com

Books

• Systemantics: how systems work and especially how they fail | Fun wisdom

• How to Focus: A Monastic Guide for an Age of Distraction

• A Philosophy of Software Design | John Ousterhoust

• Getting Real | 37signals | Canonical argument for less

TL;DR - It’s a modular monolith built on this stack:

1. Backend: Ruby on Rails

2. HTTP server: Puma (Ruby web server)

3. Edge: Nginx

4. Reverse proxy: Go service (Workhorse)

5. Background jobs: Sidekiq (Ruby framework)

6. DB — primary: PostgreSQL

7. DB — connection pooling: PgBouncer

8. DB — high availability: Patroni

9. Cache: Redis

10. Git: Custom gRPC repo interface (Git & Gitaly)

11. Blob: AWS S3

12. Frontend — rendering: Haml & Vue

13. Frontend — state: Pinia (Vue store), Immer (immutable cache),

14. API: GraphQL (Apollo) + REST

15. Observability: Prometheus & Grafana

16. Error tracking: Sentry & OpenTelemetry

17. Deployments: GitLab Omnibus (Omnibus fork)

GitLab is what happens when you commit to the monolith. Its CI/CD, security scanner, issue tracker, and deployment toolchain are bundled into one product rather than five held together with webhook glue. The stack isn’t flashy either: a large Ruby on Rails app, Postgres, Redis, and a handful of Go services.

No Kafka, event-sourcing, mesh of 200 microservices. Instead, GitLab maintains its modularity through discipline and radical transparency.

This article traces the path from your git push command to the DB and back, ending with an explainer of how GitLab monitors all of it in prod, and where the architecture is headed.

What Is GitLab?

If you have never used GitLab (shame), here’s the gist.

It’s a platform for software teams. At its core, it hosts Git repos. On top of that, it adds CI/CD pipelines (automated build, test, and deploy workflows), merge requests, issue tracking, container registries, and security scanning, all in one application.

Unlike GitHub, which relies on a marketplace of third-party integrations for CI/CD, GitLab ships all of this natively. This “one-app-to-rule-them-all” stance is what shapes many of the architectural choices described in this article.

It’s also open-source.

The Glossary

GitLab introduces a lot of proper nouns. Before going further, here’s the cheat sheet. We will cover each of these in depth later.

+-------------+-----------------------------------------------------------+---------+
| Name        | What It Does                                              | Built In|
+-------------+-----------------------------------------------------------+---------+
| Puma        | Multi-threaded web server. The main application           | Ruby    |
|             | engine: receives web requests, runs business logic,       |         |
|             | reads and writes the database.                            |         |
+-------------+-----------------------------------------------------------+---------+
| Workhorse   | Go reverse proxy sitting in front of Puma. Handles        | Go      |
|             | slow, heavy I/O so Ruby does not have to: git push/pull   |         |
|             | over HTTPS, large file uploads, CI log streaming,         |         |
|             | artifact downloads.                                       |         |
+-------------+-----------------------------------------------------------+---------+
| GitLab Shell| Handles SSH connections for git push/pull. Authenticates  | Go/Ruby |
|             | the user, then hands the git operation to Gitaly.         |         |
+-------------+-----------------------------------------------------------+---------+
| Gitaly      | gRPC service that owns all Git data on disk. Replaced     | Go      |
|             | a brittle NFS-shared filesystem. Every git read/write     |         |
|             | goes through Gitaly RPC calls — nothing touches the git   |         |
|             | repos directly.                                           |         |
+-------------+-----------------------------------------------------------+---------+
| Praefect    | Load-balancing replication proxy for Gitaly. Sits in      | Go      |
|             | front of a cluster of Gitaly nodes, enforces quorum       |         |
|             | writes, and handles automatic failover.                   |         |
+-------------+-----------------------------------------------------------+---------+
| Consul      | Service registry. Holds the current topology of the       | Go      |
|             | Postgres cluster so that PgBouncer can find the primary   |         |
|             | after a failover without any human intervention.          |         |
+-------------+-----------------------------------------------------------+---------+
| Geo         | Multi-site replication. Secondary sites mirror the        | Ruby/Go |
|             | primary for read performance and disaster recovery.       |         |
+-------------+-----------------------------------------------------------+---------+
| GitLab      | Standalone Go binary that organizations deploy            | Go      |
| Runner      | wherever jobs should run: their own servers, a cloud      |         |
|             | account, a Kubernetes cluster. Polls GitLab for           |         |
|             | pending CI jobs and executes them.                        |         |
+-------------+-----------------------------------------------------------+---------+

The Full Picture

This is the end-to-end architecture. Every section below zooms into one layer of it.

Layer 1: Client to Ingress

Most traffic comes in over HTTPS and goes through the same ingress path. The exception is SSH, which we will discuss at the end of this section.

Cloudflare: The Front Door

Every request to GitLab.com hits Cloudflare first, which gives them a firewall, DDoS protection, a CDN, and TLS termination. We’ll break down Cloudflare in its own issue, so let’s leave it at that for now =]

Nginx: The Gateway

Nginx receives HTTP requests from the load balancer and routes them to Workhorse. It handles a few things here:

• TLS termination for self-managed installs where Cloudflare is not in front

• Static file serving for anything that lives on disk, like uploads that have not yet been moved to object storage

• Rate limiting headers, enforced in conjunction with GitLab’s Redis backend rate limiter

• Request routing — Nginx sends everything to Workhorse, which will decide what to do next

Nginx is intentionally thin here; it is not doing complex business logic. Its job is to be a fast, reliable conduit to the Workhorse.

The SSH Path: GitLab Shell

To retain the real hackers, GitLab also allows SSH connections, which bypass Nginx entirely. A git push git@gitlab.com:user/repo.git connects directly to GitLab Shell on port 22.

GitLab Shell is a small Go app that acts as a restricted SSH shell. When your SSH client connects, it reads your public key, calls the GitLab API to verify the key belongs to an authorized user, then hands off the git pack data to Gitaly for storage. It never actually lets you get a general shell prompt — it only accepts git-specific commands (lame).

Layer 2: Ingress to Core

Once a request passes through Nginx, it reaches the core application layer — the part that actually understands what the user is trying to do.

Workhorse: The Smart Proxy

Workhorse is a Go service that sits between Nginx and Puma. In addition to normal reverse proxy duties, it has a specific job: keep slow I/O away from Ruby threads.

Ruby has a Global Interpreter Lock (GIL) that limits true parallelism. Tying up a Ruby thread on a slow upload or a long git-pack operation is expensive, preventing that thread from serving other requests while it waits. Workhorse exists to prevent this.

The communication between Workhorse and Puma is slightly unusual.

While the standard reverse proxy request goes like this:

client -> proxy -> app -> proxy -> client

…adding Workhorse into the mix makes it go like this:

client -> Workhorse -> Puma -> Workhorse -> app -> Workhorse -> client

Workhorse forwards the request to Puma. Puma processes the request and then replies with special headers (X-Sendfile, X-GitLab-Git-HTTP-Session). Workhorse uses those to hijack the response and send it directly to the client.

Basically, Puma tells Workhorse what to do via response headers, and Workhorse takes it from there.

Workhorse is responsible for:

• Git HTTP operations — git push and pull over HTTPS. It streams the pack data directly to and from Gitaly via gRPC, without routing the raw bytes through Ruby

• File uploads — when you upload an attachment, Workhorse buffers the multipart upload and either stores it temporarily on disk or streams it directly to object storage, then notifies Puma that the upload is complete.

• Artifact downloads — when you download a CI artifact, Workhorse fetches it from object storage and streams it to your client, again without involving Puma

• CI log tailing — while a job runs, Workhorse streams log chunks from Redis to the browser in real time.

This is weird, but it frees the Puma thread immediately instead of making it wait for I/O to finish.

Puma: The Rails Application Engine

Puma is where the actual GitLab application runs. It is a multi-threaded Ruby HTTP server running the GitLab Rails app — a large codebase that handles:

• Authentication & authorization (who is this user, can they see the code)

• Reading and writing to PostgreSQL (user records, merge requests, issue data)

• Reading and writing to Redis (sessions, locks, cache, real-time events)

• Calling Gitaly via gRPC for repository operations (list branches, read a file, get a diff)

• Enqueuing Sidekiq background jobs for any work that does not need to happen synchronously

• Rendering the HTML, JSON, or GraphQL response

Puma scales horizontally: add more Puma nodes behind the load balancer, and throughput increases linearly (because Puma nodes are stateless). Sessions live in Redis. Nothing lives on the Puma host that cannot be reconstructed.

The Rails app is organized in a monolithic codebase, but it’s nevertheless modular. Each domain manages its own models, services, and API endpoints. This provides the DX benefits (easy to run locally, shared tooling, fewer network hops) while keeping the blast radius of any given change small.

Sidekiq: The Background Worker

Not everything needs to happen before the user sees a response. Sending a notification email, updating a downstream pipeline status, archiving a CI log to object storage — these are all better handled asynchronously.

Sidekiq is GitLab’s background job processor for this type of work. It runs as a separate process (or many processes at scale), reads job payloads from Redis queues, and processes them independently from the web request cycle.

When Puma wants to fire off async work, it enqueues a job — a small JSON payload describing a Ruby class and its arguments — into a Redis list. Sidekiq workers are sitting in a loop pulling from those lists. They pick up the job, execute the Ruby code, and either mark it complete or retry it on failure.

Puma (on a request):                   
  1. Process MR comment creation            
  2. Write comment to PostgreSQL           
  3. Enqueue NotificationWorker ---->    Sidekiq (background):
  4. Return 201 to user                  1. Pull NotificationWorker job from Redis 
     (~40ms)                             2. Load recipients from PostgreSQL                                          
                                         3. Send emails via SMTP
                                         4. Mark job complete
                                         (~2 seconds, who cares)                        

At GitLab.com scale, Sidekiq is not one process — it’s a fleet of workers, each configured with specific queue assignments. CI state-transition workers run on dedicated Sidekiq fleets so that a slow email-delivery worker does not hold up the CI queue (a lesson learned the hard way).

Layer 3: Core to External Services

Remember that the app tier does not hold state. It delegates to a set of purpose-built external services. Let’s learn what each of these star players does.

PostgreSQL: The System of Record

Every structured piece of app data lives in Postgres: users, projects, groups, issues, merge requests, CI pipelines, job records, artifact metadata, permissions — all of it. It’s the authoritative source of truth.

GitLab.com doesn’t connect to Postgres directly, however. It goes through PgBouncer, a connection pooler that multiplexes thousands of connections down to a manageable pool. Without it, a Puma/Sidekiq fleet of hundreds of processes would each try to hold open a database connection, quickly exhausting Postgres’s connection limit.

GitLab also uses read replicas — secondary Postgres nodes that receive a streaming copy of all writes from the primary. Read-heavy operations like loading a project’s issue list or rendering a dashboard can be routed to replicas, offloading the primary for write-heavy work.

Redis

GitLab uses Redis as eight logically distinct stores, each with different operational requirements and eviction policies.

+------------------------+-------------------------------------------+------------------+
| Instance               | Stores                                    | OK to Lose?      |
+------------------------+-------------------------------------------+------------------+
| queues                 | Sidekiq job payloads, retry state         | No - jobs lost   |
| shared_state           | Distributed locks, exclusive leases       | No - race conds  |
| sessions               | User login sessions                       | No - logged out  |
| actioncable            | WebSocket pub/sub for real-time UI        | Mostly fine      |
| trace_chunks           | In-flight CI job log bytes                | Mostly fine      |
| cache                  | Computed Rails.cache data                 | Yes - rebuildable|
| rate_limiting          | Request counters per IP/user              | Yes - rebuildable|
| repository_cache       | Branch and tag name lists                 | Yes - rebuildable|
+------------------------+-------------------------------------------+------------------+

Cache instances use an LRU eviction policy: when they fill up, the least-recently-used keys are evicted. That is the correct behavior for a cache. (Imagine you applied LRU eviction to the queues instance: Sidekiq jobs would silently disappear from the queue before being processed!)

On a small self-managed instance, one Redis process handles everything. On GitLab.com, these are fully separate clusters, sized and scaled independently. This ensures that when the CI system generates a spike in trace chunk writes, it does not compete with session storage.

Gitaly: RPC for Git

Early GitLab stored git repositories on a shared NFS filesystem. Every Puma and Sidekiq node would mount the NFS volume and call git commands directly on disk. This worked, up to a point. As scale grew, the limitations became painful: NFS latency spikes, split-brain risks on failover, no way to observe what git operations were running or how long they took.

Gitaly eventually replaced NFS. It is a gRPC service — you call it via a typed remote procedure call (RPC) interface, not by shelling out git on a shared filesystem. Every git operation in GitLab now translates to a Gitaly RPC:

• To get a MR diff, Rails calls Gitaly.CommitDiff(repo, from_sha, to_sha). Gitaly streams diff hunks back as gRPC messages.

• To list branches, Rails calls Gitaly.FindAllBranches(repo). Gitaly reads from disk, returns structured branch data

• To git push over HTTP: Workhorse calls Gitaly.PostReceivePack(repo, pack_data). Gitaly writes objects, updates refs, runs server-side hooks.

Because every git operation is a discrete, typed RPC, Gitaly can be monitored, load-balanced, and independently scaled in a way that NFS never could have.

Object Storage: Keeping Blobs Separate

Not everything fits neatly in a Postgres row. CI artifacts (binaries, test reports, logs), Git LFS files, container images, uploaded attachments, and archived job logs are binary blobs that can be gigabytes in size.

Storing this in Postgres, while technically possible, would inflate table sizes, make backups slow, and force blob I/O through the same connection pool as structured queries.

GitLab sanely puts all blobs in object storage — GCS on GitLab.com, S3 on self-managed installs. The DB holds only a pointer (a storage path and some metadata). When you download a CI artifact:

Browser ──► Workhorse ──► Rails (auth check only)
                              │
                              │  "This artifact lives at
                              │   gs://artifacts-bucket/job/42/artifact.zip"                    "
                              │
                              ▼
                         Workhorse generates a presigned URL
                         or proxies the stream directly from GCS
                         to the browser

The auth check is instantaneous. The actual data transfer happens between Workhorse and GCS, with no Puma thread blocked waiting for bytes. Another example of using Puma for what it’s good at (HTTP) and avoiding it for I/O.

The AI Gateway: Model Routing for Duo

GitLab Duo is the umbrella name for AI features: code suggestions, chat, root-cause analysis for failed CI pipelines, security vulnerability explanations, and so on. All of these features funnel through a centralized AI Gateway — a standalone Python/FastAPI service that routes requests to the right model.

Rails/Sidekiq
      │
      ▼
AI Gateway (FastAPI)
      ├──► Anthropic Claude   (code generation, chat)
      ├──► Google Vertex AI   (various tasks)
      └──► Self-hosted models (for enterprise customers)

Why a gateway rather than Rails calling model providers directly? A few reasons. The Gateway creates one point of control for rate limiting, cost accounting, model versioning, and provider fallback across all deployment types. When GitLab wants to swap one model provider for another or A/B test two models, they change it in one place. Self-managed customers do not need to configure cloud provider credentials for every AI feature separately; they just connect to the Gateway.

External Services: The Outside World

GitLab also talks to a set of external systems that are not “part of GitLab” per se but are deeply integrated:

• SMTP servers — for sending email notifications (Sidekiq-dispatched)

• OAuth providers — Google, GitHub, SAML, etc for SSO

• LDAP / Active Directory — for enterprise users provisioning and group sync

• Kubernetes API — for the GitLab agent for k8s, used for deployments and cluster management

• Container registries — GitLab’s built-in registry uses a separate service gitlab-registry), based on Docker.

• Webhook targets — when you configure a GitLab webhook, Sidekiq makes an outbound HTTP request to your configured endpoint on relevant events

We’ve covered the main building blocks of GitLab. The following explains the connective tissue and peripheral bits that make everything come together in prod.

High Availability

How GitLab eliminates single points of failure at every layer of the stack.

Application HA

Stateless nodes. Puma and Sidekiq nodes hold no local state. User sessions live in Redis. Uploaded files live in object storage. If a Puma node dies, the load balancer routes traffic to the remaining nodes before users ever notice — their session is still valid because it was in Redis all along. Adding capacity is equally painless: spin up more nodes and register them with the load balancer. No data migration needed.

Praefect adds high-availability (HA) on top of Gitaly. It is a proxy that sits in front of a cluster or Gitaly nodes. On a write, Praefect sends the operation to all nodes and waits for a quorum to confirm before returning success. This is “strong consistency,” meaning you can read from any Gitaly replica immediately after a write and get the correct data. When a Gitaly node fails, Praefect promotes a replica automatically.

DB HA: Patroni + PgBouncer + Consul

Patroni runs on each Postgres node. It manages leader election, monitors health, and handles automatic failure. If the primary becomes unreachable, Patroni elects a new primary within seconds. It broadcasts the result to Consul, the service registry. PgBouncer watches Consul, so it knows to point new connections at the primary without any restart. Application nodes never need to know which host is primary; they always talk to PgBouncer.

Failover typically completes in under 30 seconds. Applications experience a brief pause as PgBouncer re-routes, then continue normally.

Gitaly HA: Praefect Cluster with Quorum Writes

Praefect is a router and transaction manager for Gitaly.

On writes, Praefect sends the operation to all Gitaly nodes and waits for a quorum (majority) to confirm before reporting success. This means you can read from any replica immediatly after a write and see the latest data — no stale reads. If the primary Gitaly node fails, Praefect promotes a replica and resumes service automatically.

Praefect maintains its own small Postgres DB (separate from GitLab’s main Postgres) to track replication state. A full HA deployment ends up with two Postgres clusters: one for GitLab app data, one for Praefect.

Redis HA: Sentinel Mode

Each Redis instance in a HA deployment runs with Redis Sentinel: a primary Redis with replicas, monitored by at least three Sentinel processes. If the primary fails, Sentinels elect a new primary and notify clients to reconnect. Application nodes are configured to discover the current primary through Sentinel rather than connecting to a fixed host.

One constraint worth knowing: Redis Cluster (Redis’s own horizontal-sharding mode) is not supported for all GitLab Redis uses. The trace_chunks instance uses a data structure that is incompatible with Redis Cluster’s key-slot sharding. All GitLab Redis instances therefore use Sentinel mode, not Cluster mode.

Site Level Recovery: GitLab Geo

What happens if the entire primary data center goes offline? How do developers in Asia-Pacific stop having 300ms latency every time they run git clone against a server in Virginia?

GitLab’s answer: Geo, a warm-standby for distributed teams. Geo provides local caches that can be placed geographically close to remote teams, which can serve read requests. This can reduce the time it takes to clone and fetch large repositories.

The secondary site gets Postgres via standard streaming replication — a continuous, near-real-time copy of every write. Git repos are synced by the Geo Log Cursor daemon, a separate process that reads the log and enqueues background jobs to clone or fetch updated repos from the primary via HTTPS.

Replication is async — there is always some lag, usually measured in seconds. For disaster recovery, planned failover waits for full synchronization before promoting the secondary. Zero data loss. Emergency failover can be forced immediately. In this case, some data written in the last few seconds may be lost.

Monitoring: How GitLab Knows When Things Break

A system with this many parts will have failures. The question is how quickly they can be detected, and how much context you’ll have to start troubleshooting.

GitLab’s observability stack covers three domains: metrics (what is happening now), logs (what happened when), and traces (specific request tracking).

Metrics: Prometheus and Grafana

Every GitLab component exposes a Prometheus metrics endpoint. The in-process GitLab Exporter scrapes the running Rails app and produces application-level metrics. Prometheus collects and stores all of these on a regular scrape interval. Grafana provides dashboards and alert routing.

Key metrics the team monitors:

+----------------------------+-----------------------------------------------+
| Metric                     | What It Tells You                             |
+----------------------------+-----------------------------------------------+
| HTTP request rate/latency  | Is the web tier healthy? Latency spikes       |
| by controller action       | point to slow DB queries or Gitaly RPCs       |
+----------------------------+-----------------------------------------------+
| Sidekiq queue depth        | Are jobs piling up? Indicates under-capacity  |
| by queue name              | or a worker that is stuck                     |
+----------------------------+-----------------------------------------------+
| Sidekiq job latency        | Time from enqueue to start. High latency      |
| by worker class            | = queue starvation                            |
+----------------------------+-----------------------------------------------+
| Gitaly RPC latency         | Git operation performance. Spikes indicate    |
| by RPC name                | disk I/O issues on Gitaly nodes               |
+----------------------------+-----------------------------------------------+
| PostgreSQL connection      | Is PgBouncer saturated? Are we running out    |
| pool saturation            | of Postgres connections?                      |
+----------------------------+-----------------------------------------------+
| CI queue depth             | Pending jobs waiting for a runner. High       |
| by runner type             | values = runner fleet under-capacity          |
+----------------------------+-----------------------------------------------+
| Redis command latency      | Cache/queue slowness. Usually indicates       |
|                            | memory pressure or network issues             |
+----------------------------+-----------------------------------------------+
| Error rate by service      | Rolling count of 5xx responses. The canary    |
|                            | for most application-level problems           |
+----------------------------+-----------------------------------------------+

Logs

Every component in GitLab emits structured JSON logs. The critical piece is the correlation ID: a UUID assigned to each incoming request by Workhorse, stamped into every log generated anywhere in the call chain — Rails, Gitaly, Sidekiq, all of ‘em.

{
  "method": "POST",
  "path": "/api/v4/jobs/request",
  "status": 200,
  "duration_s": 0.043,
  "correlation_id": "01HXKG4P8B3QV7N6J2MZ", // <-- important
  "user_id": 1234,
  "meta.caller_id": "Ci::RegisterJobService",
  "db_duration_s": 0.012,
  "redis_calls": 3,
  "gitaly_calls": 0,
  "queue_duration_s": 0.001
}

When an on-call engineer sees a latency spike on a Grafana dashboard, they grab the correlation ID of an affected request and use it as a filter key across all log sources — Elasticsearch, Splunk, Loki, whatever the install uses.

Tracing: OTEL

Metrics tell you what is slow. Logs tell you when the slowness happened. Distributed traces tell you why by showing the exact sequence of service calls and how long each one took.

GitLab uses OpenTelemetry-compatible tracing. The correlation ID doubles as the trace ID here. A slow pipeline creation request might produce a trace like:

POST /api/v4/projects/123/pipeline   847ms total
  ├── Auth check (Redis)              3ms
  ├── Project load (Postgres)         8ms
  ├── YAML fetch from Gitaly         22ms    ← Gitaly.GetFile RPC
  ├── YAML parse and rule eval        41ms
  ├── Pipeline write (Postgres)       15ms
  ├── Stage+job row inserts          180ms   ← this is the bottleneck today
  ├── Sidekiq enqueue (Redis)         4ms
  └── Response serialization         12ms

Interestingly, this distributed tracing is still considered “experimental” at the time of this article.

Error Tracking

Unhandled exceptions in Rails are captured and routed to GitLab’s error tracking system. GitLab uses its own built-in Error Tracking feature (which integrates with Sentry-compatible backends) as well as self-hosted Sentry for some teams. Errors are deduplicated, grouped by fingerprint, and surfaced with full stack traces and the request context (user, path, params) that caused them.

Error tracking events also carry the correlation ID, so you can jump from an error event directly to all logs from that specific request.

Scaling PostgreSQL

For those familiar with these real-world apps, things probably seem pretty straightforward up to this point; GitLab hasn’t tried to do anything cute or uniquely ambitious on the engineering side.

At their current scale, however, the PostgreSQL situation is getting untenable. They started by allowing horizontal cloning by making Puma & Sidekiq stateless. Then they decomposed CI tables into a separate DB, a multi-year effort that required forbidding cross-database JOINs in the Rails model layer and rewriting hundreds of queries.

Now they’re partitioning data by customer. This can’t be done via traditional sharding, because Postgres has no native sharding. CitusDB was evaluated and ruled out. Application-level sharding requires every query to know which shard to target, and makes cross-shard queries effectively impossible.

GitLab’s answer is Cells (interestingly, the same name Discord used for its similar abstraction). Instead of sharding rows in a database, they’ll run multiple independent copies of the entire application stack — each called a Cell — with a shared routing layer in front.

An org is assigned to exactly one Cell. The global router resolves which Cell handles each request based on the org in the URL. Each Cell is essentially a complete, isolated GitLab instance. Adding capacity means adding a new Cell — no resharding or cross-shard coordination.

The hard part is cross-cell operations: a user who belongs to orgs on two different cells, or a fork from a project on Cell 1 to Cell 2. These require a global identity layer and router that is still being designed.

They’re refreshingly transparent about this effort: you can track the project status, ADRs, and proposed architecture on their public handbook page (handbook.gitlab.com).

The “Boring” Takeaway

Perhaps you’ve been waiting for more complexity, the kind of juicy architectural optimization that makes the front page of HackerNews.

If that’s the case, GitLab ain’t for you.

The most interesting part about GitLab’s architecture is what isn’t there. No Kafka. No event sourcing. No distributed saga coordinator. No fancy message bus (yet). Their Z-axis scaling is being solved not with sharing but with what is essentially “run more copies of the whole app” lol.

Its monolithic codebase and the conventional tools that support it are simple, lean, pragmatic. They’ll extract and abstract, but only when warranted:

• Gitaly was created when NFS-backed git storage became a liability

• Workhorse was born when Ruby’s I/O throughout became the bottleneck for git operations

• The CI DB was split when CI tables started consuming 40% of write traffic.

The monolith survives because it genuinely serves the product well: it is fast to develop, easy to test locally, straightforward to reason about, and naturally consistent within a single DB transaction.

This is a story of engineering sanity, not architectural fashion. And for 30 million users, it’s working just fine.

References

1. GitLab Architecture Overview — GitLab Docs | Canonical component diagram and service descriptions.

2. Reference Architectures — GitLab Docs | Tier-by-tier HA topologies from single node to 50k+ users.

3. Gitaly Cluster / Praefect — GitLab Docs | Replication proxy, quorum writes, and automatic failover.

4. Redis Development Guidelines — GitLab Docs | Multiple Redis instances, eviction policies, and use case separation.

5. Geo — GitLab Docs | Multi-site replication, the Geo Log Cursor, and failover procedures.

6. PostgreSQL HA with Patroni — GitLab Docs | Patroni, PgBouncer, and Consul HA topology in depth.

7. GitLab Scalability — GitLab Docs | AKF Scale Cube framing — X/Y/Z axes, current state, and gaps.

8. Multiple Databases — GitLab Docs | CI database separation and cross-database query restrictions.

9. Cells Architecture Blueprint — GitLab Handbook | Design, goals, and current status of the Cells initiative.

10. AI Architecture — GitLab Docs | AI Gateway design, Duo routing, self-hosted model support.

11. Find Relevant Log Entries with a Correlation ID — GitLab Docs | Correlation ID flow through Workhorse, Rails, Gitaly, and Sidekiq.

12. Log System — GitLab Docs | Structured JSON logging reference — all log files and their schemas.

13. Error Tracking — GitLab Docs | GitLab’s built-in Sentry-compatible error tracking integration.

14. GitLab Workhorse — GitLab Docs | Design rationale and feature set of the smart proxy layer.

15. Decomposing the GitLab Backend Database — GitLab Blog | The story of separating the CI database from main.

16. EE Feature Guidelines — GitLab Docs | How CE/EE code separation works at the Rails autoloader level.

17. How We Designed GitLab Reference Architectures — GitLab Blog | History and rationale of the reference architecture tiers.

18. Scaling Sidekiq at GitLab.com — GitLab Blog Lessons from specialized queue configuration at production scale.

19. GitLab Shell — GitLab Repository | SSH session handler.

20. AKF Scale Cube — AKF Partners | 2024 | The original X/Y/Z axes of scalability model GitLab references.

Critiquing the bad is useful, so is celebrating the good.

That’s why this issue highlights everyday simplicity.

Engineering simplicity

1. CI/CD scripts in Bash only. No Terraform, subscriptions, config hell, version updates. Just Bash.

2. Claude skill for simplifying (SKILL.md). “Favor minimal, legible implementations. Fewer lines and clearer structure make code easier to understand and maintain.”

3. Typescript on server and client. Strong-typing for AI. Easier to run and read code for devs.

4. Monolith instead of microservices. Easier for normal-sized teams to manage. “[It] doesn’t pretend to provide a failsafe architectural road to glory. That’s a fool’s errand. Many programmers suffering under many oppressive influences will turn any architecture made with any tool into a big pile of mud.” (signalvnoise)

5. Markdown files. Portable, straightforward. These files lack fancy embeds and database linking, but they work. In fact, an entire note-taking company is built on this file type (obsidian.md).

6. Peter Level’s vanilla stack: HTML, CSS, vanilla JavaScript, jQuery, PHP, and SQLite — all running on a single VPS (indiehackers.com).

7. Dashes and lowercase for files. That’s it. This saves so much time, trust me.

   taxes-2025.pdf
   taxes-2025-amended.pdf
   statement-checking-2025-01.pdf
   statement-checking-2025-02.pdf

8. SQLite in production. Notion’s early days and plenty of indie apps run SQLite fine. One file, zero config, no separate server process (sqlite.org/whentouse).

9. ENV files for config. A flat text file beats a secrets manager, a config service, and three IAM policies. Works locally, works in prod, understandable.

10. Cron jobs instead of queues. Before Kafka, SQS, or BullMQ — there was cron. Scheduled tasks, no broker, no consumer group, no dead-letter queue. If your job runs once an hour, you don’t need a message bus.

AFK Simplicity

1. Two-tier pricing models. Free or premium. Free or usage-based. Starter or enterprise. The end.

2. Dotted grid paper. The best of grid paper + lined paper + freeform paper (leuchturm1917.us)

3. Capsule wardrobe. A small collection of thoughtfully chosen, easy-to-mix pieces that allows you to create a wide range of outfits with fewer items. Less decision fatigue, more time (modernminimalism.com).

4. Crossfit’s diet recommendation: “Eat meat and vegetables, nuts and seeds, some fruit, little starch, no sugar.”

5. Aldi’s 25-cent deposit to use a cart. “This […] ultimately saves our customers money because we don’t have to hire extra staff to collect grocery carts.” Yep.

6. My grandma’s “as much as you want” blue cheese rice soup recipe.

1. Buy Uncle Ben’s seasoned “long grain + wild rice” original
2. Cook according to directions
3. Add plain yogurt (not Greek) (as much as you want)
4. Add crumbled blue cheese (as much as you want)
5. Heat thoroughly
6. If you want to make it a 1 dish meal, add browned hamburger

7. Apple Notes for tracking workouts. I used to have three workout apps. Now I have three notes.

8. The Big Ass Calendar. An entire year on one wall. No flipping, loading screens, or subscriptions (thebigasscalendar.com).

9. My old gym’s response to complaints about the playlist:

   

10. SF street names engraved in the sidewalk. Fewer signs.

Once you start looking for the beautifully simple solutions to problems, you’ll see them everywhere.

I hope these inspire you to simplify your own stack and life.

Reply with an example or two of your own!

The Depth of Knowledge (DOK) model categorizes thought into four levels of complexity:

1. Recall. Memorizing, matching, reciting, quoting

2. Skill. Inferring, categorizing, interpreting, predicting

3. Strategy. Critiquing, developing, comparing, concluding

4. Extend. Design, connect, synthesize, create

DOK helps educators create a curriculum with varied cognitive rigor. A good math exam would require students to identify basic formulas, choose the right one, analyze data, and solve a real-world problem. It also helps explain why some students excel at certain assignments but struggle with others.

We can look at our system design skills through a similar lens.

In keeping with our theme of simplicity, I propose just three levels:

1. Facts. The difference between SQL and NoSQL.

2. Heuristics. SQL for structured data, NoSQL for unstructured data.

3. Inventions. Distributed SQL.

Our Discord case study displays engineering at all three levels of complexity.

1. Facts: Knowing enough about programming to make the app fast.

2. Heuristics: Adopting Elixir, Rust, and ScyllaDB, despite their newness.

3. Inventions: Creating a “Super-Disk” abstraction to guarantee both speed (SSDs) and reliability (hard drives).

Discord’s inventions are the most impressive.

“But we’re not at Discord scale — our apps don’t need to support trillions of messages or millions of concurrent users. Plus, we can pass most system design interviews with memorization alone. Why not just move on and build stuff instead of worrying about creating more abstractions?”

I get it.

But it’s a shame to build a strong base during interview season, only to let it atrophy after day one on the job, obediently accepting the precedents forever.

Invention is worthy of pursuit. Although you’ll never be assigned a ticket to “Invent New Abstraction,” doing so is one of the most high-impact things you can do as an engineer.

Invention case study

Carl Hewitt’s Actor Model introduced a communication protocol of messages and mailboxes that solves race conditions. Although the model is a beautiful invention, creating something so esoteric feels a bit out-of-reach for those of us who don’t have a PhD from MIT.

Thankfully, we don’t have to introduce a new paradigm to invent; plenty of opportunities arise within our current stack. The Ergo Framework is a perfect example of how regular engineers can invent abstractions that simplify rather than complicate.

The problem

Ergo is an implementation of the Actor Model in Go, bridging the gap between a 50-year-old theory and a modern programming language. The basics of the actor model are perfectly compatible with the Go language — queues for mailboxes, trees for supervisors.

However, there’s one scenario where the language conflicts with the model: I/O (file reads, listening to OS signals, TCP accept). The Actor Model requires a balance of async delivery with sync processing. That’s because Actors work sequentially. One message arrives, gets processed, completes. Next message. This strict workflow is what eliminates race conditions.

Blocking I/O breaks this model.

In Ergo, a normal Process is an actor — a lightweight entity that handles messages sequentially in its own goroutine. There’s one process per goroutine. Processes can make sync calls and create monitors — standard actor stuff. Since there’s one goroutine, it must handle everything. If it blocks on I/O, message processing stops.

Call net.Listener.Accept() in a handler and watch the goroutine freeze while it waits for connections. Messages pile up until the system times out.

Let’s try solving the problem with a simple heuristic.

“Compute is cheap, add more. Moore’s Law. Just spawn a new goroutine for Accept().”

This bails you out of the immediate bottleneck. But now the two goroutines access the actor’s state concurrently. We just traded a perf problem for a race condition problem.

“Add a lock” (level one).

Sure, but now we have to add a bunch of code to handle the lock’s complexity.

The only way we can preserve the guarantees of concurrency that come with the Actor Model and the I/O use-case in Go is to invent a more elegant abstraction (level three).

The invention

The Ergo authors created “Meta-processes” for this situation.

Unlike normal processes, which have one goroutine per process, Meta-processes have two: a Reader and a Handler. The reader is for blocking operations — Accept() loops, ReadFrom() calls. The handler does everything except I/O via sequential messages in traditional Actor fashion. The Reader runs continuously from spawn until termination. The Handler comes and goes based on message traffic.

Meta-processes separate concerns. The Reader handles I/O, while the Handler processes messages. The Reader blocks reading while the Handler blocks writing. Two blocking operations, two goroutines, neither interrupting the other.

Meta-processes mediate the purity of the Actor Model with the messiness of the real I/O world. They turn blocking operations into async messages and messages into blocking commands. Regular processes implement everything else.

It’s a wonderfully simple distinction that a lay-developer can implement, even if they don’t appreciate the mechanism.

The takeaway

To escape a bind, add constraints.

To escape locks in concurrent programming, the Actor Model added communication constraints: messages and actors. Authors need to follow the rules by only sending messages and not sharing state, but they get guaranteed concurrency.

To escape blocking I/O in single-processes, Ergo added process constraints: one for reading, one for writing. Authors need to discern whether they’re working with I/O, but they get to keep concurrency in Go.

It’s counterintuitive to consider adding more rules in the pursuit of simplicity. It’s also easy to add the wrong rules and get more complexity without extra benefits. Tolerating this tension is a prerequisite to finding a level three invention. It’s the kind of challenge that comes from both understanding the theory and working in the slimy guts of a problem.

You can’t memorize or skim your way out of these situations.

That’s why they require level-three thinking.

Keep memorizing definitions, skimming, and interviewing. But also pursue answers that aren’t so accessible. Our industry is increasingly outsourcing its thinking; we need those who continue to exercise their cognition more than ever.

Bonus: More inventions

MapReduce: map() + reduce()

Containers: cgroups + namespaces

Git: immutable snapshots

CRDTs: sync + merge without locks

Virtual DOM: diffs against the real DOM

Promises: Async that reads like sync

Resources

Depth of Knowledge: Difficulty and Complexity (webbalign.org)

DOK chart (pdesas.org)

The Actor Model: Messages and Mailboxes (fullstack.zip)

Discord: A Case Study in Performance Optimization (fullstack.zip)

Ergo Basics (ergo.services)

Perhaps you’ve shrugged it off as an inevitable side-effect of progress.

Is it, though?

Increasing the number of components makes a system more complicated.

But entangling the individual components makes it complex.

A Swiss watch with its many gears is complicated. But there’s a manual. A watchmaker can repair it reliably.

The unpredictable behavior between components is what makes a system complex. Weather is complex. Cities are complex.

The problem isn’t the number of parts; it’s that the parts can’t be understood in isolation; you can’t pull one out without disturbing the others.

More components, more complicated.

More relationships between components, more complex.

Complication is an inevitable side-effect of solving harder problems.

On the other hand, complexity is a choice — one that is often made unconsciously while solving a problem. The irony is that by intertwining in the pursuit of an immediate solution, we shorten the solution’s lifespan.

We only need to peek at a few open-source projects to see this reality at work.

Dependencies reveal complexity

Naming is hard, but we got it right by calling packages “dependencies.” The more dependencies a project has, the more relationships, the more complexity.

If it’s true that complexity kills, then we should expect that the projects that survive have fewer dependencies.

I sampled 19 popular open-source web libraries and apps and found just that.

For scale, the average project in this list is 6.5 years old and has anywhere from

• 72k to 3M lines of code

• 11 to 340 dependencies

Lines of Code per Dependency

• VSCode: 23,579

• SuperProductivity: 21,173

• Ionic Framework: 11,518

• OpenClaw: 10,632

• Posthog: 8,970

• Playwright: 8,562

• react-hook-form: 6,562

• webpack: 6,050

• social-analyzer: 6,022

• excalidraw: 5,579

• Cal.com: 4,039

• Ionic Capacitor: 2,637

• Remix: 1,795

• Logseq: 1,772

• drawdb: 1,495

• Compass Calendar: 1,383

• Vitest: 602

• Formik: 566

The official winner is VSCode, with over 23,000 lines of code for each of its dependencies. As a low-level tool backed by Microsoft and the default app for every web dev for the past decade, this makes sense. There’s a lot of money and engineering time spent on keeping it lean-ish. But close behind is a productivity app with 614k lines of code and 29 dependencies backed by … one guy in Germany whose been working on it part-time for the last nine years!

SuperProductivity has endured because it has resisted complexity.

Compass is similar to SuperProductivity in spirit. Yet it’s at the bottom of the list because its developer has eaten from the forbidden fruit of the npm tree (and suffered greatly as a result).

The OGs at Remix are wiser; they’ve made the avoidance of dependencies a core principle:

Avoid Dependencies. Dependencies lock you into somebody else’s roadmap. Choose them wisely, wrap them completely, and expect to replace most of them with our own package eventually. The goal is zero.
— Remix’s README

This relationship between complexity and longevity is not unique to the open-source projects I cherry-picked. Just look at your own team’s graveyard of archived repos. I’ll bet the ones that didn’t make it have a lot of complex relationships.

We can choose differently now

Now that software is cheap, we’re forced to confront the question, “Do things still need to be this complex?” A triple-decker of abstractions might’ve been justifiable in a big codebase maintained by 500 engineers, each of whom can only keep 5% of the system in their head. But if that same team “reorgs” to 100 engineers and 400 coding agents, each of which can hold 100% of the system in context, could we travel more lightly?

Yes, and it’s already happening. Cloudflare rebuilt Next.js sans opinionated build system with one engineer in one week (cloudflare.com). Anthropic built a C compiler from scratch with 16 agents and $20k (anthropic.com).

The abundance of subsidized tokens encourages the creation of unadulterated projects like these headline-grabbers. But it also offers a unique opportunity to right our wrongs, to absolve the complexity we’ve accrued during times of weakness.

These days, I wake up with foaming bloodlust towards my package.json, for I know that the more complexity I expunge, the longer the software can survive.

There’s an irony here, one that mirrors the opportunity presented by social media.

As we’re bombarded with cheap dopamine, it’s never been easier to settle into chronic distraction. It’s also never been easier to compete: just read a book for five minutes to become a top 84% reader in the US (bls.gov).

We’re always one prompt away from a PR that introduces more complexity. But we’re also always one prompt away from a PR that cleans up the old.

What about the model is so special? There are entire books that explain this in detail. For this edition, we’re zeroing in on two core ideas: messages and mailboxes. These are the mechanisms that constrain each actor while preserving the power of the system.

Messages

The key to the actor model is messages, which are:

• The bundles of data — typically objects — that actors pass between one another.

• The inputs and outputs of every actor. Rather than sharing state directly, actors communicate exclusively through messages.

• Immutable. Once a message is created, it cannot be changed. Think of it like sending a sealed letter: the recipient gets exactly what you dropped off, and no one can alter it in transit. This one rule eliminates an entire class of bugs.

What actually happens when an actor receives a message?

One of four things:

1. The actor updates its own state

2. It creates more actors

3. It sends a message to another actor

4. It changes its behavior (how it will respond to future messages)

That’s it. Four activities summarize every interaction in an actor-based system.

Actor messages are perfect for Domain Driven Design (DDD).

DDD models code around an org’s entities and rules. It’s a natural design for our sprawling distributed systems. Anywhere the business logic is rich, stateful, and central to the product, DDD tends to shine.

Actors are a natural way to implement DDD entities — the core objects that have identity and evolve. In classic DDD, these entities exchange one of four messages:

1. Command: Tells an actor to change state (”Process this payment”)

2. Event: Records that a state change happened (”PROCESSED”)

3. Query: Requests information (”What’s the current balance?”)

4. Results: Response to a query ($247)

Four rules that detail how nodes communicate. Sound familiar? The terms vary, but both the Actor Model and its DDD embodiment use communication as a constraint.

Mailboxes

Where do actors process messages? That’s where mailboxes come in.

A mailbox is like an in-memory queue attached to each actor. Messages arrive and wait their turn. The actor works through them one at a time.

The key point here is that the mailbox is separate from the actor. The actor doesn’t manage its own queue — that’s handled by the library (like Akka in Java) or by your own implementation.

This separation seems unintuitive at first. If there is only one mailbox for each actor, why add that abstraction in the first place?

Same reason why none of our email apps support tasks: delivering and processing are two very different jobs. This separation of concerns allows the mailbox to focus on delivering the message and the actor to focus on reacting to it. Practically, it also makes scaling the system easier, as actors and their mailboxes can exist in separate containers.

Bringing this together

The actor model comes with a powerful guarantee: state can always be updated safely. The actor itself can only access its own state; other actors can only request that state by sending a message; only one message can be processed at a time.

Even though actors are constantly communicating, state is mediated through messages. Nothing gets corrupted by a parallel operation.

This is only possible thanks to the addition of communication rules. These keep nodes independent enough to do their jobs, while keeping the system trustworthy enough to grow.

The Remote Company

Imagine you’re working at an in-person company that doesn’t believe in SaaS or documentation. All decisions are verbal. When there are five of you, that works fine — it’s actually kind of fun. Once there are fifty people, everything breaks down. Shared resources get locked up. It takes forever to get people aligned. Constant interruptions undermine productivity.

Then the company reinvents itself. It goes fully remote, mandates email-only communication, and bans multi-tasking. You work through your inbox, one message at a time. Suddenly, agendas are written, conferences rooms are reserved, and everyone gets updates. It’s less chaotic fun, but things are actually getting done again.

The communication rules were needed to ensure the system could flow again.

That’s the actor model.

Takeaways

Why does this matter?

As our systems grow more complex, one of the most valuable skills we can develop is recognizing elegant simplicity. Alan Kay put it well: “You get simplicity by finding a slightly more sophisticated building block to build your theories out of.”

The actor model is exactly that kind of building block. It doesn’t introduce an impressive new algorithm or chip architecture. Its genius lives at the transport layer. We’ve learned to obsess over the computation layer’s complexity at the expense of the simple solutions in plain sight.

The better we get at recognizing this kind of structural simplicity, the better we can protect it from the slow erosion that complexity brings. And eventually, if we keep exercising this muscle, we’ll start designing more elegant models ourselves.

Forward this to a fellow actor on your team.

Bonus points if you use DDD Command syntax.

Keep Exploring

Actor Fundamentals (akka.io)

Actors: a model of concurrent computation in distributed systems (archive.org)

The Actor Model (wikipedia)

A Feature Model of Programming Languages (sciencedirect)

The Actor Model Whiteboard Session with Carl Hewitt (youtube)

The Actor Model, Behind the Scenes with XState (youtube)

Why Did the Actor Model Not Take Off? (reddit)

I find the opposite to be true.

The people who understand something most deeply are usually the ones working hardest to make it simpler.

Smart people simplify

Tim Berners-Lee understood the web as a common place for data to be shared. He could’ve flaunted his intellect by building this place with Turing-complete programming languages like Haskell or C++. But those would’ve made it harder to work with and share information. Simpler languages, on the other hand, were easier to understand and write. You can combine a handful of HTML tags to have a website. CSS won’t store your data, but a few intuitive keywords will add some visual pop.

The internet grew because its languages were simple enough for normal people to use.

Indeed, on the Web, the least powerful language that’s suitable should usually be chosen.

— Tim Berners-Lee, The Rule of Least Power (w3)

BTW - Tim documented seven principles of design for himself. Five of them are about simplicity (w3).

Frank Lloyd Wright’s architecture adhered to nine principles. There were two for eliminating the unnecessary (walls and decorations) and seven for integrating the essentials (nature, the foundation, heating, and lighting). These simple rules forced him to ‘stretch’ his design vocabulary without losing coherence. Rather than imitating the eclecticism of the times, he harmonized the essentials. The varied expression of a few pure ideas is why his designs still feel modern a century later.

To reduce the number of necessary parts of the house and the separate rooms to a minimum, and make all come together as enclosed space - so divided that light, air, and vista permeated the whole with a sense of unity.

— Frank’s first principle.

The French aviator and literary star Antoine de Saint-Exupéry shared Wright’s appreciation for unity. He believed the role of engineers was to remove and refine away until only a cohesive whole remained. He witnessed this process with his planes. During his early flying career in the 1920s, the cockpits were a chaotic mess of raw gauges, exposed levers, and instruments scattered without much visual logic. By the time he wrote his memoir in 1939, they had smoothed so much complexity that he marveled at how he now spent more time thinking about flying and less on gadgetry. Seeing this transformation undoubtedly led to his now-famous anthem of simplicity.

Have you ever thought, not only about the airplane but about whatever man builds, that all of man’s industrial efforts, all his computations and calculations, all the nights spent over working draughts and blueprints, invariably culminate in the production of a thing whose sole and guiding principle is the ultimate principle of simplicity? [...] If anything at all, perfection is finally attained not when there is no longer anything to add, but when there is no longer anything to take away, when a body has been stripped down to its nakedness.

— Wind Sand And Stars

Simplicity is not a philosophical ideal that only appeals to artists or children.

The creators of our physical and digital world recognize its power.

Make no mistake, though — the road through complexity is unglamorous and difficult.

We’ll be exploring how to navigate our way towards simple solutions in the next few Offline editions.

One thing to carry with you into the week:

» If your solution still feels complicated, you probably haven’t finished thinking yet.

For me, “fan-in” and “fan-out” didn’t make sense until I found some visuals and real-world examples.

If those terms seem nebulous to you, read on.

Fan-out splits.

Fan-in consolidates.

Fan-out: one thing triggers many downstream operations (broadcast / scatter).

Fan-in: many things consolidate into fewer results (gather / reduce)

Fan-out splits a large job into smaller sub-tasks run in parallel

Fan-in consolidates outputs from those tasks into a single result.

Inputs are fanned out.

Outputs are fanned in.

Fan-out improves performance by running tasks in parallel.

Fan-in can improve performance by preventing unnecessary work. (But it can also hurt performance in the form of a bottleneck or queue.)

Fan-out happens first (scatter-gather).

Fan-in happens as a result of a fan-out. (But it can happen independently, like log aggregation.)

Fan-out examples

A message in the #announcements Discord channel is fanned out to members.

A user uploads a video, and YouTube triggers tasks for audio extraction, thumbnail generation, and content moderation.

A light switch fans-out the “on” operation for three light bulbs.

When you click “Place Order,” Amazon fans-out tasks for processing the payment, updating warehouse inventory, and sending a confirmation email.

When you tap “Schedule Pickup,” Uber fans-out notifications to dozens of drivers.

When you type “best pizza,” Google fans-out your query to many shards.

Fan-in examples

An API fans-in duplicate queries so that only one is sent to the DB (request coalescing).

Google Flights collects individual search results from multiple airlines and presents a single list to the user.

After each cluster returns its result, Google Search fans-in the results and serves the top 10.

When to fan

Fan-out when: work is independent, parallelizable, and latency matters.

Beware of thundering herd, cost spikes, downstream overload, and partial failures.

Fan-in when: you need a single answer, dedup, or batching.

Beware of hotspots, head-of-line blocking, slowest-task wins (“tail latency”), and complex retries.

(Check out our Discord case-study for a real-world example of these trade-offs.)

A good principle is to minimize the very high fan-outs and fan-ins.

The examples above focus on data flow, but this also applies to data modeling.

Would you rather maintain the fanned-out (purple) blog?

Hopefully these terms feel a little less weird now.

Forward this to someone on your team who is also combating fan jargon.

10 years ago, I found a scholarship that sends emerging leaders to Asia for a year.

They find them housing and even a good job.

I was hungry for adventure and validation, so I applied.

I made it to the final round, but didn't get selected.

I applied the next year, confident I'd get it this time.

Rejected.

I applied for a third year.

Rejected.

Instead of traveling the world, I got an IT job down the street.

It hurt to be denied permission to do something exciting.

It felt like the universe had conspired against me for three years.

Until I realized something so obvious now:

>> I could just go to Asia on my own <<

I didn't need any committee's permission.

A few months later, I quit my job and hopped on a one-way flight to Bangkok.

I spent the next three months rock climbing around Southeast Asia.

My Asia trip was nothing like how I planned, but that's what made it so special.

I'm sharing this story partly to flex on you all.

But also because I know that rejection sucks, whether it comes from employers, committees, or dating apps.

And prolonged rejection has a way of narrowing our imagination.

If you're still knocking on the front door, here's a friendly reminder:

>> There’s usually more than one way in. <<

Discord is a permanent, invite-only space where people can hop between voice, video, and text. On the surface, it seems like “just another chat app.” Take a closer look, and you’ll see that it’s really a finely-tuned system that delivers speed, scale, and reliability — the consumer app hat-trick.

Every time you send a message, join a voice channel, or watch a stream, Discord has to route the event to the right place, notify tons of clients, and do so fast enough that it feels instantaneous. That’s easy when your server has 50 people. It’s insane when it has 19 million.

This is the story of the creative optimizations that keep Discord snappy at scale.

Part I: The Actor Model

Before digging into Discord’s implementation details, we need to understand the architectural pattern upon which it’s built: The Actor Model.

Carl Hewitt introduced this idea in his 1973 paper (pdf). Alan Kay embraced it for passing messages in the 80s. Gul Agha formalized its relevance for distributed systems in 1985. Since then, many teams and tools have adopted the model. If you’ve ever read an elaborate sequence diagram or worked in an “event-driven architecture,” you can thank the Actor Model.

What problem does the model address?

In shared-memory, multiple threads use the same state, which quickly results in race conditions. You can prevent this by adding a data-access constraint: locks. But locks come with their own bugs, like when multiple threads wait for the other to release the lock, resulting in a permanent freeze (deadlock). As systems grow, these problems become bottlenecks.

The Actor Model allows data to be updated more easily in distributed systems.

An actor is an agent with a mail address and a behavior. Actors communicate through messages and carry out their actions concurrently. Instead of locks, the Actor Model ensures safe concurrency with communication constraints. The Actor Model can be summarized in four rules:

1. Each actor owns its state (no one else can directly mutate it)
2. Actors only communicate through messages
3. Actors process messages one at a time (no race conditions)
4. In response to a message, an actor can:
   - Change its state
   - Send messages
   - Create child actors

Here’s an actor who follows the rules.

actor.start();
actor.send(message);          // send to another
actor.subscribe(s => { ...}); // listen to another
actor.getSnapshot();          // cached result from .subscribe

As you can see, actors alone are actually pretty simple, almost pure.

As long as you play by the rules, you won’t have race conditions and lock spaghetti. You also get a few other benefits:

• Location independence. The interface means that you can be confident each actor will behave consistently regardless of its location. Doesn’t matter if one actor is on localhost and another is remote. Two actors can even share the same thread (no thread management!).

• Fault tolerance. If one actor fails, its manager can revive it or pass its message to an available actor.

• Scalability. Actors are easy to instantiate, making them compatible with microservices and horizontal scaling.

• Composability. They encourage atomic over monolithic architecture.

Actors in the wild

There was a quiet period after Hewitt dropped his paper, but adoption has recently taken off in response to our growing data footprint.

Here are a few modern examples of the actor model:

Video editing software sends each setting change so that they are reflected in the draft immediately.

Trading platforms (Robinhood) treat each withdrawal as an isolated actor, which has a function to update an account. Say you have $10 in your account. If you and your wife both try to buy $10 of $GME simultaneously, your app will process the request that arrives first. When the second gets out of the queue, it’ll run the checkBalance statement, see that it’s now $0, and deny it.

AI agents. An agent is an actor. It passes messages (prompts), has internal state (context), and spawns other agents (actors). AI Agents are the perfect use case for the pattern, as Hewitt anticipated. His original paper was called “A Universal Modular ACTOR Formalism for Artificial Intelligence,” after all.

If you squint past the trendy design and jargon on Akka’s site, it looks like the same pitch Hewitt made five decades ago.

Event-driven fabric

Multi-step agents

Persisted state through snapshots and change events

Isolated execution units that maintain their own state

Parallel processing without shared bottlenecks

Unbounded data flows

The next time you see these keywords on a landing page, remember to get out your Actor Pattern Buzzword card and yell “Bingo!”

This concept is relevant on the frontier as well as in the verticals. Cursor’s self-driving codebase experiment took a first-principled journey from unconstrained sharing to a well-defined communication flow between planner and workers. Sounds very similar to actors and managers, doesn’t it?

Is it silly that we’re pretending to be pioneers by rejecting the idea of a global state? Yes, but at least we’re not the only ones who need to rediscover old problems: the agents in Cursor’s experiment also tried and failed to make locks work.

Agents held locks for too long, forgot to release them, tried to lock or unlock when it was illegal to, and in general didn’t understand the significance of holding a lock on the coordination file. — Wilson Lin @ Cursor

Thankfully, the Actor Pattern still works regardless of our willingness to recognize it.

What’s the catch?

It’s really cool that the Actor Pattern has gained relevance over the decades. But the model isn’t without trade-offs. Things become more complicated when you add production necessities, such as a manager, callbacks, promises, initial state, and recovery. The ease of composition will inevitably lead some teams into microservice hell, where they’ll get lost in boilerplate and hairball graphs.

Debugging dataflow bugs can be harder. Although each actor has well-isolated logging, it can be harder to trace a bug across multiple services.

Price. Agents that create more agents is a dream for the engineering team. If that process isn’t constrained, however, it’ll turn into a nightmare for the finance team.

Finally, implementing the actor model in a big org requires education. Pure functions, state machines, event-driven architecture — these are unfamiliar concepts to many. It took me days of research before I “got it.” Many orgs won’t want to dedicate the time to get everyone thinking in a new paradigm, so they’ll fall back to their monolithic habits.

Thankfully, the industry has started bridging the gap between the usefulness of this Actor Model and its adoption complexity by creating languages and frameworks. These preserve the actors’ tenets while making them easier to implement.

Summary

The actor model makes it easier to avoid locks and race conditions in distributed systems. It does this by standardizing communication and data access.

Letting too many things share the same data and chat freely leads to chaos. Think: startup engineer who has to handle info from Slack, hallway requests from the PM, email, and standup. That guy is going to overcommit (deadlock), forget (data loss), and struggle to organize (recovery).

Using an actor model is like requiring everyone to communicate over email. Everyone follows the rules of SMTP (recipient, subject, body) and can only respond to one email at a time (concurrency). In this system, the communication constraints minimize mistakes and conflicts.

All this adds up to a faster, more reliable system at scale. Everyone knows how to talk to each other. They know how to ask for and deliver things. This helps them work autonomously without blocking others.

Having an efficient pattern becomes more important the more distributed a system becomes. As Gul predicted in 1985, more time is spent on “communication lags than on primitive transformations of data.” A team that knows that all too well is Discord, which has successfully instrumented the Actor Model to process trillions of messages without data loss or latency.

Let’s see how.

Part II: How Discord Processes Trillions of Messages

Everything is an “actor.” Every single Discord server, WebSocket connection, voice call, screenshare, etc... distributed using a consistent hash ring.

It’s an incredibly great model for these things. We’ve been able to scale this system from hundreds to hundreds of millions of users with very little changes to the underlying architecture.

— Jake, Principal Engineer @ Discord

Thanks to Discord’s initial target user, gamers, speed has always been an unquestioned requirement. When a message is sent, others need to see it immediately. When someone joins a voice channel, they should be able to start yapping right away. A delayed message or laggy chat can ruin a match.

Discord needed a smooth way to turn plain text/voice data into internal messages and then route them to the correct guild (AKA: Discord server) in real-time.

How data flows

Guilds and users talk over Elixir and WebSockets.

1. Users connect to a WebSocket and spin up an Elixir session process, which then connects to the guild.

2. Each guild has a single Elixir process, which acts as a router for all guild activity.

3. A Rust data service deduplicates API queries before sending them to ScyllaDB.

4. Background communication happens over PubSub. For example, the Elasticsearch worker consumes events, batches with others, and starts indexing.

How the fan-out works

“Fan-out” refers to the act of sending a message to multiple destinations in parallel without requiring a response. This is exactly what Discord needed to implement to make their chat feel real-time. When a user comes online, they connect to a guild, and the guild publishes presence to all other connected sessions. Each guild and connected user has one long-lived Erlang process. The guild’s process keeps track of client sessions and sends updates to them. When a guild receives a message, it fans it out to all client sessions. Finally, each session process pushes the update over WebSocket to the client.

In other words:

User ↔ WebSocket ↔ Session ↔ Guild

Improving the fan-out

Elixir’s functional implementation of the Actor Pattern allowed it to handle a lot of processes with ease compared to other languages.

# Publishing to other guilds in 4 lines of Elixir. Pretty neat.
def handle_call({:publish, message}, _from, %{sessions: sessions}=state) do
  Enum.each(sessions, &send(&1.pid, message))
  {:reply, :ok, state}
end

The language’s affordances helped Discord get started easily. But as usage grew, it became clear that they’d need to do more to stay responsive. If 1,000 online members each said, “hello, world” once, Discord would have to process 1 million notifications.

10,000 messages → 100 million notifications.

100,000 → 10 billion.

Given that each guild is one process, they needed to max out the throughput of each. Here’s how they did that:

1. Splitting the work across multiple threads using a relay

2. Tuning the Elixir in-memory database

3. Using workers processes to offload operations (maintenance, deploys)

4. Delegating the fan-out to a separate “sender,” offloading the main process

Thanks to these efforts, routing the notifications to users in a massive guild no longer became the crippling bottleneck it easily could’ve.

But when one bottleneck is removed, another takes its place. In Discord’s case, their next biggest problem shifted from the messaging layer down to the database.

Part III: Discord’s Hot Partitions

Problem: The Cassandra database was causing slowness

With notifications being routed to the correct guild, and Discord’s API converting those payloads into queries, the last step was to get the data back from their Apache Cassandra cluster. As a NoSQL database optimized for horizontal scaling, Cassandra should’ve scaled linearly without degrading performance. Yet, after adding 177 nodes and trillions of messages, reads were slowing in popular servers.

Why??

In a word: partitions.

Discord partitioned their messages data across Cassandra using channel ids and a 10-day window, and partitions were replicated across three Cassandra nodes.

CREATE TABLE messages (
    channel_id bigint,
    bucket int,        // static 10-day time window
    message_id bigint,
    author_id bigint,
    content text,
    PRIMARY KEY ((channel_id, bucket), message_id)
) WITH CLUSTERING ORDER BY (message_id DESC);

That’s fine, but two other factors were at play:

1. Popular guilds get orders of magnitude more messages than smaller ones. Midjourney has 19 million members, of whom over 2 million could be online at any time.

2. Reads (search, fetching chat history) are more expensive than writes in Cassandra, due to its need to query its memtable and SSTables.

Lots of reads led to hot partitions, which slowed the messages reads, which slowed the guild. What’s worse, the (valid) use of replication and quorums made other parts of Discord slow, like the upstream nodes that queried the hot partitions. Running maintenance on a backed-up cluster also became tricky due to Cassandra’s high garbage collection, which resulted in many “stop the world” events. Cassandra’s endless config options — flags, gc algorithms, heap sizes — only made it harder to mitigate these issues.

  // messages in a channel
  [{
    "channel_id": 1072891234567890123,
    "bucket": 19783, // #announcements channel
    "message_id": 1072934567890123456,
    "author_id": 856781234567890123, // Midjourney dev
    "content": "Hey @everyone, v2 is live. Try it out!"
  },
  {
    "channel_id": 1072891234567890123, // snowflake id
    "bucket": 19783,
    "message_id": 1072934567890123455,
    "author_id": 923456789012345678, // user
    "content": "woah, so cool"
  }]

Why the partition scheme wasn’t the problem

I instinctively pointed my finger at that 10-day static bucket window and thought, “That’s silly. There’s gotta be a better way to organize the data.”

But is there?

• If they dropped the bucket altogether, the hot partitions would get even hotter.

• If they added a random shard to the partition key, they’d trade hot partition problems with fan-out problems (extra queries, more latency).

• If they made the buckets smaller (1-day or 1-hour), they’d have to coordinate more round trips.

• What if they kept Cassandra for what it’s good at (writes) and moved reads to another solution (cache + index + log)? Sounds complicated.

• Partitioning by message_id would eliminate hot partitions altogether. But Discord’s primary use case for reads is to get the most recent messages in a channel. Fetching “the last 50 messages in #general” now becomes a huge hunt across every node in the cluster.

Every option either makes it harder to target the data or trades hot partition problems with other problems. So, searching for a more capable database actually seems like a sound first step. If they find one that can cool down the partitions enough to avoid a bottleneck, they could avoid adding more business logic to the data layer.

Thankfully, they found what they were looking for

Solution 1: Switch to ScyllaDB

ScyllaDB is a hard-to-pronounce C++ NoSQL data store that advertises itself as “The Cassandra you really wanted.”

Zing.

On paper, it seems to back that claim up. It’s faster. It’s cheaper. It’s simpler.

How? Here is the gist:

• Per-core sharding → more efficient CPU

• Per-query cache bypassing → faster queries

• Per-row repair (vs partition) → faster maintenance

• Better algorithms for scheduling, compiling, drivers, and transactions

• No garbage collection

After reading the full Cassandra vs ScyllaDB comparison, I actually felt bad for Cassandra. Like, this was supposed to be a friendly scrimmage — take it easy, guys.

Similar to their early investment in the then-unproven Elixir language, their early bet on Scylla still demanded some work. They had to work with the Scylla team to improve its reverse queries, for example, which were initially insufficient for scanning messages in ascending order. Eventually, Scylla got good enough to support Discord’s workflows, and Discord got good enough at Scylla to casually write the 392-page book on it.

# Optimization scorecard

Better fan-out (service layer): ✔️
Better DB (data layer): ✔️

What else did Discord do to maintain speed at scale?

Turns out, everything.

Let’s look at two more in detail.

Solution 2: Request Coalescing

Regardless of how compatible your DB is with your app, overriding it with requests will cause latency. Discord encountered this problem with its Python API, which other services could call directly to fetch data from the database. Unfortunately, the message fan-out mechanism resulted in lots of duplicate requests in popular guilds.

This duplicate request problem isn’t unique to Discord, Python, or NoSQL. It’s been around ever since lots of people wanted to access the same information. If you are serving thousands of hits per second, the queue of waiting requests can get huge.

This introduces two problems:

1. The CPU releases thousands of threads, sending the load sky high (AKA: “thundering herd”).

2. Users don’t like waiting.

To protect itself from a herd of redundant messages queries after someone spams @everyone in #announcements, Discord introduced a new Rust service, their Data Service Library. When a request comes into the Data Service Library, it subscribes to a worker thread. This worker makes the query and then sends the result over gRPC to all the subscribers. If 100 users request the same message simultaneously, there will only be one DB request.

Request -> Python API -> DB
Request -> Python API -> Data Service Library -> DB

This technique successfully reduced inbound query rate by 10-50x for popular channels and scaled DB connections linearly (one per worker, not per concurrent request). In addition to speeding up UX, it also increased Discord’s confidence in shipping concurrent code, which undoubtedly improved product velocity and reduced bugs.

Most importantly, however, it let them say they rewrote it in Rust.

Meme cred is very important.

— Bo Ingram, Senior Software Engineer @ Discord

Not a team to rest on their meme laurels, they further optimized request handling by routing each message according to its channel_id. For example, all requests for an #announcements channel go to the same message service. This sounds like it’d be a bad thing, but apparently Rust’s magic lets it handle this case without a sweat. This also makes the request routing from the above point much simpler, as all queries for channel_id=123 hit the same instance’s in-memory cache.

Was the Rust coalescing service overkill? Should they have just used their Redis cache and moved on?

I don’t think so. Caching is about serving slightly stale content. Instead of reducing the total number of requests in the herd, it would’ve simply made each request faster. It also would’ve come with more code to manage.

Caching as a standalone solution to the thundering herd works for apps that don’t need to feel “real-time.” For example, a news site might serve a slightly outdated version of its main page immediately to prevent visitors from having to wait to read something. This buys them time to fetch and render the latest article. The result is faster perceived load times. If they didn’t do this, a thundering herd problem would emerge if 10,000 visitors loaded a trending article at the same time.

Here’s how one HTTP proxy implemented this caching strategy using a simple grace value:

// Vinyl Cache (2010-2014)
// <https://vinyl-cache.org/docs/6.1/users-guide/vcl-grace.html#grace-mode>
sub vcl_hit {
     if (obj.ttl >= 0s) {
          // A pure unadulterated hit, deliver it
          return (deliver);
     }
     if (obj.ttl + obj.grace > 0s) {
          // Object is in grace, deliver it
          // Automatically triggers a background fetch
          return (deliver);
     }
     // fetch & deliver once we get the result
     return (miss);
}

Key takeaway: When queries are expensive (DB round-trip, API call), in-flight deduplication is cheaper than a distributed cache.

For the record, Discord utilizes this type of caching with their CDN and Redis, but not in the Rust Data Service. (If you’re still hungry for more request optimization nuances, read up on Request Hedging and comment whether you think it would’ve helped Discord’s thundering herd problem.)

Clearly, the request layer was prime territory for optimization, which makes sense for an app that sends many small payloads (a few hundred bytes each). What might be more surprising is where Discord unlocked its next big performance win.

# Optimization scorecard
Better fan-out (service layer): ✔️
Better DB (data layer): ✔️
Better queries (API layer): ✔️
Memes (vibes layer): ✔️
Better disks (hardware!): ??

Solution 3: Super-Disk

Eventually, the biggest impact on Discord’s database performance became disk operation latency. This latency resulted in an ever-growing queue of disk reads, eventually causing queries to time out.

Discord runs on GCP, which offers SSDs that can operate with microseconds. How is this even an issue?

Remember that Discord needs speed and reliability. The SSDs are fast, but they failed Discord’s second requirement during internal testing. If a disk holds data in memory and then fails, the data is unrecoverable.

GCP + SSD = 🙅‍♂️ (too unreliable)

What about GCP’s Persistent Disk option? These are great for duplicating and recovering data. But they’re slow: expect a couple of milliseconds of latency, compared to half a millisecond for SSDs.

GCP + Persistent Disk = 🙅‍♂️ (too slow)

No worries, surely GCP offers a disk abstraction that delivers the best of both SSD and Persistent Disk.

Nope.

Not only does GCP make you pick between fast and reliable, but their SSDs max out at 375 GB — a non-starter for the Discord DB instances that require 1 TB.

Once again, the Discord engineers were on their own.

They had to find a way to

1. Stay within GCP

2. Continue snapshotting for backups

3. Get low-latency disk reads

4. Maintain uptime guarantees

So what creative solution did they cook up this time? The “Super-Disk”: A disk abstraction that involves Linux, a write-through cache, and RAID arrays. They wrote a breakdown with the details, so here’s the summary: GCP + SSD + Persistent Disk = 👍 (fast, reliable).

Bonus: 5 More Solutions

We walked through the juicy optimization solutions to the hot partition problem. But Discord didn’t view performance as an isolated priority. Instead, they took every opportunity they found to make things fast. Here is a final roundup of other interesting techniques they used.

Request routing by PIDs (messaging). They optimized how messages are passed between nodes by grouping PIDs by their remote node and then hashing them according to the number of cores. Then they called it Manifold and open-sourced it.

Sortable Ids (data). They used Twitter’s “Snowflake” id format, which is based on timestamps. This let them generate ids without a DB, sort by creation time, and infer when a message was sent by its id alone. The Snowflake project is zipped up in a public archive, but I dug up the good part for you. Notice how the timestamp is used.

  protected[snowflake] def nextId(): Long = synchronized {
    var timestamp = timeGen()

    if (timestamp < lastTimestamp) {
      exceptionCounter.incr(1)
      log.error("clock is moving backwards.  Rejecting requests until %d.", lastTimestamp);
      throw new InvalidSystemClock("Clock moved backwards.  Refusing to generate id for %d milliseconds".format(
        lastTimestamp - timestamp))
    }

    if (lastTimestamp == timestamp) {
      sequence = (sequence + 1) & sequenceMask
      if (sequence == 0) {
        timestamp = tilNextMillis(lastTimestamp)
      }
    } else {
      sequence = 0
    }

    lastTimestamp = timestamp
    ((timestamp - twepoch) << timestampLeftShift) |
      (datacenterId << datacenterIdShift) |
      (workerId << workerIdShift) | 
      sequence
  }

Of course, Discord had to add its own touch to the ID. Instead of using the normal epoch (1970), they used the company’s epoch: the first second of 2015. I’d normally roll my eyes at this kind of unnecessary complexity. But that’s pretty cool, so instead I grant +5 meme points.

Elasticsearch abstraction (search). Although they don’t store messages in Elasticsearch, they store message metadata and user DMs for indexing and fast retrieval. To support the use case of searching through all of a user’s DMs, they grouped their Elasticsearch clusters in “cells.” This helped them avoid some bottlenecks during indexing. It also helped them index a message by user_id (instead of guild_id), which enabled cross-DM search.

WebRTC tuning (entry). I stayed away from WebRTC here because I covered the basics in the Google Meet breakdown on our YouTube channel. However, Discord does do some interesting stuff at this layer: using the Opus codec, circumventing the auto-ducking on Windows in favor of their own volume control, voice detection, and reducing bandwidth during silence.

Passive sessions (entry): Turns out that 90% of large server sessions are passive; the users aren’t actively reading or writing in them. The team realized they could improve latency by simply sending the data to the right users at the right time. If a user doesn’t have a guild tab open, they won’t receive all of the new messages. But once they tab into the guild, their passive session will be “upgraded” into a normal session, and they’ll receive the full firehose of messages in real-time. This led to a 20% reduction in bandwidth. Amazing example of a big perf win that isn’t technically complicated.

Emojis and lists on Android (client): They were committed to React Native. Then they saw that the custom emojis weren’t rendering well on low-grade Androids. So, they abandoned their original plan and wrote the emoji feature in Kotlin while also maintaining React Native. The worst of both worlds: their architecture is now split between iOS (React Native) and Android (native).

Similar story for rendering lists. They needed to create their own library. Then they adopted Shopify’s. When that stopped working, they created another library that used the native RecyclerView. All for rendering lists quickly on Android.

# Optimization scorecard
Fan-out (service layer): ✔️
DB (data layer): ✔️
Queries (API layer): ✔️
Disk (hardware): ✔️
Routing: ✔️
Ids: ✔️
WebRTC: ✔️
Sessions: ✔️
Emojis: ✔️

This team will tackle any performance problem in the stack with a calm recklessness that I can’t help but admire.

Part IV: Lessons for Engineers

While most of us won’t have to handle trillions of messages, there are some elegant engineering principles we can learn from Discord’s success over the last decade.

Build for simplicity in v1, but design for change.

We examined the migration from Cassandra → ScyllaDB. But they actually started with a DB that was even more unfit for their eventual scale: Mongo. They picked it during the early days because it bought them the time to experiment and learn what they needed. As they suspected, Mongo eventually gave them enough problems (like incorrectly labeling states) to warrant moving to Cassandra. When their Cassandra problems got serious enough, they moved to Scylla.

This is the right way the stack should evolve.

They could’ve never anticipated the bottlenecks at the onset, let alone the right solutions to them (ScyllaDB wasn’t even a viable alternative at the time). Instead of wasting time over-projecting future problems, they picked the tool that helped them serve users and focused on their current bottleneck. They also didn’t overfit their systems according to their current DB, which would’ve slowed down the inevitable migration.

Anticipate the future headaches, but don’t worry about preventing them in v1. Create the skeleton of your solution. Fill in the details with a hacky v0. When the pain happens, find your next solution.

Learn the fundamentals.

Language fundamentals.

Deeply understanding a language helps you pick the right one. When Discord started evaluating the functional programming language Elixir, it was only three years old. Elixir’s guarantees seemed too good to be true. As they learned more about it, however, they found it easier to trust. As they scaled, so did their need to master Elixir’s abstractions for concurrency, distribution, and fault-tolerance. Nowadays, frameworks handle these concerns, but the language provided the building blocks for capable teams like Discord to assemble their own solutions. After over a decade, it seems like they’re happy they took a bet on an up-and-coming language:

What we do in Discord would not be possible without Elixir. It wouldn’t be possible in Node or Python. We would not be able to build this with five engineers if it was a C++ codebase. Learning Elixir fundamentally changed the way I think and reason about software. It gave me new insights and new ways of tackling problems.

— Jake Heinz, Lead Software Engineer @ Discord

There is a world where the 2016 Discord engineers were unwilling to experiment with an unproven language, instead opting for the status quo stack. No one would’ve pushed back at the time. Perhaps that’s also why no one can match their performance in 2026.

Garbage collection fundamentals.

GC’s unpredictability hampered Discord’s performance on multiple dimensions.

We really don’t like garbage collection

— Bo Ingram, Senior Software Engineer @ Discord

On the DB layer, Cassandra’s GC caused those “stop-the-world” slowdowns. On the process layer, BEAM’s default config value for its virtual binary heap was at odds with Discord’s use case. Although it was a simple config fix, a lot of debugging and head-scratching were required (discord).

Unfortunately, our usage pattern meant that we would continually trigger garbage collection to reclaim a couple hundred kilobytes of memory, at the cost of copying a heap which was gigabytes in size, a trade-off which was clearly not worth it.

— Yuliy Pisetsky, Staff Software Engineer @ Discord

At the microservice layer, they had a ‘Read States’ service written in Go, whose sole purpose was to keep track of which channels and messages users had read. It was used every time you connected to Discord, every time a message was sent, and every time a message was read. Surprise, surprise — Go’s delayed garbage collection led to CPU spikes every two minutes. They had to dig into Go’s source code to understand why this was happening, and then had to roll out a duct-tape config solution that worked OK enough…until Rust, void of a garbage collector, arrived to save the day.

Even with just basic optimization, Rust was able to outperform the hyper hand-tuned Go version.

— Jesse Howarth, Software Engineer @ Discord (medium)

The team had to go on so many wild garbage-collecting goose hunts that I’m sure they’re now experts in the topic. While learning more about gc wouldn’t have prevented the problems from happening, I suspect it would’ve made the root cause analysis smoother.

As MJ said, “Get the fundamentals down and the level of everything you do will rise.” If Michael Jordan and The Michael Jordan of Chat Apps can appreciate the basics, then so can I.

Define your requirements early.

The high expectations of their early gamer users solidified the need to prioritize speed and reliability from day one. This gave the young team clarity: they knew they’d need to scale horizontally and obsess over performance.

From the very start, we made very conscious engineering and product decisions to keep Discord well suited for voice chat while playing your favorite game with your friends. These decisions enabled us to massively scale our operation with a small team and limited resources.

— Jozsef Vass, Staff Software Engineer, Discord

At scale, even a “small” feature can turn into a massive undertaking. Calling @here or sending ... typing presence in a 1M user Midjourney guild isn’t trivial. Discord’s need to deliver first-class speed forced it to keep the UX refined and focus on the endless behind-the-scenes witchcraft we discussed here. Had they compromised their performance principle by chasing trends or rushed into other verticals, they would’ve eroded their speed advantage.

This is why, eleven years since the Discord epoch, the app still doesn’t feel bloated or slow.

Although every feature feels like it’s just a prompt away from prod, clarifying your value prop is still crucial. Discord’s was voice chat for gamers.

What is yours?

If you don’t have a good answer, you’ll either ship so slow that you’ll get outcompeted or go so fast that your app becomes slop.

Build a good engineering culture.

Step 1: Have good engineers.

I haven’t seen anyone question this team’s competence. But here’s an example for any lurking doubters.

In 2020, Discord’s messaging system had over 20 services and was deployed to a cluster of 500 Elixir machines capable of handling millions of concurrent users and pushing dozens of millions of messages per second. Yet the infra team responsible only had five engineers. Oh, and none of them had experience with Elixir before joining the company.

Clearly, there were no weak links on this team.

These people are rarely incubated by altruistic companies. Instead, their obsession with juicy technical problems drives their mastery. Then they find the companies that have challenging technical problems and the right incentives. Clearly, Discord has both.

If you don’t have traction, frontier problems, or cash, you’ll have to compete for talent on different vectors. The engineers in your funnel probably won’t be as good as Discord’s, but the takeaway is the same: hire the best you can get.

Step 2: Let them be creative.

Good engineers come up with creative solutions to hard problems.

We saw this in the multi-pronged attack on their hot partition problem. When faced with database issues, lesser engineers would’ve blamed everything on Cassandra and the old engineers who chose it. Then they’d have given hollow promises about how all their problems would fix themselves once they switched to a new DB. Discord instead got curious about how they could make things easier for whatever DB they used, which led them to their Data Service Library for request coalescing.

Same story with the Super-Disk. Investigating the internal disk strategy of their GCP instances isn’t a project that a VP would assign a team in a top-down fashion. Instead, management let the engineers explore the problem space until a unique solution emerged.

Having a culture where engineers can figure stuff out is the corollary benefit to getting clear on your core value. Ideas come from the bottom up, people get excited about the work, and unique solutions emerge. I commend both the Discord engineers for embracing the challenges we discussed and the managers who let them cook.

How creative is your team in the face of tough problems? How can you give them the clarity and space they need to venture beyond the trivial solutions?

Don’t seek complexity, but accept it when it finds you.

Complexity is not a virtue. Every engineer learns this the hard way after their lil’ refactor causes downtime and gets them called into the principal’s manager’s office. The true motivation behind a lot of these changes is, “I’m bored, and this new thing seems fun.” (Looking at you, devs who claim users need an agentic workflow to fill out a form). Complex is challenging. I like challenges. Therefore, let’s do it.

No.

But complexity also isn’t the enemy. For Discord, the enemy is latency. When they run out of simple ways to make their app fast, they come up with creative solutions that, yes, are often complex. Their tolerance for the right complexity is what makes them great. They do the complex stuff not for themselves, but for their users.

It’s OK to complicate things…as long as it helps the users.

As we’ve seen, Discord’s need for speed led it across every layer of the stack, from cheap Androids to expensive data centers. This article documents plenty of well-executed perf techniques along the way — some textbook, some creative. What it really highlights, though, is the power available to teams who commit to helping their users at scale.

Recommendations

The Actor Model

Carl Hewitt (wikipedia)

The Actor Model (wikipedia)

A Feature Model of Programming Languages (sciencedirect)

The Actor Model Whiteboard Session with Carl Hewitt (youtube)

The Actor Model, Behind the Scenes with XState (youtube)

Why Did the Actor Model Not Take Off? (reddit)

Performance Optimization

Fan-Out Definition and Examples (dagster)

How Discord Indexes Trillions of Messages (discord • hackernews • youtube)

How Discord Supercharges Network Disks for Extreme Low Latency (discord)

Pushing Discord’s Limits with a Million+ Online Users in a Single Server (discord)

How Discord Handles Two and Half Million Concurrent Voice Users using WebRTC (discord)

In Google Calendar’s Secret Engineer Weapon: Restraint, I praised Google for unapologetically saying “no” to the temptations of modernity.

"Use a frontend framework."
No.
JavaScript.

"Use Typescript."
No.
Javascript.

"Use a trendy CSS tool."
No.
CSS classnames.

"Give us a native desktop app."
No.
Browser.

"Give us dark mode."
No.

"Make it really fast."
No.

Their commitment to simplicity has helped GCal to defend its position as the default calendar for billions of regular people.

For API clients, however, that same simplicity makes life tragically complicated.

How that complexity creeps in, and what we can learn from it as builders.

The Problem: Keeping Data In-Sync

You’re building a cool new calendar called Cool Calendar.

To help users get started, you allow them to import their existing Google Calendar.

This’ll require access to your users’ Google Calendar data.

You look at the GCal API and see the typical suspects:

“Cool,” you think. “I just gotta get the calendarId, then get all the events for that calendar. When I need to update an event, I’ll use the eventId from the GET events response.”

You could do all this with a few simple functions:

It works! All the events are in your DB, and they also show up in the user’s GCal after a user updates an event from your UI.

You open the GCal app, create a new event called “Celebrate at Denny’s,” and look forward to your cinnamon roll pancakes.

Cool Cal app is open, but the new event doesn’t show up.

Your empty stomach sinks as you realize you only implemented half the solution (CoolCal → GCal).

You still need to ensure that GCal changes propagate to Cool Calendar (GCal → CoolCal).

Denial

You know that others have encountered this use-case before, so you revisit the GCal API docs.

Thankfully, you find a guide called “Synchronize resources effectively.”

The guide is short, which puts you at-ease.

It looks like you just need to add a few more calls.

Since you don’t know how many events were updated or when they were, you know you’ll need a loop to fetch all of them.

Makes sense.

Thankfully, the docs include a full snippet of this in Java.

It even uses the do-while looping strategy, which makes you feel smart.

Surely you can just convert that to JavaScript and move on.

Anger

It’s been an hour, and you’ve successfully converted the snippet to a JavaScript function called syncGcal() and updated your API.

It works!

Whenever you run syncGcal(), a user’s new GCal data is added to CoolCal.

You resolve to include the New York Style Cheesecake for your troubles.

As you’re on your way out, a few questions arise in your tired brain…

1. The syncToken isn’t persisted anywhere

2. syncGcal() only works when it’s called manually

How often should I run syncGcal()?

Every hour?

Every minute?

Where will I store the token?

Local storage?

The user collection?

A new collection?

What happens if the token becomes invalid?

Can I keep retrying?

Would I need to do a full import as a backup?

What about quota limits?

What if a user makes a change during this process?

You become enraged — at the guide for not warning you about these edge cases, then at your naiveté for assuming it should.

Bargaining

“I’m not alone — I can reach out for help,” you say to yourself, just like you practiced in therapy.

You’re confident that the API docs will have a method called syncResources() that you overlooked.

Or it’ll be offered as a service? Like some YC company that’ll magically sync all your data for you.

At the very least, there’ll be a Richard Stallman disciple who has an MIT repo you can point Claude to.

> clone the two-way sync stuff from github.com/stallman-cal. no bugs.

Depression

The official docs are bare.

All the YC companies are building agents.

Stallman doesn’t trust your for-profit calendar app.

(There’s an MIT project called Compass Calendar, but it only has 200 stars. Can’t be trusted.)

No one cares about your two-way sync feature.

You are, in fact, all alone.

A single tear falls onto your mechanical keyboard as you realize you won’t be going to Denny’s today.

Acceptance

After homemade French toast and a postprandial walk, you come to terms with your destiny to act like a real engineer and do it yourself.

A few weeks later, it’s finally done.

But you still can’t stop wondering

“Why did GCal do it this way?”

“Did it have to be this complicated?”

The Requirements

You put yourself in Google’s shoes by reverse-engineering the requirements in the form of a Product Requirements Doc (PRD).

The Solution

That was fun, so you document the solution to meet those requirements as a Technical Design Doc (TDD).

Analysis

Finally, you feel qualified to say whether GCal’s “Give ‘em a token” solution was a good one.

Strengths of Token Pagination

Opaque tokens protect the client from overwhelm. They can hide the complexity from the client so they don’t need to understand everything (multi-region replication, for example)

Cleanly handles deletions. By controlling when tokens expire, GCal can ensure clients always see deletions during incremental sync. Timestamp-based approaches can’t make this guarantee.

Easier implementation changes. Google can completely change its storage backend, sharding strategy, or topology without affecting clients. This surely made the migration from NoSQL (Megastore) to NewSQL (Spanner) easier in the 2010s (highscalability.com).

Weaknesses

Push is disconnected from sync. The webhook tells you something changed, but not what. This means every push notification triggers a new incremental sync call. This gets chatty when power-users make frequent tweaks. Why not include a lightweight diff in the push payload itself, or at least indicate which event IDs changed?

Some filters are incompatible. Incremental syncs are incompatible with the timeMin and timeMax parameters. You must choose one method or the other. This is a bummer for humans, who think in terms of time more easily than in tokens.

No partial sync recovery. The page tokens are ephemeral. If a client crashes mid-pagination (after page 3 of 7), they must restart the entire sync. For large calendars, this is painful. An alternative would be durable page tokens that survive across sessions, or checkpointing the sync progress.

Alternatives (and why they weren’t used)

Bottom Line

The sync token + cursor pagination approach is a pragmatic, battle-tested pattern that works. It’s the same pattern used by Microsoft Graph (delta tokens), Stripe (pagination cursors), and most enterprise APIs.

Google chose it because it lets them hide internal complexity while providing “good enough” sync semantics for 99% of calendar use cases.

The alternatives (event sourcing, CRDTs, OT) would only make sense if GCal required real-time collaboration or true offline-first P2P sync.

The Full Picture

If Google’s goal with GCal were to generate revenue, they’d offer more docs, videos, and webinars. There’d be a handsome Developer Advocate, tiered API pricing, and a Client Success Team. I’d see billboards around SF that just say,

npm i gcal

But Google’s goal with GCal is less ambitious:

>> Don’t annoy regular users so much that they download Apple Calendar.

It’s not fair to expect so much out of a free API.

It’s amazing that you can integrate with GCal in the first place

GCal honors its responsibility as a foundational part of your app by prioritizing reliability over adoption convenience:

• Not many failures occur.

• The API does what its docs say, even if they don’t say much.

• They don’t surprise you with breaking changes

Still, there is a gap between the developer’s expectations and reality when they first think, “I know, I’ll just integrate with GCal.”

Currently, the developer has to close that gap with time, effort, and a loss of innocence.

But Google could reduce the pain by providing more resources.

For example, the docs could offer more implementation guidance.

The code samples could evolve from bare repos to demo applications.

On balance, keeping the API simple was the right move.

A primitive API might annoy power users.

But if it doesn’t affect UX, revenue, retention, or the brand, then keep it simple.

Takeaways for Builders

When creating a public sync API

Keep it boring: use existing standards

We’ve seen here how GCal’s API is built on cursor pagination and sync tokens. Its event model is also built around established standards: CalDAV, iCalendar, and RRULEs.

Instead of inventing their optimized versions, they stuck with the basics. This is a big reason why there’ve been so many successful GCal integrations since their API went public in 2006 (unlike Yahoo).

Use cursor pagination when the data is a moving target

When writes happen while you read, you need consistency across pages to avoid duplicates and missed rows.

Examples: Calendars, feeds, audit logs, append-only streams.

Use offset pagination when the data is stable

If data is static and doesn’t have concurrent writes, you can get away with a simpler approach.

Example: Search results with page numbers, admin table for small data.

Don’t oversell the API

You know how it feels to get excited in the morning to finish a PR and then for a hike … only to realize that it’s more complicated than you thought and not see daylight until the next afternoon.

Google’s API doesn’t promise to do everything for you.

The end product will sell the API: spend your time delighting your users with your product before worrying about delighting the devs who integrate with it

Don’t break things

If your integration is foundational to another person’s product, then go slow.

Spend more time testing the migration.

Get feedback on the docs before launching them.

Send “heads up” emails and terminal messages so no one is surprised.

This requires an identity shift from “crackhead vibe coder” who breaks things in prod to “disciplined software engineer.”

You won’t always get it right, but take the responsibility seriously.

One breaking change will do more damage than three features will do good.

GCal’s API is stable, so my sleep score thankfully hasn’t dropped since using it.

When integrating a public sync API

Don’t start with an integration

“My users won’t use my app unless it integrates with the tools they’re already using.”

Maybe, but let them tell you that.

Adding an integration is an expensive side quest on your product-market fit journey.

If the value prop is good enough, they’ll have no problem creating a new account and adding data from scratch.

Accept that Pareto is right

In other words, the last 20% of tasks will take 80% of the time.

The surface for edge cases and gotchas widens as you implement.

Google’s docs aren’t the only ones that will tempt you into thinking that things can be simple AND free.

Temper their promises by considering retries, idempotency, fallback, delta syncs, offline, token invalidation, quotas, and conditional requests (ETags).

You don’t need all of those for your MVP, but expect that they’ll come up eventually.

Simplicity for one party means complexity for another

Simple UI → complex backend

Good for regular users → difficult for power-users

Simple pricing → complex R&D

Simple API → complex integration

The next time you think, “I’ll just use this free API, it looks straightforward,” remember:

It’s either free or simple; Someone always pays.

Now go get your pancakes.

Doing fewer things is a great place to start, because it’ll give you more room for error as you figure out what works best for you.

The next best method is to commit to the goal: saying you’ll do something, and then doing it.

There are many ways to make commitments.

• The founder promises to go back to corporate if he doesn’t get his app profitable within 3 months.

• Nathan Berry said he’d create an app that makes 5k MRR within 6 months (he did it).

• Elon committed to raising Tesla’s market cap to 8.5T in 10 years.

What no one tells you is that how you commit is even more important than what you commit to.

Anatomy of a commitment

A commitment enforces constraints.

Here’s a chart that explains what constraint to select and how to communicate it.

Analysis

The more constraints, the more difficult the commitment

Getting ripped is harder than getting strong because it requires effort in the gym and the kitchen.

Profit is harder than revenue because it requires spending discipline.

Getting promoted is harder than completing a course because it requires approval.

Before committing to a bold outcome like becoming a millionaire within 5 years, focus on an input commitment like deep-working for 2 hrs a day.

Commit to the right audience.

Your Instagram audience doesn’t care that you work 70-hr weeks; they care about seeing cool pictures in their feed.

The users of your app don’t care about your margins; they care about how well your app solves their problem.

A good commitment shared with the wrong audience shows that you don’t know what people want.

Standup Example

Last year, we had three senior developers working on Compass.

Our yearly internal commitment was to become profitable.

Each week, we made individual commitments that moved us closer to that.

Here were our requirements for a commitment.

Basically, we just had to commit to closing a set of issues each week.

Since we don’t track hours, these were internal, outcome commitments.

This forced us to scope the work ahead of the meeting and allowed others to push back.

Most importantly, it made it clear whether we were executing well.

If everyone agreed that our commitments were realistic, but we kept missing them, then we knew we had an issue.

This cut through all the B.S. and helped us make corrections quicker than if we never put our work on the line in the first place.

Traps to avoid

Commitments also came with traps.

If you treat all commitments as equal, you’ll be less effective.

Which of these matters most for an unemployed junior software engineer?

1. Waking up at 5 am and building his portfolio for 5 hours

2. Submitting 20 quality jobs applications/week

3. Getting hired within 3 months

Answer: #3

If that engineer realizes he works best at night, he should drop the 5 am commitment.

If he hasn’t gotten one interview after 100 cold-applications, he should drop that approach and consider another, like networking.

If you make a big commitment, you’ll be less willing to change it as you get new data.

For example, you committed to writing a book about Framework X. But then AI happened, making Framework X irrelevant. You’ll want to keep writing instead of adapting to the new world.

If you make a public commitment and fail, you’ll want to pretend it never happened.

Sure, most people already forgot.

But you haven’t.

Staying quiet will do more damage to your psyche and reputation than owning it.

Have you ever filled out an internal survey and then realized that they never shared the results? It’s because they didn’t like the results, and they were too afraid to face the root problems.

BTW - We failed to become profitable last year. We owned it, made a lot of changes, and are moving forward (see, not that hard).

Why this work

It takes guts to put your reputation on the line.

It takes discipline to keep doing the work when life gets hectic.

The better you get at this, the more confidence you’ll have.

Your commitments will get more ambitious.

You’ll get better at keeping them.

People will notice that your words predict reality, which will make them more valuable.

Code is getting cheaper, but your credibility isn’t.

We treat “understanding” like the goal.

But the world doesn’t reward us for comprehension.

It rewards us for helping people.

That’s it.

Stories from history

When Edison was asked what he thought of “the Einstein theory,” his reply shocked everyone:

I don’t think anything of Einstein’s theory because I can’t understand it.

Einstein’s papers were dense and challenging, and the practical applications of relativity weren’t yet clear.

Edison could’ve stopped everything for weeks to study the paper until he could give a better answer and seem smart.

Instead, he continued to focus on creating a domestic source of rubber, saving the phonograph economy, and establishing a succession plan for his company — all problems that he could actually solve.

A lawyer was grilling Henry Ford on the witness stand with random facts in an attempt to make him look uneducated.

“Who was Benedict Arnold?”

“How many soldiers did the British send over to America to put down the Rebellion of 1776?”

Ford finally snapped and said,

“WHY should I clutter up my mind with general knowledge, for the purpose of being able to answer questions, when I have men around me who can supply any knowledge I require?”

In real life

We can snicker at that reporter and lawyer for missing the plot.

Yet we’re the ones who refuse to apply for a job until we have 3 more portfolio pieces … even though no one cares about your portfolio.

We refuse to launch our app on Product Hunt until it’s visually perfect … even though it solves user problems better than the incumbent.

We refuse to touch AWS before getting certified …even though no one asked us to be certified.

Root cause

Our early education followed the rhythms of time.

A 10-week quarter.

12 grades before high school graduation.

4 years of undergrad.

2 more years for the master’s.

Since these slots were pre-allocated, the task was to learn as much as possible and outperform others on the test.

We hoarded knowledge to outperform when we should’ve been using knowledge to help.

You might’ve learned enough about multiplication after 3 weeks of arithmetic to create a calculator app.

The app would’ve worked, but because you didn’t memorize the time tables, you would’ve only gotten a 70/100 on the test. So you put your creation on hold and kept learning memorizing instead.

This pattern of continuing to learn past the point of usefulness is what causes us engineers to feel like we’re never ready.

It’s why we force ourselves to finish the 200-page data structure book even though we got what we needed on page 20.

It’s why we optimize for performance before a user ever notices the lag

It’s why we avoid the hard problems at work because we don’t know how to solve them (yet).

The fix

Stop thinking in terms of exam scores (0-100).

Start thinking in terms of problems solved (Solved / Not solved).

Except there’s a catch:

You have to start with other people’s problems.

Once you’ve successfully solved someone’s problem, then you can ship it.

“But what if I don’t know enough to solve the problem?”

Then take a course or read a book….but stop once you have enough knowledge to solve the problem.

The task isn’t completing the course with a high grade.

The task is solving someone else’s problem.

An English student only needs to understand Macbeth enough to write a thesis that makes the teacher go, “Huh, interesting.”

A Junior Developer only needs to code well enough to close the issue assigned to them.

The Principal Architect only needs to know enough about distributed system design to write a doc for the team to start prototyping.

Final words

Whatever you want in life, you’ll get it faster by giving more.

Once you realize that the only thing stopping you from getting what you want is a bunch of other people’s problems, it’ll feel silly to delay the fix in the pursuit of obscure knowledge.

No one will ever say, “Your app would solve my problem perfectly, but I don’t think you understand Dijkstra’s algorithm, so I’m going to pass.”

“But I like learning.”

If amassing knowledge beyond the point of utility is fun, and you have time, then have fun.

But don’t trick yourself: maxing out your knowledge doesn’t automatically help others.

Unfortunately, productivity really is this simple:

Just help people.

For a deep-dive on how to set up a system to help others, see my latest video:

It’s a careful integration of the WebRTC framework into a reliable product that makes it successful.

TL;DR

How real-time video conferencing works

1. A cloud server helps two users make a connection

2. The two users share data privately, with some oversight from the cloud server

3. After the call, the users disconnect & the cloud server processes the data

WebRTC: The Foundation

How to use an open-source protocol to stream real-time video

Peer-to-peer connection is an important foundation of open real-time communication (RTC). The WebRTC framework makes that happen by establishing a standard way to connect and then share data … without needing to fuss about plugins, licenses, or apps. (Remember the Flash-era video tools that required clunky downloads?)

WebRTC handles:

• Connecting to peers

• Sharing audio, video, and chat data.

• IPs, ports, and exchange it with other WebRTC clients

• client resolution & codecs

// sample for creating a P2P connection
async function createOffer() {
const offer = await peerConnection.createOffer(offerOptions);

await peerConnection.setLocalDescription(offer);

outputTextarea.value = offer.sdp;

const peerConnection = new RTCPeerConnection(null);

const tracks = parseInt(numAudioTracksInput.value);

}

There are a couple of pieces that make all that possible

• A signaling server that acts like a match-maker between two clients

• A STUN/TURN relay server that keeps track of client IPs (even through NATs & firewalls!)

• The MediaStream API, which synchronizes camera and mic inputs

• Packets that are encoded and decoded so quickly that it appears “real-time”

  

How Google Meet Brings It Together

Having a capable open-source framework is one thing, but turning that into a product for billions is another.

Here are some of the things that the Google Meet team had to do to make your call run smoothly

Maintaining a strong connection

• Using its Selective Forwarding Unit (SFU) to avoid P2P mesh overload.

• Creating its own API to multiplex multiple virtual streams with large groups (WebRTC alone would crumble)

• Switching codes, IPs, and stream quality to respond to restrictive networks

• Reporting errors and closing sessions

Offering a lightweight client

• Minimal code: CSS, HTML, JS, libs are surprisingly straightforward

• Minimal Storage: mostly cookies for network info, LocalStorage for CPU stats

Other

• Using the Google Events API and Pub/Sub to respond to the meeting events

• Uses an in-house AI library (now powered by Gemini) to differentiate between subject/background to support the filters feature

Product

Tech aside — creating a unified video product for the Google Workspace was an accomplishment in product coordination. Google launched Hangouts in 2013. They rebranded it as Meet in 2020 to compete with Zoom. They incorporated their VoIP product (Duo) in 2022. They added Gemini in 2025. All while maintaining performance and keeping the UX light.

Takeaways

Start with the primitives, then solve the problems as they come. If you were to make a competing product, I’d suggest getting it working with WebRTC alone. Then continue to develop your signaling server algorithms as problems arise. A lot of “boring” work will come up, like NAT traversal, firewall rules, device permissions, and echo cancellation. Instead of trying to anticipate all these at the start, build upon a simple yet flexible foundation.

Don’t go cross-platform unless it’s necessary. Much like Google Calendar, Meet has resisted the urge to bloat the product with features or release a desktop app. This makes it easier to update and keeps it quick.

Use the platform’s APIs. The WebRTC docs are clunky. It’s an old and unsexy framework. But it works. Instead of rolling out their own protocol or plugins, Meet wisely built upon something reliable and focused on the UX. The web might not evolve as quickly as AI or mobile, but it has a lot to offer.

Stay out of the way. What makes Meet great is what it doesn’t do. No forced installs. No performance issues. No complex UX or onboarding. It’s integrated into Gmail and Google Calendar, making it seamless to set up a meeting.

Users don’t care whether the app is a pure-play like Meet or whether it uses native codes and a heavy WebAssembly media stack (like Zoom).

But the former will always be easier to make it seem like it “just works.”

It already takes a lot to make it seem like you do so little, so why make it harder than it has to be?

Further Reading

WebRTC: https://web.dev/articles/webrtc-basics

Mozilla’s guide: https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API

Google’s WebRTC Code: https://webrtc.googlesource.com/src/

What app would you like to learn about next?