Or just keep skimming this =]

These examples follow the playbook that Elon runs across his companies, officially called “The Algorithm”:

1. Question every requirement

2. Delete as many as possible

3. Simplify

4. Go faster

5. Automate

Last issue explained how Elon resorted to this playbook to solve bottlenecks at PayPal, Starlink, and SolarCity.

The same framework also works proactively, as we see with Tesla’s Optimus robot.

The one requirement Elon mandated was that it should resemble a human. But the designers, already students of The Algorithm, knew to eliminate. When designing the hand, they initially tried to omit the pinky, which seemed like overkill. They quickly realized that it made the robot too creepy, so they added it back.

This satisfied Elon’s corollary that 10% of eliminated parts should be added back (otherwise you’re not cutting out enough).

Estimating costs early also helped the team surface unnecessary complexity. When they reported that the actuators for the wrist would balloon costs by 54%, Elon pushed back hard:

"This is a shitty design. Use the damn lift gate actuators from our cars, which we know how to make cheaply.”

They eventually settled on a more refined design that included two finger joints (not like our three), two degrees of wrist rotation (unlike our three), and cheaper parts.

Takeaway for engineers

It’s never too early to push back against complexity. Although many of the Optimus meetings indulged in distant-future planning scenarios (like robots on Mars), Elon used cost to bring things back to earth.

We engineers usually don’t have as much visibility into costs, but we know how long things take. So when a PM proposes something complicated to placate a demanding client, it’s on us to push back about how much time that extra feature will take. Even better: explain how the real solution to going faster is not to add more parts, but to remove them and prevent more from being added so carelessly.

More

Elon keeps it simple (fullstack.zip)

Elon Musk, Walter Isaacson (amazon.com)

How Elon Works, Founders Podcast (youtube.com)

Or just keep reading =]

How Elon protects simplicity

Elon created “The Algorithm” as a playbook to accomplish great things quickly at low cost. It goes like this:

1. Question every requirement. Requirements are recommendations. The only immutable ones were those decreed by physics.

2. Delete any part or process you can. You may have to add them back later. In fact, if you don’t add back at least 10% of them, then you didn’t delete enough.

3. Simplify and optimize.

4. Accelerate cycle time. In other words, “Go faster.”

5. Automate.

Remove, simplify what remains, then automate the process. It seems trite in the abstract, but a few examples will show how radically committed Elon is to this philosophy in practice.

During the early Paypal days, opening a financial account required the user to fill out long forms and hand over their SSN and address. Elon rewrote his site to get the fewest number of keystrokes to open an account by eliminating those forms and even the requirement to have a username (using email as the userid).

When Starlink’s satellites became too bulky and expensive to manufacture, Elon fired the leaders and led marathon meetings to challenge every little requirement. A rat’s nest of parts was turned into a simple flat satellite, and costs decreased by orders of magnitude.

Life got way easier because we culled those parts. I was too chicken to propose that, but Elon made us try it.
— Mark Juncosa, Starlink

When SolarCity’s panels lagged, Elon fired the leaders and eliminated the requirement that they must work around every vent and chimney pipe. Instead, they decided to shear them off and put the panels on top. The 240 roof parts were halved. The website was simplified to offer just three roofs: small, medium, and large.

When SpaceX’s Raptor engine proved incapable of a Mars mission, Elon instructed his team to focus on a lean design, one reminiscent of a gnarly post-skinned cat. He suggested some extreme ideas, like deleting the whole gas manifold and booster skirt. Then he emphasized his commitment to leanness with a flurry of emails.

We are on a deletion *rampage*!!

Nothing is sacred. Any remotely questionable tubes, sensors, manifolds, etc. will be deleted tonight. Please go ultra-hardcore on deletion and simpli cation.

Takeaways for software engineers

There’s a clear pattern throughout the book: When Elon sees a bottleneck, he removes rather than adds. Instead of adding more employees and materials, he fires people and removes parts.

This instinct is worth appreciating for us engineers, who have gotten used to adding with impunity. When a senior engineer imports a library, adds a framework, and integrates with a third-party, no one blinks an eye. Code is cheap, and it’s only getting cheaper.

But just because adding lines of code is easier than adding rocket ship parts doesn’t mean it’s free. There is a cost to every abstraction. For us, that cost is time. It takes time to dig through convoluted business logic, to wait for long CI runs to finish, and to tiptoe around a brittle part of the codebase.

When things become bloated, slow, or pricey, we should resist the urge to optimize and instead question whether we need the thing in the first place.

Removing abstractions, imports, and requirements will poke some egos, but so will becoming a team that’s incapable of shipping anything good.

More

Elon Musk, Walter Isaacson (amazon.com)

How Elon Works, Founders Podcast (youtube.com)

What people don’t know is that his books are written on a DOS machine running WordStar 4.0, a VBA word processor that dominated in the 80s thanks to its portability and WYSIWYG interface. Slow revisions, kludgey code, and corporate drama slowed progress long enough for Word to dethrone WordStar as the corporate standard. After a failed merger and relaunch, WordStar was officially abandoned in 1992.

So why does he still use it thirty years later?

Because it works.

It does everything I want a word processing program to do, and it doesn’t do anything else. I don’t want any help. I hate some of these modern systems where you type a lowercase letter and it becomes a capital. I don’t want a capital. If I wanted a capital, I would’ve typed a capital. I know how to work the shift key!
— GRRM.

Lessons

George’s preference for less is refreshing in an era obsessed with more.

Young carpenters might lose some sleep researching the benefits of Milwaukee vs DeWalt tools, while mechanics compare Gearwrench to Ryobi. But then they wake up, buy a set, and use those tools until they die.

We digital craftsmen, on the other hand, will try five IDEs and LLMs in a week. It’s easy to rationalize this schizophrenic adoption as an attempt to keep our skills sharp. But that mindset reveals a mistaken understanding of our role in the world.

It’s not about the tools, it’s about what you create with them.

Carpenters translate ideas into houses. They can do that with a nail and hammer, or a cordless track saw with an attached vacuum bag.

Writers translate ideas into words. They can do that with a DOS machine running WordStar 4.0, a phone with the Notes app, or pen and paper.

Coders translate ideas into code. They can do that with an agent window, vi and a keyboard, or ChatGPT and their voice.

Examples

Pieter Levels continued to write plain HTML, CSS, SQLite, and PHP in Sublime Text long after VS Code became the standard IDE. His friends made fun of him for his outdated tools. His followers pointed out the superiority of TypeScript and Docker. Yet he focused on his users and products long enough to generate $3M ARR with an ancient stack.

René Rebe codes in a single terminal with vi and basic motions. That elementary setup helped him focus on the code long enough to write a USB driver from scratch in three hours.

$LOUD_ENGINEER_ON_X uses NeoVim, configured through LazyVim, with a fully customized setup that auto-provisions via dotfiles managed by Chezmoi, stored in a private self-hosted Forgejo Git server running on a home Kubernetes cluster. His workflow is so “optimal” that he sells a course to young engineers on how to replicate it. (He hasn’t shipped anything helpful for normal people in years.)

Which type of engineer do you want to be?

If you want to be like the last engineer, then keep jumping on whatever new VC-backed tool shows up on your feed this week.

If you want to be like the first two, your path is much simpler:

1. Use simple tools that do a few things well. Really, just pick any IDE, calendar, task app, and OS and don’t overthink it.

2. Keep using that tool until it breaks or your craft suffers.

“BuT tEcH mOvEs So FaSt. I nEeD to sTaY rElEvAnT!”

Yes, occasionally a truly innovative tool appears that is so superior it’d be questionable to resist it (think: computer vs. typewriter). Between those paradigm shifts, a constant stream of “new” tools appears, each overhyping its benefits, downplaying its costs, and distracting you from your craft.

You’ll know when a paradigm shift is happening, and you’ll have plenty of time to transition. You’re not jeopardizing your career by moving to the right of the adoption curve.

The real risk is maintaining a greater allegiance to novelty than to your craft.

When GRRM’s old computer finally died, he had his assistant build a state-of-the-art computer for him….so that he could run DOS and WordStar 4.0.

WordStar shipped seven versions after 4.0. Instead of using this computer upgrade as an opportunity to try the latest version from 1994, he stuck with the 1987 program that he already mastered.

One of the most successful living writers demands simplicity. He wants his tool to do a few things well and let him handle the rest.

Let’s make George proud.

Close social media.

Open whatever IDE you’re already used to.

And get to work.

More

Interview with James Corey (lightspeedmagazine) | GRRM’s assistant

The best engineer doesn’t have the best setup (fullstack.zip)

A Tour of Word Star 4.0 (youtube)

WordStar (wikipedia)

The simplest $30k playbook

It was called “Bootstrapping Your E-Commerce Empire Side Hustle With No Money,” led by Jonathan Spaulding. He walked us through how to find a niche, set up a no-frills store, and ship, all without spending a nickel.

There were some pragmatic cost-saving tactics, like 3D-printing custom products and offering only small items to minimize shipping. What I enjoyed the most, however, was how simple he kept his operations.

His go-to-market involved walking around his house with a cardboard box, adding things that he didn’t need, and listing them on eBay that day.

For market research, he asks coworkers what small items they recently bought and makes them for less with his 3D printer.

To fulfill orders, he gets a bundle of USPS boxes and puts them in his home mailbox.

For support, he responds through the eBay mobile app during downtime.

However indignant the e-com gurus may feel about these elementary strategies, they work for him. This side hustle nets Jonathan $30k of yearly profit.

That headline isn’t going to make him HackerNews famous. But so what? That’s a year’s tuition. Runway for a startup idea. $66k in 10 years at 8% growth. An epic vacation. Enough Claude credits to create a custom C compiler. It also fits his lifestyle by allowing him to work from his phone and stay home with his family.

Why simplicity sells

Our feeds are so good at convincing us that we need more information before we’re ready to do things in the world. We need to watch the next video, finish the course, or get a few more years of experience before starting something new.

People like Jonathan prove them wrong.

You can just do things.

Yes, right now.

Really, it’s that simple.

Not easy, but simple.

This is especially true for engineers, who have more agency than ever. We don’t need to learn a new framework, refactor the API (again), or wait around for a job offer.

We just need to walk around our metaphorical house with our metaphorical cardboard box and sell things to people who want to buy.

There are two ways to be loved as a software engineer:

1. Solve the most complex problems so others don’t have to

2. Make complex problems simple enough for regular engineers to solve themselves

The first path is for master craftsmen maxxing out their engineering skill. The second is for those who enjoy teaching as much as coding.

Let’s look at a few examples of the second type of person.

Jeff Delaney & YouTube

Jeff Delaney is the man behind the Fireship, a YouTube channel for devs with over 4M subscribers. Although the videos are well-produced and sprinkled with the dry jokes devs enjoy, there is a deeper reason why developers love the channel: It makes them feel smart, quickly.

After watching a regular technical video, I often feel even more overwhelmed than when I started the video 20 minutes earlier. Jeff’s videos, on the other hand, are so straightforward and down-to-earth that I actually understand the core idea. His flagship 100 Seconds of Code series, for example, makes CS topics like Big-O notation and GraphQL feel refreshingly approachable.

Jeff probably isn’t the best engineer, nor does he try to be. Instead, he combines his “good enough” engineering with a world-class ability to cut out complexity.

I always ask myself, ‘How much can I cut out of this video and still teach the concept effectively?’

I try to distill the concepts into a very potent shot of whiskey.

— Jeff Delaney, creator of Fireship

Rich Hickey & Clojure

Rich is a great engineer — he created the Clojure language after all. Instead of trying to become an even better engineer for its own sake, though, he spends a lot of time teaching others. During the first few years after Clojure’s launch, Rich went on a conference tour, spreading the good word about the language. In his classic Simple Made Easy talk, he defined the problem of complexity so well that a generation of programmers started valuing simplicity.

Rob & CS

I felt imposter syndrome during my first two years as a junior engineer. During that time, our lead engineer, Rob, gave lunch talks about engineering. He’d whiteboard draw simple diagrams about the basics of computing: transistors, data structures, the difference between OOP and functional languages. Every time I’d walk back to my desk with a lil’ pep in my step, because for just a few minutes it felt like I understood what was going on and that I belonged. That’s why everyone loved Rob.

If you’re the type of engineer who wants to balance both hard and soft skills, consider teaching your peers. We reward those who make things simple: they make the world less scary, which makes us feel more capable. Whether in a PR review or a big conference presentation, the most popular message will be the one that shows the audience that they have more agency than they realized.

You’re a non-technical founder selecting between two Founding Engineer candidates. This hire will be responsible for creating your MVP, scaling the engineering team, and solving the hardest technical problems along the way. They’ll set the bar for engineering excellence, and everyone will try to mimic how this person works and thinks.

You shadow both candidates for a day while they work on a passion project. Although you won’t understand the details, you’re hoping to observe their workflow and the tools they use, since they’ll likely set something similar up for you.

Engineer 1

This engineer runs Fedora on a Framework laptop, with Sway as his tiling window manager, but tmux layered on top as a second window manager inside the terminal. His editor is Neovim, configured through LazyVim, with a fully customized setup that auto-provisions via dotfiles managed by Chezmoi, stored in a private self-hosted Forgejo Git server running on a home Kubernetes cluster. All development happens inside dev containers via DevPod, so there’s always a container runtime running.

Every tool is picked to keep workflows in the terminal and on his hardware. He uses w3m as a terminal browser to avoid trackers. He uses Aerc for email, an open-source terminal client. He also operates his own databases. Claude is kept in a container so Anthropic can’t see his notes or dotfiles.

His setup lets him sit down at any machine and be fully operational in minutes. As the owner of a DevOps education business, his workflow also demonstrates to prospects what’s possible if they commit to privacy and the terminal.

Engineer 2

This engineer works in a single, full-screened terminal. He uses vi with only basic motions and no syntax highlighting. His “workflow” involves changing the default font to Comic Sans.

He’s decided that he needs a USB driver for his kernel. He starts by googling “apple xserve frontpanel usb driver github,” scans the repo, and concludes, “Ja, this is super simple” (German accent).

He starts by copying and pasting Linux kernel code, mumbling in C, and sipping espresso. Thirty minutes in, he leans over to boot up the physical server next to his desk, which hums loudly for the rest of the session. Then he starts writing his own code. After 2.5 hours, he calmly reports back, “So, ja. It works. The future is now.”

My pick

I’m going with Engineer 2. He gets it. It’s not about fancy tools or impressive workflows that require a paid course to understand. It’s about shipping software. If this guy is comfortable raw dogging C in vi without any AI, I’m comfortable trusting him with the codebase.

The people on YouTube agree:

“I trust this man with my life.”

“This guy is not getting replaced by AI.”

“Writing code without syntax highlighting is demon level.”

“This is engineering.”

“Every time he says ‘Yes, of course,’ I just nod along. ‘Yes, of course.’”

To be fair, I straw-maned Engineer 1 big time. He’s a DevOps pro with serious credentials. Comparing his workstation demo to a C programmer’s live stream is apples to oranges. But by Engineer 1’s own admission, his system is elaborate. After all, it took him 70 minutes to explain how it all works. There’s a coherent philosophy of protecting privacy and flow state underneath it all…but it’s still a lot.

Solving meaningful problems for customers will bring plenty of necessary complexity. I want to work with people who are content to keep everything else simple.

Who you picking? Reply to LMK.

More

Watch Linux kernel developer write a USB driver from scratch in just 3h for Apple Xserve front-panel (youtube)

My Entire Neovim + Tmux + AI Workflow (2026 Update) (youtube)

Listen to any great engineer long enough, and they’ll eventually mention the idea of “simplicity.”
They’ll say things like, “Keep it simple.”
They’ll block your PR because it adds a little dependency.
They’ll yap about how we sent a man to the moon with 64kb of code.

Why do they care so much about this idea?

They care because they’ve felt the pain of complexity.
One day, they decided they were fed up with all the BS.
They just want to build things that work and tune out the noise.

I considered expanding this post by curating data and sharing my anecdotes from building web apps over the last decade. But sometimes it’s better to stay quiet and let the real pros speak. So here’s what the truly great programmers think about simplicity.

The web is basically a very simple idea. Very simple.
— Sir Tim Berners-Lee, inventor of the internet

Keep it simple. When in doubt during design, choose the simplest solution.
— Brian Carpenter, early CERN and IBM, CS PhD

We don’t need gigabytes of code. If we can write small programs, we have a chance at understanding them.
— Joe Armstrong, Erlang inventor, CS PhD

C is a very simple language. It’s one of the reasons I enjoy it.
— Linus Torvalds, created Linux kernel & git

An idiot admires complexity, a genius admires simplicity
— Terry Davis, created TempleOS

More

Architectural Principles of the Internet (1958)

TempleOS | Down the Rabbit Hole (2019)

Making reliable distributed systems in the presence of software errors (2003)

Devs love speed, which is why we’re obsessed with benchmarks and Rust. But speed is not just about acceleration; it’s also about mass.

So what’s the heaviest part of our codebase?

Dependencies.

In other words, other people’s code that we rely on to make our code work. GitLab’s 2024 survey of 5,000 DevSecOps pros claims that over 25% of the code devs work on is from open-source. Those guys are either working on CLIs or lying. My calendar app’s codebase has 10 million lines. I only wrote 116,351 lines — just 1.2% of the product. The other 98.8%? Other people’s code.

These numbers are insane, but not anomalous. Web apps are especially bloated these days. This is why we should spend less time looking at the speedometer and more time looking at the scale.

Dependencies are weight

Every dependency adds weight. The more you carry, the more every step costs, and the slower you move. Using minimal dependencies is like traveling internationally with a backpack. You might be going to the same destination as your buddy who brought two suitcases, but you’ll have vastly different experiences.

Let’s start with the heavy traveler, the one who has a “checked” codebase. What is his life like?

Checked codebases

The repo started small and grew quickly. Each dependency felt like a good idea at the time. Of course, you need an HTTP client (even though it comes with a bulky logger). Of course you need that component library with strong opinions about theming (even though you’ll only use five of its components). Of course you should start with $FRAMEWORK_THAT_EVERYONE_IS_USING.

He started with a lot, added more along the way, and never stopped. The PMs were eager to add features, and no one slowed the sprawl.

Now the builds take forever, no one understands why half the packages are there, but they’re all too afraid to do anything about it. So instead of simplifying globally, everyone optimizes locally. He’s greasing the wheels of a 50-lb suitcase instead of taking out the 10-lb boots. He optimizes, but he’s still big and slow.

Traveling with a checked codebase is exhausting. It takes a lot of planning to do a little movement. Each implementation step is also laborious, as he has to check so many surfaces for downstream bugs. He’s forced to bring a month’s worth of clothes for a short weekend getaway hotfix. By the end of the trip, he’s more exhausted than when he left.

Carry-on codebases

The minimalist is initially scoffed at as a simpleton, an irresponsible kid unprepared for the prod life. He sticks to what works, fights complexity vigorously, and has enough clout to be trusted. As a result, his dependency tree is minimal. Everyone knows what every package does. Sure, some teammates grumble about how a modern tool would be faster. But, on the whole, things are working. The builds are fast enough, newcomers can understand the code in days instead of weeks, and things get shipped.

Traveling with a carry-on codebase is chill. A POC gets demo’d. Builds pass. Merged. Done. There are fewer hotfixes, patches, docs, and meetings. The trips to Featureville actually feel energizing, and the team looks forward to the next experiment.

When the requirements change, it’s easy to recalibrate and move forward. Coding here feels like freedom. Not from bugs or feature creep, but from the weight that used to make overcoming those challenges so draining.

The reality of travel

Traveling light has become even more important recently, because travel itself has remained complex. While the vibe coding narrative promises to collapse the SDL into a one-shot prompt, we know that there are still many steps to shipping real products.

Pull → install deps → run tests → wait for CI → deploy → retest → get design feedback → get PR feedback → repeat → deploy to prod → fix bugs → write docs.

The journey is long. At every stage, a heavier dependency graph adds time. Moreover, the dependencies themselves require attention.

We still need to upgrade them. This is like replacing a backpack with a hole at the bottom with a new one.

We’ll need to swap heavy deps for alternatives. This is like swapping a heavy cotton sweater with a light dry-fit one.

We’ll still need to remove third-party libs with built-ins. This is like leaving your glass tupperware at home when you learn the hotel provides some.

Like it or not, dealing with dependencies is still part of our lives, which means it’s still helpful to learn how to pack lightly.

A small dependency graph will not make your product automatically great. But it will make the product easier to build, easier to understand, easier to debug, and easier to adapt when reality changes.

A light traveler is a fast traveler.

Story from the road

I fly with this backpack instead of those rolly carry-ons that real adults use. It fits overhead and underneath. It lets me take the stairs at the hotel when the elevator line is long. It helps me blend in at a coffee shop. When the ski lifts shut down, it allowed me to haul my gear to the top for one last run. My fancy shoes don’t fit, which means I lose some fashion points. But I gain so much speed and lightness. Once you feel that freedom, it’s hard to go back.

More

On the Relationship between Software Complexity and Maintenance Costs (2014).

Software complexity and maintenance costs (1993) | Analyzes Cobol maintenance projects in a large commercial bank.

Software Dependencies, Work Dependencies, and Their Impact on Failures (2008) | How dependency measures relate to defects.

DevSecOps Report: Application security in the digital age (2024) | GitLab’s survey

Do you think traveling light is always better, or are there exceptions?

Anthropic accidentally leaked its Claude Code source code. Although the archived repo is easy to find, deeper analysis is not. The popular reactions are surface-level, while the technical reports read like a mindless doctor’s note written by Claude itself.

“It’s just a Bash wrapper.”

“Wait, that’s it? This is so hack-y.”

“There’s an undercover pet!”

“It’s a CLI with multiple layers of abstractions.”

I blame the lack of substance on the legal grey zone we’re in: linking the repo or sharing specifics would anger the copyright police. At least that’s what my $20/month lawyer/SWE tells me.

After reading through the repo, I nevertheless have to share this amazing fact:

Claude Code has its own skill for simplifying code (simplify.ts).

That’s not as flashy as those other headlines, but it’s still important.

Coding with Gemini, Grok, or Cursor can feel like a game of whack-a-mole. They’ll add the feature I asked for and pass the tests…but they also litter the code with unnecessary comments, dependencies, and abstractions. Claude’s submissions feel more elegant — as if it spent some extra time refining away the rough edges first.

Well, that’s exactly what they’re doing, and it’s all documented in simplify.ts. Instead of pointing you to that file, though, here is my own “simplify-code” SKILL.md, which is spiritually equivalent to Claude’s.

// Excerpt from my SKILL.md for simplifying code 
---
name: simplify-code
description: Simplifies code by minimizing complexity, eliminating duplication, and prioritizing legibility for contributors. Use when implementing features, fixing bugs, refactoring, or when the user asks to simplify, clean up, make DRY, reduce complexity, or improve maintainability.
---

# Simplify Code

Favor minimal, legible implementations. Fewer lines and clearer structure make code easier to understand and maintain.
...

You and I aren’t the only ones who recognize the importance of this simple engineering philosophy. Claude is used by a whopping 75% of the smallest companies (ie, the ones that are free to use whatever they want) and is already the overall market leader, despite being released only 11 months ago.

It seems we agree:

When it comes to code,

Less > more.

Simple > complex.

We’ve been willing to pay for simplicity with our wallets and our time all along. Now we finally know why.

A few bonus observations:

Mixed file names. You’ll see kebab-case, camel case, Pascal case, and lowercase all in the root directory alone.

A few massive files. The entry point is 4600+ lines long, for example.

Types, kinda. It’s 100% TypeScript, but they don’t rely on types very heavily. There are lots of optional properties, and types aren’t widely shared.

My instinct tells me these are less engineering conventions they settled on and more symptoms of an internal POC that rapidly evolved into a mainstream product.

What was your first reaction to the leaked code?

More

AI Tooling for Software Engineers in 2026 | Pragmatic Engineer | usage stats

The format

They’re system design case studies, featuring popular apps.

Part I: Architecture. The stack, protocols, and main players that the featured app relies on.

Part II: Feature Deep-Dive. How everything comes together in the app’s most interesting feature.

Part III: Context. Where these ideas come from, and what they mean for regular devs like you and me.

The backstory

I broke down a few apps on my personal channel in 2024, and engineers seemed to like them. I got comments like this one:

Big fan of this! Definitely wanna see more video breakdown apps! :)

Of course, I ignored that feedback and instead uploaded 100 unrelated videos. When those flopped, I finally thought, “Maybe I should just give my audience what they want.”

I created “Fullstack Engineering” to be the strictly business vehicle for this educational content.

You’re probably familiar with the Fireship channel (4.1M subs as of today). If those videos are espresso, Fullstack Engineering is pour-over. They’re longer in format, but richer in context.

My goal with this channel is to give you something to watch during your WFH Friday lunch that educates and inspires. If you go back to the IDE excited, that’s a win.

Here are the first four.

Reply to LMK what app you wanna see next.

Before coding agents, a PR went through more checkpoints. You’d suggest a design pattern in an RFC or an issue comment. Your teammate would advocate for a different pattern. You’d go back and forth on the idea before committing to a tradeoff. After a day of POCing the first approach, you realized there’s an easier way, so you’d update the RFC or have another conversation. Then you’d submit a draft PR. More feedback from teammates who skimmed it with fresh eyes. They asked for more tests and told you how you could ditch that new library for a built-in. More refinement. By the time you make the PR public and everyone approves, your code is 2X simpler than the first version.

Things are different now.

Instead of clarifying our thinking in an RFC or issue, we prompt and get a PR. An agent reviews the code and auto-fixes major bugs. Another writes the PR description.

The end.

Sure, there’s a little back-and-forth between you and the agent, but only enough to make the tests green. Even reading the code is starting to feel like an inconvenience.

Our velocity has skyrocketed, but so has the complexity of our codebases. Part of that complexity is inevitable. Yet more is a byproduct of accepting the agent’s defaults without scrutiny. The agent doesn’t want to waste tokens refining its already green solution, and we don’t want to waste time nitpicking. So extra libraries get added, nested if statements sneak in, and kludgey abstractions get overlooked.

Teams won’t care until their complexity debt starts to slow their velocity or introduce more bugs. At that point, it’ll be tempting to start from scratch with another prompt. But there might be a more pragmatic way out.

Solution: Simpler tools

Replace complex tools with simple tools. Keep the requirements and tests, but replace one heavy tool with a lighter one.

To make this practical, here are 40 swaps to consider, organized by domain.

Simpler Auth

1. Auth0 / Clerk → express-session + bcrypt + a users table. Don’t lock yourself into a vendor for a personal project.

2. OAuth → API keys. Trade the redirect dance with an API key in the header. Consider OAuth when your users can only auth with accounts they don’t control (rare).

3. Username / password → Magic link. No hashing, strength enforcement, or reset forms.

4. Per-route auth middleware → Single auth gateway. A single middleware that authenticates every request by default is easier than sprinkling requireAuth() on each route.

5. SSO → anything else. Add it when a paying customer asks for it.

6. JWT everywhere → Server sessions. Easier to revoke and doesn’t require rotation.

7. RBAC service → role column on a users table. Unless you’re an enterprise, WHERE role = ‘admin’ in your middleware is fine. Consider a better abstraction when you have 3+ distinct roles.

8. Third-party MFA service → TOTP + otplib. It’s an open spec, and otplib generates and validates codes in ~10 lines. There’s a maintenance burden, but it might be cheaper than locking yourself in too early.

9. Refresh token rotation → long-liven tokens + revocation list. If calls are inside your infra, you might not need to worry about refreshing yet.

10. passport.js → manual middleware. Passport adds 20+ files and a plugin ecosystem just to check a password and set a session. If you're not juggling multiple strategies, a verifyPassword function and a session check is 15 lines and nothing to maintain.

Simpler Storage

1. Redis queues → Postgres LISTEN/NOTIFY. Postgres has a built-in pub/sub mechanism. NOTIFY sends a message on a named channel; LISTEN opens a persistent connection that receives those messages in real time. You can store messages in a jobs table and tell workers to pick it up on NOTIFY. Or use a lib like pg-boss if you don’t wanna coordinate locks and retries yourself. This’ll last you until you get to a few hundred jobs / second.

2. Trendy DB → SQLite. A single file, zero config, and it handles tens of thousands of reads per second. Upgrade to Postgres when you need concurrent writes from multiple processes.

3. Time-series DB (InfluxDB) → Postgres timestamptz + indexes. For application metrics, user activity logs, or event streams at normal SaaS scale, a timestamptz column with a partial index covers the query patterns without running a second database.

4. Separate search service (Algolia, Elasticsearch) → Postgres full-text search.

5. ORMs (Prisma, Sequelize) → SQL + a query builder. OMRs abstract SQL until they don’t. If you’ll be writing raw queries anyway, a lightweight query builder can get the job done without hiding the DB from you.

6. MongoDB → Postgres JSONB. If you’re already on Postgres and don’t want to stress about schema design upfront, you can still use JSON for now without having to bring in another DB. IF your data is document-shaped AND you’re confident you’ll hit scale quickly, then ignore.

7. Redis cache → Postgres Materialized View. Pre-compute the result and store it in a table. Great way to preserve speed for complex queries and avoid a cache. Must be OK with some data staleness.

   1. Use-case: daily sales dashboard.

   2. See: Choosing Between Postgres Materialized Views and Redis Application Caching (leapcell.io).

8. S3 → local filesystem. If you’re on a single VPS, you don’t need to manage another round-trip, IAM policy, and subscription. If GitLab could run its git commands on a shared NFS filesystem, you’re probably fine to store some files locally.

   1. GitLab Architecture (fullstack.zip)

9. Multiple databases → one Postgres schema per domain. Running separate databases for separate domains means cross-domain queries require application-level joins, separate connection pools, and separate migration pipelines. A Postgres schema per domain (users.accounts, billing.invoices) gives you logical separation with a single connection string.

10. Key-value store → single-table Postgres with a type column. Redis shines for ephemeral data — rate limiting, leaderboards, session tokens. For “store this thing and look it up by key” use cases in a persistent store, a Postgres table with (type, key, value, expires_at) is queryable, inspectable, and doesn’t require operating a second data store.

Simpler Infra

1. Fly.io / Railway → 1 VPS. SSH in, run your app with pm2, put Caddy in front. You own the machine, you understand the machine, and you're not beholden to a platform's pricing or uptime decisions.

2. k8s → systemd + reverse proxy. Automatic startup, dependency checking, and failure recovery.

   • How to run pods as systemd services with Podman | redhat.com

   • Replacing Kubernetes with Systemd | news.ycombinator.com.

3. Docker → Bash + pm2. Replace the whale with pm2 (for Node apps), and you’ll free up a ton of RAM, storage, dependencies, and time.

4. Terraform → Bash. For a personal project or small team with one VPS, a shell script that runs apt install, creates users, and drops config files is fine. Terraform is for provisioning lots of stuff across teams.

5. Secrets manager → Env files + chmod 600.

6. Staging environment → Feature flags. It doesn’t get more real than prod, not matter how much you dress up your staging env. How many of us are acting like we have an SLA with five 9s when we don’t even have 5 customers?

7. GitHub Actions → git hooks + deploy script. A post-receive git hook on your server that pulls and restarts is a 10-line bash script that deploys on git push.

8. Autoscaling → Vertical scaling. What happened first, you spent a week configuring autoscaling or 10 minutes doubling your VPS size?

9. Nginx + Certbot → Caddy. Auto-TLS, one config file. HTTPS in Nginx usually requires installing Certbot separately, configuring it, setting up a cron job for renewal, and debugging when it gets out of sync with your Nginx config. With Caddy, you just put your domain name in the Caddyfile — Caddy obtains and renews the certificate automatically, handling HTTP-to-HTTPS redirects, OCSP stapling, and multi-domain SANs with no additional configuration. The makers of Certbot even agree that this is a better solution for everyone.

   1. Should Caddy and Traefik Replace Certbot? | eff.org

10. Multi-region deployment → backups + a restore script. Small SaaS apps don’t need 99.99% uptime. A daily Postgres dump to S3 and a restore script that gets you back in 10 minutes is more than enough.

Simpler Libraries

1. ESLint + Prettier + Husky → biome. Biome replaces that entire stack with one tool — it’s 10–25x faster than ESLint and ships as a single binary instead of 127+ npm packages. Caveat: it doesn’t cover every ESLint plugin (e.g. eslint-plugin-jsx-a11y), so teams with heavy custom rule sets may not get a clean 1:1 swap.

2. Jest/Vitest + mocking libs → node:assert. Ideal for simple backend unit tests in pure Node.js environments where minimal dependencies are a priority (nodejs.org)

3. .env + dotenv library → Native --env-file flag. Since Node 20, you can load your env file natively.

4. node-config → env files. node-config is a popular library for managing environment-specific config (development.json, production.json, etc.). For most projects, a plain .env file per environment covers the same ground with zero deps.

5. uuid → crypto.randomUUID(). Why does the uuid package get 91 million downloads per week when crypto is native and 4x faster? I think it’s just a better name.

6. nodemon → node --watch. Unless you need granular ignore patterns, this simple flag is good for anyone on Node 18+.

7. axios → fetch. Works identically in browsers. Can upgrade if you need interceptors or fancy errors.

8. moment.js → Date or Temporal. Ditch the bulky libs for a lib that the JS ecosystem spent a decade pushing through. (You’ll need a polyfill until Safari gets its act together).

9. lodash → array methods. Lodash predates modern JS. _.map, _.filter, _.groupBy, _.flatten are all native today. Another example of muscle-memory keeping a heavy lib around.

10. express → v0: node:http, v1: hono. Node for simple internal services, hono if you need a router on the edge. 0 deps.

Conclusion

It’s easy for me to tell you to “just replace this whole part of your stack.” It’s hard to do this in a company, especially when you need to convince teammates along the way. After spending the last three months doing nothing but this type of cleanup in Compass Calendar, however, I can tell you it’s worth it. You’ll have fewer dependencies, fewer lines of code, and fewer headaches. You’ll start to understand the codebase again. Best of all, you’ll develop a deep conviction that’ll help you forever separate signal from noise:

It doesn’t have to be this complicated.

More

Simple is smart (fullstack.zip)

A Plea for Simplicity (schneier.com)

Choose Boring Technology (mcfunley.com)

77 Things to avoid in 2026: Developer edition (fullstack.zip)

The Law of Leaky Abstractions (joelonsoftware.com)

“I’m {FirstName}, and I’m a complex-oholic.”

Say it.

It’s OK, we’ve all been there — prioritizing our ego ahead of our users.

Deep down, we know we were overdoing it all along.

Not ready yet?

All good, it takes time. For now, skim this list and let yourself yearn for the days of yore, when life seemed so straightforward.

Simplify your system

1. Microservices → modular monolith. If you’re a small team, a monolith with clean module boundaries gives you the same separation of concerns without the network tax (Modular Monoliths).

2. Abstract base classes → plain functions + composition. If your class has one subclass, it probably shouldn’t be a class (The Composition Over Inheritance Principle).

3. Feature flag service → Env var + if statement. LaunchDarkly and friends are powerful when you have millions of users. You don’t, so if (process.env.IS_PREMIUM_USER === 'true') is fine (How To Build Feature Flags Like A Senior Dev In 20 Minutes).

4. Semantic versioning automation → git tag + a changelog entry. (Keep a Changelog)

5. GraphQL → REST + JSON. Do multiple clients need different shapes of the same data? If not, a REST API will be easier to build, cache, and debug (You probably don’t need GraphQL).

6. Event-driven everything → direct function calls. If two things always happen together, just call one from the other and save yourself the tracing scavenger hunt (Method Calls vs Event-Driven Architecture in a Modular Monolith API?).

7. API gateway → reverse proxy routes. Gateways come with rate limits, middleware, routing, and policies. Your existing proxy can do the essentials with a few config lines (Is an API Gateway a Reverse Proxy?).

8. Pub/sub → EventEmitter. Node’s built-in can handle internal pub/sub within a single process without a broker or network hop (Redis pub/ sub vs node event emitter).

9. Polyrepo → monorepo. This’ll cut down your CI configs, versions, PRs and repos. Go further by skipping Nx/Turborepo in favor of a simple Makefile (Understanding Monorepos).

10. Multi-tenancy → multiple single-tenants. Every unicorn engineer will proudly talk about their journey from single-tenant to multi-tenant. Every tech lead of a graveyard startup will reluctantly admit how they spent too much time optimizing instead of iterating (Choosing the right SaaS architecture: Multi-Tenant vs. Single-Tenant).

Did you reject these indignantly? Perhaps you bristled at the thought of a vet like you doing something so elementary.

Well, hopefully you’ll scale so quickly that you outgrow these script-kiddie tactics in a few months. None of the fancy things is wrong; they’re just wrong now, at this scale.

Just ship and help your users, then let the real problems tell you what to add back.

Complexity has its place.

Just not yet.

This article follows the steps that GitLab takes for you after you push a commit.

TL;DR

• Pipelines are declared in gitlab-ci.yml, then compiled into an internal job graph and persisted in Postgres.

• Jobs move through states like created → pending → running → success

• Merged results pipelines validate your changes against the current main.

• Merge trains validate your changes against the eventual main.

• This is all based on dependency graphs, producer-consumer queues, and optimistic validation.

YAML → Postgres

Step 1 — Git event received. GitLab Shell (SSH) or Workhorse (a Go reverse proxy) receives the pack data, stores it via Gitaly (gRPC for git data), and triggers a post-receive hook.

Step 2 — Ci::CreatePipelineService runs. Rails fetches the YML via a Gitaly GetFile RPC. The YAML goes through several passes:

• include: directives are resolved — local files, project templates, even remote URLs. This is recursive and can get deep.

• Variables like $CI_COMMIT_SHA AND $CI_PROJECT_NAME are interpolated.

• rules:, only:, and except: are evaluated against the pipeline context (branch, commit message, changed files, schedule) to determine which jobs actually apply.

• needs: relationships are resolved into the DAG (more on that later)

Step 3 — Everything is written to PostgreSQL in one transaction:

After GitLab parses .gitlab-ci.yml, it stops treating the file as the source of truth. Instead, it compiles that config into relational rows. One row represents the pipeline, a few rows for its stages, one row per job represents runnable work, and extra rows for dependencies.

In other words, GitLab turns your YAML into a small execution graph stored in PostgreSQL. That graph is what the scheduler and runners operate on later.

Here’s pseudo data to help visualize what things are like in Postgres:

PIPELINE
  5001  main@abc123  status=created

STAGES
  6001  build   pipeline=5001
  6002  test    pipeline=5001
  6003  deploy  pipeline=5001

JOBS
  7001  compile  stage=6001  status=created
  7002  rspec    stage=6002  status=created
  7003  jest     stage=6002  status=created
  7004  ship     stage=6003  status=created

NEEDS
  7002 needs compile
  7003 needs compile
  7004 needs rspec
  7004 needs jest

Here is what a simplified ci_builds row looks like at creation:

SELECT id, pipeline_id, name, stage, status, runner_id, yaml_variables
FROM ci_builds
WHERE pipeline_id = 99801;

-- id      | pipeline_id | name           | stage | status  | runner_id
-- --------|-------------|----------------|-------|---------|----------
-- 4410023 | 99801       | lint           | test  | pending | NULL
-- 4410024 | 99801       | unit-tests     | test  | pending | NULL
-- 4410025 | 99801       | docker-build   | build | created | NULL  ← waiting on stage order

Note that a job in created is still invisible to runners at this point; it exists in the DB but is not yet in the queue.

Step 4 — Ci::InitialPipelineProcessWorker is enqueued.

This Sidekiq job calls ProcessPipelineService, which evaluates the DAG, finds all jobs whose dependencies are satisfied, and transitions them from created → pending.

A job in pending is now on the ticket rail, so runners can see it.

Postgres → Runner

Runners poll POST /api/jobs/request every ~3 seconds. On the Rails side, this hits Ci::RegisterJobService, which:

1. Queries ci_builds for pending jobs matching this runner’s tags and project access

2. Attempts to atomically transition the selected job from pending to running using optimistic locking

3. If the update succeeds, it returns the full job payload — env variables, Docker image, scripts, artifact definitions, etc.

4. If another process won the race, it returns 204 No Content, prompting the runner to retry next cycle.

Runner → Green Check

Workflow 1: Merge Requests

Merge request pipelines answer the baseline CI question: “Did my change work?” When you push, GitLab ensures you can build, test, lint, and deploy that change in isolation by testing a temporary merge commit that combines source and main. Nothin’ fancy here.

Workflow 2: Merge Trains

“Will Cory’s code break mine if it gets in first, though?”

Now you’re asking the right question.

When you’re working on your side project, you don’t need to worry about this. You’re the only one merging into main, so if it passes on your branch, it’ll pass in main. On a collaborative project with many changes landing at once, however, this isn’t guaranteed: two branches that each pass CI individually might conflict when combined.

Merge trains address this by testing the result of merging your branch into the target, rather than your branch in isolation.

A merge train is a queue of Merge Requests (MRs), where each is compared to earlier ones to ensure they all work together.

For example:

Four merge requests (M1-M4) are added to a merge train in order, which creates four merged results pipelines that run in parallel:

1. The first pipeline runs on the changes from M1 combined with main.

2. The second pipeline runs on the changes from M1, M2, and main.

3. The third pipeline runs on the changes from M1, M2, M3, and main.

4. The fourth pipeline runs on the changes from M1, M2, M3, M4, and main.

If M1 completes successfully, it merges into main, while the other pipelines continue to run. Any new MRs added to the train include the M1 changes from the target branch, along with the other changes from the merge train.

That’s the happy path.

Here’s where it gets fun.

If the pipeline for M3 fails:

• M1 and M2 continue to run.

• M3 fails and is removed from the train.

• A new pipeline starts for the changes from M1, M2, M4 and main (without M3).

The payoff: M4’s dev got the changes from M1 and M2 without manually restarting their pipeline. Without merge trains, M4’s dev would either be waiting around for M1 and M2 or removing M3’s crappy code.

While this “speculative thing on top of another speculative thing” got the banks in trouble in 2008, GitLab thankfully hedges risk a little more responsibly. Think: automatic rebasing, hiding speculative future states, fast forwarding, juggling Git and MR refs, capping parallel pipelines (to 20). It required years of behind-the-scenes work.

This train flow puts more pressure on the dev team to maintain a reliable test suite, as a single flaky test can make the difference between a smooth and bumpy ride.

The payoff is worth it, though: developers can be confident that their changes will work against both the present and the near future of a shared branch.

(For the record, you don’t get all this for free; Merge trains are part of GitLab’s Premium tier.)

Historical Context

The ideas that support GitLab’s CI originally popped up in the 50s and 60s. Understanding their history will make all of this feel less magical (in a good way!).

Pipes

The most obvious precedent is the pipe. In 1964, Douglas McIlroy wrote an internal Bell Labs memo proposing that programs be linked “like a garden hose — screw in another segment when it becomes necessary to massage data in another way.” He envisioned software as interchangeable components connected through data streams. The idea sat dormant for nearly a decade before Ken Thompson implemented pipes as a core Unix feature in 1973.

ps aux | grep chrome  # is chrome running?
ls -1 | wc -l         # how many files in cwd?

This idea had the power to extend further. Imagine your email and projects being piped into your calendar, and then to your team’s combined data lake. However, platforms are disincentivized to share: Google Drive and Dropbox don’t want to share the same files. Perhaps AI will revive this idea. You could imagine that all apps tap into a shared phone context, rather than each having its own data.

The idea that complex work should be expressed as a chain of discrete, single-purpose steps where the output of one becomes the input of the next is the foundation of every CI pipeline ever run. The stages: keyword is the same idea in different clothes: break your build process into named phrases, run them in order, and let each phase’s artifacts flow into the next.

DAG Scheduling

GitLab’s needs: keyword lets you write pipelines like this:

build:
  stage: build

unit-tests:
  needs: [build]

integration-tests:
  needs: [build]

deploy:
  needs: [unit-tests, integration-tests]

This is a Directed Acyclic Graph (DAG) — a graph of dependencies with no cycles, where the sort of nodes gives you a valid execution order. Multi-step projects are represented as directed graphs so that the longest sequence of dependent tasks (the “critical path”) becomes clear. This allows operators to find where work could be parallelized without delaying the overall timeline. The strategy was used to schedule chemical plants and coordinate the Polaris missile program.

GitLab’s job scheduler does the same thing. ProcessPipelineService evaluates the job DAG on every status change, unblocking downstream jobs the moment their dependencies complete.

The specific algorithm is Kahn’s topological sort, published in 1962: repeatedly select nodes with no incoming edges, emit them, remove them from the graph, repeat. Every CI scheduler does some variant of this.

Optimistic Concurrency Control

What if hundreds of runners simultaneously call POST /api/jobs/request, causing multiple Rails processes to receive these requests concurrently? How can they ensure that the same pending job is not handed to two runners at once?

The naive, pessimistic answer is a global lock: before selecting a job, lock the table, pick one, mark it running, unlock. This works, but at scale, it creates a queue of Rails processes waiting for the lock, and throughput tanks. Even if deadlocks are avoided, this locking limits concurrency.

Optimistic Concurrency Control (OCC), formalized in 1981, is an alternative that works well in environments with low data contention. The premise: instead of locking preemptively, assume conflicts are rare, proceed without locking, and detect conflicts at commit time.

In practice, GitLab’s RegisterJobService does this:

UPDATE ci_builds
SET    status = 'running',
       runner_id = $1,
       lock_version = lock_version + 1
WHERE  id = $2
  AND  status = 'pending'
  AND  lock_version = $3;  -- ← the OCC check

If another Rails process already picked this job (incrementing lock_version), this UPDATE matches zero rows. The service detects the miss and returns 204 No Content to the runner, which simply tries again on the next poll cycle. No deadlock, global lock, or serialization. Conflicts are rare, and when they do occur, only one poll cycle is wasted.

Where things go from here

Everything here assumes a human pushes code, waits for a runner, and sits around waiting for a green check.

That workflow is starting to feel antiquated.

If n agents are committing to n worktrees simultaneously, doesn’t it seem odd for them to poll and coordinate complex trains?

I think the primitives — pipes, DAGs, OCC — will stay, but the workflows built on top of them will look very different.

Whatever happens, one thing is certain: having a reliable and fast way to evaluate code is always going to be important.

Recommended Reading

GitLab’s Architecture | fullstack.zip | Stack and overview

Unix Pipeline History | Wikipedia

The Bell System Technical Journal | 1978 | McIlroy’s philosophy

Directed Acyclic Graph | Wikipedia

On Optimistic Methods for Concurrency Control | Harvard | The original 1981 paper.

Optimistic Concurrency Control | Wikipedia

Ultimate vs Immediate Goals

In How to Focus: A Monastic Guide for an Age of Distraction, young monks ask a master how to reconcile their tendency to chase goals with their vocation to focus on the present. The master differentiates between an ultimate goal (telos) and an immediate goal (scopos). Pursuing the ultimate goal provides purpose, while pursuing the immediate goal provides focus.

The merchant’s ultimate goal of profit keeps them centered when trouble happens at sea. The soldiers’ ultimate goal of honor helps them face the danger of their campaigns. So what’s the monks’ ultimate goal? Salvation. Makes sense, but too abstract for the young monks to act on daily. So the master recommends pursuing salvation by focusing on the immediate goal of “clarity of heart,” achieved through solitude, fasts, labor, and reading.

This empowered the young monks to channel their existing ambition into daily habits rather than attempting to remove their ambition.

5th-century monks aren’t the only ones who can direct the power of flawed nature toward good. Although our goals are less noble, engineers also have the opportunity to work with our nature — or against it.

I’ll recommend a goal at the end. But first, we have to start with the question:

What we do naturally

We build things. Not because we have to, but because we can’t stop. 52% of devs code for fun even after coding all day.

We automate the boring stuff. Give a dev a mindlessly repetitive task and watch him spend three times as long building a buggy script to eliminate it.

We learn obsessively. Yes, partly to keep up with the industry’s pace. But things can only move so fast because so many of us refuse to wait for permission to pick up a new skill. We’ll find the docs, read the src, and figure things out regardless.

We give a sh*t about craft. Tech debt is the top frustration at work, cited by 62% of us — ahead of tool reliability, deployment pipelines, and security issues. We hate living in a mess (especially if we didn’t create it).

We give back. 25% of developers actively contribute to open-source, which doesn’t include those who volunteer to help with issues, docs, sponsorship, or organizing.

We’re drawn to hard problems. Two out of three developers say they are motivated by a passion for learning new things, and the problems that excite us most are the ones nobody has cleanly solved yet.

What we naturally resist

Context switching. The average digital worker toggles between apps and websites nearly 1,200 times per day and spends 4 hours per week reorienting themselves. I don’t know the numbers for devs, but I bet they’re worse. It requires sustained focus to hold the codebase in our working memory, which is why everyone hates silly meetings and interruptions.

Devs in Japan are 4X less likely to struggle with context switching than those in the UK. (IDK what to do with that fact yet, but it’s interesting.)

Waiting for CI. Slow feedback loops also break the flow.

Hollow metrics. We know productivity KPIs are BS: 66% of developers don’t believe or aren’t sure that current metrics reflect their real contribution.

Fixing other people’s messes. We hate spending time debugging legacy code, untangling connective tissue between APIs, and working around workarounds. Unless we made the mess, then it’s fine.

Estimation theater. Estimating costs and timelines while requirements change every other day and everything is on fire seems silly.

Meetings. Most engineers spend nearly a third of their week in meetings. We resist this instinctively, because we know exactly what we could have shipped instead.

Put it all together, and you get a group of creative, crafty builders who are motivated by hard problems and are allergic to inefficiency. We just want to ship things that matter and leave the codebase better than we found it.

Application

Now that we know ourselves, we can pick goals that work with our nature rather than against it.

This is easy for your day job: your ultimate goal is to maximize value — generate more revenue or reduce expenses (cringe, I know). Feel free to adopt a more altruistic goal on your employer’s behalf, but don’t complain when you get cut during the next layoff.

For your side project, pick whatever ultimate goal resonates.

Some ideas:

• Fix a personal pain point

• Learn a tool that excites you

• Make enough to replace your main income

• Build the tool you always wished existed

The important point is that you commit to one ultimate goal. Don’t optimize for four things simultaneously.

The traps hide in the immediate goal, as it often leads to complexity. Initially, complex feels like progress. It’s challenging, exciting. It gives you an excuse to learn and write more code. More features, more services, more dependencies.

But complex things don’t last.

That’s why I humbly recommend making your immediate goal to preserve simplicity.

Fix your problem with a simple stack. Learn a new thing, simply. Make enough $ to replace your main job ... using simple workflows.

This works because it embraces our tendency to find creative solutions. Instead of adding, however, it forces us to use that creativity to subtract. Instead of dragging kludgy abstractions along, it asks us to find cleaner ones. Deleting a thousand lines of code that you spent months writing doesn’t feel great, but replacing 950 lines with 50 feels amazing.

Simple software is natural software. It’s easier to scan at 6 am, easier to write at 6 pm, and easier to fix at 1 am. That's why it has a chance to survive long enough for you to reach your ultimate goal.

What’s another thing we do naturally that I left out? Reply to LMK.

Recommendations

Data

• Developer Ecosystem 2025 | JetBrain’s survey of 25k devs

• Developers want more, more, more | Stack Overflow’s 2024 survey

• State of AI 2025 | How devs use AI

• The Context Switching Crisis | Hivel AI 2025

• What Motivates Open Source Software Contributors | Opensource.com

Books

• Systemantics: how systems work and especially how they fail | Fun wisdom

• How to Focus: A Monastic Guide for an Age of Distraction

• A Philosophy of Software Design | John Ousterhoust

• Getting Real | 37signals | Canonical argument for less

TL;DR - It’s a modular monolith built on this stack:

1. Backend: Ruby on Rails

2. HTTP server: Puma (Ruby web server)

3. Edge: Nginx

4. Reverse proxy: Go service (Workhorse)

5. Background jobs: Sidekiq (Ruby framework)

6. DB — primary: PostgreSQL

7. DB — connection pooling: PgBouncer

8. DB — high availability: Patroni

9. Cache: Redis

10. Git: Custom gRPC repo interface (Git & Gitaly)

11. Blob: AWS S3

12. Frontend — rendering: Haml & Vue

13. Frontend — state: Pinia (Vue store), Immer (immutable cache),

14. API: GraphQL (Apollo) + REST

15. Observability: Prometheus & Grafana

16. Error tracking: Sentry & OpenTelemetry

17. Deployments: GitLab Omnibus (Omnibus fork)

GitLab is what happens when you commit to the monolith. Its CI/CD, security scanner, issue tracker, and deployment toolchain are bundled into one product rather than five held together with webhook glue. The stack isn’t flashy either: a large Ruby on Rails app, Postgres, Redis, and a handful of Go services.

No Kafka, event-sourcing, mesh of 200 microservices. Instead, GitLab maintains its modularity through discipline and radical transparency.

This article traces the path from your git push command to the DB and back, ending with an explainer of how GitLab monitors all of it in prod, and where the architecture is headed.

What Is GitLab?

If you have never used GitLab (shame), here’s the gist.

It’s a platform for software teams. At its core, it hosts Git repos. On top of that, it adds CI/CD pipelines (automated build, test, and deploy workflows), merge requests, issue tracking, container registries, and security scanning, all in one application.

Unlike GitHub, which relies on a marketplace of third-party integrations for CI/CD, GitLab ships all of this natively. This “one-app-to-rule-them-all” stance is what shapes many of the architectural choices described in this article.

It’s also open-source.

The Glossary

GitLab introduces a lot of proper nouns. Before going further, here’s the cheat sheet. We will cover each of these in depth later.

+-------------+-----------------------------------------------------------+---------+
| Name        | What It Does                                              | Built In|
+-------------+-----------------------------------------------------------+---------+
| Puma        | Multi-threaded web server. The main application           | Ruby    |
|             | engine: receives web requests, runs business logic,       |         |
|             | reads and writes the database.                            |         |
+-------------+-----------------------------------------------------------+---------+
| Workhorse   | Go reverse proxy sitting in front of Puma. Handles        | Go      |
|             | slow, heavy I/O so Ruby does not have to: git push/pull   |         |
|             | over HTTPS, large file uploads, CI log streaming,         |         |
|             | artifact downloads.                                       |         |
+-------------+-----------------------------------------------------------+---------+
| GitLab Shell| Handles SSH connections for git push/pull. Authenticates  | Go/Ruby |
|             | the user, then hands the git operation to Gitaly.         |         |
+-------------+-----------------------------------------------------------+---------+
| Gitaly      | gRPC service that owns all Git data on disk. Replaced     | Go      |
|             | a brittle NFS-shared filesystem. Every git read/write     |         |
|             | goes through Gitaly RPC calls — nothing touches the git   |         |
|             | repos directly.                                           |         |
+-------------+-----------------------------------------------------------+---------+
| Praefect    | Load-balancing replication proxy for Gitaly. Sits in      | Go      |
|             | front of a cluster of Gitaly nodes, enforces quorum       |         |
|             | writes, and handles automatic failover.                   |         |
+-------------+-----------------------------------------------------------+---------+
| Consul      | Service registry. Holds the current topology of the       | Go      |
|             | Postgres cluster so that PgBouncer can find the primary   |         |
|             | after a failover without any human intervention.          |         |
+-------------+-----------------------------------------------------------+---------+
| Geo         | Multi-site replication. Secondary sites mirror the        | Ruby/Go |
|             | primary for read performance and disaster recovery.       |         |
+-------------+-----------------------------------------------------------+---------+
| GitLab      | Standalone Go binary that organizations deploy            | Go      |
| Runner      | wherever jobs should run: their own servers, a cloud      |         |
|             | account, a Kubernetes cluster. Polls GitLab for           |         |
|             | pending CI jobs and executes them.                        |         |
+-------------+-----------------------------------------------------------+---------+

The Full Picture

This is the end-to-end architecture. Every section below zooms into one layer of it.

Layer 1: Client to Ingress

Most traffic comes in over HTTPS and goes through the same ingress path. The exception is SSH, which we will discuss at the end of this section.

Cloudflare: The Front Door

Every request to GitLab.com hits Cloudflare first, which gives them a firewall, DDoS protection, a CDN, and TLS termination. We’ll break down Cloudflare in its own issue, so let’s leave it at that for now =]

Nginx: The Gateway

Nginx receives HTTP requests from the load balancer and routes them to Workhorse. It handles a few things here:

• TLS termination for self-managed installs where Cloudflare is not in front

• Static file serving for anything that lives on disk, like uploads that have not yet been moved to object storage

• Rate limiting headers, enforced in conjunction with GitLab’s Redis backend rate limiter

• Request routing — Nginx sends everything to Workhorse, which will decide what to do next

Nginx is intentionally thin here; it is not doing complex business logic. Its job is to be a fast, reliable conduit to the Workhorse.

The SSH Path: GitLab Shell

To retain the real hackers, GitLab also allows SSH connections, which bypass Nginx entirely. A git push git@gitlab.com:user/repo.git connects directly to GitLab Shell on port 22.

GitLab Shell is a small Go app that acts as a restricted SSH shell. When your SSH client connects, it reads your public key, calls the GitLab API to verify the key belongs to an authorized user, then hands off the git pack data to Gitaly for storage. It never actually lets you get a general shell prompt — it only accepts git-specific commands (lame).

Layer 2: Ingress to Core

Once a request passes through Nginx, it reaches the core application layer — the part that actually understands what the user is trying to do.

Workhorse: The Smart Proxy

Workhorse is a Go service that sits between Nginx and Puma. In addition to normal reverse proxy duties, it has a specific job: keep slow I/O away from Ruby threads.

Ruby has a Global Interpreter Lock (GIL) that limits true parallelism. Tying up a Ruby thread on a slow upload or a long git-pack operation is expensive, preventing that thread from serving other requests while it waits. Workhorse exists to prevent this.

The communication between Workhorse and Puma is slightly unusual.

While the standard reverse proxy request goes like this:

client -> proxy -> app -> proxy -> client

…adding Workhorse into the mix makes it go like this:

client -> Workhorse -> Puma -> Workhorse -> app -> Workhorse -> client

Workhorse forwards the request to Puma. Puma processes the request and then replies with special headers (X-Sendfile, X-GitLab-Git-HTTP-Session). Workhorse uses those to hijack the response and send it directly to the client.

Basically, Puma tells Workhorse what to do via response headers, and Workhorse takes it from there.

Workhorse is responsible for:

• Git HTTP operations — git push and pull over HTTPS. It streams the pack data directly to and from Gitaly via gRPC, without routing the raw bytes through Ruby

• File uploads — when you upload an attachment, Workhorse buffers the multipart upload and either stores it temporarily on disk or streams it directly to object storage, then notifies Puma that the upload is complete.

• Artifact downloads — when you download a CI artifact, Workhorse fetches it from object storage and streams it to your client, again without involving Puma

• CI log tailing — while a job runs, Workhorse streams log chunks from Redis to the browser in real time.

This is weird, but it frees the Puma thread immediately instead of making it wait for I/O to finish.

Puma: The Rails Application Engine

Puma is where the actual GitLab application runs. It is a multi-threaded Ruby HTTP server running the GitLab Rails app — a large codebase that handles:

• Authentication & authorization (who is this user, can they see the code)

• Reading and writing to PostgreSQL (user records, merge requests, issue data)

• Reading and writing to Redis (sessions, locks, cache, real-time events)

• Calling Gitaly via gRPC for repository operations (list branches, read a file, get a diff)

• Enqueuing Sidekiq background jobs for any work that does not need to happen synchronously

• Rendering the HTML, JSON, or GraphQL response

Puma scales horizontally: add more Puma nodes behind the load balancer, and throughput increases linearly (because Puma nodes are stateless). Sessions live in Redis. Nothing lives on the Puma host that cannot be reconstructed.

The Rails app is organized in a monolithic codebase, but it’s nevertheless modular. Each domain manages its own models, services, and API endpoints. This provides the DX benefits (easy to run locally, shared tooling, fewer network hops) while keeping the blast radius of any given change small.

Sidekiq: The Background Worker

Not everything needs to happen before the user sees a response. Sending a notification email, updating a downstream pipeline status, archiving a CI log to object storage — these are all better handled asynchronously.

Sidekiq is GitLab’s background job processor for this type of work. It runs as a separate process (or many processes at scale), reads job payloads from Redis queues, and processes them independently from the web request cycle.

When Puma wants to fire off async work, it enqueues a job — a small JSON payload describing a Ruby class and its arguments — into a Redis list. Sidekiq workers are sitting in a loop pulling from those lists. They pick up the job, execute the Ruby code, and either mark it complete or retry it on failure.

Puma (on a request):                   
  1. Process MR comment creation            
  2. Write comment to PostgreSQL           
  3. Enqueue NotificationWorker ---->    Sidekiq (background):
  4. Return 201 to user                  1. Pull NotificationWorker job from Redis 
     (~40ms)                             2. Load recipients from PostgreSQL                                          
                                         3. Send emails via SMTP
                                         4. Mark job complete
                                         (~2 seconds, who cares)                        

At GitLab.com scale, Sidekiq is not one process — it’s a fleet of workers, each configured with specific queue assignments. CI state-transition workers run on dedicated Sidekiq fleets so that a slow email-delivery worker does not hold up the CI queue (a lesson learned the hard way).

Layer 3: Core to External Services

Remember that the app tier does not hold state. It delegates to a set of purpose-built external services. Let’s learn what each of these star players does.

PostgreSQL: The System of Record

Every structured piece of app data lives in Postgres: users, projects, groups, issues, merge requests, CI pipelines, job records, artifact metadata, permissions — all of it. It’s the authoritative source of truth.

GitLab.com doesn’t connect to Postgres directly, however. It goes through PgBouncer, a connection pooler that multiplexes thousands of connections down to a manageable pool. Without it, a Puma/Sidekiq fleet of hundreds of processes would each try to hold open a database connection, quickly exhausting Postgres’s connection limit.

GitLab also uses read replicas — secondary Postgres nodes that receive a streaming copy of all writes from the primary. Read-heavy operations like loading a project’s issue list or rendering a dashboard can be routed to replicas, offloading the primary for write-heavy work.

Redis

GitLab uses Redis as eight logically distinct stores, each with different operational requirements and eviction policies.

+------------------------+-------------------------------------------+------------------+
| Instance               | Stores                                    | OK to Lose?      |
+------------------------+-------------------------------------------+------------------+
| queues                 | Sidekiq job payloads, retry state         | No - jobs lost   |
| shared_state           | Distributed locks, exclusive leases       | No - race conds  |
| sessions               | User login sessions                       | No - logged out  |
| actioncable            | WebSocket pub/sub for real-time UI        | Mostly fine      |
| trace_chunks           | In-flight CI job log bytes                | Mostly fine      |
| cache                  | Computed Rails.cache data                 | Yes - rebuildable|
| rate_limiting          | Request counters per IP/user              | Yes - rebuildable|
| repository_cache       | Branch and tag name lists                 | Yes - rebuildable|
+------------------------+-------------------------------------------+------------------+

Cache instances use an LRU eviction policy: when they fill up, the least-recently-used keys are evicted. That is the correct behavior for a cache. (Imagine you applied LRU eviction to the queues instance: Sidekiq jobs would silently disappear from the queue before being processed!)

On a small self-managed instance, one Redis process handles everything. On GitLab.com, these are fully separate clusters, sized and scaled independently. This ensures that when the CI system generates a spike in trace chunk writes, it does not compete with session storage.

Gitaly: RPC for Git

Early GitLab stored git repositories on a shared NFS filesystem. Every Puma and Sidekiq node would mount the NFS volume and call git commands directly on disk. This worked, up to a point. As scale grew, the limitations became painful: NFS latency spikes, split-brain risks on failover, no way to observe what git operations were running or how long they took.

Gitaly eventually replaced NFS. It is a gRPC service — you call it via a typed remote procedure call (RPC) interface, not by shelling out git on a shared filesystem. Every git operation in GitLab now translates to a Gitaly RPC:

• To get a MR diff, Rails calls Gitaly.CommitDiff(repo, from_sha, to_sha). Gitaly streams diff hunks back as gRPC messages.

• To list branches, Rails calls Gitaly.FindAllBranches(repo). Gitaly reads from disk, returns structured branch data

• To git push over HTTP: Workhorse calls Gitaly.PostReceivePack(repo, pack_data). Gitaly writes objects, updates refs, runs server-side hooks.

Because every git operation is a discrete, typed RPC, Gitaly can be monitored, load-balanced, and independently scaled in a way that NFS never could have.

Object Storage: Keeping Blobs Separate

Not everything fits neatly in a Postgres row. CI artifacts (binaries, test reports, logs), Git LFS files, container images, uploaded attachments, and archived job logs are binary blobs that can be gigabytes in size.

Storing this in Postgres, while technically possible, would inflate table sizes, make backups slow, and force blob I/O through the same connection pool as structured queries.

GitLab sanely puts all blobs in object storage — GCS on GitLab.com, S3 on self-managed installs. The DB holds only a pointer (a storage path and some metadata). When you download a CI artifact:

Browser ──► Workhorse ──► Rails (auth check only)
                              │
                              │  "This artifact lives at
                              │   gs://artifacts-bucket/job/42/artifact.zip"                    "
                              │
                              ▼
                         Workhorse generates a presigned URL
                         or proxies the stream directly from GCS
                         to the browser

The auth check is instantaneous. The actual data transfer happens between Workhorse and GCS, with no Puma thread blocked waiting for bytes. Another example of using Puma for what it’s good at (HTTP) and avoiding it for I/O.

The AI Gateway: Model Routing for Duo

GitLab Duo is the umbrella name for AI features: code suggestions, chat, root-cause analysis for failed CI pipelines, security vulnerability explanations, and so on. All of these features funnel through a centralized AI Gateway — a standalone Python/FastAPI service that routes requests to the right model.

Rails/Sidekiq
      │
      ▼
AI Gateway (FastAPI)
      ├──► Anthropic Claude   (code generation, chat)
      ├──► Google Vertex AI   (various tasks)
      └──► Self-hosted models (for enterprise customers)

Why a gateway rather than Rails calling model providers directly? A few reasons. The Gateway creates one point of control for rate limiting, cost accounting, model versioning, and provider fallback across all deployment types. When GitLab wants to swap one model provider for another or A/B test two models, they change it in one place. Self-managed customers do not need to configure cloud provider credentials for every AI feature separately; they just connect to the Gateway.

External Services: The Outside World

GitLab also talks to a set of external systems that are not “part of GitLab” per se but are deeply integrated:

• SMTP servers — for sending email notifications (Sidekiq-dispatched)

• OAuth providers — Google, GitHub, SAML, etc for SSO

• LDAP / Active Directory — for enterprise users provisioning and group sync

• Kubernetes API — for the GitLab agent for k8s, used for deployments and cluster management

• Container registries — GitLab’s built-in registry uses a separate service gitlab-registry), based on Docker.

• Webhook targets — when you configure a GitLab webhook, Sidekiq makes an outbound HTTP request to your configured endpoint on relevant events

We’ve covered the main building blocks of GitLab. The following explains the connective tissue and peripheral bits that make everything come together in prod.

High Availability

How GitLab eliminates single points of failure at every layer of the stack.

Application HA

Stateless nodes. Puma and Sidekiq nodes hold no local state. User sessions live in Redis. Uploaded files live in object storage. If a Puma node dies, the load balancer routes traffic to the remaining nodes before users ever notice — their session is still valid because it was in Redis all along. Adding capacity is equally painless: spin up more nodes and register them with the load balancer. No data migration needed.

Praefect adds high-availability (HA) on top of Gitaly. It is a proxy that sits in front of a cluster or Gitaly nodes. On a write, Praefect sends the operation to all nodes and waits for a quorum to confirm before returning success. This is “strong consistency,” meaning you can read from any Gitaly replica immediately after a write and get the correct data. When a Gitaly node fails, Praefect promotes a replica automatically.

DB HA: Patroni + PgBouncer + Consul

Patroni runs on each Postgres node. It manages leader election, monitors health, and handles automatic failure. If the primary becomes unreachable, Patroni elects a new primary within seconds. It broadcasts the result to Consul, the service registry. PgBouncer watches Consul, so it knows to point new connections at the primary without any restart. Application nodes never need to know which host is primary; they always talk to PgBouncer.

Failover typically completes in under 30 seconds. Applications experience a brief pause as PgBouncer re-routes, then continue normally.

Gitaly HA: Praefect Cluster with Quorum Writes

Praefect is a router and transaction manager for Gitaly.

On writes, Praefect sends the operation to all Gitaly nodes and waits for a quorum (majority) to confirm before reporting success. This means you can read from any replica immediatly after a write and see the latest data — no stale reads. If the primary Gitaly node fails, Praefect promotes a replica and resumes service automatically.

Praefect maintains its own small Postgres DB (separate from GitLab’s main Postgres) to track replication state. A full HA deployment ends up with two Postgres clusters: one for GitLab app data, one for Praefect.

Redis HA: Sentinel Mode

Each Redis instance in a HA deployment runs with Redis Sentinel: a primary Redis with replicas, monitored by at least three Sentinel processes. If the primary fails, Sentinels elect a new primary and notify clients to reconnect. Application nodes are configured to discover the current primary through Sentinel rather than connecting to a fixed host.

One constraint worth knowing: Redis Cluster (Redis’s own horizontal-sharding mode) is not supported for all GitLab Redis uses. The trace_chunks instance uses a data structure that is incompatible with Redis Cluster’s key-slot sharding. All GitLab Redis instances therefore use Sentinel mode, not Cluster mode.

Site Level Recovery: GitLab Geo

What happens if the entire primary data center goes offline? How do developers in Asia-Pacific stop having 300ms latency every time they run git clone against a server in Virginia?

GitLab’s answer: Geo, a warm-standby for distributed teams. Geo provides local caches that can be placed geographically close to remote teams, which can serve read requests. This can reduce the time it takes to clone and fetch large repositories.

The secondary site gets Postgres via standard streaming replication — a continuous, near-real-time copy of every write. Git repos are synced by the Geo Log Cursor daemon, a separate process that reads the log and enqueues background jobs to clone or fetch updated repos from the primary via HTTPS.

Replication is async — there is always some lag, usually measured in seconds. For disaster recovery, planned failover waits for full synchronization before promoting the secondary. Zero data loss. Emergency failover can be forced immediately. In this case, some data written in the last few seconds may be lost.

Monitoring: How GitLab Knows When Things Break

A system with this many parts will have failures. The question is how quickly they can be detected, and how much context you’ll have to start troubleshooting.

GitLab’s observability stack covers three domains: metrics (what is happening now), logs (what happened when), and traces (specific request tracking).

Metrics: Prometheus and Grafana

Every GitLab component exposes a Prometheus metrics endpoint. The in-process GitLab Exporter scrapes the running Rails app and produces application-level metrics. Prometheus collects and stores all of these on a regular scrape interval. Grafana provides dashboards and alert routing.

Key metrics the team monitors:

+----------------------------+-----------------------------------------------+
| Metric                     | What It Tells You                             |
+----------------------------+-----------------------------------------------+
| HTTP request rate/latency  | Is the web tier healthy? Latency spikes       |
| by controller action       | point to slow DB queries or Gitaly RPCs       |
+----------------------------+-----------------------------------------------+
| Sidekiq queue depth        | Are jobs piling up? Indicates under-capacity  |
| by queue name              | or a worker that is stuck                     |
+----------------------------+-----------------------------------------------+
| Sidekiq job latency        | Time from enqueue to start. High latency      |
| by worker class            | = queue starvation                            |
+----------------------------+-----------------------------------------------+
| Gitaly RPC latency         | Git operation performance. Spikes indicate    |
| by RPC name                | disk I/O issues on Gitaly nodes               |
+----------------------------+-----------------------------------------------+
| PostgreSQL connection      | Is PgBouncer saturated? Are we running out    |
| pool saturation            | of Postgres connections?                      |
+----------------------------+-----------------------------------------------+
| CI queue depth             | Pending jobs waiting for a runner. High       |
| by runner type             | values = runner fleet under-capacity          |
+----------------------------+-----------------------------------------------+
| Redis command latency      | Cache/queue slowness. Usually indicates       |
|                            | memory pressure or network issues             |
+----------------------------+-----------------------------------------------+
| Error rate by service      | Rolling count of 5xx responses. The canary    |
|                            | for most application-level problems           |
+----------------------------+-----------------------------------------------+

Logs

Every component in GitLab emits structured JSON logs. The critical piece is the correlation ID: a UUID assigned to each incoming request by Workhorse, stamped into every log generated anywhere in the call chain — Rails, Gitaly, Sidekiq, all of ‘em.

{
  "method": "POST",
  "path": "/api/v4/jobs/request",
  "status": 200,
  "duration_s": 0.043,
  "correlation_id": "01HXKG4P8B3QV7N6J2MZ", // <-- important
  "user_id": 1234,
  "meta.caller_id": "Ci::RegisterJobService",
  "db_duration_s": 0.012,
  "redis_calls": 3,
  "gitaly_calls": 0,
  "queue_duration_s": 0.001
}

When an on-call engineer sees a latency spike on a Grafana dashboard, they grab the correlation ID of an affected request and use it as a filter key across all log sources — Elasticsearch, Splunk, Loki, whatever the install uses.

Tracing: OTEL

Metrics tell you what is slow. Logs tell you when the slowness happened. Distributed traces tell you why by showing the exact sequence of service calls and how long each one took.

GitLab uses OpenTelemetry-compatible tracing. The correlation ID doubles as the trace ID here. A slow pipeline creation request might produce a trace like:

POST /api/v4/projects/123/pipeline   847ms total
  ├── Auth check (Redis)              3ms
  ├── Project load (Postgres)         8ms
  ├── YAML fetch from Gitaly         22ms    ← Gitaly.GetFile RPC
  ├── YAML parse and rule eval        41ms
  ├── Pipeline write (Postgres)       15ms
  ├── Stage+job row inserts          180ms   ← this is the bottleneck today
  ├── Sidekiq enqueue (Redis)         4ms
  └── Response serialization         12ms

Interestingly, this distributed tracing is still considered “experimental” at the time of this article.

Error Tracking

Unhandled exceptions in Rails are captured and routed to GitLab’s error tracking system. GitLab uses its own built-in Error Tracking feature (which integrates with Sentry-compatible backends) as well as self-hosted Sentry for some teams. Errors are deduplicated, grouped by fingerprint, and surfaced with full stack traces and the request context (user, path, params) that caused them.

Error tracking events also carry the correlation ID, so you can jump from an error event directly to all logs from that specific request.

Scaling PostgreSQL

For those familiar with these real-world apps, things probably seem pretty straightforward up to this point; GitLab hasn’t tried to do anything cute or uniquely ambitious on the engineering side.

At their current scale, however, the PostgreSQL situation is getting untenable. They started by allowing horizontal cloning by making Puma & Sidekiq stateless. Then they decomposed CI tables into a separate DB, a multi-year effort that required forbidding cross-database JOINs in the Rails model layer and rewriting hundreds of queries.

Now they’re partitioning data by customer. This can’t be done via traditional sharding, because Postgres has no native sharding. CitusDB was evaluated and ruled out. Application-level sharding requires every query to know which shard to target, and makes cross-shard queries effectively impossible.

GitLab’s answer is Cells (interestingly, the same name Discord used for its similar abstraction). Instead of sharding rows in a database, they’ll run multiple independent copies of the entire application stack — each called a Cell — with a shared routing layer in front.

An org is assigned to exactly one Cell. The global router resolves which Cell handles each request based on the org in the URL. Each Cell is essentially a complete, isolated GitLab instance. Adding capacity means adding a new Cell — no resharding or cross-shard coordination.

The hard part is cross-cell operations: a user who belongs to orgs on two different cells, or a fork from a project on Cell 1 to Cell 2. These require a global identity layer and router that is still being designed.

They’re refreshingly transparent about this effort: you can track the project status, ADRs, and proposed architecture on their public handbook page (handbook.gitlab.com).

The “Boring” Takeaway

Perhaps you’ve been waiting for more complexity, the kind of juicy architectural optimization that makes the front page of HackerNews.

If that’s the case, GitLab ain’t for you.

The most interesting part about GitLab’s architecture is what isn’t there. No Kafka. No event sourcing. No distributed saga coordinator. No fancy message bus (yet). Their Z-axis scaling is being solved not with sharing but with what is essentially “run more copies of the whole app” lol.

Its monolithic codebase and the conventional tools that support it are simple, lean, pragmatic. They’ll extract and abstract, but only when warranted:

• Gitaly was created when NFS-backed git storage became a liability

• Workhorse was born when Ruby’s I/O throughout became the bottleneck for git operations

• The CI DB was split when CI tables started consuming 40% of write traffic.

The monolith survives because it genuinely serves the product well: it is fast to develop, easy to test locally, straightforward to reason about, and naturally consistent within a single DB transaction.

This is a story of engineering sanity, not architectural fashion. And for 30 million users, it’s working just fine.

References

1. GitLab Architecture Overview — GitLab Docs | Canonical component diagram and service descriptions.

2. Reference Architectures — GitLab Docs | Tier-by-tier HA topologies from single node to 50k+ users.

3. Gitaly Cluster / Praefect — GitLab Docs | Replication proxy, quorum writes, and automatic failover.

4. Redis Development Guidelines — GitLab Docs | Multiple Redis instances, eviction policies, and use case separation.

5. Geo — GitLab Docs | Multi-site replication, the Geo Log Cursor, and failover procedures.

6. PostgreSQL HA with Patroni — GitLab Docs | Patroni, PgBouncer, and Consul HA topology in depth.

7. GitLab Scalability — GitLab Docs | AKF Scale Cube framing — X/Y/Z axes, current state, and gaps.

8. Multiple Databases — GitLab Docs | CI database separation and cross-database query restrictions.

9. Cells Architecture Blueprint — GitLab Handbook | Design, goals, and current status of the Cells initiative.

10. AI Architecture — GitLab Docs | AI Gateway design, Duo routing, self-hosted model support.

11. Find Relevant Log Entries with a Correlation ID — GitLab Docs | Correlation ID flow through Workhorse, Rails, Gitaly, and Sidekiq.

12. Log System — GitLab Docs | Structured JSON logging reference — all log files and their schemas.

13. Error Tracking — GitLab Docs | GitLab’s built-in Sentry-compatible error tracking integration.

14. GitLab Workhorse — GitLab Docs | Design rationale and feature set of the smart proxy layer.

15. Decomposing the GitLab Backend Database — GitLab Blog | The story of separating the CI database from main.

16. EE Feature Guidelines — GitLab Docs | How CE/EE code separation works at the Rails autoloader level.

17. How We Designed GitLab Reference Architectures — GitLab Blog | History and rationale of the reference architecture tiers.

18. Scaling Sidekiq at GitLab.com — GitLab Blog Lessons from specialized queue configuration at production scale.

19. GitLab Shell — GitLab Repository | SSH session handler.

20. AKF Scale Cube — AKF Partners | 2024 | The original X/Y/Z axes of scalability model GitLab references.

Critiquing the bad is useful, so is celebrating the good.

That’s why this issue highlights everyday simplicity.

Engineering simplicity

1. CI/CD scripts in Bash only. No Terraform, subscriptions, config hell, version updates. Just Bash.

2. Claude skill for simplifying (SKILL.md). “Favor minimal, legible implementations. Fewer lines and clearer structure make code easier to understand and maintain.”

3. Typescript on server and client. Strong-typing for AI. Easier to run and read code for devs.

4. Monolith instead of microservices. Easier for normal-sized teams to manage. “[It] doesn’t pretend to provide a failsafe architectural road to glory. That’s a fool’s errand. Many programmers suffering under many oppressive influences will turn any architecture made with any tool into a big pile of mud.” (signalvnoise)

5. Markdown files. Portable, straightforward. These files lack fancy embeds and database linking, but they work. In fact, an entire note-taking company is built on this file type (obsidian.md).

6. Peter Level’s vanilla stack: HTML, CSS, vanilla JavaScript, jQuery, PHP, and SQLite — all running on a single VPS (indiehackers.com).

7. Dashes and lowercase for files. That’s it. This saves so much time, trust me.

   taxes-2025.pdf
   taxes-2025-amended.pdf
   statement-checking-2025-01.pdf
   statement-checking-2025-02.pdf

8. SQLite in production. Notion’s early days and plenty of indie apps run SQLite fine. One file, zero config, no separate server process (sqlite.org/whentouse).

9. ENV files for config. A flat text file beats a secrets manager, a config service, and three IAM policies. Works locally, works in prod, understandable.

10. Cron jobs instead of queues. Before Kafka, SQS, or BullMQ — there was cron. Scheduled tasks, no broker, no consumer group, no dead-letter queue. If your job runs once an hour, you don’t need a message bus.

AFK Simplicity

1. Two-tier pricing models. Free or premium. Free or usage-based. Starter or enterprise. The end.

2. Dotted grid paper. The best of grid paper + lined paper + freeform paper (leuchturm1917.us)

3. Capsule wardrobe. A small collection of thoughtfully chosen, easy-to-mix pieces that allows you to create a wide range of outfits with fewer items. Less decision fatigue, more time (modernminimalism.com).

4. Crossfit’s diet recommendation: “Eat meat and vegetables, nuts and seeds, some fruit, little starch, no sugar.”

5. Aldi’s 25-cent deposit to use a cart. “This […] ultimately saves our customers money because we don’t have to hire extra staff to collect grocery carts.” Yep.

6. My grandma’s “as much as you want” blue cheese rice soup recipe.

1. Buy Uncle Ben’s seasoned “long grain + wild rice” original
2. Cook according to directions
3. Add plain yogurt (not Greek) (as much as you want)
4. Add crumbled blue cheese (as much as you want)
5. Heat thoroughly
6. If you want to make it a 1 dish meal, add browned hamburger

7. Apple Notes for tracking workouts. I used to have three workout apps. Now I have three notes.

8. The Big Ass Calendar. An entire year on one wall. No flipping, loading screens, or subscriptions (thebigasscalendar.com).

9. My old gym’s response to complaints about the playlist:

   

10. SF street names engraved in the sidewalk. Fewer signs.

Once you start looking for the beautifully simple solutions to problems, you’ll see them everywhere.

I hope these inspire you to simplify your own stack and life.

Reply with an example or two of your own!

The Depth of Knowledge (DOK) model categorizes thought into four levels of complexity:

1. Recall. Memorizing, matching, reciting, quoting

2. Skill. Inferring, categorizing, interpreting, predicting

3. Strategy. Critiquing, developing, comparing, concluding

4. Extend. Design, connect, synthesize, create

DOK helps educators create a curriculum with varied cognitive rigor. A good math exam would require students to identify basic formulas, choose the right one, analyze data, and solve a real-world problem. It also helps explain why some students excel at certain assignments but struggle with others.

We can look at our system design skills through a similar lens.

In keeping with our theme of simplicity, I propose just three levels:

1. Facts. The difference between SQL and NoSQL.

2. Heuristics. SQL for structured data, NoSQL for unstructured data.

3. Inventions. Distributed SQL.

Our Discord case study displays engineering at all three levels of complexity.

1. Facts: Knowing enough about programming to make the app fast.

2. Heuristics: Adopting Elixir, Rust, and ScyllaDB, despite their newness.

3. Inventions: Creating a “Super-Disk” abstraction to guarantee both speed (SSDs) and reliability (hard drives).

Discord’s inventions are the most impressive.

“But we’re not at Discord scale — our apps don’t need to support trillions of messages or millions of concurrent users. Plus, we can pass most system design interviews with memorization alone. Why not just move on and build stuff instead of worrying about creating more abstractions?”

I get it.

But it’s a shame to build a strong base during interview season, only to let it atrophy after day one on the job, obediently accepting the precedents forever.

Invention is worthy of pursuit. Although you’ll never be assigned a ticket to “Invent New Abstraction,” doing so is one of the most high-impact things you can do as an engineer.

Invention case study

Carl Hewitt’s Actor Model introduced a communication protocol of messages and mailboxes that solves race conditions. Although the model is a beautiful invention, creating something so esoteric feels a bit out-of-reach for those of us who don’t have a PhD from MIT.

Thankfully, we don’t have to introduce a new paradigm to invent; plenty of opportunities arise within our current stack. The Ergo Framework is a perfect example of how regular engineers can invent abstractions that simplify rather than complicate.

The problem

Ergo is an implementation of the Actor Model in Go, bridging the gap between a 50-year-old theory and a modern programming language. The basics of the actor model are perfectly compatible with the Go language — queues for mailboxes, trees for supervisors.

However, there’s one scenario where the language conflicts with the model: I/O (file reads, listening to OS signals, TCP accept). The Actor Model requires a balance of async delivery with sync processing. That’s because Actors work sequentially. One message arrives, gets processed, completes. Next message. This strict workflow is what eliminates race conditions.

Blocking I/O breaks this model.

In Ergo, a normal Process is an actor — a lightweight entity that handles messages sequentially in its own goroutine. There’s one process per goroutine. Processes can make sync calls and create monitors — standard actor stuff. Since there’s one goroutine, it must handle everything. If it blocks on I/O, message processing stops.

Call net.Listener.Accept() in a handler and watch the goroutine freeze while it waits for connections. Messages pile up until the system times out.

Let’s try solving the problem with a simple heuristic.

“Compute is cheap, add more. Moore’s Law. Just spawn a new goroutine for Accept().”

This bails you out of the immediate bottleneck. But now the two goroutines access the actor’s state concurrently. We just traded a perf problem for a race condition problem.

“Add a lock” (level one).

Sure, but now we have to add a bunch of code to handle the lock’s complexity.

The only way we can preserve the guarantees of concurrency that come with the Actor Model and the I/O use-case in Go is to invent a more elegant abstraction (level three).

The invention

The Ergo authors created “Meta-processes” for this situation.

Unlike normal processes, which have one goroutine per process, Meta-processes have two: a Reader and a Handler. The reader is for blocking operations — Accept() loops, ReadFrom() calls. The handler does everything except I/O via sequential messages in traditional Actor fashion. The Reader runs continuously from spawn until termination. The Handler comes and goes based on message traffic.

Meta-processes separate concerns. The Reader handles I/O, while the Handler processes messages. The Reader blocks reading while the Handler blocks writing. Two blocking operations, two goroutines, neither interrupting the other.

Meta-processes mediate the purity of the Actor Model with the messiness of the real I/O world. They turn blocking operations into async messages and messages into blocking commands. Regular processes implement everything else.

It’s a wonderfully simple distinction that a lay-developer can implement, even if they don’t appreciate the mechanism.

The takeaway

To escape a bind, add constraints.

To escape locks in concurrent programming, the Actor Model added communication constraints: messages and actors. Authors need to follow the rules by only sending messages and not sharing state, but they get guaranteed concurrency.

To escape blocking I/O in single-processes, Ergo added process constraints: one for reading, one for writing. Authors need to discern whether they’re working with I/O, but they get to keep concurrency in Go.

It’s counterintuitive to consider adding more rules in the pursuit of simplicity. It’s also easy to add the wrong rules and get more complexity without extra benefits. Tolerating this tension is a prerequisite to finding a level three invention. It’s the kind of challenge that comes from both understanding the theory and working in the slimy guts of a problem.

You can’t memorize or skim your way out of these situations.

That’s why they require level-three thinking.

Keep memorizing definitions, skimming, and interviewing. But also pursue answers that aren’t so accessible. Our industry is increasingly outsourcing its thinking; we need those who continue to exercise their cognition more than ever.

Bonus: More inventions

MapReduce: map() + reduce()

Containers: cgroups + namespaces

Git: immutable snapshots

CRDTs: sync + merge without locks

Virtual DOM: diffs against the real DOM

Promises: Async that reads like sync

Resources

Depth of Knowledge: Difficulty and Complexity (webbalign.org)

DOK chart (pdesas.org)

The Actor Model: Messages and Mailboxes (fullstack.zip)

Discord: A Case Study in Performance Optimization (fullstack.zip)

Ergo Basics (ergo.services)

Perhaps you’ve shrugged it off as an inevitable side-effect of progress.

Is it, though?

Increasing the number of components makes a system more complicated.

But entangling the individual components makes it complex.

A Swiss watch with its many gears is complicated. But there’s a manual. A watchmaker can repair it reliably.

The unpredictable behavior between components is what makes a system complex. Weather is complex. Cities are complex.

The problem isn’t the number of parts; it’s that the parts can’t be understood in isolation; you can’t pull one out without disturbing the others.

More components, more complicated.

More relationships between components, more complex.

Complication is an inevitable side-effect of solving harder problems.

On the other hand, complexity is a choice — one that is often made unconsciously while solving a problem. The irony is that by intertwining in the pursuit of an immediate solution, we shorten the solution’s lifespan.

We only need to peek at a few open-source projects to see this reality at work.

Dependencies reveal complexity

Naming is hard, but we got it right by calling packages “dependencies.” The more dependencies a project has, the more relationships, the more complexity.

If it’s true that complexity kills, then we should expect that the projects that survive have fewer dependencies.

I sampled 19 popular open-source web libraries and apps and found just that.

For scale, the average project in this list is 6.5 years old and has anywhere from

• 72k to 3M lines of code

• 11 to 340 dependencies

Lines of Code per Dependency

• VSCode: 23,579

• SuperProductivity: 21,173

• Ionic Framework: 11,518

• OpenClaw: 10,632

• Posthog: 8,970

• Playwright: 8,562

• react-hook-form: 6,562

• webpack: 6,050

• social-analyzer: 6,022

• excalidraw: 5,579

• Cal.com: 4,039

• Ionic Capacitor: 2,637

• Remix: 1,795

• Logseq: 1,772

• drawdb: 1,495

• Compass Calendar: 1,383

• Vitest: 602

• Formik: 566

The official winner is VSCode, with over 23,000 lines of code for each of its dependencies. As a low-level tool backed by Microsoft and the default app for every web dev for the past decade, this makes sense. There’s a lot of money and engineering time spent on keeping it lean-ish. But close behind is a productivity app with 614k lines of code and 29 dependencies backed by … one guy in Germany whose been working on it part-time for the last nine years!

SuperProductivity has endured because it has resisted complexity.

Compass is similar to SuperProductivity in spirit. Yet it’s at the bottom of the list because its developer has eaten from the forbidden fruit of the npm tree (and suffered greatly as a result).

The OGs at Remix are wiser; they’ve made the avoidance of dependencies a core principle:

Avoid Dependencies. Dependencies lock you into somebody else’s roadmap. Choose them wisely, wrap them completely, and expect to replace most of them with our own package eventually. The goal is zero.
— Remix’s README

This relationship between complexity and longevity is not unique to the open-source projects I cherry-picked. Just look at your own team’s graveyard of archived repos. I’ll bet the ones that didn’t make it have a lot of complex relationships.

We can choose differently now

Now that software is cheap, we’re forced to confront the question, “Do things still need to be this complex?” A triple-decker of abstractions might’ve been justifiable in a big codebase maintained by 500 engineers, each of whom can only keep 5% of the system in their head. But if that same team “reorgs” to 100 engineers and 400 coding agents, each of which can hold 100% of the system in context, could we travel more lightly?

Yes, and it’s already happening. Cloudflare rebuilt Next.js sans opinionated build system with one engineer in one week (cloudflare.com). Anthropic built a C compiler from scratch with 16 agents and $20k (anthropic.com).

The abundance of subsidized tokens encourages the creation of unadulterated projects like these headline-grabbers. But it also offers a unique opportunity to right our wrongs, to absolve the complexity we’ve accrued during times of weakness.

These days, I wake up with foaming bloodlust towards my package.json, for I know that the more complexity I expunge, the longer the software can survive.

There’s an irony here, one that mirrors the opportunity presented by social media.

As we’re bombarded with cheap dopamine, it’s never been easier to settle into chronic distraction. It’s also never been easier to compete: just read a book for five minutes to become a top 84% reader in the US (bls.gov).

We’re always one prompt away from a PR that introduces more complexity. But we’re also always one prompt away from a PR that cleans up the old.

What about the model is so special? There are entire books that explain this in detail. For this edition, we’re zeroing in on two core ideas: messages and mailboxes. These are the mechanisms that constrain each actor while preserving the power of the system.

Messages

The key to the actor model is messages, which are:

• The bundles of data — typically objects — that actors pass between one another.

• The inputs and outputs of every actor. Rather than sharing state directly, actors communicate exclusively through messages.

• Immutable. Once a message is created, it cannot be changed. Think of it like sending a sealed letter: the recipient gets exactly what you dropped off, and no one can alter it in transit. This one rule eliminates an entire class of bugs.

What actually happens when an actor receives a message?

One of four things:

1. The actor updates its own state

2. It creates more actors

3. It sends a message to another actor

4. It changes its behavior (how it will respond to future messages)

That’s it. Four activities summarize every interaction in an actor-based system.

Actor messages are perfect for Domain Driven Design (DDD).

DDD models code around an org’s entities and rules. It’s a natural design for our sprawling distributed systems. Anywhere the business logic is rich, stateful, and central to the product, DDD tends to shine.

Actors are a natural way to implement DDD entities — the core objects that have identity and evolve. In classic DDD, these entities exchange one of four messages:

1. Command: Tells an actor to change state (”Process this payment”)

2. Event: Records that a state change happened (”PROCESSED”)

3. Query: Requests information (”What’s the current balance?”)

4. Results: Response to a query ($247)

Four rules that detail how nodes communicate. Sound familiar? The terms vary, but both the Actor Model and its DDD embodiment use communication as a constraint.

Mailboxes

Where do actors process messages? That’s where mailboxes come in.

A mailbox is like an in-memory queue attached to each actor. Messages arrive and wait their turn. The actor works through them one at a time.

The key point here is that the mailbox is separate from the actor. The actor doesn’t manage its own queue — that’s handled by the library (like Akka in Java) or by your own implementation.

This separation seems unintuitive at first. If there is only one mailbox for each actor, why add that abstraction in the first place?

Same reason why none of our email apps support tasks: delivering and processing are two very different jobs. This separation of concerns allows the mailbox to focus on delivering the message and the actor to focus on reacting to it. Practically, it also makes scaling the system easier, as actors and their mailboxes can exist in separate containers.

Bringing this together

The actor model comes with a powerful guarantee: state can always be updated safely. The actor itself can only access its own state; other actors can only request that state by sending a message; only one message can be processed at a time.

Even though actors are constantly communicating, state is mediated through messages. Nothing gets corrupted by a parallel operation.

This is only possible thanks to the addition of communication rules. These keep nodes independent enough to do their jobs, while keeping the system trustworthy enough to grow.

The Remote Company

Imagine you’re working at an in-person company that doesn’t believe in SaaS or documentation. All decisions are verbal. When there are five of you, that works fine — it’s actually kind of fun. Once there are fifty people, everything breaks down. Shared resources get locked up. It takes forever to get people aligned. Constant interruptions undermine productivity.

Then the company reinvents itself. It goes fully remote, mandates email-only communication, and bans multi-tasking. You work through your inbox, one message at a time. Suddenly, agendas are written, conferences rooms are reserved, and everyone gets updates. It’s less chaotic fun, but things are actually getting done again.

The communication rules were needed to ensure the system could flow again.

That’s the actor model.

Takeaways

Why does this matter?

As our systems grow more complex, one of the most valuable skills we can develop is recognizing elegant simplicity. Alan Kay put it well: “You get simplicity by finding a slightly more sophisticated building block to build your theories out of.”

The actor model is exactly that kind of building block. It doesn’t introduce an impressive new algorithm or chip architecture. Its genius lives at the transport layer. We’ve learned to obsess over the computation layer’s complexity at the expense of the simple solutions in plain sight.

The better we get at recognizing this kind of structural simplicity, the better we can protect it from the slow erosion that complexity brings. And eventually, if we keep exercising this muscle, we’ll start designing more elegant models ourselves.

Forward this to a fellow actor on your team.

Bonus points if you use DDD Command syntax.

Keep Exploring

Actor Fundamentals (akka.io)

Actors: a model of concurrent computation in distributed systems (archive.org)

The Actor Model (wikipedia)

A Feature Model of Programming Languages (sciencedirect)

The Actor Model Whiteboard Session with Carl Hewitt (youtube)

The Actor Model, Behind the Scenes with XState (youtube)

Why Did the Actor Model Not Take Off? (reddit)

I find the opposite to be true.

The people who understand something most deeply are usually the ones working hardest to make it simpler.

Smart people simplify

Tim Berners-Lee understood the web as a common place for data to be shared. He could’ve flaunted his intellect by building this place with Turing-complete programming languages like Haskell or C++. But those would’ve made it harder to work with and share information. Simpler languages, on the other hand, were easier to understand and write. You can combine a handful of HTML tags to have a website. CSS won’t store your data, but a few intuitive keywords will add some visual pop.

The internet grew because its languages were simple enough for normal people to use.

Indeed, on the Web, the least powerful language that’s suitable should usually be chosen.

— Tim Berners-Lee, The Rule of Least Power (w3)

BTW - Tim documented seven principles of design for himself. Five of them are about simplicity (w3).

Frank Lloyd Wright’s architecture adhered to nine principles. There were two for eliminating the unnecessary (walls and decorations) and seven for integrating the essentials (nature, the foundation, heating, and lighting). These simple rules forced him to ‘stretch’ his design vocabulary without losing coherence. Rather than imitating the eclecticism of the times, he harmonized the essentials. The varied expression of a few pure ideas is why his designs still feel modern a century later.

To reduce the number of necessary parts of the house and the separate rooms to a minimum, and make all come together as enclosed space - so divided that light, air, and vista permeated the whole with a sense of unity.

— Frank’s first principle.

The French aviator and literary star Antoine de Saint-Exupéry shared Wright’s appreciation for unity. He believed the role of engineers was to remove and refine away until only a cohesive whole remained. He witnessed this process with his planes. During his early flying career in the 1920s, the cockpits were a chaotic mess of raw gauges, exposed levers, and instruments scattered without much visual logic. By the time he wrote his memoir in 1939, they had smoothed so much complexity that he marveled at how he now spent more time thinking about flying and less on gadgetry. Seeing this transformation undoubtedly led to his now-famous anthem of simplicity.

Have you ever thought, not only about the airplane but about whatever man builds, that all of man’s industrial efforts, all his computations and calculations, all the nights spent over working draughts and blueprints, invariably culminate in the production of a thing whose sole and guiding principle is the ultimate principle of simplicity? [...] If anything at all, perfection is finally attained not when there is no longer anything to add, but when there is no longer anything to take away, when a body has been stripped down to its nakedness.

— Wind Sand And Stars

Simplicity is not a philosophical ideal that only appeals to artists or children.

The creators of our physical and digital world recognize its power.

Make no mistake, though — the road through complexity is unglamorous and difficult.

We’ll be exploring how to navigate our way towards simple solutions in the next few Offline editions.

One thing to carry with you into the week:

» If your solution still feels complicated, you probably haven’t finished thinking yet.