Your next startup: Immutable Linux as a service

Immutability.

For software engineers managing Linux systems, this is the North star. DevOps and Site Reliability Engineers will talk your ear off about it.

Unfortunately, Linux is not at all immutable by default. In fact, when you consider all of the codebases that compile together to produce its binaries, a typical Linux root file system is probably the most mutable configuration space most developers encounter in their careers.

At scale, Linux's lack of immutability is a problem. Tech companies are willing to spend billions per year to fix it.

Any startup that can earn a slice of that revenue stream as a new flavor of "immutability provider" will have struck gold: and all they'd have to do to out-compete their rivals is offer a better immutability solution for bare metal data centers.

What is immutable Linux?

As many of my fellow infrastructure engineers will happily explain, the "immutability" we're all looking for is a misnomer. After all, we do need to be able to deploy new versions of our software from time to time.

Instead, the real definition of "immutable" in Linux systems administration is a bit more nuanced. In different contexts, and from different people, you might hear things like:

• "We want our server infrastructure to be composed of disposable instances"
• "We need our Linux distributions to be reprovisionable, with anti-hysteresis mechanisms"
• "We want our deployment systems to be what we CRAVE"

Essentially, we want to be able to:

• completely version our Linux operating system's root file system.
• update Linux's file system contents atomically, safely, and deliberately in production.
• wipe the whole root file system clean and start over from our starting state whenever necessary.

Why do companies want immutable Linux?

Mutable Linux systems present a very large configuration space. Linux users can change files all across the file system, reconfiguring or installing all sorts of new behavior in the process. At large scales, with hundreds or thousands of hosts, this flexibility becomes less of an asset and more of a liability.

Having such a large mutable space that users can change at will in such a granular fashion makes it very difficult to track changes or standardize software and configuration versions. This is a reliability nightmare.

Linux's inherent mutability also presents serious challenges for the sorts of software security efforts often required by compliance regimes. The larger the space where changes can be made in production systems, the larger the risk surface for unpatched software vulnerabilities and malicious software.

As the mutable space grows and scales, companies end up needing increasingly complex tools and infrastructure to manage their internal operations. These tools, and the engineering know-how it takes to implement, deploy, and maintain them, are very expensive.

Adopting immutable Linux lowers operating costs for modern software companies as they scale.

How do companies get immutable Linux today?

One way or another, companies end up spending lots of money to wrangle Linux's naturally mutable nature. Due to the difficult and specialized nature of the work, most companies pay another business to make Linux immutable on their behalf.

For a prospective startup, this is good news! Companies are already in the habit of shelling out lots of dough to solve this problem. Today, they do this in two main ways: by paying for either private cloud or public cloud solutions.

Either they pay for public clouds...

Public clouds, like AWS, GCP, and Azure, are the most common ways for companies to acquire immutable infrastructure. You don't have to implement your infrastructure immutably on these platforms, but it is considered a best practice to do so.

And it's easy to do: all three major public clouds have virtual machine platforms that let you manage your Linux operating system in an immutable fashion. You can take image snapshots of your disk volumes in the management console, and can then clone, replace and destroy identical-looking Linux machines on a whim.

But all this convenience comes with a notoriously high price tag. AWS makes over half its revenue from EC2 instances alone. Their customers regularly feel the pain of those cloud costs—there's an entire mini industry of small businesses dedicated to helping businesses lower their public cloud spend.

... or they build private clouds

Private clouds are like AWS, but hosted in-house, exclusively to serve the company that built them. Before public clouds appeared on the scene ten or fifteen years ago, building your own private cloud was the only game in town. Even today, private clouds remain popular, especially within large enterprises.

One of the most popular ways to run immutable Linux systems in a private cloud is to buy VMWare licenses and run VMWare's VSphere hypervisor on top of your own hardware. With this model, you can use the virtualization layer to abstract away the base operating system, running all your software in virtual machines instead.

Another, increasingly popular way to run immutable Linux on a private cloud is to run specialized Linux distributions like CoreOS to push as much mutable complexity out of the root file system and up into containers as possible. You can also use bare metal deployment automation like MaaS or Tinkerbell to redeploy the entire operating system on a whim.

But both of these private cloud implementation strategies are expensive. With VMWare, you pay a fortune for their software licenses. With open source solutions, you pay dearly, too: on the salaries you'll ultimately need to pay for new engineering teams to deploy and scale the "free" software.

What about companies that don't make Linux immutable?

Of course, there is a third option for software companies today: don't pursue immutable Linux at all. These companies just carry on with bog standard mutable Linux and hope for the best.

These days, this usually isn’t done out of a lack of recognition for the value of immutability. Instead, many companies don't pursue immutability simply because they can't afford to.

These companies can't pay the premium to run on AWS, so they end up running their software in co-located data centers. For that same reason, they don't have the engineering resources internally to acquire private cloud style immutability either.

Eventually, if these companies are successful, they end up needing to invest their hard-won, robust revenue streams in making their systems as immutable as possible after the fact.

As anyone involved in one of these immutability retrofits can attest, they are often much more expensive to complete than it would have cost to embrace immutability up front in the first place.

That's where your startup comes in.

How can your startup help?

Your startup should sell software that runs on top of your customers' existing co-located server hardware. It should provide a way to install Linux immutably, letting your customers manage their co-located servers like virtual machines.

To do this, pick one of the open source offerings for providing Linux immutability and master its operation. Instead of building a self-service platform from scratch, start by offering it as a managed service.

As you win new business during the early phases of your company, you'll have to get your tech stack of choice working on each customers' chosen hardware. This will be tedious, but worthwhile—eventually, existing support for a reasonably large selection of common server hardware will become part of your company's moat.

As far as pricing goes, you don't have to be cheap—you just have to be cheaper than your expensive competition. With this product model, that should be easy—your customers get immutable Linux while continuing to save money by bringing their own hardware instead of renting it from AWS.

As your managed service offering achieves that market position, begin to develop and market a proprietary self-service offering on top of it. Over time, you can develop what will grow into a much more scalable product offering—one which requires minimal day-to-day support for customer operations and lends itself nicely to a software licensing model.

If you execute well, your offering should become one of the most commonly compatible ways to spin up disposable, immutable Linux installations on bare metal servers. An open-core software model might help speed up this development and adoption process.

In other words, in the long run you're trying to build the next VMware—but this time, without the VMs.

Would this startup idea work?

I don't know—but this idea has a few things going for it.

First of all, it has lots of potential customers who have already demonstrated a willingness to spend lots of money for solutions to the problem the startup is trying to solve. That helps.

Second, this startup idea scales incrementally, and ought to need relatively little initial funding. The bootstrapping-style business growth model increases your odds of success—it gives you a climb that looks more like a staircase than a cliff, with more chances to learn from customers and iterate on your product along the way.

Lastly, the space itself is growing rapidly. Mutable Linux in co-locations is commonplace—and co-location giant Equinix alone did $7.2B in revenue in 2022. And they are growing quickly, as are their competitors. We have not yet reached peak demand for data center capacity.

But of course, in the end no one can actually know for certain if this startup could succeed. You'll have to found it yourself to find out.