How to think about scalability as a startup engineer

Big-O notation for building a company

Will it scale?

At some point, every startup engineer gets asked that question.

To answer, many instinctively reach for Big-O notation. They start by analyzing the runtime complexity of their sorting algorithms, like they learned to do in university.

Unfortunately, when people ask about startup scalability, that's usually not what they want to hear. In startup land, people want to know if your product can scale—the algorithmic complexity is at most one piece of the puzzle.

And yet, engineers can still use big-O notation to structure their answer precisely and effectively. To do that, we just need to refactor how we think about big-O notation.

What is startup engineering?

Startups are businesses that are not yet profitable. They are trying to develop themselves so that as they grow, costs increase more slowly than revenue, driving them towards profitability.

Startup engineering is engineering within that context.

Making the company financially sustainable is the ultimate goal. Who cares about the fastest way to sort a million 32-bit integers if you can reach profitability without ever needing to sort more than a dozen of them?

Startup engineers need to answer scalability questions about the products they build from the perspective of company-level scalability, not the technology's capacity for scale in isolation.

What is big-O notation?

Big-O notation is a mathematical approach computer scientists use to characterize the upper bound of a function's output as its input trends towards a particular value (typically infinity). When learning big-O notation, students consider three interconnected things at once:

1. A function
2. How much that function costs to run
3. An input of increasing size

Computer scientists use Big-O notation to categorize a function's behavior into one of several behavior families, or some combination of those families. Different sorting algorithms have different performance characteristics, and computer scientists use big-O notation to describe those characteristics.

For a typical computer science undergraduate studying merge sort, the function is the sorting algorithm, the cost of running the function is its execution time, and the input of increasing size is the list of integers that the student is trying to sort.

Big-O notation for product scalability

Startup engineers can use big-O notation as a framework for evaluating product scalability. With a bit of planning and estimation, the same core concepts can help determine whether an engineered product is headed in a valuable direction or a dangerously expensive one.

When applied to product scalability, big-O notation's three interconnected things become:

1. The product
2. How much the product costs to run
3. The startup's increasing scale

Scale is the input

Because startups are not yet profitable, they have to scale up to survive. There is no room for negotiation with this financial reality. Engineers can't automate away their startup's need for increased scale—all they can do is optimize for cost.

Thus, the startup's overall scale should be the input to the function in our repurposed big-O notation. Since "scale" is pretty abstract, startups typically choose a metric to measure it concretely.

Entire books have been written on what the scaling metric should be. For the sake of brevity, let's assume the metric is revenue.

While not the perfect scalability metric for every startup, revenue tends to be a pretty good one. As long as its products scale effectively, it is difficult for a startup to go bankrupt if its revenue is trending towards infinity.

Product is the function

It doesn't matter if something is affordable at scale if you're going to throw it away before you get there. Thus, for scalability analyses, it's best to consider problems that will last for the lifetime of the company. These are the engineering efforts whose scalability is worthy of examination.

Because of this, startup engineers should analyze the scalability of their products, not their projects.

Projects are features that get built and shipped. They have an end date.

Products, on the other hand, do not—unless the startup fails or pivots, products tend to last for years, and often, the entire life of the company.

Products can be either internal or external engineering efforts. Whether designing your CI/CD architecture, production database schema, or web app front end architecture, scalability matters. In their own ways, each of those products and more will probably need to scale if the startup is going to succeed.

Regardless of the product, when it comes to scalability at a startup, the question is the same: in the long run, will costs increase sub-linearly relative to company revenue?

Cost is the curve

Cost is how much the startup must spend to produce, maintain, and operate the product. To estimate cost, startup engineers should start from revenue targets and work backwards to predict how costs will change over time.

It's probably impossible to forecast this with complete accuracy. Product cost estimation is part of the art of startup engineering. Within the big-O notation framework, however, that's OK—the math doesn't require an exact calculation, just a rough estimate of the worst case scenario.

For most startups, engineering teams incur three types of cost—development, maintenance, and infrastructure.

Development cost

A product's development cost is the cost to build all the projects that product will require over time. For software products, this is mostly engineering hours, spent in bursts over time as engineers work on each individual project over the product's lifetime.

Some projects to improve a product are customer-facing features. Startups take these projects on to win new business and keep existing customers happy, so the startup can grow. Is your customer base going to demand a long list of third party integrations as your product grows? Add that cost to the curve.

Other projects are internal improvements, necessitated by increasing company scale. Customers don't see these improvements, but they're necessary for the system to continue functioning. If your backend is a monolith, will you have to refactor once your startup's engineering team quadruples in size? Add that cost to the curve, too.

Once startup engineers have added in all the projected development costs looming in a product's future, the cost curve generally looks like a step function. The product incurs cost increases in short bursts as both internal and customer-facing projects start and end.

Maintenance cost

A product's maintenance cost is the additional development cost that it incurs over time as engineers are forced to fix bugs and service products after they're released in order to keep things functioning properly.

Engineering hours spent fixing bugs are maintenance costs. This is difficult to predict exactly, but higher overall system complexity typically correlates to more bugs. Keep things as simple as possible—and when your designs get complicated, account for that in your product's cost projection.

Manual toil designed into the system also contributes significantly to maintenance cost for many startup products. This is often overlooked, but if you keep an eye out for it, it's easy to quantify. Are there manual operations that engineers need to perform regularly in order to keep the product running? How long do they take, and how much do engineers cost per hour?

Infrastructure cost

Infrastructure cost is the cost incurred for the underlying resources necessary to run the product. This is usually the easiest of the three types of costs to compute, and the one engineers are typically most comfortable estimating.

The rise of public cloud infrastructure has made infrastructure cost curve projections remarkably easy to quantify. How much will the app cost to run per month, once you deploy it in AWS? As company scale increases, how much more will you have to spend in AWS in order to keep up with however much additional load that additional scale places on the application?

How to answer the scalability question

Once you've estimated cost curves for these three categories, the rest of the analysis is straightforward. If you sum the curves, what does the overall cost function look like?

This is where big-O notation shines. The estimate doesn't need to be exact—it's the functional behavior of its upper bound that answers the scalability question. It's a simple judgment—the product's cost has to increase sub-linearly relative to company revenue.

With N equal to revenue, is the product's cost curve below O(N)? Then it scales. Otherwise, it does not.

Startup scalability calculus

Some products are more important than others, and no startup can afford to build every idea it dreams up. But while startups often "do things that don't scale", the startup itself can't be one of those "things". Early stage sales, marketing, and customer relationship-building strategies don't need to scale, but in the long run, the company has to.

This calculus sits at the center of any startup's existence. If a startup's overall cost curve increases at or above a linear rate relative to revenue, it cannot scale its way to profitability. Startup engineering teams should leverage this basic mathematical understanding of scalability when architecting the products they build for their company.