Little's Law: The Formula Behind Every Good System Design Interview

You're in a system design interview and you're asked to design a banking application.

Customers can:

• Login

• View accounts

• Check transactions

• Transfer money

• Download statements

The interviewer adds one detail:

"We have 1M customers."

Most candidates hear that and start reaching for servers, load balancers, and database shards. That instinct is understandable, and it's wrong.

Here's the thing: 1M users is almost meaningless as a design input. What matters is the workload those users generate: how many are active, how frequently they make requests, how long those requests take, and what resources they consume.

One of the tools that helps translate that workload into something measurable is a 60-year-old equation with three variables: Little's Law.

---

The "1M Users" Trap

The interviewer asks:

"Our system has 1M users. Can you design a system that supports them?"

Your first question shouldn't be:

"How many servers do we need?"

Instead, ask:

"How many of those users are active at the same time?"

Because:

1M registered users ≠ 1M active users ≠ 1M concurrent users

Suppose the interviewer clarifies:

"At peak time, we have 100K concurrent users."

That's useful, but concurrent users still don't tell us how much traffic the backend receives.

Users spend most of their time reading, thinking, scrolling, typing, or waiting. They don't continuously send requests.

Therefore:

100K concurrent users ≠ 100K requests/sec

The next question is:

"How frequently do those users make requests?"

That leads us to RPS.

---

From Users to RPS

RPS means Requests Per Second.

Suppose the interviewer says:

"At peak time, the application receives 20K requests per second."

Now we have 100K concurrent users generating 20K RPS.

These are different dimensions of the workload.

And we care about peak RPS, not just average traffic. A banking system might normally receive 5K RPS but reach 20K RPS during predictable traffic surges.

A simplified traffic pattern might look like:

RPS
 ^
 |                                  /\
 |                                 /  \
 |                                /    \
 |                               /      \       Peak RPS
 |                              /        \      ≈ 20K
 |                             /          \
 |                            /            \
 |                           /              \
 |__________________________/                \________________
 |       Normal traffic ≈ 5K RPS
 |
 +----------------------------------------------------------------> Time
   Midnight    Morning       Noon       Afternoon       Night
                              Salary Day

A simple planning target is:

Target capacity ≈ Expected peak × Safety factor

For example:

$$20K \times 1.5 = 30K \text{ RPS}$$

The safety margin is illustrative. The right amount depends on growth, failures, scaling behavior, and cost.

But even knowing the peak RPS doesn't tell us how much infrastructure we need.

Why?

Because requests have a lifetime.

---

Little's Law: From RPS to Concurrency

Suppose requests arrive at 20K RPS and spend 100 ms on average inside the system.

A request enters the system, consumes resources while it is processed, and eventually leaves. While it is being processed, it is in flight.

This is where Little's Law becomes useful.

Little's Law is commonly written as:

$$L = \lambda W$$

Where:

• L = Average number of items in the system
• λ = Average arrival rate
• W = Average time spent in the system

For an HTTP application:

• L = Average concurrent requests
• λ = Requests per second
• W = Average request latency in seconds

Therefore:

$$\text{Average concurrent requests} = \text{RPS} \times \text{Average latency}$$

Let's apply it. With 20K RPS and 100 ms (0.1 seconds) average latency:

$$L = 20{,}000 \times 0.1 = 2{,}000$$

So, on average, approximately 2K requests are in flight.

Concurrent Users vs. RPS vs. Concurrent Requests

People mix these three up all the time, but they're really not the same thing at all.

	Concurrent Users	RPS	Concurrent (In-Flight) Requests
What it measures	People actively using the app right now	Rate at which requests arrive at the backend	Requests currently being processed inside the system
Unit	Users	Requests / second	Requests
Driven by	Product engagement, session length	User behavior × concurrent users	RPS × average latency
Formula	— (measured directly)	Requests / time window	\(L = \lambda W\)
In our example	100K	20K	2K
What increases it	More people opening the app	More frequent actions per user, or more users	Slower backend (higher latency), not just more traffic
What it tells infrastructure	Roughly bounds session/auth load	Bounds request-handling throughput needed	Bounds how many threads, connections, and memory slots must be held at once

A useful mental model for how the three connect:

Concurrent Users → (activity rate) → RPS → (× latency) → Concurrent Requests

Mixing these up is a classic mistake. Sizing your connection pool off the user count, or your server fleet off concurrent users instead of concurrent requests, can throw your whole capacity plan off. In our example these number::chatgpt-content-reference{index="0"}s are worlds apart (100K vs. 20K vs. 2K), and each one drives a different part of the architecture.

---

Why latency changes everything

Now imagine the database becomes slow.

The traffic hasn't changed. RPS remains 20K.

But average latency increases from 100 ms to 1 second.

Little's Law gives us:

$$L = 20{,}000 \times 1 = 20{,}000$$

The average number of in-flight requests increases 10X.

Latency isn't only a user-experience problem. It also determines how much work remains inside the system.

Real systems also have tail latency. Slow requests can hold threads, execution slots, database connections, and memory for longer, and can trigger timeouts and retries.

Before going further, it's worth pausing on what "average latency" actually hides.

When people talk about latency, they often report percentiles rather than a single average. A percentile tells you how many requests fall below a given latency value.

• p95 latency means 95% of requests completed faster than this value (5% were slower).
• p99 latency means 99% of requests completed faster than this value (only 1% were slower).

Percentiles matter because averages hide outliers. A system can have a low average latency while still leaving a meaningful slice of users with a painfully slow experience. In a banking application, that slow 1% might be exactly the money transfers you care most about getting right.

This distinction matters for Little's Law too, because the law depends on the mean time in system, not a percentile.

---

From In-Flight Requests to Capacity

What do those in-flight requests consume?

Requests
   │
   ▼
Application
   │
   ├── CPU
   ├── Memory
   ├── Threads / execution slots
   ├── DB connections
   ├── Network bandwidth
   └── Downstream resources

Suppose load testing shows that one application instance can safely handle 2K RPS while maintaining <200 ms p95 latency for the tested workload.

Our expected peak is 20K RPS and with headroom the target capacity becomes 30K RPS, as calculated earlier.

Under those benchmark conditions:

$$\frac{30K}{2K} = 15 \text{ instances}$$

But this result depends on the request mix, instance size, downstream performance, and resource limits.

And not all requests cost the same.

A balance lookup might be a simple cached read. A money transfer may involve authentication, validation, account lookups, a transaction, ledger updates, auditing, and event publishing.

Therefore: 1 RPS of balance requests ≠ 1 RPS of money transfers

RPS tells us how much traffic arrives. It does not tell us how expensive that traffic is.

For realistic capacity planning, we need the workload mix.

More importantly, money transfers require correctness guarantees such as atomic updates, idempotency, concurrency control, and durable transaction records.

A system that processes more transfers per second but occasionally transfers money twice is not a successful banking system.

---

When Capacity Is Exceeded

Every component has a processing limit.

Suppose a queue receives 10K jobs per second, but its consumers can process only 5K jobs per second.

The backlog grows:

10K jobs/sec
     │
     ▼
  ┌───────────┐
  │   Queue   │
  │ █████████ │
  └─────┬─────┘
        │
        ▼
     5K jobs/sec

As long as arrival rate exceeds processing capacity, the queue continues to grow.

Arrival rate > Processing capacity
        ↓
     Queue grows
        ↓
   Waiting time grows
        ↓
    Latency grows

This is where backpressure matters.

A healthy system should avoid allowing unlimited work to accumulate in a component that cannot keep up. Depending on the workload, that can mean bounded queues, rate limiting, load shedding, consumer scaling, request throttling, or carefully designed retries.

The important point is simple:

If a component cannot process work as quickly as it arrives, adding more work only makes the backlog worse.

This also explains why a slow dependency can become a system-wide problem.

Database slows down
        ↓
Request latency increases
        ↓
Requests remain in flight longer
        ↓
Connections and execution slots stay occupied
        ↓
Queues grow
        ↓
Requests wait longer
        ↓
Latency increases further

This can lead to connection-pool exhaustion, timeouts, retry storms, and cascading failures.

---

The Bottleneck Determines Capacity

Consider a simplified architecture:

                   30K RPS
                      │
                      ▼
                Load Balancer
                      │
             ┌────────┼────────┐
             ▼        ▼        ▼
          API #1   API #2    API #N
             │        │        │
             └────────┼────────┘
                      │
                ┌─────┴─────┐
                ▼           ▼
              Cache      Database
                            │
                       ┌────┴────┐
                       ▼         ▼
                    Primary    Replica

Suppose testing gives us:

Component	Tested capacity
Load balancer	50K RPS
Application instance	2K RPS
Application Tier	30K RPS
Cache	100K operations/sec
Database	15K queries/sec

The application tier may handle 30K RPS, and the cache may handle 100K operations per second.

But if the database cannot sustain the required workload, adding more application servers won't solve the problem.

It may simply send more work toward the database.

Capacity planning is ultimately about finding and managing bottlenecks.

---

The Interview Approach

When the interviewer says:

"Our system has 1M users."

Don't immediately choose servers.

Walk through the problem:

1M users
   ↓
How many are active?
   ↓
How frequently do they make requests?
   ↓
What's the peak RPS?
   ↓
What's the request mix?
   ↓
What's the average latency?
   ↓
How many requests are in flight?
   ↓
What resources does each request consume?
   ↓
What is the capacity of each component?
   ↓
Where is the bottleneck?
   ↓
How much headroom do we need?

This is a much stronger approach than:

"1M users? Let's deploy 20 servers."

---

The Real Lesson

Scalability doesn't begin with choosing more servers.

It begins with understanding the workload.

A system with 1M registered users might need relatively little infrastructure if only a small fraction are active. A system with far fewer users might need substantial capacity if those users generate frequent, expensive requests.

The goal is to translate vague requirements into measurable quantities:

• How much traffic arrives?

• When does it peak?

• How long do requests take?

• How much work is in flight?

• What resources does that work consume?

• Which component limits throughput?

• How much capacity is needed for growth and failures?

Little's Law gives us a powerful starting point:

$$L = \lambda W$$

But the equation is only useful when we define the system boundary, measure the right workload, and understand its assumptions.

The goal of capacity planning isn't to guess how many servers to deploy. It's to understand what limits the system, measure that limit, and design enough capacity to meet the workload reliably.