A Disk Usage Alert Led Me Down the OpenStack Rabbit Hole
An engineer's journey through OpenStack Swift, clustering, and cloud architecture.
It Started With a Disk Usage Alert
While going through our monitoring dashboards, I noticed that one of our backup object storage servers had been experiencing high disk usage for quite some time. Since the issue had remained unresolved, I decided to take ownership of the investigation.
Our existing disaster recovery setup consisted of two independent single-node OpenStack Swift deployments. Instead of native replication, data was periodically copied from the primary server to the backup server using a custom synchronization process, where objects were downloaded, staged on the backup server, and then uploaded back into its own Swift instance.
After reviewing the existing architecture and previous discussions, I found that several approaches had already been proposed. These included using rsync or rclone for more efficient synchronization, inotifywait for near real-time replication, and even introducing a Virtual IP (VIP) with Keepalived to improve failover. Each proposal addressed a specific operational challenge and aimed to improve disaster recovery.
The more I read, however, the more I realized that we were continuously adding new components and operational logic around keeping two completely independent object storage systems in sync.
Rather than deciding which approach to implement, I stepped back and asked a simpler question:
There had to be a simpler way!
That question eventually led me to explore OpenStack Swift's native clustered architecture, where many of these challenges are already solved by design.
Discovering OpenStack Swift
Swift wasn't designed as two independent servers periodically copying data between each other. It was designed as a distributed object storage system, where multiple servers work together as a single storage cluster.
The more I read, the more one question kept coming back:
Why weren't we using the architecture it was originally designed for?
I don't know the answer. Every organization has historical decisions, constraints, deadlines, and trade-offs.
But discovering the intended architecture was one of those rare engineering moments where you stop and think:
"Wait... this already exists?"
How a Swift Cluster Works
A typical Swift deployment consists of two main components:
- Proxy Servers: The entry point for all client requests. Every upload, download, and delete operation goes through a proxy.
- Storage Nodes: These store the actual account, container, and object data. A cluster can contain many storage nodes distributed across different servers, racks, or even data centers.
When a client uploads an object, it doesn't need to know which storage server should store it. It simply sends the request to a proxy server.
The proxy takes care of everything else.
Naturally, the next question is:
How does the proxy know where every object belongs?
The Ring: The Brain of Swift
The answer is the Swift Ring.
Instead of maintaining a central metadata database that records the location of every object, Swift uses a data structure called the Ring.
The Ring uses consistent hashing to determine exactly which storage devices should hold every object. Every proxy server has a copy of the Ring, so any proxy can independently determine where an object should be stored or retrieved.
For example, imagine a cluster with three storage nodes and a replication factor of three.
Client
│
▼
+--------------+
| Proxy Server |
+--------------+
│
▼
+--------------+
| Swift Ring |
+--------------+
│ │ │
▼ ▼ ▼
Node A Node B Node C
The proxy simply consults the Ring, which determines the correct storage nodes for that object.
There is no central lookup service and no single metadata server that becomes a bottleneck or a single point of failure.
Replication Is Part of the Platform
In our existing setup, replication was an external process. We had to download objects from one Swift deployment and upload them into another using custom synchronization logic.
A Swift cluster works very differently.
When an object is uploaded, Swift immediately stores multiple replicas across different storage nodes according to the replication policy defined in the Ring.
Object Upload
│
▼
Proxy Server
│
▼
Swift Ring
│
├──── Replica 1 ───► Node A
├──── Replica 2 ───► Node B
└──── Replica 3 ───► Node C
There is no dedicated "primary" storage server and no separate "backup" server.
Every replica is an equal member of the cluster, and every storage node participates in serving requests.
If one node fails, requests are automatically served from the remaining replicas.
Meanwhile, Swift's background replicator processes continuously scan the cluster and restore any missing or outdated replicas until the desired replication level is achieved again.
Replication isn't an add-on or a scheduled task.
It's built into the storage platform itself.
Looking back, I realized I had been trying to solve the problem from the wrong direction.
I was comparing rsync, rclone, inotifywait, and other synchronization strategies.
Those are all excellent tools, but they solve synchronization at the filesystem level.
OpenStack Swift solves it at the architecture level.
Instead of asking administrators to keep independent storage systems synchronized, it treats every storage node as part of a single distributed cluster, where replication, data placement, and failure recovery are built into the platform from day one.
That was probably the biggest takeaway from this entire investigation.
Putting It to the Test
Reading about the architecture wasn't enough, so I built a small OpenStack Swift cluster as a proof of concept on a few Azure virtual machines.
I configured a proxy node, multiple storage nodes, and the Swift Ring, then tested uploads, replication, and node failures. Watching the cluster automatically distribute objects and continue serving data despite individual node failures made the architecture click for me.
That Rabbit Hole Became OpenStack
Naturally, curiosity won.
What started as learning about OpenStack Swift quickly turned into learning about OpenStack itself.
That's when I realized Swift is just one service in a much larger ecosystem.
OpenStack isn't simply an object storage solution. It's an open-source cloud platform that provides the building blocks needed to build your own private or public cloud, much like the services offered by AWS or Azure.
Many of its core projects have direct counterparts in the public cloud world:
| OpenStack | Purpose | AWS Equivalent | Azure Equivalent |
|---|---|---|---|
| Nova | Virtual machines | Amazon EC2 | Azure Virtual Machines |
| Neutron | Networking | Amazon VPC | Azure Virtual Network |
| Swift | Object storage | Amazon S3 | Azure Blob Storage |
| Cinder | Block storage | Amazon EBS | Azure Managed Disks |
| Keystone | Identity & access management | AWS IAM | Microsoft Entra ID (Azure AD) |
| Horizon | Web management dashboard | AWS Management Console | Azure Portal |
| Heat | Infrastructure as Code | AWS CloudFormation | Azure Resource Manager (ARM) |
OpenStack vs Public Clouds
One realization that helped everything click was understanding that OpenStack isn't competing with AWS or Azure in the traditional sense.
AWS and Azure are cloud providers. You consume their services.
OpenStack is a cloud platform. You deploy and operate it yourself.
Think of it this way:
| Public Cloud | OpenStack |
|---|---|
| Rent infrastructure from a provider | Build and operate your own cloud |
| Provider manages the platform | You manage the platform |
| Pay as you go | Run on your own hardware |
| Best for public cloud workloads | Best for private, hybrid, or sovereign clouds |
Once I looked at it from that perspective, the similarities between the services made much more sense. Nova isn't trying to replace EC2 as a business. It's providing the compute building block that lets you build a cloud platform with capabilities similar to what EC2 offers.
A Surprisingly Interesting Origin Story
The story behind OpenStack is just as interesting as the technology itself.
OpenStack was launched in 2010 through a collaboration between NASA and Rackspace. Yes, that NASA.
At the time, cloud computing was taking off, with providers like AWS leading the market. Instead of building another proprietary cloud, NASA and Rackspace set out to create an open-source cloud platform that anyone could deploy, modify, and contribute to.
The project began by combining NASA's Nova compute project with Rackspace's object storage technology. Over the years, it grew into one of the world's largest open-source infrastructure projects, supported by thousands of contributors and hundreds of organizations.
Today, OpenStack powers private and public clouds worldwide and is governed by the OpenInfra Foundation, now part of the Linux Foundation, continuing its mission of making cloud infrastructure open and accessible.
The Best Part
This whole journey started because of a disk usage alert.
I wasn't trying to learn about cloud architecture.
I wasn't researching OpenStack.
I certainly wasn't planning to spend hours reading about distributed object storage.
I was simply following a problem.
Sometimes that's how the best learning happens.
You pull on one thread, and suddenly you're exploring an entire ecosystem that has been quietly solving problems for more than a decade.
What's Next?
I'm planning to explore more of OpenStack over the coming weeks, especially how its different services work together to build a complete cloud platform.
If you've never looked into it before, I'd highly recommend spending some time exploring its projects and architecture.
Who knows?
You might start by investigating a simple monitoring alert and end up learning how an entire cloud platform is built.
Just be warned: building your own cloud sounds surprisingly achievable right up until you realize you're about to recreate a small part of AWS in your spare time. That's usually when you gain a whole new level of respect for the engineers who build and operate these systems every day.
Sometimes the best learning doesn't start with a tutorial. It starts with a production issue and a bit of curiosity. Happy exploring!
