Healthchecks.io Now Uses Self-Hosted Object Storage

Self Hosted object storage looks neat!

For this project, where you have 120GB of customer data, and thirty requests a second for ~8k objects (0.25MB/s object reads), you’d seem to be able to 100x the throughput vertically scaling on one machine with a file system and an SSD and never thinking about object storage. Would love to see why the complexity

I'm sure it's a lot better now but everytime I see btrfs I get PTSD.

I don't get it, if it's running on the same (mentioning "local") machine, why does it even need the S3 API? Could just be plain IO on the local drive(s)

Self Hosted object storage looks neat!

The complexity for that is almost always for redundancy and for ease of deploys.

I'm sure it's a lot better now but everytime I see btrfs I get PTSD.

I'd worry about file create, write, then fsync performance with btrfs, but not about reliability or data-loss.

But a quick grep across versitygw tells me they don't use Sync()/fsync, so not a problem... Any data loss occurring from that is obviously not btrfs fault.

I don't get it, if it's running on the same (mentioning "local") machine, why does it even need the S3 API? Could just be plain IO on the local drive(s)

The complexity for that is almost always for redundancy and for ease of deploys.

I'd worry about file create, write, then fsync performance with btrfs, but not about reliability or data-loss.

But a quick grep across versitygw tells me they don't use Sync()/fsync, so not a problem... Any data loss occurring from that is obviously not btrfs fault.

The S3 API doesn't work like normal filesystem APIs.

Part of it is that it follows the object storage model, and part of it is just to lock people into AWS once they start working with it.

seperate machine I think given the quoted point at the end:

> The costs have increased: renting an additional dedicated server costs more than storing ~100GB at a managed object storage service. But the improved performance and reliability are worth it.

So you don't need to refactor your code?

The S3 API doesn't work like normal filesystem APIs.

Part of it is that it follows the object storage model, and part of it is just to lock people into AWS once they start working with it.

Apart from all these other products that implement s3? MinIO, Ceph (RGW), Garage, SeaweedFS, Zenko CloudServer, OpenIO, LakeFS, Versity, Storj, Riak CS, JuiceFS, Rustfs, s3proxy.

What kind of vendor lock-in do you even talk about. Their API is public knowledge, AWS publishes the spec, there are multiple open source reference client implementations available on GitHub, there are multiple alternatives supporting the protocol, you can find writings from AWS people as high in hierarchy as Werner Vogels about internals. Maybe you could say that some s3 features with no alternative implementation in alternative products are a lock-in. I would consider it a „competitive advantage”. YMMV.

I'm 100% aware of how S3 works. I was questioning why the S3 API is needed when the service is using local storage.

> part of it is just to lock people into AWS once they start working with it.

This is some next-level conspiracy theory stuff. What exactly would the alternative have been in 2006? S3 is one of the most commonly implemented object storage APIs around, so if the goal is lock-in, they're really bad at it.

Apart from all these other products that implement s3? MinIO, Ceph (RGW), Garage, SeaweedFS, Zenko CloudServer, OpenIO, LakeFS, Versity, Storj, Riak CS, JuiceFS, Rustfs, s3proxy.

seperate machine I think given the quoted point at the end:

> The costs have increased: renting an additional dedicated server costs more than storing ~100GB at a managed object storage service. But the improved performance and reliability are worth it.

Years of serious corruption bugs.

I'm 100% aware of how S3 works. I was questioning why the S3 API is needed when the service is using local storage.

Sometimes API compatibility is an important detail.

I've worked at a few places where single-node K8s "clusters" were frequently used just because they wanted the same API everywhere.

So you don't need to refactor your code?

And when/if you decide to head back to a 3rd party it requires no refactoring again.

> part of it is just to lock people into AWS once they start working with it.

Sometimes API compatibility is an important detail.

I've worked at a few places where single-node K8s "clusters" were frequently used just because they wanted the same API everywhere.

And when/if you decide to head back to a 3rd party it requires no refactoring again.

yeah, sure, those 5-10 different API calls would surely be a huge toll to refactor... I'd rather run an additional service to reimplement the S3 API mapping to my local drive /s

> What exactly would the alternative have been in 2006?

Well, WebDAV (Document Authoring and Versioning) had been around for 8 years when AWS decided they needed a custom API. And what service provider wasn't trying to lock you into a service by providing a custom API (especially pre-GPT) when one existed already? Assuming they made the choice for a business benefit doesn't require anything close to a conspiracy theory.

And it worked as a moat until other companies and open source projects started cloning the API. See also: Microsoft.

Years of serious corruption bugs.

Gluster was that for me

yeah, sure, those 5-10 different API calls would surely be a huge toll to refactor... I'd rather run an additional service to reimplement the S3 API mapping to my local drive /s

> What exactly would the alternative have been in 2006?

And it worked as a moat until other companies and open source projects started cloning the API. See also: Microsoft.

Gluster was that for me

Ah, another one! Yep, also same, before ceph days at least (although I've had my own, albeit self-inflicted, nightmare there too).

Yup, still get nightmares about glusterfs.... still have one customer running on it.

Ah, another one! Yep, also same, before ceph days at least (although I've had my own, albeit self-inflicted, nightmare there too).

Yup, still get nightmares about glusterfs.... still have one customer running on it.

I heard it got better, but we ran into the BOTF (billions of tiny files) issue around 2016. (For a genealogy startup this was a serious issue)

Healthchecks.io ping endpoints accept HTTP HEAD, GET, and POST request methods. When using HTTP POST, clients can include an arbitrary payload in the request body. Healthchecks.io stores the first 100kB of the request body. If the request body is tiny, Healthchecks.io stores it in the PostgreSQL database. Otherwise, it stores it in S3-compatible object storage. We recently migrated from a managed to a self-hosted object storage. Our S3 API is now served by Versity S3 Gateway and backed by a plain simple Btrfs filesystem.

The Managed Options

In 2022, while implementing ping request body offloading to object storage, I was evaluating which object storage provider to use.

AWS S3 has per-request pricing, which would make it expensive-ish for Healthchecks.io usage patterns (frequent PutObject S3 operations, one operation per every large-enough ping request). Also, AWS being subject to the CLOUD Act, Healthchecks.io would need to encrypt data before handing it off to AWS, which would add complexity.

OVHcloud is what I picked initially. There are no per-request fees, OVHcloud is an EU company, and the performance seemed good. Unfortunately, over time, I saw an increasing amount of performance and reliability issues. As my experience got worse and worse, I looked for alternatives.

In 2024, I migrated to UpCloud. Same as OVHcloud, it has no per-request fees and is an EU company. There was a clear improvement in the quality of service: the S3 operations were quicker, and there were fewer server errors or timeouts. Unfortunately, over time, the performance of UpCloud object storage deteriorated as well. There were periods where all operations would become slow and hit our timeout limits. The S3 DeleteObjects operations in particular were getting slower and slower over time. So I looked for alternatives again, including self-hosted.

Requirements

Our current (April 2026) object usage is:

14 million objects, 119GB
Object sizes range from 100 bytes to 100’000 bytes. The average object size is 8KB.
30 upload operations per second on average, with regular spikes to 150 uploads/second.
Constant churn of uploaded/deleted objects.

Our candidate object storage system would need to be able to support this usage and have room to grow. Luckily, we are still at the scale where everything can easily fit on a single system, and operations like taking a full backup can be reasonably quick. Everything would be more complicated if we had many-terabyte requirements.

Availability and durability: for the Healthchecks.io use cases, the object storage is not as mission-critical as our primary data store, the PostgreSQL database. If the database goes down, the service is completely broken, and monitoring alerts stop going out. If the object storage goes down, then users cannot inspect ping bodies through the web interface or through the API, but the system otherwise still functions. If some ping bodies get permanently lost, that is bad, but not as bad as losing any data going into the PostgreSQL database.

Latency: the quicker, the better. There are places in code where Healthchecks.io does S3 operations during the HTTP request/response cycle. Individual S3 operations taking multiple seconds could choke the web server processes. While using UpCloud, I had to add some load-shedding logic to prevent slow S3 operations from escalating into bigger issues.

The Self-Hosted Options

I ran local experiments with Minio, SeaweedFS, and Garage. My primary objection to all of them was the operational complexity. It is not too hard to follow the “get started” instructions and get a basic cluster up and running. But, for a production-ready setup, I would need, as a minimum:

automate the setup of the cluster nodes,
learn and test the update procedure,
learn and test the procedure of replacing a failed cluster node,
set up monitoring and alerting for cluster-specific health issues.

Since I’m a one-person team, and I already run self-hosted Postgres, self-hosted HAProxy load balancers, and self-hosted email, I would really like to avoid taking up the responsibility of running another non-trivial system. Something simple would be much preferred.

Versity S3 Gateway

Versity S3 Gateway turns your local filesystem into an S3 server. An S3 PutObject operation creates a regular file on the filesystem, an S3 GetObject operation reads a regular file from the filesystem, and an S3 DeleteObject operation deletes a file from the filesystem. It does not need a separate database for metadata storage. You can use any backup tool to take backups. The upgrade procedure is: replace a single binary and restart a systemd service. It is written in Go, and is being actively developed. The one bug I found and reported was fixed in just a few days.

The big obvious caveat with Versity S3 Gateway and the filesystem as the backing store is, of course, availability and durability. The objects live on a single system, which can fail at any point of time without any prior warning. I need to be ready for this scenario.

The Setup

In March 2026, I migrated to self-hosted object storage powered by Versity S3 Gateway.

S3 API runs on a dedicated server. It listens on a private IP address. Application servers talk to it over Wireguard tunnels.
Objects are stored on the server’s local drives (two NVMe drives in RAID 1 configuration).
Objects are stored on a Btrfs filesystem. With Btrfs, unlike ext4, there is no risk of running out of inodes when storing lots of tiny files.
Every two hours, a rsync process synchronizes the added and deleted files to a backup server.
Every day, the backup server takes a full backup, encrypts it, and stores it off-site. We keep full daily backups for the last 30 days.

With this setup, if both drives on the object storage server fail at the same time, the system could lose up to 2 hours of not yet backed-up ping request bodies. This can be improved, as usual, with the cost of extra complexity.

The Results

After switching to self-hosted object storage, the S3 operation latencies dropped:

The queue of ping bodies waiting to be uploaded to object storage shrank:

There have been no availability issues yet, but the new system has been live for only a couple of weeks.

The list of our data sub-processors now has one less entry.

The costs have increased: renting an additional dedicated server costs more than storing ~100GB at a managed object storage service. But the improved performance and reliability are worth it.

I am cautiously optimistic about the new system, and I think it is an improvement over the old one. But I am also open to migrating again if I find a system with better tradeoffs.

Thanks for reading, and happy monitoring,
–Pēteris

Hacker Times