Can SQLite run on it?

> NFSv4 but it seems like nobody actually uses that

Hurry up and you might be able to adopt it before its 30th birthday!

How about CephFs?

I'm glad you noticed this, I thought this was a wildly insane thing to do. Its like the satanic inversion of 9P protocol

pjdtest result of JuiceFS -> https://juicefs.com/en/blog/engineering/posix-compatibility-...

> There we did see some file corruption creep in

Did you figure out what caused corruption? Was minio losing blocks or was juicefs corrupted even though minio was consistent?

> poor locking support (this sounds like it works better)

File locking on Unix is in general a clusterf*ck. (There was a thread a few days ago at https://news.ycombinator.com/item?id=46542247 )

> no manual fence support; a bad but common way of distributing workloads is e.g. to compile a test on one machine (on an NFS mount), and then use SLURM or SGE to run the test on other machines. You use NFS to let the other machines access the data... and this works... except that you either have to disable write caches or have horrible hacks to make the output of the first machine visible to the others. What you really want is a manual fence: "make all changes to this directory visible on the server"

In general, file systems make for poor IPC implementations. But if you need to do it with NFS, the key is to understand the close-to-open consistency model NFS uses, see section 10.3.1 in https://www.rfc-editor.org/rfc/rfc7530#section-10.3 . Of course, you'll also want some mechanism for the writer to notify the reader that it's finished, be it with file locks, or some other entirely different protocol to send signals over the network.

FUSE is full of gotchas. I wouldn't replace NFS with JuiceFS for arbitrary workloads. Getting the full FUSE set implemented is not easy -- you can't use sqlite on JuiceFS, for example.

The meta store is a bottleneck too. For a shared mount, you've got a bunch of clients sharing a metadata store that lives in the cloud somewhere. They do a lot of aggressive metadata caching. It's still surprisingly slow at times.

> * The bloody .nfs000000 files. I think this might be fixed by NFSv4 but it seems like nobody actually uses that. (Not helped by the fact that CentOS 7 is considered "modern" to EDA people.)

Unfortunately, NFSv4 also has the silly rename semantics...

JuiceFS can be scaled to hundreds of PB by design, also is verified by thousands of users in production [1].

[1] https://juicefs.com/en/blog/company/2025-recap-artificial-in...

Agree, don't recommend POSIX proxy on S3 for complex workload, like S3FS. In the design of JuiceFS, S3 is like raw disk, JuiceFS metadata engine is like partition table, compare with local file system.

Show me an operating system built on top of a browser that can be used to solve real-world problems like JuiceFS.

We developed Object Mount (formerly cunoFS) (https://www.storj.io/object-mount?hn=1) specifically to not rely on any metadata storage other than S3 AND preserve 1:1 mapping of objects to files AND support for POSIX. We have a direct mode that uses LD_PRELOAD to keep everything in userspace so no FUSE overhead.

This approach isn't right for every use case and juice might be a better fit for this sort of 'direct block store', but wanted to include it here for folks that might want something like Juice but without having to maintain a metadata store.

(Disclosure: I work at Storj that develops Object Mount)

JuiceFS metadata engine comparison -> https://juicefs.com/docs/community/metadata_engines_benchmar...

Redis is as reliable as the storage you persist it to. If you're running Redis right, it's very reliable. Not S3 reliable, though. But if you need S3 reliable, you would turn to something else.

I expect that most folks looking at this are doing it because it means:

1. Effectively unbounded storage

2. It's fast

3. It's pretty darn cheap

4. You can scale it horizontally in a way that's challenging to scale other filesystems

5. All the components are pretty easy to set up. Many folks are probably already running S3 and Redis.

It says MySQL can be used instead of Redis for the metadata

I think they should replace Redis with Valkey or even better use rocksdb.

https://github.com/tractordev/apptron ?

My criticism is of the basic architecture, not usability or fitness for a particular purpose.

If a distributed file system is useful, then a properly architectured one is 100x more useful and more performant.

OrbitDB?

AFAIU the NFSv4 protocol in principle allows implementing unlinking an open file without silly rename, but the Linux client still does the silly rename dance.

Thanks, good to see these. Looks like these only show metadata operations though. But I guess that's probably enough to extrapolate that the system is going to be roughly 2-4x slower with other metadata stores.

Why are you building an operating system on top of a browser?

Pretty cool

> 4. You can scale it horizontally in a way that's challenging to scale other filesystems

Easy to scale on RDS, along with everything else. But there’s no Kubernetes operator. Is there a better measure of “easy” or “challenging?” IMO no. Perhaps I am spoiled by CNPG.

It was definitely MinIO related, I probably should have made that clearer. We noticed that with zero fault tolerance, MinIO objects would randomly become corrupted, which MinIO would present as "you're making too many requests, please slow down". We were certainly not making too many requests.

> FUSE is full of gotchas

I want to go ahead and nominate this for the understatement of the year. I expect that 2026 is going to be filled with people finding this out the hard way as they pivot towards FUSE for agents.

> In general, file systems make for poor IPC implementations.

I agree but also they do have advantages such as simplicity, not needing to explicitly declare which files are needed, lazy data transfer, etc.

> you'll also want some mechanism for the writer to notify the reader that it's finished, be it with file locks, or some other entirely different protocol to send signals over the network.

The writer is always finished before the reader starts in these scenarios. The issue is reads on one machine aren't guaranteed to be ordered after writes on a different machine due to write caching.

It's exactly the same problem as trying to do multithreaded code. Thread A writes a value, thread B reads it. But even if they happen sequentially in real time thread B can still read an old value unless you have an explicit fence.

I am currently looking for a way to take a legacy application that uses the filesystem as it's database and needs to support locking (flock) on FreeBSD and scale it horizontally (right now we only scale vertically and rebuilding from a corrupted FS and/or re-pulling our, backup, data from S3 takes too long if we lose a machine). We investigated NFS but the FreeBSD NFS performance was 1/10th or worse than on Linux and switching to Linux is not in the cards for us right now.

Does Object Mount support file locks (flock specifically) and FreeBSD? I see some mention of FreeBSD but I can't find anything on locking.

For context, we are working with a large number of small (<10KB if not <4KB) files normally.

Is this open source? I am a happy Storj customer, would love to use it if it's open source.

Redis isn't durable unless you drastically reduce the performance.

Filesystems are pretty much by definition durable.

> Redis is as reliable as the storage you persist it to.

For a single node, if you tank performance by changing the configuration, sure. Otherwise, no, not really.

I don't get why you'd want a file system that isn't durable, but to each their own.

But how would RocksDB work with S3? It needs support for append that generic S3 buckets do not provide and for checkpoints and backup it assumes the support for hardlinks that S3 does not have at all.

I think so.

Can run MySQL and PG, but don't recommend, not good performance for production. but for temporary it's OK. Here's a case study -> https://juicefs.com/en/blog/user-stories/xiachufang-mysql-ba...

And here's a POSIX compatibility comparison with other cloud file system, like AWS EFS. https://juicefs.com/en/blog/engineering/posix-compatibility-...

> ZeroFS supports running multiple instances on the same storage backend: one read-write instance and multiple read-only instances.

Well that's a big limiting factor that needs to be at the front in any distributed filesystem comparison.

Though I'm confused, the page says things like "ZeroFS makes S3 behave like a regular block device", but in that case how do read-only instances mount it without constantly getting their state corrupted out from under them? Is that implicitly talking about the NBD access, and the other access modes have logic to handle that?

Edit: What I want to see is a ZeroFS versus s3backer comparison.

Edit 2: changed the question at the end

Respect to your work on ZeroFS, but I find it kind of off-putting for you to come in and immediately put down JuiceFS, especially with benchmark results that don't make a ton of sense, and are likely making apples-to-oranges comparisons with how JuiceFS works or mount options.

For example, it doesn't really make sense that "92% of data modification operations" would fail on JuiceFS, which makes me question a lot of the methodology in these tests.

Let's remember that JuiceFS can be setup very easily to not have a single point of failure (by replicating the metadata engine), meanwhile ZeroFS seems to have exactly that.

If I was a company I know which one I'd prefer.

Looks like the underdog beats it handily and easier deployment to boot. What's the catch?

does having to maintain the slatedb as a consistent singleton (even with write fencing) make this as operationally tricky as a third party db?

The magnitude of performance difference alone immediately makes me skeptical of your benchmarking methodology.

In large-scale metadata scenarios, JFS recommends using a distributed key-value store to host metadata, such as TiKV or FoundationDB. Based on my experience with large JFS users, most of them choose TiKV.

Disclaimer: I'm the co-founder of TiKV.

I'm betting this is on the front page today (as opposed to any other day; Juice is very neat and doesn't need us to hype it) because of our Sprites post, which goes into some detail about how we use Juice (for the time being; I'm not sure if we'll keep it this way).

The TL;DR relevant to your comment is: we tore out a lot of the metadata stuff, and our metadata storage is SQLite + Litestream.io, which gives us fast local read/write, enough systemwide atomicity (all atomicity in our setting runs asymptotically against "someone could just cut the power at any moment"), and preserves "durably stored to object storage".

> It also writes files in it's own uninterpretable format to object storage, so if you lose the metadata store, you lose your data.

That's so confusing to me I had to read it five times. Are you saying you lose the metadata, or that the underlying data is actually mangled or gone, or merely that you lose the metadata?

One of the greatest features of something like this to me would be the ability to durable even beyond JuiceFS access to my data in a bad situation. Even if JuiceFS totally messes up, my data is still in S3 (and with versioning etc even if juicefs mangles or deletes my data, still). So odd to design this kind of software and lose this property.

Did you happen to look into CephFS? CERN (folks that operate Large Hadron Collider) use it to store ~30PB of scientific data. Their analysis cluster is serving ~30GB/s reads

Nothing around content addressable storage? Has anyone used something like IPFS / Kubo in production at that kind of scale?

(for those who don't know IPFS, I find the original paper fascinating: https://arxiv.org/pdf/1407.3561)

For a proper comparison, also significant to note that JuiceFS is Apache-2.0 licensed while ZeroFS is dual AGPL-3.0/commercial licensed, significantly limiting the latter's ability to be easily adopted outside of open source projects.

What is your plan after MinIO enters maintenance mode?

yes, support more 10 options, include redis, SQL-like DB, TiKV, FoundationDB, and more. see here -> https://juicefs.com/docs/community/databases_for_metadata

According to this link:

https://vonng.com/en/pg/pgfs/

> The magic is: JuiceFS also supports using PostgreSQL as both metadata and object data storage backend! This means you only need to change JuiceFS’s backend to an existing PostgreSQL instance to get a database-based “filesystem.”

Sounds ideal for your kind of situation where a filesystem is being abused as a database :-)

Not open source. What would be your use case?

No BSD support unfortunately. Just Linux, Mac, windows.

SeaweedFS has a RocksDB metadata backend for instance.

Mind helping us all out ahead of time by expanding on what kind of gotchas FUSE is full of?

> The writer is always finished before the reader starts in these scenarios. The issue is reads on one machine aren't guaranteed to be ordered after writes on a different machine due to write caching.

In such a case it should be sufficient to rely on NFS close-to-open consistency as explained in the RFC I linked to in the previous message. Closing a file forces a flush of any dirty data to the server, and opening a file forces a revalidation of any cached content.

If that doesn't work, your NFS is broken. ;-)

And if you need 'proper' cache coherency, something like Lustre is an option.

Enabling the WAL doesn't make Redis slow. It's slower than the default, but it's still exceptionally fast.

> Filesystems are pretty much by definition durable.

Where do you think Redis persists its data to

With the wal redis runs with perfectly reasonable performance. Of course you're not going to have the performance of an in-memory only DB if you're flushing to disk on every write.

There's no vacuuming, there's no need for indexing. You can see the time complexity of most operations. Key-value operations are mostly O(1). You'll never get that kind of performance with other databases because they intentionally don't give you that granularity.

The metadata of the filesystem isn't the performance bottleneck in most cases.

ZeroFS is a single-writer architecture and therefore has overall bandwidth limited by the box it's running on.

JuiceFS scales out horizontally as each individual client writes/reads directly to/from S3, as long as the metadata engine keeps up it has essentially unlimited bandwidth across many compute nodes.

But as the benchmark shows, it is fiddly especially for workloads with many small files and is pretty wasteful in terms of S3 operations, which for the largest workloads has meaningful cost.

I think both have their place at the moment. But the space of "advanced S3-backed filesystems" is... advancing these days.

I have very limited experiences with object storage, but my humble benchmarks with juicefs + minio/garage [1] showed very bad performance (i.e. total collapse within a few hours) when running lots of small operations (torrents).

I wouldn't be surprised if there's a lot of tuning that can be achieved, but after days of reading docs and experimenting with different settings i just assumed JuiceFS was a very bad fit for archives shared through Bittorrent. I hope to be proven wrong, but in the meantime i'm very glad zerofs was mentioned as an alternative for small files/operations. I'll try to find the time to benchmark it too.

[1] https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1021

I'm always curious about the of the option space. I appreciate folks talking about the alternative s. What's yours?

Yea, that is a big caveat to ZeroFS. Single point of failure. It is like saying I can write a faster etcd by only having a single node. Sure, that is possible, but the hard part of distributed systems is the coordination, and coordination always makes performance worse.

I personally have went with Ceph for distributed storage. I personally have a lot more confidence in Ceph over JuiceFS and ZeroFS, but realize building and running a ceph cluster is more complex, but with that complexity you get much cheaper S3, block storage, and cephfs.

It’s not great UX on that angle. I am currently working on coordination (through s3, not node to node communication), so that you can just spawn instances without thinking about it.

Litestream.io is amazing. Using sqlite as a DB in a typical relational data model where objects are related mean most read then write transactions would have to one node, but if the using it for blobs as first class objects(e.g. video uploads or sensor data) which are independent probably means you can shard and scale your set up the wazoo right?

Truth. It supports very large volume, more than 100PiB and 10B files in a single volume.

Disclosure: I'm co-founder of JuiceFS

I'm not an expert in any way, but i personally benchmarked [1] juiceFS performance totalling collapsing under very small files/operations (torrenting). It's good to be skeptical, but it might just be that the bar is very low for this specific usecase (IIRC juiceFS was configured and optimized for block sizes of several MBs).

https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1021

The latency and bandwidth really isn't there for HPC.

> but I find it kind of off-putting for you to come in and immediately put down JuiceFS, especially with benchmark results that don't make a ton of sense, and are likely making apples-to-oranges comparisons with how JuiceFS works or mount options.

The benchmark suite is trivial and opensource [1].

Is performing benchmarks “putting down” these days?

If you believe that the benchmarks are unfair to juicefs for a reason or for another, please put up a PR with a better methodology or corrected numbers. I’d happily merge it.

EDIT: From your profile, it seems like you are running a VC backed competitor, would be fair to mention that…

[1] https://github.com/Barre/ZeroFS/tree/main/bench

It backs its metadata up to S3. You do need metadata to map inodes / slices / chunks to s3 objects, though.

Tigris has a one-to-one FUSE that does what you want: https://github.com/tigrisdata/tigrisfs

Sure, so the use case I have requires elastic storage and elastic compute. So CephFS really isn't a good fit in the cloud environment for that case. It would get prohibitively expensive.

As I understand it, if the metadata is lost then the whole filesystem is lost.

I think this is a common failure mode in filesystems. For example, in ZFS, if you store your metadata on a separate device and that device is destroyed, the whole pool is useless.

Why would this matter if you're just using the database?

Redis using the WAL for durable data can only be as fast as you can fsync on every write, just like every other database. The time to fsync is going to dominate the write path for Redis, and you'll sacrifice almost all of Redis's speed to achieve it. If you're running Redis in always fsync mode it I'd suggest there are better storage systems to use.

> Where do you think Redis persists its data to

This is sort of the point, Redis isn't for persistence. I mean it can, you can use the WAL, it's handy for not having to recompute when you relaunch Redis etc, but Redis is fundamentally an in-memory system and should be treated as such. It is designed for use-cases that necessitate in-memory performance, and that don't require durability.

> With the wal redis runs with perfectly reasonable performance. Of course you're not going to have the performance of an in-memory only DB if you're flushing to disk on every write.

But that's not what gets benchmarked against.

> You'll never get that kind of performance with other databases because they intentionally don't give you that granularity.

Okay but you also won't get horizontal scaling while maintaining consistency in Redis.

> The metadata of the filesystem isn't the performance bottleneck in most cases.

Okay, but they do show that other metadata stores are 2-4x slower than Redis, so it would be great to see whole-system benchmarks that use those.

Our product is Archil [1], and we are building our service on top of a durable, distributed SSD storage layer. As a result, we have the ability to: (a) store and use data in S3 in its native format [not a block based format like the other solutions in this thread], (b) durably commit writes to our storage layer with lower latency than products which operate as installable OSS libraries and communicate with S3 directly, and (c) handle multiple writers from different instances like NFS.

Our team spent years working on NFS+Lustre products at Amazon (EFS and FSx for Lustre), so we understand the performance problems that these storage products have traditionally had.

We've built a custom protocol that allows our users to achieve high-performance for small file operations (git -- perfect for coding agents) and highly-parallel HPC workloads (model training, inference).

Obviously, there are tons of storage products because everyone makes different tradeoffs around durability, file size optimizations, etc. We're excited to have an approach that we think can flex around these properties dynamically, while providing best-in-class performance when compared to "true" storage systems like VAST, Weka, and Pure.

[1] https://archil.com

Some users use JuiceFS with CephFS RADOS, as alternative with Ceph MDS.

We're looking at alternatives, I've made some previous comments on that front. Sadly MinIO was the only option with sufficient performance for this particular situation. Thankfully we're not using any MinIO-specific features, so at least the migration path away is clear.

It wasn't my job so I didn't look into this fully, but the main issue we had was clients claiming that files didn't exist when they did. I just reread the NFS man page and I guess this is the issue:

> To detect when directory entries have been added or removed on the server, the Linux NFS client watches a directory's mtime. If the client detects a change in a directory's mtime, the client drops all cached LOOKUP results for that directory. Since the directory's mtime is a cached attribute, it may take some time before a client notices it has changed. See the descriptions of the acdirmin, acdirmax, and noac mount options for more information about how long a directory's mtime is cached.

> Caching directory entries improves the performance of applications that do not share files with applications on other clients. Using cached information about directories can interfere with applications that run concurrently on multiple clients and need to detect the creation or removal of files quickly, however. The lookupcache mount option allows some tuning of directory entry caching behavior.

People did talk about using Lustre or GPFS but apparently they are really complex to set up and maybe need fancier networking than ethernet, I don't remember.

It depends on what level of FUSE you're working with.

If you're running a FUSE adapter provided by a third party (Mountpoint, GCS FUSE), odds are that you aren't going to get great performance because it's going to have to run across a network super far away to work with your data. To improve performance, these adapters need to be sure to set fiddly settings (like using Kernel-side writeback caching) to avoid the penalty of hitting the disk for operations like write.

If you're trying to write a FUSE adapter, it's up to you to implement as much of the POSIX spec that you need for the programs that you want to run. The requirements per-program are often surprising. Want to run "git clone", then you need to support the ability to unlink a file from the file system and keep its data around. Want to run "vim", you need the ability to do renames and hard links. All of this work needs to happen in-memory in order to get the performance that applications expect from their file system, which often isn't how these things are built.

Regarding agents in particular, I'm hopeful that someone (which is quite possibly us), builds a FUSE-as-a-service primitive that's simple enough to use that the vast majority of developers don't have to worry about these things.

I replaced a GlusterFS cluster with JuiceFS some years ago and it's been a relief. Just much easier to manage.

> The benchmark suite is trivial and opensource.

The actual code being benchmarked is trivial and open-source, but I don't see the actual JuiceFS setup anywhere in the ZeroFS repository. This means the self-published results don't seem to be reproducible by anyone looking to externally validate the stated claims in more detail. Given the very large performance differences, I have a hard time believing it's an actual apples-to-apples production-quality setup. It seems much more likely that some simple tuning is needed to make them more comparable, in which case the takeaway may be that JuiceFS may have more fiddly configuration without well-rounded defaults, not that it's actually hundreds of times slower when properly tuned for the workload.

(That said, I'd love to be wrong and confidently discover that ZeroFS is indeed that much faster!)

Ceph is more something you build your own cloud with than something you run on someone else's cloud.

Yes, I'm working in the space too. I think it's fine to do benchmarks, I don't think it's necessary to immediately post them any time a competitor comes up on HN.

I don't want to see the cloud storage sector turn as bitter as the cloud database sector.

I've previously looked through the benchmarking code, and I still have some serious concerns about the way that you're presenting things on your page.

metadata backup is very important. don't forget.

It doesn’t, you are free to use ZeroFS for commercial and closed source products.

FUSE generally has low overall performance because of an additional data transfer process between the kernel space and user space, which is less than ideal for AI training.

Ceph. The answer is always Ceph.

I did set up GPFS tadam... almost exactly 20 years ago. I wouldn't say it absolutely required fancy networking (infiniband) or was extraordinary complex to set up, certainly on par with NFS when you hit its quirks (which was the reason we went off experimenting with gpfs and whatnot).

> you need to support the ability to unlink a file from the file system and keep its data around. Want to run "vim", you need the ability to do renames and hard links

Those seem like pretty basic POSIX filesystem features to be fair. Awkward, sure... there's also awkwardness like symlinks, file locking, sticky bits and so on. But these are just things you have to implement. Are there gotchas that are inherent to FUSE itself rather than FUSE implementations?

> presenting things

I don’t have a dog in this race, have to say thou the vagueness of the hand waving in multiple comments is losing you credibility

The MLPerf performance of JuiceFS last year, FUSE client, and TCP/IP network. There're some analysis about performance and bottleneck. -> https://juicefs.com/en/blog/engineering/mlperf-storage-v2-ai...

This clarification is helpful, thanks! The README currently implies a slightly different take, perhaps it could be made more clear that it's suitable for use unmodified in closed source products:

> The AGPL license is suitable for open source projects, while commercial licenses are available for organizations requiring different terms.

I was a bit unclear on where the AGPL's network-interaction clause draws its boundaries- so the commercial license would only be needed for closed-source modifications/forks, or if statically linking ZeroFS crate into a larger proprietary Rust program, is that roughly it?

These are basic POSIX features, but I think the high-level point that Kurt is trying to make is that building a FUSE file system signs you up for a nearly unlimited amount of compatibility work (if you want to support most applications) whereas their approach (just do a loopback ext4 fs into a large file) avoids a lot of those problems.

My expectations are that in 2026 we will see more and more developers attempt to build custom FUSE file systems and then run into the long tail of compatibility pain.

Also worth noting (as a sibling comment pointed out) that despite these assurances the untested legal risks of AGPL-licensed code may still cause difficulties for larger, risk-averse companies. Google notably has a blanket policy [1] banning all AGPL code entirely as "the risks outweigh the benefits", so large organizations are probably another area where the commercial license comes into play.

[1] https://opensource.google/documentation/reference/using/agpl...

> so the commercial license would only be needed for closed-source modifications/forks

Indeed.

> just do a loopback ext4 fs into a large file

How does that work with multiple clients though?

tl;dr it doesn't. I'm not sure what they're planning in this capacity (I haven't checked out sprites myself), but I would guess that it's going to be a function of "snapshots" as a mechanism to give multiple clients ephemeral write access to the same disk.