> Pull requests are disabled. Coding agents make it too easy to send a large,
> low-context change that costs maintainers more time than it saves.
> Thoughtful contributions are welcome; please understand the code,
> keep the patch focused, and respect the review time you are asking for.
>
> Send a git format-patch attachment to ...celld's README states "Object-storage compare-and-swap ensures that exactly one node owns a cell at a time," but I'm skeptical this actually holds at the point where data is written to storage - I had an AI skim through the code with me, and the actual segment writes looked like plain, unconditional PUTs with no epoch check.
In many cases, I think using Cloudflare's Durable Objects is probably the right call instead.
It worked really well! Excited to see more options outside of Cloudflare.
CloudFlare OS: https://blog.cloudflare.com/cloudflare-os/ https://news.ycombinator.com/item?id=49182996
And yes, celld is isolates! The very lightweight within v8 isolation boundary. Hence the "very low idle cost". The deno team rolled their own new runtime! This one without deno_core! Shout out to this excellent 2022 post: https://deno.com/blog/roll-your-own-javascript-runtime https://news.ycombinator.com/item?id=35819990 https://news.ycombinator.com/item?id=35819990
It's super super exciting having an age where many pieces of software can all run with low profiles. Very timely, again, for the unbelievably order of magnitude (and thats 10 to the not 2 to the, for you fellow computer nerds) ish ram crunch we are in. Next up, at some point, ideally we get some v8-like runtimes where we can share libraries across multiple isolates! Separating the shared code from the shared data, so we have multiple instances of the library, seems much harder. (It feels like wasm has some/much promise here and I eagerly await clearer signals that code-sharing while sandboxing is indeed possible)
So happy to see support for running durable objects outside of one provider. Upvoted.
The "durable object" concept has been repeatably demonstrated to be a valuable abstraction.
"Each object is its own SQLite database, addressed by name and replicated to an S3-compatible bucket you own" -- this concept can take you a long way, both in its power and simplicity.
i would prefer a thing that was more self-contained, not dependent on a black box service layer underneath.
much apologies if i just have a poor understanding.
It's just making it way easier to spam maintainers with code.
Good contributors are probably worth even more now.
cloudflare gives you that instant geo-sync across the world that is hard to beat
with Celld do i need to buy bare metal in major continents
... but that S3 _is_ the control plane and consensus layer, no? You're just pushing this down the stack to whoever runs that S3 clone.
> At this time, we are not seeking outside contribution.
> AI has made writing code easy. The hard part, today, is not writing the code, but reviewing it, making sure quality stays high, and keeping the product coherent. In that light, unfortunately, external code contributions are "donating" the easy part of the job, while creating more of the hard work.
Feels very weird, but is logically sound: owners know exactly what they want and so they can work with Claude et al to iterate on features faster than with most drive-by contributors.
It reads like an "end of an era" but I imagine the steady state will be somewhere in the middle: high trust, high context contributors will still be able to contribute meaningful work.
I'm probably missing your point, but aren't wasm memories achieving exactly this? Just an example from wasmtime: https://docs.wasmtime.dev/examples-multimemory.html
https://www.microsoft.com/en-us/research/wp-content/uploads/...
More practical example (serverless WebSockets) https://youtu.be/FgWVoryZ8PU
celld is the full distributed system (albeit single tenant). It distributes DOs (cells) across any number of VMs. The databases for each DO are in object storage with RPO=0 guarantee.
Short writeup: https://crabmusket.net/2024/durable-execution-versus-session...
Summary:
I think the way to decide is, do you want to program with "objects" or "processes"?
I'd use Durable Objects (or what I called Session Backends in my article, after Jamsocket) if I wanted to model an "entity" that lasts indefinitely (e.g. a Figma doc, a user, a concert/event). I'd use a temporal/restate function for something that has a linear sequence of events and eventually comes to an end.
This isn't a hard rule. You can get the same outcome out of both technologies. (For example, Cloudflare built Workflows on top of Durable Objects; Rivet did the reverse.) You can use either one to implement the other, but you're going to be going out of your way based on what APIs are provided.
So where it's overkill, this is where Celld is a good fit. It's a 100% open play, despite mentioning three hosted APIs: S3, Cloudflare Workers, and Durable Objects. The thing is, that Deno tried making similar ones. Deno Deploy is analagous to S3, and Deno KV is analagous to Durable Objects. However, with Celld the only dependencies are being able to run a binary and some S3-compatible storage. Cloudflare provides S3 compatible storage, but this README doesn't mention that, except in an example. Perhaps Ryan is leaving the door open for Deno to be acquired by Cloudflare.
FWIW here's an old Deno blog post comparing Durable Objects to other stuff: https://denoland.medium.com/deno-kv-vs-cloudflare-workers-kv...
on the other hand, deno land folks stole from (used open source from) the best. the coordination layer/control-plane is all S3 CAS of dumb json files, which is a fantastic common-mode infra requirement for most orgs anyways, & perhaps a durable control-plane substrate you'd feel comfortable having someone else run (such as aws or others), while you run the data-plane (workers) yourself. and then for the durable objects themselves, they used litestream, which is a pretty top pick, excellent way to get radical distribution (but actual topology not included, some assembly required)! https://hn.algolia.com/?q=litestream
there's no reason this wouldn't run fine on most hosting, you definitely don't need bare metal. the virtues of v8 sandboxing / isolates! no need for vm's at all, no nested vm difficulties if you are trying to host on a shared host! but if you're asking questions like this, i want to again point you back to my top point.
the stuff is great when it works but Durable Objects can be quite expensive. whenever i get too excited about em all it takes is a little time trying to price it out to calm me down.
> do i need to buy bare metal
my first idea would be to run celld on AWS Kubernetes deployed to local zones https://docs.aws.amazon.com/eks/latest/userguide/auto-local-...
it’s not “region: earth” like cloudflare but perhaps worth the trade off.
It's like fixing all the problems with democracy by putting everyone in charge of their own 1-person election.
There was also Jamsocket and Plane, though it looks like they've shut down after acquisition: https://github.com/jamsocket/plane
This seems to be the first one which is providing drop-in compatibility for Cloudflare's JS-side APIs, including JSRPC etc.
https://github.com/cloudflare/workerd/pull/6780
But honestly I love that there are multiple implementations now.
thats hard to beat
ah well, we have celld!
Self-hosted, distributed Durable Objects.
celld is an open-source daemon that runs Cloudflare Workers and Durable Objects on your own machines. Each object is its own SQLite database. celld addresses an object by name and replicates it to a bucket that you own. The bucket can be S3-compatible or Google Cloud Storage. The nodes coordinate through that bucket alone, with no control plane and no consensus. Because every object is its own small database, applications shard by construction — the contention and blast-radius failures of one shared database are designed out, not managed. A cell that no node holds is inactive, and an inactive cell costs nearly nothing. Learn more at celld.dev or read the documentation.
Every celld node embeds V8 and executes Wrangler bundles. The fleet shares
one bucket, which contains deployments, cell state, and small ownership
records. The bucket can be S3-compatible or Google Cloud Storage. Object-storage compare-and-swap ensures that exactly one node owns a
cell at a time, without a membership protocol, failure detector, or consensus
service.
celld continuously replicates each cell's SQLite database to the bucket. When a cell moves, or when an inactive cell activates, its new owner restores that database and resumes execution. The bucket is the durable source of truth; nodes are replaceable.
The installer downloads the celld binary (provenance is verifiable with
gh attestation verify):
curl -fsSL https://celld.dev/install.sh | sh
Put ~/.local/bin on your PATH if the installer asks you to.
Worker projects deployed with celld deploy need
esbuild on PATH; asset-only projects do not.
The installer keeps each release under ~/.local/lib/celld/releases and points
one symlink at the current one. To remove celld, delete the symlink and the
releases:
rm `which celld` && rm -rf ~/.local/lib/celld
The release image contains the celld binary and is published for Linux
x86-64 and ARM64:
docker run --rm ghcr.io/denoland/celld --version
Persist the runtime's local state and pass the standard AWS credential environment through:
docker volume create celld-state
docker run --rm --network host \
-e AWS_ACCESS_KEY_ID \
-e AWS_SECRET_ACCESS_KEY \
-e AWS_SESSION_TOKEN \
-e CELLD_WATCH=/var/lib/celld/state \
-v celld-state:/var/lib/celld \
ghcr.io/denoland/celld \
--bucket s3://my-cells-bucket \
--endpoint https://ACCOUNT.r2.cloudflarestorage.com \
--region auto \
--listen 0.0.0.0:8080 \
--internal-listen 10.0.0.12:8081 \
--advertise node-a.internal:8081
Drop --endpoint and --region for AWS S3. Expose port 8080 through the load
balancer, and keep port 8081 on the private network.
celld uses the standard AWS credential chain. Deploy to an S3-compatible bucket, then start celld against the same bucket:
celld deploy . \
--bucket s3://my-cells-bucket
celld \
--bucket s3://my-cells-bucket \
--listen 0.0.0.0:8080 \
--internal-listen 10.0.0.12:8081 \
--advertise 10.0.0.12:8081
Use --endpoint for another S3-compatible service and --region when it
cannot be inferred. A gs:// bucket selects Google Cloud Storage. celld then
uses the Cloud Storage XML API with generation preconditions. Authentication
uses Application Default Credentials. celld rejects an S3 --endpoint for a
gs:// bucket, and it ignores the storage region:
celld deploy . --bucket gs://my-cells-bucket
celld --bucket gs://my-cells-bucket --listen 0.0.0.0:8080 \
--internal-listen 10.0.0.12:8081 --advertise 10.0.0.12:8081
A fleet runs one application, and every node loads its
latest successfully committed deployment from deploy/current.json. Run
celld --help for the complete command line.
Deployment objects use the documented types in crates/celld/protocol.rs. celld deploy invokes esbuild from PATH for Worker code, accepts the supported
Wrangler config subset—including co-deployed or asset-only static
assets—and writes those objects directly. Every node discovers owners and
peers from bucket leases; there is no account or join service.
Peer HTTP and the operator API use the internal listener. Put every advertised
address on a trusted private network or an encrypted overlay such as WireGuard
or Tailscale. Do not publish the internal port. celld rejects a literal public
IP unless you supply --unsafe-public-advertise. An explicit advertised
address requires an explicit internal-listener address. celld cannot verify a
hostname or a translated port, so you must route the advertised address to the
internal listener. The first current node creates fleet/peer-auth.json in the
bucket. All peer requests are
protocol-versioned, body-bound, HMAC-authenticated, clock-bounded, and
replay-protected with that fleet secret. Treat access to the bucket and its
credentials as fleet administrator access.
celld diagnose enumerates every node lease by default, then performs a signed
direct probe of each live peer:
celld diagnose --bucket s3://my-cells-bucket
The report keeps checking after an individual failure and distinguishes
expired records, malformed or unsafe advertise addresses, unreachable peers,
and incompatible protocols. It also prints each node's coarse resident-cell,
WebSocket, RSS, CPU, file-descriptor, pressure, and shedding sample. Pass one
or more --peer NODE_ID options to restrict the check.
Set a hard resident-cell limit on each loaded node:
CELLD_MAX_RESIDENT_CELLS=1000 \
celld --bucket s3://my-cells-bucket --listen 0.0.0.0:8080 \
--internal-listen 10.0.0.12:8081 --advertise node-a.internal:8081
celld enables a memory threshold at 80% of the available memory by default. Set
CELLD_MAX_RSS_MB to change the threshold, or set it to 0 to disable memory
pressure shedding. celld measures the memory that the cells hold, and not the
resident set size of the process. The two differ, because the memory allocator
keeps some freed pages instead of returning them to the operating system.
Shedding a cell cannot return those pages, so a threshold on the resident set
size holds a node in pressure after the node gives every cell back. The /state
route reports both numbers.
celld also applies an absolute cap to the resident set size of the process. The cap is 95% of the available memory. It protects the node when the allocator holds memory that shedding cannot return, because the operating system stops a process that uses more memory than the machine has. The node logs a warning when this cap applies.
The cap is a share of the machine, and celld does not derive it from the
threshold. A CELLD_MAX_RSS_MB at or above 95% of the available memory therefore reaches
the cap. The cap is then the effective limit. The node decides on its resident
set size, and celld reports this at startup. CELLD_MAX_RSS_MB=0
disables the threshold and the cap together. When celld cannot read the size of
the available memory, it applies a cap of 125% of an explicit threshold.
Under pressure, celld durably replicates and fences the least-recently used idle cells. It then publishes the cells as unowned without resetting their epochs. Those cells become inactive, and celld refuses to reacquire new unowned cells.
Each limit releases separately. The threshold releases when the memory in use falls to 80% of the threshold. The cap releases when the resident set size falls to 80% of the cap. A crossing of one limit therefore does not hold the node against the other.
A spare receives no assignment. It acquires a released cell through the same bucket protocol when normal traffic reaches it. celld does not shed a cell with active work or a live host WebSocket.
Pull requests are disabled. Coding agents make it too easy to send a large, low-context change that costs maintainers more time than it saves. Thoughtful contributions are welcome; please understand the code, keep the patch focused, and respect the review time you are asking for.
Send a git format-patch attachment to ry@deno.com.
Contributor License Agreement: By emailing a patch, you certify that you have the right to submit it and assign to Deno Land Inc. all rights in the patch that you can assign. Where a right cannot be assigned, you grant Deno Land Inc. a perpetual, irrevocable, worldwide, royalty-free, transferable, sublicensable license to use, modify, combine, relicense, redistribute, or publish the patch, in whole or in part, with or without attribution.
See the limitations and security pages before operating a public fleet.
The goal of this new design is to scale to a cluster while being operationally very easy to set up. Ideal for self-hosting.
NFSv4 is an easy first step, convenient because it's broadly understood, has many implementations, and requires no client libraries. I could imagine a follow-up to support LiteFS instead of NFS could make a lot of sense, though it'll get more complicated.
But yes, celld is definitely ahead of us here. No doubt about that.
Dual-licensed under the GNU AGPL v3 (fully featured, for open source use)
and a commercial license.