The road to ACID transactions in Cassandra 6

Atomic batches in 2013, single-partition Paxos compare-and-swap later that year, and now strictly serializable cross-partition transactions with Accord in unreleased Cassandra 6. We explore each on a three-node cluster with an accounting workload.

By Phil EatonAugust 16, 2026 Focus

You are getting early access to this article as a subscriber. Your support makes articles like this possible. Thank you.

Cassandra is a compelling data system. It is one of extremely few vendor-neutral, open-source databases supporting a SQL-like query language with builtin sharding and builtin replication. A desirable combination. And a reason Cassandra has so many (large) users including Apple, eBay, Bloomberg, and Netflix.

Cassandra has evolved significantly since its first release. From an eventually consistent data model without transactions and a schemaless, NoSQL interface over Thrift to where (in the upcoming 6.0 release) it stands as an ACID transactional SQL-like database (granted: severe SQL limitations, transactions are non-interactive, we’ll get to that later).

Meanwhile the lack of joins plus automatic sharding (and a limited secondary index story) means a key characteristic has stayed the same: you model tables based on queries. And as a result your application might end up denormalizing, turning a single write into multiple writes in order to position the database to efficiently answer different queries later on.

In this article we’ll set up a three-node Cassandra cluster on one machine, running the cassandra-6.0 branch (a pre-release state) to test out some transactional workloads across four of Cassandra’s transactional options: none (the default), BATCH updates, Lightweight transaction (LWT) updates, and Accord (i.e. ACID) updates. Accord transactions will become available only when Cassandra 6 is released (perhaps later this year), which is why we are using the pre-release branch.

Setting up a cluster#

Install Java 21 and the ant build system, and gcc and Go for our concurrent test runner Monastery.

sudo apt-get install -y openjdk-21-jdk ant gcc golang git clone https://github.com/theconsensuslabs/monastery cd monastery CGO_ENABLED=1 go build -buildmode=plugin -o cql.so ./plugins/cql CGO_ENABLED=1 go build -o monastery .

Then grab and build Cassandra.

git clone https://github.com/apache/cassandra cd cassandra git checkout cassandra-6.0 ant artifacts -Dcheck.skip=true -Dant.gen-doc.skip=true -Dno-javadoc=true

Set up directories and configuration for three nodes, giving them unique IP addresses and JMX ports.

for i in 1 2 3; do n=node$i # run `killall java` first and then this will clean up the data directories for clean re-runs. rm -rf /etc/cassandra/$n /var/log/cassandra/$n /var/lib/cassandra mkdir -p /etc/cassandra/$n /var/log/cassandra/$n cp -r ~/cassandra/conf/* /etc/cassandra/$n/ echo " cassandra_storagedir=\"/var/lib/cassandra/$n\" JVM_OPTS=\"\$JVM_OPTS -Dcassandra.jmx.local.port=7${i}99\"" >> /etc/cassandra/$n/cassandra-env.sh echo " cluster_name: 'theconsensus-lab' listen_address: 127.0.0.$i rpc_address: 127.0.0.$i seed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: "127.0.0.1:7000" accord: enabled: true" >> /etc/cassandra/$n/cassandra.yaml # Set up max memory usage. echo " -Xms4G -Xmx4G" >> /etc/cassandra/$n/jvm-server.options done

Now start up the three nodes one at a time. (-R allows us to run as root.)

CASSANDRA_CONF=/etc/cassandra/node1 CASSANDRA_LOG_DIR=/var/log/cassandra/node1 /root/cassandra/bin/cassandra -R >> /var/log/cassandra/node1/console.log 2>&1

Wait for the node to come up (you’ll get connection refused errors for a few seconds until the node comes fully up). Eventually you’ll see this:

$ /root/cassandra/bin/nodetool -p 7199 status Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 127.0.0.1 72.73 KiB 16 100.0% 6d194555-f6eb-41d0-c000-000000000001 rack1

Where “UN” means “Up” and “Normal”.

Now let’s add node2.

CASSANDRA_CONF=/etc/cassandra/node2 CASSANDRA_LOG_DIR=/var/log/cassandra/node2 /root/cassandra/bin/cassandra -R >> /var/log/cassandra/node2/console.log 2>&1

And once it’s up, nodetool status will eventually look like this.

$ /root/cassandra/bin/nodetool -p 7299 status Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 127.0.0.1 74.46 KiB 16 100.0% 6d194555-f6eb-41d0-c000-000000000001 rack1 UN 127.0.0.2 79.89 KiB 16 100.0% 6d194555-f6eb-41d0-c000-000000000002 rack1

Now start the final node.

CASSANDRA_CONF=/etc/cassandra/node3 CASSANDRA_LOG_DIR=/var/log/cassandra/node3 /root/cassandra/bin/cassandra -R >> /var/log/cassandra/node3/console.log 2>&1

And wait for it to join.

$ /root/cassandra/bin/nodetool -p 7399 status Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 127.0.0.1 163.14 KiB 16 64.7% 6d194555-f6eb-41d0-c000-000000000001 rack1 UN 127.0.0.2 173.47 KiB 16 59.3% 6d194555-f6eb-41d0-c000-000000000002 rack1 UN 127.0.0.3 76.7 KiB 16 76.0% 6d194555-f6eb-41d0-c000-000000000003 rack1

Since this is a view of the cluster you’d get these same results querying the nodetool status on any node in the cluster (in this fault-less environment anyway).

Here’s the current view from node2.

$ /root/cassandra/bin/nodetool -p 7299 status Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 127.0.0.1 163.14 KiB 16 64.7% 6d194555-f6eb-41d0-c000-000000000001 rack1 UN 127.0.0.2 173.47 KiB 16 59.3% 6d194555-f6eb-41d0-c000-000000000002 rack1 UN 127.0.0.3 76.7 KiB 16 76.0% 6d194555-f6eb-41d0-c000-000000000003 rack1

Let’s get transactional!

Accounting#

Let’s say we have an accounts table that tracks balances after transfers. We’ll generate transfers and apply them to Cassandra using every method we have available (plain Cassandra, BATCH, LWT per row, conditional BATCH with LWT, and Accord). We’ll partition our accounts table by customer ID and order by account ID.

We will have only two accounts, with starting balances of 1,000 each. We’ll have two writers produce transfers between the two accounts concurrently. On top of the first axis (e.g. LWT vs Accord) we’ll have a second axis where one variant will do a read-modify-write to produce the transfers and one variant of the workload will do blind writes (no reads involved) to produce the transfers.

While the two writers are concurrently writing, we’ll have a third thread reading concurrently and asserting that the sum of balances between both accounts is 2,000.

When the concurrent writes complete and the workload ends we’ll assert that the sum of balances is still 2,000. For the read-modify-write workload variants we’ll also assert that the end balance of both accounts is a well-known number (because the workload’s intent is deterministic).

And we’ll have a third and final axis. One set of workloads will cross partition boundaries by transferring between accounts belonging to different customers. And the other set of workloads will not cross partition boundaries by only transferring between accounts belonging to the same customer.

The tables will also have two op columns for operations that are capable of doing conditional writes (LWT and Accord) to use as idempotency keys. It isn’t cheating that plain updates and non-LWT BATCH updates won’t use the idempotency columns because they can’t use the idempotency columns.

I’ll explain more about this all as we go.

Plain updates#

Monastery allows us to script concurrent operations on a database by a fixed number of clients. Monastery will run the script against the database for us.

We start off by defining a setup section of the script.

CREATE KEYSPACE IF NOT EXISTS lab WITH replication = {'class':'NetworkTopologyStrategy','datacenter1':3}; DROP TABLE IF EXISTS lab.accounts; CREATE TABLE lab.accounts (customer int, account_id int, balance int, PRIMARY KEY (customer, account_id)); INSERT INTO lab.accounts (customer, account_id, balance) VALUES (1, 1, 1000); INSERT INTO lab.accounts (customer, account_id, balance) VALUES (1, 2, 1000);

blind-plain-same.cql

Since the replication factor is 3 and there are only 3 nodes in the cluster, sharding will effectively not happen. But if we added more nodes to the cluster and kept the replication factor at 3, sharding would meaningfully happen.

Then we specify a concurrent section for our two writers and one reader. Each client will do an action repeatedly. The writers will send blind updates repeatedly transferring units between accounts. And the readers will repeatedly try to assert that the balance of the two accounts is constant.

--- concurrent w1: repeat 400 as x { UPDATE lab.accounts SET balance = {x} WHERE customer = 1 AND account_id = 1; -- assert ok UPDATE lab.accounts SET balance = 2000 - {x} WHERE customer = 1 AND account_id = 2; -- assert ok } w2: repeat 400 as x { UPDATE lab.accounts SET balance = 2000 - {x} WHERE customer = 1 AND account_id = 1; -- assert ok UPDATE lab.accounts SET balance = {x} WHERE customer = 1 AND account_id = 2; -- assert ok } r1: repeat 1500 { SELECT balance FROM lab.accounts WHERE customer = 1; -- assert sum(0) = 2000 or error }

blind-plain-same.cql

The last section of the script is a final check stage where we can make any final queries and assertions.

--- check: SELECT balance FROM lab.accounts WHERE customer = 1; -- assert sum(0) = 2000

blind-plain-same.cql

There’s no balance for account 1 and 2 that we could assume this will come to since they’re both completely in competition. However, the overall invariant remains that no money should be gained or lost.

When we run this script with Monastery we will often see isolation violated (which is expected) in the concurrent section (i.e. the balances don’t sum to 2,000). We may or may not see the final assertion succeed, but if it succeeds it is because of luck not a guarantee.

$ ./monastery cql '127.0.0.1?consistency=quorum' blind-plain-same.cql COUNT CLIENT ASSERTION GOT 939 r1 sum(0) = 2000 or error ({1847}, {1850}) +776 more f48ab59a-62cb-4061-bb0e-5883b369ef7d 939 assertion(s) failed

So, in this run, the concurrent reader saw mismatched balances 939 of 1500 times. But the ending writes end up balanced again. (These are blind writes so this is more possible than read-modify-writes which would amplify inconsistency.)

Ok, so plain Cassandra doesn’t make for a great bank. At least not in this particular data model. But we have other options! Let’s see BATCH next.

BATCH updates#

Batches, completed in their current form by Cassandra 1.2 (January 2013), let you combine a number of statements into one mutation per partition that is applied isolated and atomically. If the batch spans partitions, it also becomes a guarantee that the statements are eventually applied even in the face of node failures.

All statements in a batch share the same timestamp, where otherwise each statement has its own timestamp. Conflicts are decided by timestamp per cell, not per row. So when two conflicting batches carry different timestamps, the later batch wins every cell that both wrote, and no column ends up holding a value from a different batch than its neighbour. But timestamps are client-generated, and two clients can tie. Cassandra breaks a timestamp tie per cell by keeping the greater value, so two tied batches can each win some columns and lose others. We’ll see this happen shortly.

If we take our blind-plain-same.cql and wrap updates as a BATCH then we’ll actually end up somewhere consistent.

blind-batch-same.cql

Give it a run.

$ ./monastery cql '127.0.0.1?consistency=quorum' blind-batch-same.cql no assertion failures a98088c0-5aec-421a-8ef7-10e15642b243

That’s great! At least for a single partition.

Mostly, anyway. Most runs come back clean like that. But run it a few more times and every few runs you’ll catch a bad sum.

$ ./monastery cql '127.0.0.1?consistency=quorum' blind-batch-same.cql COUNT CLIENT ASSERTION GOT 6 r1 sum(0) = 2000 ({1897}, {1893}) +3 more 252c3bcc-35b0-4586-9741-239abf1b577c 6 assertion(s) failed

Look at the two balances. Both of them are large. 1,897 and 1,893 come to 3,790, well over 2,000. Landing on two large numbers takes the large half of one batch next to the large half of the other.

This is the timestamp caveat from earlier rather than a bug. With both writers updating two rows so frequently, their client-generated timestamps collide fairly often. On a tie Cassandra compares the values themselves and keeps the greater one, cell by cell. Account 1 resolves to the larger of x and 2000 - x, and so does account 2. Both cells keep the big number.

We can even see this by hand. Set the same timestamp explicitly on two batches and have them disagree on both rows.

DROP TABLE IF EXISTS lab.tie; CREATE TABLE lab.tie (customer int, account_id int, balance int, PRIMARY KEY (customer, account_id)); INSERT INTO lab.tie (customer, account_id, balance) VALUES (1, 1, 1000) USING TIMESTAMP 1755300000000000; INSERT INTO lab.tie (customer, account_id, balance) VALUES (1, 2, 1000) USING TIMESTAMP 1755300000000000; BEGIN BATCH USING TIMESTAMP 1755400000000000 UPDATE lab.tie SET balance = 100 WHERE customer = 1 AND account_id = 1; UPDATE lab.tie SET balance = 1900 WHERE customer = 1 AND account_id = 2; APPLY BATCH; BEGIN BATCH USING TIMESTAMP 1755400000000000 UPDATE lab.tie SET balance = 1800 WHERE customer = 1 AND account_id = 1; UPDATE lab.tie SET balance = 200 WHERE customer = 1 AND account_id = 2; APPLY BATCH; SELECT customer, account_id, balance, WRITETIME(balance) FROM lab.tie WHERE customer = 1;

tie.cql

Neither batch wrote (1800, 1900), but that's what we get.

$ /root/cassandra/bin/cqlsh 127.0.0.1 -f tie.cql customer | account_id | balance | writetime(balance) ----------+------------+---------+-------------------- 1 | 1 | 1800 | 1755400000000000 1 | 2 | 1900 | 1755400000000000 (2 rows)

But again this is documented last-write-wins conflict resolution.

BATCH updates, cross-partition#

Let’s tweak our workload slightly to transfer units across partitions: between customers.

blind-batch-cross.cql

Give it a run.

$ ./monastery cql '127.0.0.1?consistency=quorum' blind-batch-cross.cql COUNT CLIENT ASSERTION GOT 231 r1 sum(0) = 2000 ({10}, {1989}) +230 more b8bb3045-fd26-423b-a67e-44bbf1543bd5 231 assertion(s) failed

And we indeed see atomicity preserved (each batch's writes were eventually applied together) but not isolation (concurrent reads saw mismatched balances). The final check passed too, though after what we saw with tied timestamps that part is not quite guaranteed: if the last two batches tie, the durable end state can also mix. Again, this is what the docs tell us will happen.

But let’s go back to working with the same partition and look at another limitation of BATCH updates: read-modify-write workloads.

BATCH updates, read-modify-write#

Let’s change up our workload slightly, keeping the schema the same. This time we’ll have two writers both incrementing units from one account and decrementing units from another. Since both writers are incrementing and decrementing accounts in the same direction, there is a logical ending balance for each account.

CREATE KEYSPACE IF NOT EXISTS lab WITH replication = {'class':'NetworkTopologyStrategy','datacenter1':3}; DROP TABLE IF EXISTS lab.accounts; CREATE TABLE lab.accounts (customer int, account_id int, balance int, op1 int, op2 int, PRIMARY KEY (customer, account_id)); INSERT INTO lab.accounts (customer, account_id, balance, op1, op2) VALUES (1, 1, 1000, 0, 0); INSERT INTO lab.accounts (customer, account_id, balance, op1, op2) VALUES (1, 2, 1000, 0, 0); --- concurrent w1: repeat 200 { p = SELECT balance - 1 FROM lab.accounts WHERE customer = 1 AND account_id = 1; q = SELECT balance + 1 FROM lab.accounts WHERE customer = 1 AND account_id = 2; BEGIN BATCH \ UPDATE lab.accounts SET balance = {p} WHERE customer = 1 AND account_id = 1; \ UPDATE lab.accounts SET balance = {q} WHERE customer = 1 AND account_id = 2; \ APPLY BATCH; } w2: repeat 200 { p = SELECT balance - 1 FROM lab.accounts WHERE customer = 1 AND account_id = 1; q = SELECT balance + 1 FROM lab.accounts WHERE customer = 1 AND account_id = 2; BEGIN BATCH \ UPDATE lab.accounts SET balance = {p} WHERE customer = 1 AND account_id = 1; \ UPDATE lab.accounts SET balance = {q} WHERE customer = 1 AND account_id = 2; \ APPLY BATCH; } r1: repeat 1000 { SELECT balance FROM lab.accounts WHERE customer = 1; -- assert sum(0) = 2000 } --- check: SELECT balance FROM lab.accounts WHERE customer = 1; -- assert sum(0) = 2000 check: SELECT balance FROM lab.accounts WHERE customer = 1 AND account_id = 1; -- assert ({600}) check: SELECT balance FROM lab.accounts WHERE customer = 1 AND account_id = 2; -- assert ({1400})

rmw-batch-same.cql

According to the grammar, we can’t even put SELECTs inside the BATCH. So the read stage is not even part of the “transaction”. So there’s basically no consistency we can provide for read-modify-write with BATCH alone. But let’s try it out and see.

$ ./monastery cql '127.0.0.1?consistency=quorum' rmw-batch-same.cql COUNT CLIENT ASSERTION GOT 831 r1 sum(0) = 2000 ({797}, {1210}) +166 more 1 check ({1400}) ({1210}) 1 check ({600}) ({797}) 1 check sum(0) = 2000 ({797}, {1210}) fb0dcd27-a93c-4f6b-9db0-35a4944a58ef 834 assertion(s) failed

Not great! But again, this is documented. And we’ve still got lightweight transactions!

Lightweight transactions#

Lightweight transactions (LWT) came out in Cassandra 2.0 (September 2013) which gave us atomic compare-and-swap built on Paxos. We cannot atomically SELECT and then UPDATE, but we can at least atomically conditionally UPDATE.

Also, LWT timestamps are derived from Paxos and are unique per partition, so timestamp ties that we saw in the BATCH workloads are just not possible when using LWT.

One limitation of LWT is that while there is a way to know that a conditional update definitely failed, there’s no way if the LWT times out to know if it succeeded or not. So in the LWT workload we’ll make use of the op fields to store an idempotency token. Each writer gets its own op field. And each writer loops, retrying the LWT that inserts a unique op value, until it gets back the op value it sent in.

Additionally, while LWT goes through Paxos, reads by default do not. The client executes CONSISTENCY SERIAL to indicate it wants SELECTs to go through Paxos. These reads can fail on a timeout as well so we assert the reads sum to 2,000 or that the read errors.

Let’s rewrite rmw-batch-same.cql in terms of LWT.

CREATE KEYSPACE IF NOT EXISTS lab WITH replication = {'class':'NetworkTopologyStrategy','datacenter1':3}; DROP TABLE IF EXISTS lab.accounts; CREATE TABLE lab.accounts (customer int, account_id int, balance int, op1 int, op2 int, PRIMARY KEY (customer, account_id)); INSERT INTO lab.accounts (customer, account_id, balance, op1, op2) VALUES (1, 1, 1000, 0, 0); INSERT INTO lab.accounts (customer, account_id, balance, op1, op2) VALUES (1, 2, 1000, 0, 0); --- concurrent w1: repeat 200 as op { retry { a, p = SELECT balance, balance - 1 FROM lab.accounts WHERE customer = 1 AND account_id = 1; b, q = SELECT balance, balance + 1 FROM lab.accounts WHERE customer = 1 AND account_id = 2; BEGIN BATCH \ UPDATE lab.accounts SET balance = {p}, op1 = {op} \ WHERE customer = 1 AND account_id = 1 IF balance = {a} AND op1 < {op}; \ UPDATE lab.accounts SET balance = {q} WHERE customer = 1 AND account_id = 2 IF balance = {b}; \ APPLY BATCH; -- assert ok or error SELECT op1 FROM lab.accounts WHERE customer = 1 AND account_id = 1; -- assert ({{op}}) } } w2: repeat 200 as op { retry { a, p = SELECT balance, balance - 1 FROM lab.accounts WHERE customer = 1 AND account_id = 1; b, q = SELECT balance, balance + 1 FROM lab.accounts WHERE customer = 1 AND account_id = 2; BEGIN BATCH \ UPDATE lab.accounts SET balance = {p}, op2 = {op} \ WHERE customer = 1 AND account_id = 1 IF balance = {a} AND op2 < {op}; \ UPDATE lab.accounts SET balance = {q} WHERE customer = 1 AND account_id = 2 IF balance = {b}; \ APPLY BATCH; -- assert ok or error SELECT op2 FROM lab.accounts WHERE customer = 1 AND account_id = 1; -- assert ({{op}}) } } r1: CONSISTENCY SERIAL; r1: repeat 1000 { SELECT balance FROM lab.accounts WHERE customer = 1; -- assert sum(0) = 2000 or error } --- check: SELECT balance FROM lab.accounts WHERE customer = 1; -- assert sum(0) = 2000 check: SELECT balance FROM lab.accounts WHERE customer = 1 AND account_id = 1; -- assert ({600}) check: SELECT balance FROM lab.accounts WHERE customer = 1 AND account_id = 2; -- assert ({1400})

rmw-lwt-same.cql

Give it a run.

$ ./monastery cql '127.0.0.1?consistency=quorum' rmw-lwt-same.cql no assertion failures 0e1d7cf1-c67a-4f3a-ad0a-9ce4c29bc06c

Very nice. And LWT can still run the old blind-write workload just fine too.

blind-lwt-same.cql

Run it.

$ ./monastery cql '127.0.0.1?consistency=quorum' blind-lwt-same.cql no assertion failures 54e91e0c-8d9b-4028-9601-f5300807e483

Fantastic!

But LWT only works on a single partition. Let’s write the blind-write workload with LWT, but this time transferring units between customers.

blind-lwt-cross.cql

And run it

$ ./monastery cql '127.0.0.1?consistency=quorum' blind-lwt-cross.cql COUNT CLIENT ASSERTION ERROR 400 w1 ok Batch with conditions cannot span multiple partitions 400 w2 ok Batch with conditions cannot span multiple partitions 8842fdf5-dfc9-4415-9c78-9fb68aa20791 800 assertion(s) failed

So we’ve got LWT which can get us consistent read-modify-write within a single partition, but it doesn’t work at all across partitions. And then we’ve got batches which are not isolated across partitions.

This is why the folks at Apple and University of Michigan created Accord.

Full transactions with Accord#

Accord is the EPaxos-inspired leaderless consensus protocol that enables tables in Cassandra to be marked as transactional_mode='full'. All read and write operations on these tables go through the Accord consensus. And we finally get actual ACID transactions, albeit non-interactive ones.

In LWT, the value we read to use in the compare-and-swap would often be stale. The LWT would fail and we’d have to retry it. But a failed LWT doesn’t always mean the write didn't happen. For example, it might indicate that the client timed out while the actual write (eventually) succeeded. Writing, and guarding against, the op column allowed us to make sure we didn’t apply the same write twice (or more).

In Accord, the condition is evaluated at the same time as the read, so the read is not stale and the main reason a client would see a failure is due to a client timeout or a node failure. Both are unlikely in our happy localhost environment. The idempotency key would still be useful in a real system, but we’ll drop it in our lab environment.

CREATE KEYSPACE IF NOT EXISTS lab WITH replication = {'class':'NetworkTopologyStrategy','datacenter1':3}; DROP TABLE IF EXISTS lab.accounts; CREATE TABLE lab.accounts (customer int, account_id int, balance int, op1 int, op2 int, PRIMARY KEY (customer, account_id)) WITH transactional_mode = 'full'; INSERT INTO lab.accounts (customer, account_id, balance, op1, op2) VALUES (1, 1, 1000, 0, 0); INSERT INTO lab.accounts (customer, account_id, balance, op1, op2) VALUES (1, 2, 1000, 0, 0); --- concurrent w1: repeat 200 { BEGIN TRANSACTION \ LET x = (SELECT balance FROM lab.accounts WHERE customer = 1 AND account_id = 1); \ IF x.balance >= 1 THEN \ UPDATE lab.accounts SET balance -= 1 WHERE customer = 1 AND account_id = 1; \ UPDATE lab.accounts SET balance += 1 WHERE customer = 1 AND account_id = 2; \ END IF \ COMMIT TRANSACTION; -- assert ok } w2: repeat 200 { BEGIN TRANSACTION \ LET x = (SELECT balance FROM lab.accounts WHERE customer = 1 AND account_id = 1); \ IF x.balance >= 1 THEN \ UPDATE lab.accounts SET balance -= 1 WHERE customer = 1 AND account_id = 1; \ UPDATE lab.accounts SET balance += 1 WHERE customer = 1 AND account_id = 2; \ END IF \ COMMIT TRANSACTION; -- assert ok } r1: repeat 1000 { BEGIN TRANSACTION \ LET x = (SELECT balance FROM lab.accounts WHERE customer = 1 AND account_id = 1); \ LET y = (SELECT balance FROM lab.accounts WHERE customer = 1 AND account_id = 2); \ SELECT x.balance, y.balance; \ COMMIT TRANSACTION; -- assert sum(0, 1) = 2000 or error } --- check: SELECT balance FROM lab.accounts WHERE customer = 1; -- assert sum(0) = 2000 check: SELECT balance FROM lab.accounts WHERE customer = 1 AND account_id = 1; -- assert ({600}) check: SELECT balance FROM lab.accounts WHERE customer = 1 AND account_id = 2; -- assert ({1400})

rmw-accord-same.cql

And run it.

$ ./monastery cql '127.0.0.1?consistency=quorum' rmw-accord-same.cql no assertion failures 471d1ba8-ae5b-4781-b881-d00a2cd9adcc

Ok, that’s cool, but within the same partition it’s also what was possible in the LWT version. Let’s try out the cross-partition workload.

CREATE KEYSPACE IF NOT EXISTS lab WITH replication = {'class':'NetworkTopologyStrategy','datacenter1':3}; DROP TABLE IF EXISTS lab.accounts; CREATE TABLE lab.accounts (customer int, account_id int, balance int, op1 int, op2 int, PRIMARY KEY (customer, account_id)) WITH transactional_mode = 'full'; INSERT INTO lab.accounts (customer, account_id, balance, op1, op2) VALUES (1, 1, 1000, 0, 0); INSERT INTO lab.accounts (customer, account_id, balance, op1, op2) VALUES (2, 1, 1000, 0, 0); --- concurrent w1: repeat 200 { BEGIN TRANSACTION \ LET x = (SELECT balance FROM lab.accounts WHERE customer = 1 AND account_id = 1); \ IF x.balance >= 1 THEN \ UPDATE lab.accounts SET balance -= 1 WHERE customer = 1 AND account_id = 1; \ UPDATE lab.accounts SET balance += 1 WHERE customer = 2 AND account_id = 1; \ END IF \ COMMIT TRANSACTION; -- assert ok } w2: repeat 200 { BEGIN TRANSACTION \ LET x = (SELECT balance FROM lab.accounts WHERE customer = 1 AND account_id = 1); \ IF x.balance >= 1 THEN \ UPDATE lab.accounts SET balance -= 1 WHERE customer = 1 AND account_id = 1; \ UPDATE lab.accounts SET balance += 1 WHERE customer = 2 AND account_id = 1; \ END IF \ COMMIT TRANSACTION; -- assert ok } r1: repeat 1000 { BEGIN TRANSACTION \ LET x = (SELECT balance FROM lab.accounts WHERE customer = 1 AND account_id = 1); \ LET y = (SELECT balance FROM lab.accounts WHERE customer = 2 AND account_id = 1); \ SELECT x.balance, y.balance; \ COMMIT TRANSACTION; -- assert sum(0, 1) = 2000 or error } --- check: SELECT balance FROM lab.accounts WHERE customer IN (1, 2); -- assert sum(0) = 2000 check: SELECT balance FROM lab.accounts WHERE customer = 1 AND account_id = 1; -- assert ({600}) check: SELECT balance FROM lab.accounts WHERE customer = 2 AND account_id = 1; -- assert ({1400})

rmw-accord-cross.cql

And run it.

$ ./monastery cql '127.0.0.1?consistency=quorum' rmw-accord-cross.cql no assertion failures f5c6ce4a-dbfa-48dd-b6c1-3367ef563df4

That is entirely new in Cassandra 6 (or will be, when it is released). Very cool.

Possible bug?#

While using Accord, I occasionally saw the concurrent reader report balances that didn’t sum to 2,000. This only ever happened in the same-partition workloads. And while in the RMW workload it seems slightly more possible it was an issue in the workload itself, the invalid sums happened in the simpler blind-write workload as well.

Here’s the blind-write workload.

blind-accord-same.cql

And if we run it a few times we’ll pretty consistently see errors in r1.

$ ./monastery cql '127.0.0.1?consistency=quorum' blind-accord-same.cql COUNT CLIENT ASSERTION GOT 1 r1 sum(0, 1) = 2000 or error ({1851, 1853}) 70352e3f-f960-4036-8475-140a9314ce81 1 assertion(s) failed

However, I have never seen an error in the end result. There might be an isolation bug in concurrent transactions even while the durable result is not wrong.

Even if this is a bug, it’s not particularly damning. Distributed systems have bugs. And Cassandra 6 is not even released yet.

Parting thoughts#

This was my first exposure to Cassandra. I like it a lot. I like the builtin replication and builtin sharding. I like the novel consensus protocol and the strict serializability. It’s interesting to see how it has evolved over the years. And it will be interesting to see them continue to push toward being a more general-purpose database system. Interactive transactions would be cool.

And lastly, I’m looking forward to getting help from the ASF JIRA on if these are actual bugs or if they’re just mistakes in my own code.

Edit (August 17, 2026): C. Scott Andreas from Apple confirmed that we found an actual bug in Cassandra.

Noticed a mistake? Have a question or comment? Write to the editor.

Hacker Times

Hacker Times

The road to ACID transactions in Cassandra 6

Discussion

Discussion