How to Fix Corrupted SSTables in Apache Cassandra: A Complete Guide

When Apache Cassandra detects a corrupted SSTable file during startup, you might see an error like this:

org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /var/lib/cassandra/data/keyspace/table-UUID/me-1-big-Data.db
Exiting forcefully due to file system exception on startup, disk failure policy "stop"

This can be alarming — your node refuses to start and your database appears unavailable. In this article we’ll explain what SSTables are, why corruption happens, and step-by-step methods to recover a node safely. We’ll also include preventive best practices.

What Are SSTables?

SSTables (Sorted String Tables) are immutable files on disk where Cassandra stores data. Each write to Cassandra eventually becomes an SSTable on disk. They consist of:

• Data files: contain the actual rows and columns
• Index files: map partition keys to locations in the data files
• Bloom filters and compression info: speed up reads and manage storage

Because SSTables are immutable, Cassandra normally never edits them. Corruption usually indicates a hardware or file system issue rather than a software bug.

Common Causes of SSTable Corruption

• Disk hardware failures (bad sectors, failing SSD/HDD)
• Running out of disk space during compaction
• Unclean shutdowns or power loss
• File system inconsistencies
• OS or virtualization glitches (e.g., Docker volumes not flushed)

Default Disk Failure Policy

By default, Cassandra’s disk_failure_policy in cassandra.yaml is set to stop. This means the node shuts down immediately when it encounters a file system error to protect data consistency. Other options include best_effort and ignore, but stop is safest.

Step-by-Step Recovery Procedure

1. Back Up Before Touching Anything

Navigate to the data directory on your node and copy the entire table folder to a safe location. This allows you to attempt forensic recovery later if needed.

Example on Linux:

sudo systemctl stop cassandra
cp -r /var/lib/cassandra/data/keyspace/table-UUID/ ~/backup_table_UUID

2. Identify the Corrupted SSTable

Check the Cassandra logs (/var/log/cassandra/system.log). Errors will include the exact SSTable filename, e.g.:

me-1-big-Data.db
me-1-big-Index.db
me-1-big-Summary.db

All files with the same base name (me-1-big-*) belong together.

3. Delete the Corrupted SSTable Files

Remove the corrupted SSTable and its companion files (index, summary, filter, compression). Example:

rm /var/lib/cassandra/data/keyspace/table-UUID/me-1-big-*

⚠️ Important: Delete all related files with the same prefix, not just the .db file.

4. Restart the Node

Start Cassandra again:

sudo systemctl start cassandra

Cassandra will skip the missing SSTable and rely on replicas to serve data.

5. Run a Nodetool Repair

Once the node is back online, run a repair to rebuild the missing data from other replicas:

nodetool repair keyspace_name table_name

This ensures full consistency across the cluster.

6. Verify Data Integrity

Query some known partitions to confirm that data has been restored. Use tools like cqlsh:

SELECT * FROM keyspace_name.table_name WHERE id='test_id';

Handling This Inside Docker

If Cassandra is running in a Docker container:

1. Stop the container

   docker stop cassandra-node
   

2. Remove the corrupted file inside the container

   docker run --rm -it --volumes-from cassandra-node ubuntu bash
   rm /var/lib/cassandra/data/keyspace/table-UUID/me-1-big-*
   

3. Restart the container

   docker start cassandra-node
   

Because Docker persists data in volumes, the path inside the container mirrors the host path. Always double-check before deleting.

Prevention and Best Practices

• Use RAID or high-reliability storage: Disk redundancy helps prevent data loss.
• Monitor disk space: Alert when volumes reach 80–85%.
• Graceful shutdowns: Use nodetool drain before stopping nodes to flush memtables.
• Regular backups: Use nodetool snapshot or backup tools for Cassandra.
• Upgrade Cassandra: Newer versions include more robust disk handling and checksums.
• Monitor logs: Tools like Prometheus + Grafana can alert on disk failures.

Example Recovery Scenario

Imagine you have a three-node Cassandra cluster. One node logs:

CorruptSSTableException on me-45-big-Data.db

Steps:

1. Stop Cassandra on the affected node.
2. Backup the me-45-big-* files.
3. Remove the corrupted files.
4. Restart Cassandra.
5. Run nodetool repair to rebuild from other two nodes.

Result: The node rejoins the cluster and serves data normally.

Key Takeaways

• Cassandra SSTables are immutable; corruption almost always points to underlying disk or file system issues.
• The safest recovery method is delete the corrupted SSTable + run repair.
• Prevention is better than cure: monitor, back up, and maintain your cluster proactively.