instaclustr / cassandra-sstable-tools

Tools for working with sstables

Instaclustr SSTable tools

https://www.instaclustr.com/update-released-instaclustr-sstable-analysis-tools-apache-cassandra/ https://www.instaclustr.com/instaclustr-open-sources-cassandra-sstable-analysis-tools/

Compile

$ git clone [email protected]:instaclustr/cassandra-sstable-tools.git
$ cd cassandra-sstable-tools
# Select the correct branch for major version (default is cassandra-4.1)
$ git checkout cassandra-4.1

$ mvn clean install

Install

Copy ic-sstable-tools.jar to Cassandra JAR folder, eg. /usr/share/cassandra/lib

Copy the bin/ic-sstable-tools script into your $PATH

We also offer RPM and DEB packages. In case you install them, you do not need to execute steps above, obviously.

Documentation

$ ./bin/ic-sstable-tools 
Missing required sub-command.
Usage: <main class> [-hV] [COMMAND]
  -h, --help      Show this help message and exit.
  -V, --version   Print version information and exit.
Commands:
  cfstats   Detailed statistics about cells in a column family
  pstats    Partition size statistics for a column family
  purge     Statistics about reclaimable data for a column family
  sstables  Print out metadata for sstables that belong to a column family
  summary   Summary information about all column families including how much of the data is repaired

If you want invoke help command for each subcommand, do it like:

$ ./bin/ic-sstable-tools help cfstats

summary

Provides summary information about all column families. Useful for finding the largest column families and how much data has been repaired by incremental repairs.

Usage

ic-sstable-tools summary

Output

Column	Description
Keyspace	Keyspace the column family belongs to
Column Family	Name of column family
SSTables	Number of sstables on this node for the column family
Disk Size	Compressed size on disk for this node
Data Size	Uncompressed size of the data for this node
Last Repaired	Maximum repair timestamp on sstables
Repair %	Percentage of data marked as repaired

sstables

Print out sstable metadata for a column family. Useful in helping to tune compaction settings.

Usage

ic-sstable-tools sstables <keyspace> <column-family>

Output

Column	Description
SSTable	Data.db filename of sstable
Disk Size	Size of sstable on disk
Total Size	Uncompressed size of data contained in the sstable
Min Timestamp	Minimum cell timestamp contained in the sstable
Max Timestamp	Maximum cell timestamp contained in the sstable
Duration	The time span between minimum and maximum cell timestamps
Min Deletion Time	The minimum deletion time
Max Deletion Time	The maximum deletion time
Level	Leveled Tiered Compaction sstable level
Keys	Number of partition keys
Avg Partition Size	Average partition size
Max Partition Size	Maximum partition size
Avg Column Count	Average number of columns in a partition
Max Column Count	Maximum number of columns in a partition
Droppable	Estimated droppable tombstones
Repaired At	Time when marked as repaired by incremental repair

pstats

Tool for finding largest partitions. Reads the Index.db files so is relatively quick.

Usage

ic-sstable-tools pstats [-n <num>] [-t <snapshot>] [-f <filter>] <keyspace> <column-family>

-h	Display help
-b	Batch mode. Uses progress indicator that is friendly for running in batch jobs.
-n	Number of partitions to display
-t	Snapshot to analyse. Snapshot is created if none is specified.
-f	Comma separated list of Data.db sstables to filter on

Output

Summary: Summary statistics about partitions

Column	Description
Count (Size)	Number of partition keys on this node
Total (Size)	Total uncompressed size of all partitions on this node
Total (SSTable)	Number of sstables on this node
Minimum (Size)	Minimum uncompressed partition size
Minimum (SSTable)	Minimum number of sstables a partition belongs to
Average (Size)	Average (mean) uncompressed partition size
Average (SSTable)	Average (mean) number of sstables a partition belongs to
std dev. (Size)	Standard deviation of partition sizes
std dev. (SSTable)	Standard deviation of number of sstables for a partition
50% (Size)	Estimated 50th percentile of partition sizes
50% (SSTable)	Estimated 50th percentile of sstables for a partition
75% (Size)	Estimated 75th percentile of partition sizes
75% (SSTable)	Estimated 75th percentile of sstables for a partition
90% (Size)	Estimated 90th percentile of partition sizes
90% (SSTable)	Estimated 90th percentile of sstables for a partition
95% (Size)	Estimated 95th percentile of partition sizes
95% (SSTable)	Estimated 95th percentile of sstables for a partition
99% (Size)	Estimated 99th percentile of partition sizes
99% (SSTable)	Estimated 99th percentile of sstables for a partition
99.9% (Size)	Estimated 99.9th percentile of partition sizes
99.9% (SSTable)	Estimated 99.9th percentile of sstables for a partition
Maximum (Size)	Maximum uncompressed partition size
Maximum (SSTable)	Maximum number of sstables a partition belongs to

Largest partitions: The top N largest partitions

Column	Description
Key	The partition key
Size	Total uncompressed size of the partition
SSTable Count	Number of sstables that contain the partition

SSTable Leaders: The top N partitions that belong to the most sstables

Column	Description
Key	The partition key
SSTable Count	Number of sstables that contain the partition
Size	Total uncompressed size of the partition

SSTables: Metadata about sstables as it relates to partitions.

Column	Description
SSTable	Data.db filename of SSTable
Size	Uncompressed size
Min Timestamp	Minimum cell timestamp in the sstable
Max Timestamp	Maximum cell timestamp in the sstable
Level	Leveled Tiered Compaction level of sstable
Partitions	Number of partition keys in the sstable
Avg Partition Size	Average uncompressed partition size in sstable
Max Partition Size	Maximum uncompressed partition size in sstable

cfstats

Tool for getting detailed cell statistics that can help identify issues with data model.

Usage

ic-sstable-tools cfstats [-n <num>] [-t <snapshot>] [-f <filter>] <keyspace> <column-family>

-h	Display help
-b	Batch mode. Uses progress indicator that is friendly for running in batch jobs.
-n	Number of partitions to display
-t	Snapshot to analyse. Snapshot is created if none is specified.
-f	Comma separated list of Data.db sstables to filter on

Output

Summary: Summary statistics about partitions

Column	Description
Count (Size)	Number of partition keys on this node
Rows (Size)	Number of clustering rows
(deleted)	Number of clustering row deletions
Total (Size)	Total uncompressed size of all partitions on this node
Total (SSTable)	Number of sstables on this node
Minimum (Size)	Minimum uncompressed partition size
Minimum (SSTable)	Minimum number of sstables a partition belongs to
Average (Size)	Average (mean) uncompressed partition size
Average (SSTable)	Average (mean) number of sstables a partition belongs to
std dev. (Size)	Standard deviation of partition sizes
std dev. (SSTable)	Standard deviation of number of sstables for a partition
50% (Size)	Estimated 50th percentile of partition sizes
50% (SSTable)	Estimated 50th percentile of sstables for a partition
75% (Size)	Estimated 75th percentile of partition sizes
75% (SSTable)	Estimated 75th percentile of sstables for a partition
90% (Size)	Estimated 90th percentile of partition sizes
90% (SSTable)	Estimated 90th percentile of sstables for a partition
95% (Size)	Estimated 95th percentile of partition sizes
95% (SSTable)	Estimated 95th percentile of sstables for a partition
99% (Size)	Estimated 99th percentile of partition sizes
99% (SSTable)	Estimated 99th percentile of sstables for a partition
99.9% (Size)	Estimated 99.9th percentile of partition sizes
99.9% (SSTable)	Estimated 99.9th percentile of sstables for a partition
Maximum (Size)	Maximum uncompressed partition size
Maximum (SSTable)	Maximum number of sstables a partition belongs to

Row Histogram: Histogram of number of rows per partition

Column	Description
Percentile	Minimum, average, standard deviation (std dev.), percentile, maximum
Count	Estimated number of rows per partition for the given percentile

Largest partitions: Partitions with largest uncompressed size

Column	Description
Key	The partition key
Size	Total uncompressed size of the partition
Rows	Total number of clustering rows in the partition
(deleted)	Number of row deletions in the partition
Tombstones	Number of cell or range tombstones
(droppable)	Number of tombstones that can be dropped as per gc_grace_seconds
Cells	Number of cells in the partition
SSTable Count	Number of sstables that contain the partition

Widest partitions: Partitions with the most cells

Column	Description
Key	The partition key
Rows	Total number of clustering rows in the partition
(deleted)	Number of row deletions in the partition
Cells	Number of cells in the partition
Tombstones	Number of cell or range tombstones
(droppable)	Number of tombstones that can be dropped as per gc_grace_seconds
Size	Total uncompressed size of the partition
SSTable Count	Number of sstables that contain the partition

Most Deleted Rows: Partitions with the most row deletions

Column	Description
Key	The partition key
Rows	Total number of clustering rows in the partition
(deleted)	Number of row deletions in the partition
Size	Total uncompressed size of the partition
SSTable Count	Number of sstables that contain the partition

Tombstone Leaders: Partitions with the most tombstones

Column	Description
Key	The partition key
Tombstones	Number of cell or range tombstones
(droppable)	Number of tombstones that can be dropped as per gc_grace_seconds
Rows	Total number of clustering rows in the partition
Cells	Number of cells in the partition
Size	Total uncompressed size of the partition
SSTable Count	Number of sstables that contain the partition

SSTable Leaders: Partitions that are in the most sstables

Column	Description
Key	The partition key
SSTable Count	Number of sstables that contain the partition
Size	Total uncompressed size of the partition
Rows	Total number of clustering rows in the partition
Cells	Number of cells in the partition
Tombstones	Number of cell or range tombstones
(droppable)	Number of tombstones that can be dropped as per gc_grace_seconds

SSTables: Metadata about sstables as it relates to partitions.

Column	Description
SSTable	Data.db filename of SSTable
Size	Uncompressed size
Min Timestamp	Minimum cell timestamp in the sstable
Max Timestamp	Maximum cell timestamp in the sstable
Partitions	Number of partitions
(deleted)	Number of row level partition deletions
(avg size)	Average uncompressed partition size in sstable
(max size)	Maximum uncompressed partition size in sstable
Rows	Total number of clustering rows in sstable
(deleted)	Number of row deletions in sstable
Cells	Number of cells in the SSTable
Tombstones	Number of cell or range tombstones in the SSTable
(droppable)	Number of tombstones that are droppable according to gc_grace_seconds
(range)	Number of range tombstones
Cell Liveness	Percentage of live cells. Does not consider tombstones or cell updates shadowing cells. That is it is percentage of non-tombstoned cells to total number of cells.

purge

Finds the largest reclaimable partitions (GCable). Intensive process, effectively does "fake" compactions to calculate metrics.

Usage

ic-sstable-tools purge [-n <num>] [-t <snapshot>] [-f <filter>] <keyspace> <column-family>

-h	Display help
-b	Batch mode. Uses progress indicator that is friendly for running in batch jobs.
-n	Number of partitions to display
-t	Snapshot to analyse. Snapshot is created if none is specified.

Output

Largest reclaimable partitions: Partitions with the largest amount of reclaimable data

Column	Description
Key	The partition key
Size	Total uncompressed size of the partition
Reclaim	Reclaimable uncompressed size
Generations	SSTable generations the partition belongs to

Please see https://www.instaclustr.com/support/documentation/announcements/instaclustr-open-source-project-status/ for Instaclustr support status of this project

cassandra-sstable-tools's Issues

[BUG]

Cannot install from branch 3.11
No pom.xml for 3.11 branch

To Reproduce
$ git checkout cassandra-3.11
sudo mvn clean install

[BUG] ic-pstats output showing inflated values for sizes

Describe the bug
Hello,
We are facing an issue with the ic-pstats on Cassandra V2 where the output of ic-pstats is showing table sizes at about 10x than what is displayed using "du -sh" on our servers.

To Reproduce
For example,

Sizes from linux shell:
[user@server]$ du -sh * | grep Data.db
5.3G k-t-ka-3566231-Data.db
43G k-t-ka-3566232-Data.db
8.7G k-t-ka-3566235-Data.db
9.2G k-t-ka-3566236-Data.db
281M k-t-ka-3566237-Data.db
2.3G k-t-ka-3566238-Data.db
79G k-t-ka-3566239-Data.db
697M k-t-ka-3566242-Data.db
194M k-t-ka-3566247-Data.db
496M k-t-ka-3566256-Data.db
159M k-t-ka-3566261-Data.db
48M k-t-ka-3566262-Data.db

Sizes from ic-pstats:
SSTables:
+----------------------------+----------+-------------------------+-------------------------+-------+------------+--------------------+--------------------+
| SSTable | Size | Min Timestamp | Max Timestamp | Level | Partitions | Avg Partition Size | Max Partition Size |
+----------------------------+----------+-------------------------+-------------------------+-------+------------+--------------------+--------------------+
| k-t-ka-3566231-Data.db | 47.1 GB | 2018-06-28 13:43:37.017 | 2022-12-07 18:05:09.294 | 0 | 1169960 | 40.3 kB | 76.2 MB |
| k-t-ka-3566232-Data.db | 394.7 GB | 2018-05-02 14:44:33.379 | 2022-10-31 18:34:23.681 | 0 | 8845889 | 44.6 kB | 276.7 MB |
| k-t-ka-3566235-Data.db | 79.8 GB | 2018-05-02 15:45:37.477 | 2023-04-03 18:35:59.361 | 0 | 1843560 | 43.3 kB | 34.5 MB |
| k-t-ka-3566236-Data.db | 84.1 GB | 2018-05-02 14:44:47.370 | 2023-02-04 18:16:15.122 | 0 | 1983864 | 42.4 kB | 51.2 MB |
| k-t-ka-3566237-Data.db | 2.2 GB | 2018-11-29 05:14:52.502 | 2023-03-24 12:42:03.975 | 0 | 79023 | 27.8 kB | 6.2 MB |
| k-t-ka-3566238-Data.db | 20.2 GB | 2018-07-05 04:13:49.724 | 2023-04-19 00:55:43.314 | 0 | 505155 | 40.1 kB | 11.1 MB |
| k-t-ka-3566239-Data.db | 681.2 GB | 2018-05-02 14:44:18.494 | 2021-12-15 15:19:41.094 | 0 | 21152939 | 32.2 kB | 1.1 GB |
| k-t-ka-3566242-Data.db | 6.0 GB | 2018-05-02 14:44:41.400 | 2023-04-20 18:01:43.291 | 0 | 165167 | 36.2 kB | 8.1 MB |
| k-t-ka-3566247-Data.db | 1.6 GB | 2018-09-06 13:53:53.353 | 2023-04-22 18:43:14.779 | 0 | 52510 | 29.9 kB | 4.1 MB |
| k-t-ka-3566256-Data.db | 3.9 GB | 2018-06-26 17:39:24.745 | 2023-04-24 17:43:05.252 | 0 | 125750 | 31.2 kB | 16.0 MB |
| k-t-ka-3566261-Data.db | 1.2 GB | 2019-02-08 05:14:38.296 | 2023-04-25 13:05:16.766 | 0 | 46096 | 25.8 kB | 2.7 MB |
+----------------------------+----------+-------------------------+-------------------------+-------+------------+--------------------+--------------------

Expected behavior
We were expecting the size of the data file will be the same from the linux shell and the ic-pstats output.

System and versions (please complete the following information):

• OS: Ubuntu
• Cassandra version: V2.1
• Java version: Java 8
• SSTable tools version: 2.1

java.lang.NoClassDefFoundError

when i try to run ic-summary i get the following :
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/cli/ParseException
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.cli.ParseException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more

[BUG] Droppable value displayed as a decimal but should be whole number

The column 'droppable' is reported with a decimal place in ic-sstable-tools sstables, but the value appears that it should be an integer. The decimal place is confusing as it suggest the number maybe is in KB or something similar.

Also, it might be helpful to add 'TS' as 'droppableTS:'

i.e.,
"avgPartitionSize": "23.3 kB",
"maxPartitionSize": "454.8 kB",
"avgColumnCount": 1642,
"maxColumnCount": 29521,
"droppable": "0.0"

To Reproduce
Steps to reproduce the behavior:

Expected behavior
A clear and concise description of what you expected to happen.

System and versions (please complete the following information):

• OS, Cassandra version, Java version, SSTable tools version

Additional context
Add any other context about the problem here.

cassandra.config

Could you please take a look?

10:53:47.725 [main] ERROR o.a.c.config.DatabaseDescriptor - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Expecting URI in variable: [cassandra.config]. Please prefix the file with file:/// for local files or file:/// for remote files. Aborting. If you are executing this from an external tool, it needs to set Config.setClientMode(true) to avoid loading configuration.
at org.apache.cassandra.config.YamlConfigurationLoader.getStorageConfigURL(YamlConfigurationLoader.java:73) ~[apache-cassandra-2.1.18.jar:2.1.18]
at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:84) ~[apache-cassandra-2.1.18.jar:2.1.18]
at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:161) ~[apache-cassandra-2.1.18.jar:2.1.18]
at org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:136) ~[apache-cassandra-2.1.18.jar:2.1.18]
at com.instaclustr.sstabletools.cassandra.CassandraBackend.(CassandraBackend.java:32) [ic-sstable-tools.jar:na]
at com.instaclustr.sstabletools.SSTableMetadataCollector.main(SSTableMetadataCollector.java:65) [ic-sstable-tools.jar:na]
Expecting URI in variable: [cassandra.config]. Please prefix the file with file:/// for local files or file:/// for remote files. Aborting. If you are executing this from an external tool, it needs to set Config.setClientMode(true) to avoid loading configuration.
Fatal configuration error; unable to start. See log for stacktrace.

Thanks

Update build.xml for creating a fat jar that includes all external dependencies

The project uses a bunch of other external libraries, but those dependencies are not specified in the build.xml file. It would be great if you could update the build file so that a fat jar that includes all external dependencies can be built using ANT or maven.

missing jamm-0.3.0.jar ?

I have a feeling I'm missing something obvious, but whenever I run an ic-* command I'm getting an error about missing jamm jar similar to

Error opening zip file or JAR manifest missing : /x/y/z/../lib/jamm-0.3.0.jar
Error occurred during initialization of VM
agent library failed to init: instrument

Here's what I've done so far:
git clone https://github.com/instaclustr/cassandra-sstable-tools.git
cd cassandra-sstable-tools
apt-get install ant
ant -Dcassandra.version=3.0.12 (built fine)
cp dist/3.0.12/ic-sstable-tools.jar /usr/share/cassandra/
restart cassandra
bin/ic-summary

I see a jamm jar
cassandra-sstable-tools/cassandra/3.0.12/lib/jamm-0.3.0.jar

Not working with Cassandra 3.11

+ ant -Dcassandra.version=3.11.2
Buildfile: /var/lib/jenkins/jobs/cassandra-sstable-tools/workspace/build.xml

clean:

cassandra-check:

download:
    [mkdir] Created dir: /var/lib/jenkins/jobs/cassandra-sstable-tools/workspace/cassandra/downloads
      [get] Getting: http://archive.apache.org/dist/cassandra/3.11.2/apache-cassandra-3.11.2-bin.tar.gz
      [get] To: /var/lib/jenkins/jobs/cassandra-sstable-tools/workspace/cassandra/downloads/apache-cassandra-3.11.2-bin.tar.gz

extract:
    [mkdir] Created dir: /var/lib/jenkins/jobs/cassandra-sstable-tools/workspace/cassandra/3.11.2
    [untar] Expanding: /var/lib/jenkins/jobs/cassandra-sstable-tools/workspace/cassandra/downloads/apache-cassandra-3.11.2-bin.tar.gz into /var/lib/jenkins/jobs/cassandra-sstable-tools/workspace/cassandra
     [move] Moving 5635 files to /var/lib/jenkins/jobs/cassandra-sstable-tools/workspace/cassandra

compile:
    [mkdir] Created dir: /var/lib/jenkins/jobs/cassandra-sstable-tools/workspace/build/classes
     [copy] Copying 5 files to /var/lib/jenkins/jobs/cassandra-sstable-tools/workspace/src
    [javac] Compiling 23 source files to /var/lib/jenkins/jobs/cassandra-sstable-tools/workspace/build/classes
    [javac] /var/lib/jenkins/jobs/cassandra-sstable-tools/workspace/src/com/instaclustr/sstabletools/cassandra/PurgeStatisticBackend.java:224: error: method purge in interface Row cannot be applied to given types;
    [javac]             return row.purge(purger, nowInSec);
    [javac]                       ^
    [javac]   required: DeletionPurger,int,boolean
    [javac]   found: DeletionPurger,int
    [javac]   reason: actual and formal argument lists differ in length
    [javac] /var/lib/jenkins/jobs/cassandra-sstable-tools/workspace/src/com/instaclustr/sstabletools/cassandra/PurgeStatisticBackend.java:228: error: method purge in interface Row cannot be applied to given types;
    [javac]             return row.purge(purger, nowInSec);
    [javac]                       ^
    [javac]   required: DeletionPurger,int,boolean
    [javac]   found: DeletionPurger,int
    [javac]   reason: actual and formal argument lists differ in length
    [javac] Note: /var/lib/jenkins/jobs/cassandra-sstable-tools/workspace/src/com/instaclustr/sstabletools/cassandra/CassandraBackend.java uses or overrides a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] 2 errors

Error retrieving snapshot for keyspace.columnFamily

I execute ic-cfstats,ic-pstats,ic-purge got the error

Error retrieving snapshot for keyspace.columnFamily

Error: Could not find or load main class com.instaclustr.sstabletools.SummaryCollecto

Not able to find ic-sstable-tools.jar to be copied in cassandra home and also seeing Error: Could not find or load main class com.instaclustr.sstabletools.SummaryCollecto error.

Can't get to work due to java.lang.AssertionError

Hi InstaClustr!

Trying to get work sstable-tools, but running into java.lang.AssertionError.
cassandra-sstable tools were built using ant -Dcassandra.version=3.0.11 command

ic-summary

java.lang.AssertionError: []
        at org.apache.cassandra.db.lifecycle.Tracker.getMemtableFor(Tracker.java:312)
        at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1185)
        at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:573)
        at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:421)
        at org.apache.cassandra.db.Mutation.apply(Mutation.java:210)
        at org.apache.cassandra.db.Mutation.apply(Mutation.java:215)
        at org.apache.cassandra.db.Mutation.apply(Mutation.java:224)
        at org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:566)
        at org.apache.cassandra.cql3.statements.ModificationStatement.executeInternal(ModificationStatement.java:556)
        at org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:295)
        at org.apache.cassandra.db.SystemKeyspace.updateSchemaVersion(SystemKeyspace.java:724)
        at org.apache.cassandra.config.Schema.updateVersion(Schema.java:563)
        at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:138)
        at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:126)
        at com.instaclustr.sstabletools.cassandra.CassandraBackend.<clinit>(CassandraBackend.java:33)
        at com.instaclustr.sstabletools.SummaryCollector.main(SummaryCollector.java:50)