Add sstable disk size metric about scylladb

[raphaelsc]

This metric will be very useful to understand disk usage by sstables per shard, e.g. see compaction dropping space on the fly, and also easily see data inbalance by shard due to bad modelling. Today we have to scan the directory to get that info, which is bad.

This metric sounds useful. How and when do we update the metrics? E.g., background scan at a low frequency (10s), or update the metric immediately when a sstable is added to / removed from the system. We need to consider file stream and sstables in upload directory too.

SSTable manager knows when a sstable is added / closed, we can easily calculate it on the fly without any scanning. So it covers all sstables in the system. We can consider only sealed sstables.

[avikivity]

It doesn't work as a metric, you can't look at tables, you can't get distributions.

Better a virtual table. We could have a virtual table for sstables, or a virtual table for files under data/.

[raphaelsc]

It doesn't work as a metric, you can't look at tables, you can't get distributions.

Better a virtual table. We could have a virtual table for sstables, or a virtual table for files under data/.

today we have bloom filter memory size, and you know the percentage of it in comparison to ram. a disk size metric will allow us to quickly get inbalance. I am investigated a problem where this metric would help a lot.

an advance of metric over virtual table is that you understand behavior over time, and I need that for correlating disk usage activity (per shard) with compaction.

[raphaelsc]

For example, right now, I am looking at number of sstables per shard, bloom filter size per shard, I need sstable disk size per shard, and I only have a node wide one. Today disk usage metric as provided by OS is suboptimal when we need correlation with activity in a single shard

[raphaelsc]

I think it will be valuable. And I wish we had this metric now.

[raphaelsc]

What if we add both virtual table and metric? They're not mutually exclusive. They both have its pros / cons. If I am proven wrong, then we remove the metric.

[raphaelsc]

I put more thought on this and I think metric is valuable. After all, this issue was opened in the context of an investigation and the lack of metric made my life much harder for understanding compaction behavior in a shard. You're right it doesn't provide distribution per table but it already provides distribution per shard (which is enough for detecting imbalance as a first step; today you cannot see it through monitoring) and it's the only way to see behavior over time (something virtual table cannot provide and it's important for correlation with activity in a shard). @avikivity Do you agree? Virtual table can still be added in addition to the metric.

[asias]

This metric will be very useful to understand disk usage by sstables per shard, e.g. see compaction dropping space on the fly, and also easily see data inbalance by shard due to bad modelling. Today we have to scan the directory to get that info, which is bad.

This metric sounds useful. How and when do we update the metrics? E.g., background scan at a low frequency (10s), or update the metric immediately when a sstable is added to / removed from the system. We need to consider file stream and sstables in upload directory too.

[changkhothuychung]

Hi, is this issue still active? I am new to this project, and would like to take this issue if possible. Thanks!

[mykaul]

Hi, is this issue still active? I am new to this project, and would like to take this issue if possible. Thanks!

Tanks @changkhothuychung - it's not actively being worked on - feel free to take this.