Table Partition Garbage Collection about faust

[ask]

This is only a problem with the in-memory store, right? If so I think it's OK to not clean up as I consider the memory store a development backend, not suited for production use.

[vineetgoel]

Wait so if we use rocksdb, do we not keep anything in memory at that point? Is everything just in rocks?

[ask]

Yeah, everything is in RocksDB!† Removing a partition is as easy as deleting the database directory for that partition, which is something that we can do when using RocksDB but I'm unsure if deleting
it is such a good idea. I guess it depends on how fast the data spreads to all nodes in the cluster,
but maybe the partitioner should also take into consideration the partitions a node has data for?

It would be neat if the partitioner did this:

1. Chose a node that already have a copy of the partition in a Rocks db. I guess the nodes could
   report list of partitions and checkpoints as metadata.
2. If there are no nodes with the data, assign a fresh node so that it will transfer the data.

If the partitioner implemented this, I think you would rarely come across a situation
where scenario 2) happens, and then you have a more efficient implementation
for recovery, without the needed for cleanup.

† Apart from the Key->Partition index, but that's a LRU cache of max 10k objects by default.