Feature: retention period about strfry

[hoytech]

Neat idea on the max DB size: The tough thing is figuring out a policy on which events to purge.

If you're OK with the time-based policy, you could put strfry delete --age $MAX_AGE_SECONDS into a cron-job.

[kroese]

Great, did not know that command existed! That will do for now..

[kroese]

Actually it didn't went like I expected.

When running the delete command which removed 5 million events, the database size was INCREASED with 5 GB afterwards. I am not sure how this works, maybe it causes some internal shuffling and lmdb needs to allocate more space to rearrange everything during a delete operation?

So next I did a compact which luckily decreased the size finally. But this cannot be run from the daily cron because A) it takes hours B) it requires to take strfry offline because the database files need to be swapped.

I will just keep the daily delete cron and hopefully it will keep the database the same size from now on. As lmdb is supposed to re-use the space from the deleted records for new data.

[hoytech]

I think this behaviour makes sense: When you delete values from LMDB, it doesn't actually shrink the DB size, it just leaves it as free-space that will get used for subsequent writes.

So I think doing the compact one time is probably sufficient, and the DB size will grow up to the "high water mark" (the largest amount stored at any one given time), and stay there.

[kroese]

Yes. I didnt expect it to shrink it by the deletes, I was just surprised it growing so much by just deleting stuff.