Controlling the number of files generated by Cache about python-diskcache

[grantjenks]

You'll have to test your application to know the impact. Writes to files are done to reduce the time taken writing to the database. But writes don't block reads. If your writes are rare enough, it won't matter. Just measure.

Eviction does not refer to the use of files. When eviction takes place, the item is deleted from the cache entirely.

[grantjenks]

There’s no functionality for that exactly built in today.

One option is to increase the minimum disk file size to something larger or very large. By doing so, more values will be stored directly in sqlite itself and no extra files will be used.

A more complex option is to create a custom eviction scheme. This would require inheriting from Cache and implementing the logic for culling items. You would need to read through the code to understand how. One challenge in this design is knowing the number of files used. I don’t believe that’s tracked currently.

Another option would be to create a custom Disk that somehow packs files together. Maybe you could put a zip file in subdirectories and pack values together in a single zip file. No idea what the performance would be like for reads/writes with that approach.

[GDeLaurentis]

The first option you mention might be a good and easy solution for now. Is there any downside to increasing the minimum disk file size? E.g. slower current reads/writes, or more frequent corruption.

Regarding the eviction scheme, am I correct in saying that it decides how to move entries between the *.val and cache.db files? It sounds like the "none" policy would have the same effect as a very large minimum disk file size, potentially keeping everything in cache.db. However, it seems that changing the eviction logic would postpone the problem, but not fix it.

I'm afraid zipping would be too slow, at least for my application.

Thanks for the clarifications.