Add Cache iteritems Method about python-diskcache

[grantjenks]

Deployed at 2.6.3

The default Index uses a Cache with the "none" eviction policy. If you want to limit the space the cache uses then you'll need to use Index.fromcache like:

import diskcache as dc
cache = dc.Cache('/tmp/cache', size_limit=int(1e9))
index = dc.Index.fromcache(cache)

You can add other parameters to Cache as necessary. Just remember that items may then be evicted based on space needs. (The default never evicts items.)

I haven't added docs to update the tutorial yet but the api docs at http://www.grantjenks.com/docs/diskcache/api.html#index have code samples for every method.

[grantjenks]

That is intentional. I wasn't sure what to do about expired keys. Some users want a Mapping data type and to skip expiration altogether. Other users want a cache and expiration is critical.

How are you using DiskCache? Is it as a cross-process mapping?

[pombredanne]

How are you using DiskCache? Is it as a cross-process mapping?

I have two usages in scancode-toolkit:

• a cross-process size-limited fanout cache for license matches used across invocations that you can see here: https://github.com/nexB/scancode-toolkit/blob/5aa84c1b10d6c19244f98847041a2d4b03503df2/src/licensedcode/cache.py#L153
• right I am adding a cross-process mapping that lives (and is deleted once done) for the duration of an invocation. that I am working on ATM for this ticket nexB/scancode-toolkit#267

[grantjenks]

I designed all methods on Cache to return as quickly as possible and hold locks on the underlying SQLite database as little as possible. Iteration is a corner case because it could take a long time. So I decided to permit iterating expired items. That's a bit surprising but you can call evict beforehand and explicitly remove cruft if you like.

Your code is not setting the timeout parameter so keys will expire only based on cache size. Still, just because iteration yields a key, does not mean that value is still in the cache. I'm afraid the best option is to do a get.

More thought is warranted. Cache/FanoutCache are probably more building blocks for a full-featured Mapping data type.

For now, this seems like an easy case for subclassing:

from diskcache import Cache

class MyCache(Cache):
    def iteritems(self):
        for key in self:
            try:
                yield key, self[key]
            except KeyError:
                pass

[pombredanne]

Thanks. FWIW, the FanoutCache code is not a use case for iteration but code I have not yet pushed.
And, to your point a subclass with iteritems is about right for now and I am using about the same approach (without the subclass).

[pombredanne]

And here is the code where I could possibly benefit from an iteritems like function: https://github.com/nexB/scancode-toolkit/blob/8a106d3f6ce22e381b18ec55ddfbaceb44ce4edb/src/scancode/cache.py#L133

In that use case, the are never any expired items, to I am quite fine with the behaviour of iterating over everything

[grantjenks]

I recently added diskcache.Index which has a MutableMapping interface. Does that work for you scenario? See http://www.grantjenks.com/docs/diskcache/api.html#index for details. Note that this still is missing iteritems but I would be more amenable to adding it there.

[pombredanne]

@grantjenks that would be fine! Thanks!