Comments (13)
Thanks for finding this! From what I can tell, this is what's happening:
- Attempt to Get random key
- If miss, create random data and Set with a different random key (important for some reason)
- Value is a struct pointer containing key and data as fields
Definitely a confusing one. I'm going to dig around.
from ristretto.
I believe the issue is in here.
Edit: Nevermind, using another bloom filter implementation didn't fix anything. The issue has something to do with Get increments.
from ristretto.
Spent a few hours looking at this, going to continue tomorrow. Some notes for future reference (from policy.go
):
- No memory leakage:
func (p *tinyLFU) Increment(key uint64) {
// flip doorkeeper bit if not already
//if added := p.door.AddIfNotHas(key); !added {
// increment count-min counter if doorkeeper bit is already set.
//p.freq.Increment(key)
//}
p.incrs++
if p.incrs >= p.resetAt {
p.reset()
}
}
- Memory leakage:
func (p *tinyLFU) Increment(key uint64) {
// flip doorkeeper bit if not already
//if added := p.door.AddIfNotHas(key); !added {
// increment count-min counter if doorkeeper bit is already set.
p.freq.Increment(key)
//}
p.incrs++
if p.incrs >= p.resetAt {
p.reset()
}
}
- Memory leakage:
func (p *tinyLFU) Increment(key uint64) {
// flip doorkeeper bit if not already
if added := p.door.AddIfNotHas(key); !added {
// increment count-min counter if doorkeeper bit is already set.
//p.freq.Increment(key)
}
p.incrs++
if p.incrs >= p.resetAt {
p.reset()
}
}
from ristretto.
Maybe this is due to updating a key with a large size without recalculating its cost due to the c.setBuf
contention in Set
?
if c.store.Update(i.keyHash, i.key, i.value) {
i.flag = itemUpdate
}
// attempt to send item to policy
select {
case c.setBuf <- i:
return true
default:
c.Metrics.add(dropSets, i.keyHash, 1)
return false
}
from ristretto.
@brk0v While we do have an eventually-consistent cost mechanism, the problem seems to be within the interaction between TinyLFU counter increments and Sets...
For example, forcing the value + cost update does nothing:
func (c *Cache) Set(key, value interface{}, cost int64) bool {
if c == nil || key == nil {
return false
}
i := &item{
flag: itemNew,
key: key,
keyHash: z.KeyToHash(key, 0),
value: value,
cost: cost,
}
// attempt to immediately update hashmap value and set flag to update so the
// cost is eventually updated
if c.store.Update(i.keyHash, i.key, i.value) {
i.flag = itemUpdate
}
c.setBuf <- i
return true
/*
// attempt to send item to policy
select {
case c.setBuf <- i:
return true
default:
c.Metrics.add(dropSets, i.keyHash, 1)
return false
}
*/
}
And the cost is updated here.
from ristretto.
When I print out cache.Metrics.CostAdded() - cache.Metrics.CostEvicted()
it starts out at around 1GB but keeps going lower and lower.
from ristretto.
Interesting also, if you change the Get to cache.Get("x"+k)
it also works fine. So it has something do with the gets and sets on the overlapping keys.
Scratch that, it still happens, but weirdly it happens a lot slower it seems. But if you remove the Get it doesn't happen.
from ristretto.
I'm seeing this behavior in real life as well. I'm using this ristretto test for microcache: https://github.com/erikdubbelboer/microcache/tree/ristretto
I'm using snappy so most bytes allocated come from the snappy allocation for the body buffer.
I have given ristretto 1GB but if I look at the pprof allocations I see the snappy allocations are taking up 2.5GB after 1 day. There is no other place holding on to these allocations so it all comes from ristretto. Looking at the currently allocated cost ristretto reports 1GB so it thinks it's correct but it's not 😞
pprof output:
154016: 2693161616 [591105: 11179214896] @ 0x8fc5bc 0x905179 0x907665 0x906c8f 0x90a772 0x6e06d4 0xbfdc9f 0x90e76e 0x93a5fc 0x6e06d4 0xbfe540 0x6e06d4 0x6e3b24 0x6df4c5 0x45dc71
# 154016: 2.51GB [591105: 10.41GB]
# 0x8fc5bc github.com/golang/snappy.Encode+0x22c /workspace/vendor/github.com/golang/snappy/encode.go:22
# 0x905179 github.com/erikdubbelboer/microcache.CompressorSnappy.Compress+0x1b9 /workspace/vendor/github.com/erikdubbelboer/microcache/compressor_snappy.go:16
# 0x907665 github.com/erikdubbelboer/microcache.(*microcache).store+0xb5 /workspace/vendor/github.com/erikdubbelboer/microcache/microcache.go:447
# 0x906c8f github.com/erikdubbelboer/microcache.(*microcache).handleBackendResponse+0x44f /workspace/vendor/github.com/erikdubbelboer/microcache/microcache.go:396
# 0x90a772 github.com/erikdubbelboer/microcache.(*microcache).Middleware.func1+0xe12 /workspace/vendor/github.com/erikdubbelboer/microcache/microcache.go:329
# 0x6e06d4 net/http.HandlerFunc.ServeHTTP+0x44 /usr/local/go/src/net/http/server.go:2007
# 0xbfdc9f main.main.func2.1+0x15f /workspace/cmd/content/main.go:108
# 0x90e76e github.com/julienschmidt/httprouter.(*Router).ServeHTTP+0x95e /workspace/vendor/github.com/julienschmidt/httprouter/router.go:334
# 0x93a5fc github.com/poki/mothership/pkg/logging.Middleware.func1+0x5ac /workspace/pkg/logging/middleware.go:31
# 0x6e06d4 net/http.HandlerFunc.ServeHTTP+0x44 /usr/local/go/src/net/http/server.go:2007
# 0xbfe540 main.cors.func1+0x2e0 /workspace/cmd/content/middleware.go:18
# 0x6e06d4 net/http.HandlerFunc.ServeHTTP+0x44 /usr/local/go/src/net/http/server.go:2007
# 0x6e3b24 net/http.serverHandler.ServeHTTP+0xa4 /usr/local/go/src/net/http/server.go:2802
# 0x6df4c5 net/http.(*conn).serve+0x875 /usr/local/go/src/net/http/server.go:1890
from ristretto.
I tried to reproduce this locally and couldn't. So I tried on another machine and I was able to reproduce the error. I compared the versions and realized I had an older version checked out which made me think this was a regression. Tracked it down to #75 on Oct 1st. I read the diff and spotted the error.
// Before Oct 1st commit
victims, added := c.policy.Add(item.key, item.cost)
if added {
// ...
}
for _, victim := range victims {
// ...
}
policy.Add
sometimes returns victims when added is false and the old code handled that correctly.
https://github.com/dgraph-io/ristretto/blob/master/policy.go#L168
// After Oct 1st commit
if victims, added := c.policy.Add(item.key, item.cost); added {
// ...
for _, victim := range victims {
// ...
}
}
The new code is different, causing victims to sometimes not be deleted.
Submitted PR #99
from ristretto.
With the PR it appears to be fixed on my machine. Let me know if you run into any more issues @erikdubbelboer and I'll re-open.
from ristretto.
Seems to be fixed 👍
from ristretto.
Shouldn't there be some test added for this so it doesn't happen again in the future? Ristretto not limiting it's cache to the configured size seems like a major bug to me that should never happen.
from ristretto.
@erikdubbelboer I'm working on a Test PR right now.
from ristretto.
Related Issues (20)
- benchmark about gc or plan to reduce gc HOT 1
- [BUG]: Expiration map grows without limit HOT 6
- [QUESTION]: Why did you choose ristretto as the project name ? HOT 2
- [FEATURE]: Loading or Compute If key is not present
- How many bits tinyLFU uses to count access frequency? HOT 1
- [QUESTION]: Ristretto hit ratio benchmark result HOT 2
- [BUG]: <sync.Pool GC>
- [QUESTION]: Too high missed rate?
- [FEATURE]: Jemalloc 5.3.0
- [FEATURE]: use system jemalloc
- [Non-hashed key values in Item]: onEvict should send the non hashed key values HOT 1
- [BUG]: compilation reports asm error when -buildmode=plugin
- [QUESTION]: Is there any way to implement an expired map counter with ristretto?
- [QUESTION]: How can I pre warm the cache ?
- [QUESTION]: When get v.0.1.2 tag and it's release note?
- [QUESTION]: Why is Ristretto so slow and has such a small hit ratio?
- Update projects using ristretto
- [SERIOUS BUG]: negative key cache hit problem HOT 3
- [BUG]: Can't use Google's uuid.UUID (a byte array rather than slice) as a key HOT 6
- [BUG]: cost added always differ by 56 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ristretto.