maypok86 / otter

A high performance lockless cache for Go.

High performance in-memory cache

📖 Contents

• Motivation
• Related works
• Features
• Usage
  • Requirements
  • Installation
  • Examples
• Performance
  • Throughput
  • Hit ratio
• Contribute
• License

💡 Motivation

I once came across the fact that none of the Go cache libraries are truly contention-free. Most of them are a map with a mutex and an eviction policy. Unfortunately, these are not able to reach the speed of caches in other languages (such as Caffeine). For example, the fastest cache from Dgraph labs called Ristretto, was faster than competitors by 30% at best (Otter is many times faster) but had poor hit ratio, even though its README says otherwise. This can be a problem in real-world applications, because no one wants to bump into performance of a cache library 🙂. As a result, I wanted to make the fastest, easiest-to-use cache with excellent hit ratio.

Please leave a ⭐ as motivation if you liked the idea 😄

🗃 Related works

Otter is based on the following papers

• BP-Wrapper: A Framework Making Any Replacement Algorithms (Almost) Lock Contention Free
• FIFO queues are all you need for cache eviction
• Bucket-Based Expiration Algorithm: Improving Eviction Efficiency for In-Memory Key-Value Database
• A large scale analysis of hundreds of in-memory cache clusters at Twitter

✨ Features

• Simple API: Just set the parameters you want in the builder and enjoy
• Autoconfiguration: Otter is automatically configured based on the parallelism of your application
• Generics: You can safely use any comparable types as keys and any types as values
• TTL: Expired values will be automatically deleted from the cache
• Cost-based eviction: Otter supports eviction based on the cost of each item
• Excellent throughput: Otter is currently the fastest cache library with a huge lead over the competition
• Great hit ratio: New S3-FIFO algorithm is used, which shows excellent results

📚 Usage

📋 Requirements

• Go 1.19+

🛠️ Installation

go get -u github.com/maypok86/otter

✏️ Examples

Otter uses a builder pattern that allows you to conveniently create a cache instance with different parameters.

Cache with const TTL

package main

import (
    "fmt"
    "time"

    "github.com/maypok86/otter"
)

func main() {
    // create a cache with capacity equal to 10000 elements
    cache, err := otter.MustBuilder[string, string](10_000).
        CollectStats().
        Cost(func(key string, value string) uint32 {
            return 1
        }).
        WithTTL(time.Hour).
        Build()
    if err != nil {
        panic(err)
    }

    // set item with ttl (1 hour) 
    cache.Set("key", "value")

    // get value from cache
    value, ok := cache.Get("key")
    if !ok {
        panic("not found key")
    }
    fmt.Println(value)

    // delete item from cache
    cache.Delete("key")

    // delete data and stop goroutines
    cache.Close()
}

Cache with variable TTL

package main

import (
    "fmt"
    "time"

    "github.com/maypok86/otter"
)

func main() {
    // create a cache with capacity equal to 10000 elements
    cache, err := otter.MustBuilder[string, string](10_000).
        CollectStats().
        Cost(func(key string, value string) uint32 {
            return 1
        }).
        WithVariableTTL().
        Build()
    if err != nil {
        panic(err)
    }

    // set item with ttl (1 hour)
    cache.Set("key1", "value1", time.Hour)
    // set item with ttl (1 minute)
    cache.Set("key2", "value2", time.Minute)

    // get value from cache
    value, ok := cache.Get("key1")
    if !ok {
        panic("not found key")
    }
    fmt.Println(value)

    // delete item from cache
    cache.Delete("key1")

    // delete data and stop goroutines
    cache.Close()
}

📊 Performance

The benchmark code can be found here.

🚀 Throughput

Throughput benchmarks are a Go port of the caffeine benchmarks.

Read (100%)

In this benchmark 8 threads concurrently read from a cache configured with a maximum size.

Read (75%) / Write (25%)

In this benchmark 6 threads concurrently read from and 2 threads write to a cache configured with a maximum size.

Read (50%) / Write (50%)

In this benchmark 4 threads concurrently read from and 4 threads write to a cache configured with a maximum size.

Read (25%) / Write (75%)

In this benchmark 2 threads concurrently read from and 6 threads write to a cache configured with a maximum size.

Write (100%)

In this benchmark 8 threads concurrently write to a cache configured with a maximum size.

Otter shows fantastic speed under all workloads except extreme write-heavy, but such a workload is very rare for caches and usually indicates that the cache has a very small hit ratio.

🎯 Hit ratio

Zipf

S3

This trace is described as "disk read accesses initiated by a large commercial search engine in response to various web search requests."

DS1

This trace is described as "a database server running at a commercial site running an ERP application on top of a commercial database."

P3

The trace P3 was collected from workstations running Windows NT by using Vtrace which captures disk operations through the use of device filters

P8

The trace P8 was collected from workstations running Windows NT by using Vtrace which captures disk operations through the use of device filters

LOOP

This trace demonstrates a looping access pattern.

OLTP

This trace is described as "references to a CODASYL database for a one hour period."

In summary, we have that S3-FIFO (otter) is inferior to W-TinyLFU (theine) on lfu friendly traces (databases, search, analytics), but has a greater or equal hit ratio on web traces.

👏 Contribute

Contributions are welcome as always, before submitting a new PR please make sure to open a new issue so community members can discuss it. For more information please see contribution guidelines.

Additionally, you might find existing open issues which can help with improvements.

This project follows a standard code of conduct so that you can understand what actions will and will not be tolerated.

📄 License

This project is Apache 2.0 licensed, as found in the LICENSE.

otter's Issues

List of keys in the cache

Clarification and motivation

Hi, Is there anyway to get array of all keys?
And if no, Do you have any intention to add this feature?

Acceptance criteria

I need to get list of keys exist in the cache at certain point of my program that i don't have keys.

Question about add node to Variable

Before add a node to Variable, it needs to call findBucket to find a location for the node.

func (v *Variable[K, V]) findBucket(expiration uint32) node.Node[K, V] {
	duration := expiration - v.time
	length := len(v.wheel) - 1
	for i := 0; i < length; i++ {
		if duration < spans[i+1] {
			ticks := expiration >> shift[i]
			index := ticks & (buckets[i] - 1)
			return v.wheel[i][index]
		}
	}
	return v.wheel[length][0]
}

My question is why use ticks := expiration >> shift[i] (not ticks := duration >> shift[i]) to calculate the location in v.wheel[i] ?
I think use ticks := duration >> shift[i] is more reasonable, because it represent the distance(Or time difference) from the avaliable data location(v.time>> shift[i]). expiration is a random value(unixtime.Now()+ttl), it can be anywhere in v.wheel[i], conflict with the RemoveExpired.

is there a function to return the number of hits for an "object"?

e.g. .HitCount() return the number of times the key is hit etc. possible to make it so that it returns both the value and hit count at the same time?

e.g. value, hitcount, err := cache.GetWithHitCount("key")

this will be extremely useful to check if corresponding data should be saved in disk cache etc

Add the copied map version

otter/internal/hashtable/map.go

Line 5 in 9d61c67

// https://github.com/puzpuzpuz/xsync/blob/main/mapof.go

There are many version for puzpuzpuz/xsync, i think it is better to add the source mapof version, in case there are some issues that existed in the source, but fixed in latter releases.

Cleaner api contracts

Clarification and motivation

Now otter uses an api very similar to other go cache libraries, but unfortunately it is not particularly beautiful and extensible.

For example, now we have Set(key, value) and SetWithTTL(key, value, ttl) functions, but if we want to add SetIfAbsent(key, value) function, we also need SetIfAbsentWithTTL(key, value, ttl) function, which doesn't look nice anymore. Especially since when we create the cache in the builder we already know its further behavior and whether it will need ttl or not.

We would like to create a cache with a clean api and with the functions it needs using parameters in the builder. This will allow to provide such api as Set(key, value, ttl) and SetIfAbsent(key, value, ttl). Also, since in most cases ttl is the same for all entries in the cache, we don't need to pass it every time at all, further simplifying the api. This will also allow us to use optimizations for the expiration policy, since with the same ttl we only need to maintain a simple fifo queue of items and check their expiration one by one.

Acceptance criteria

1. A more advanced version of the builder has been implemented
2. Otter provides simpler api contracts
3. Implementation tests are written

automatic permanent ttl

Clarification and motivation

Add SetWithTTL to automatically use permanent TTL when TTL is 0. People usually like to use 0ttl or -1TTL to represent permanent TTL. For example, Ristretto 0TTL is permanent TTL, and Redis -1TTL is permanent TTL.

Acceptance criteria

The ttl passed in using the SetWithTTL function is less than 0 and is automatically set to permanent tt. This is also what Ristretto does in Redis.

I have initiated a merge request #38

Eviction policy

Clarification and motivation

The eviction policy for a cache determines which items should be evicted (removed) from the cache when it becomes full and a new item needs to be added. The goal of an eviction policy is to maximize cache hit rate and minimize cache misses by making informed decisions about which items to remove.

We will use an eviction policy called S3-FIFO based on an paper from https://blog.jasony.me/system/cache/2023/08/01/s3fifo

Acceptance criteria

I'll add a basic implementation, but no tests and possibly with bugs for now

sieve

Hi @maypok86 :)

What do you think about sieve ?)
https://junchengyang.com/publication/nsdi24-SIEVE.pdf

I found a rep scalalang2.

He released sieve, and otter comparison in repo with benchmark:
https://github.com/scalalang2/go-cache-benchmark (strange sieve behavior with concurrency>1, need to retest with current version of otter)
And implementation in
https://github.com/1a1a11a/libCacheSim

can someone please do a benchmark on core utilization. e.g. 48, 196 cpu core, 384 cpu core comparison with others

curious on the lock contention and performance on multicore systems.
the original s3-fifo measured up to 16cores only.

Add growable write buffer

Clarification and motivation

Unfortunately, at the moment, otter memory consumption in small cache sizes is quite large compared to other caches. This is caused by the buffers that otter uses, and creates them too large without knowing in advance the characteristics of the workloads. Therefore, it is proposed to implement a growable write buffer that will be able to grow from minimum capacity to maximum capacity.

Invalid syntax

otter/internal/hashtable/map.go

Line 148 in c66ac1f

atomic.StorePointer(&m.table, unsafe.Pointer(t))

With this code, there is an error in my IDE, maybe atomic.StorePointer(&m.table, unsafe.Pointer(&t))?

Fix the damage to the eviction policy if the order of events is incorrect.

Clarification and motivation

Description

All implementations of caches in Go that are trying to be contention-free (ristretto, theine, otter, ccache) currently have problems with maintaining correct operation in the eventual consistency mode.

1. The most common problem (and the simplest) is the race condition with simultaneous Set operations and removal from the eviction policy, followed by removal from the hash table. This leads to zombie entries in the eviction policy and the absence of the desired entry in the hash table. This is a problem in ristretto and ccache. In theine and otter it is solved.
2. A much more terrible problem is the incorrect synchronization of the cost of each entry with the eviction policy. For example, theine updates the cost on the spot and then sends an update event, this can lead to a complete failure of the eviction policy when this entry is competitively transferred from one lru queue to another. This problem exists in theine and ristretto (complete loss of the event).
3. When sending events, they may get mixed up, Delete and Add operations to the eviction policy will be applied in the wrong order and as a result, the eviction policy will be corrupted by the zombie entry. This problem exists in all the mentioned caches.

Goal

We need to get rid of this problem in otter. Most likely, the best solution is to use a finite state machine of two states, live and dead.

Acceptance criteria

1. Fixed damage to the eviction policy in case of incorrect order of events.

Question about the hashtable.Map growThreshold

Here define a growThreshold, if the total nodes in the map table above this value, it needs to resize the map table.
growThreshold defined as growThreshold := float64(tableLen) * bucketSize * loadFactor , here tableLen (len(t.buckets))is the buckets chain number(t.buckets[i] is a chain), not the total nodes number, so we can't compare the chain number with nodes number like:
if t.sumSize() > int64(growThreshold)

Am i right?

Add delete by func

I really like otter, but what do miss (in comparison to ristretto for example) is something similar to a tag option. I need it sometimes to invalidate a part of the cache, and an easy option (in my opinion at least) to implement this would be to add a function to delete a part of the the cache depending on a function.

This would go along something like this:

// DeleteByFunc removes the association for this key from the cache if the selectFunc returns true.
func (bs baseCache[K, V]) DeleteByFunc(selectFunc func(key K, value V) bool) {
	// aggregate all keys to delete
	keysToDelete := make([]K, 0)
	bs.Range(func(key K, value V) bool {
		if selectFunc(key, value) {
	            keysToDelete = append(keysToDelete, key)
        	}
	        return true
	})
	
	for _, key := range keysToDelete {
        	bs.Delete(key)
	}
}

You probably need a mutex, but the general idea should be clear.

Any additional ideas are appreciated.

Improvement of node synchronization

Clarification and motivation

Currently otter uses spinlocks to synchronize nodes while getting values/updating information in nodes. This solution blocks reading values from the same node and slows down reads a lot. Unfortunately, we can't use sync.RWMutex because of xadd operations inside it, and since the amount of work under locks is very small, we will get the same speed, but will waste more memory than spinlock. There are several solutions to this problem:

• Using atomic.Value to store values. This solution leads to fantastic speed of reads/updates, but will require additional 16 bytes per node to store an empty interface and unnecessary allocation for each value.
• Using seqlock to synchronize nodes. This synchronization mechanism used in linux requires only two load operations on reads, which should help a lot in our case. But this method also has its problems:
  1. According to go data race detector seqlock is a data race, as it allows reading dirty data, but decides whether to give it or not by an additional check. This problem can be solved by using compile flags and compile otter with seqlock only if the -race flag was not passed, but still not very nice.
  2. As it was found out recently, the go compiler does not force barriers between two atomic load operations and rearranges value copying after the second load, which leads to the fact that such code often returns dirty data. gossa. It seems that this problem can be solved by wrapping the value copying of lock and unlock operations on a local mutex for each function, which does not lead to additional contention but triggers memory barriers.
• Using RCU operations. That is, instead of updating nodes, we will create new ones and update the eviction policy, then we don't need any synchronization when reading. This unlocks fantastic speed on reads, but slows down updates a lot.

Acceptance criteria

1. The type of node synchronization to be used in the future has been selected.
2. Implemented a new version of node synchronization.
3. Implementation tests are written

Add statistics

Clarification and motivation

A cache system needs statistics to optimize its performance and improve the efficiency of data retrieval. Statistics provide valuable information about the access patterns and usage of data, which can be used to make informed decisions regarding caching strategies and resource allocation. Here are a few reasons why statistics are important for a cache system:

1. Access patterns: Statistics help identify the most frequently accessed data items or resources. By analyzing access patterns, the cache system can prioritize caching those items that are accessed more frequently, reducing the cache miss rate and improving overall performance.

2. Cache hit rate: Statistics allow the cache system to measure the cache hit rate, which is the percentage of requests that are successfully served from the cache

Acceptance criteria

Added fast and test covered statistics package

Is it possible to set capacity by bytes?

Hi @maypok86,

First of all, thank you for making this amazing library. I didn't know S3FIFO before until now. I learned a lot from your work.

Then this issue is not a feature request because I guess setting capacity by bytes is almost impossible based on my understanding of S3FIFO which requires the maximum entry count to be set beforehand.

However, setting the maximum entry count at the beginning is hard in cases of caching data of different sizes which I believe are the most of the cases. Therefore I think it will be better for users to set the capacity by bytes.

I just want to know if you have any idea about this topic or alternatives. Thanks!

concurrent access slice

In internal/lossy/buffer.go, I think func (b *Buffer[K, V]) Add(n node.Node[K, V]) *PolicyBuffers[K, V] and func (b *Buffer[K, V]) Free() can be called at the same time,
so when runAdd( pb.Returned = append(pb.Returned, b.nodeManager.FromPointer(v))) and Free() may concurrent access the same slice because there is no locker or atomic.CompareAndSwapPointer.

Expiration algorithm

Clarification and motivation

The necessary expiration algorithm for a cache determines when and how items in the cache should be evicted or removed. The goal is to remove items that are no longer needed or have expired in order to make space for new items.

Acceptance criteria

Expiry algorithm for variable ttl is implemented

Reduce memory footprint per node

Clarification and motivation

Right now otter uses quite a few extra bytes per node (at least 48).

I have an idea how to easily reduce this to 40 bytes.

Right now the expiry time is stored as the number of seconds since January 1, 1970 UTC in uint64 and takes 8 bytes. But we can only store the time difference between the start and the node's expiry time, then we can store in uint32. I doubt a program using otter will ever run continuously for 4294967295 seconds or 136+ years. 🙂

Also otter stores a mutex per node which takes 8 bytes, but it seems we just need a simple spinlock which will take 4 bytes

Acceptance criteria

1. New version of storing expiry time and spinlock has been implemented
2. Added implementation tests

About expiration performance

I am the author of Bucket-Based Expiration Algorithm: Improving Eviction Efficiency for In-Memory Key-Value Database and today I find this project :) Really nice implementation in Go!

For curiosity, I wonder how you found this paper. Do you have a benchmark for the expiration? And is there any space I could contribute to this project? (I have >5 years experiences with Go.)

otter is one of the fastest implementation but memory usage is too high, possible to reduce?

otther completes at 26ns while phuslu completes at 29ns

./cachemain otter 10000000
otter	10000000	1216 MB	1600 MB	1431 MB
./cachemain nottl 10000000
nottl	10000000	355 MB	433 MB	438 MB

// memusage.go
package main

import (
        "fmt"
        "os"
        "runtime"
        "time"
        "strconv"

        theine "github.com/Yiling-J/theine-go"
        "github.com/cespare/xxhash/v2"
        cloudflare "github.com/cloudflare/golibs/lrucache"
        ristretto "github.com/dgraph-io/ristretto"
        freelru "github.com/elastic/go-freelru"
        hashicorp "github.com/hashicorp/golang-lru/v2/expirable"
        ccache "github.com/karlseguin/ccache/v3"
        lxzan "github.com/lxzan/memorycache"
        otter "github.com/maypok86/otter"
        ecache "github.com/orca-zhang/ecache"
        phuslu "github.com/phuslu/lru"
)

const keysize = 16

var keys []string

func main() {
        name := os.Args[1]
        cachesize, _ := strconv.Atoi(os.Args[2])

        keys = make([]string, cachesize)
        for i := 0; i < cachesize; i++ {
                keys[i] = fmt.Sprintf(fmt.Sprintf("%%0%dd", keysize), i)
        }

        var o runtime.MemStats
        runtime.ReadMemStats(&o)

        map[string]func(int){
                "nottl":      SetupNottl,
                //"phuslu":     SetupPhuslu,
                "freelru":    SetupFreelru,
                "ristretto":  SetupRistretto,
                "otter":      SetupOtter,
                "lxzan":      SetupLxzan,
                "ecache":     SetupEcache,
                "cloudflare": SetupCloudflare,
                "ccache":     SetupCcache,
                "hashicorp":  SetupHashicorp,
                "theine":     SetupTheine,
        }[name](cachesize)

        var m runtime.MemStats
        runtime.ReadMemStats(&m)

        fmt.Printf("%s\t%d\t%v MB\t%v MB\t%v MB\n",
                name,
                cachesize,
                (m.Alloc-o.Alloc)/1048576,
                (m.TotalAlloc-o.TotalAlloc)/1048576,
                (m.Sys-o.Sys)/1048576,
        )
}

func SetupNottl(cachesize int) {
        cache := phuslu.New[string, int](cachesize)
        for i := 0; i < cachesize; i++ {
                cache.Set(keys[i], i)
        }
}

/*
func SetupPhuslu(cachesize int) {
        cache := phuslu.NewTTLCache[string, int](cachesize)
        for i := 0; i < cachesize; i++ {
                cache.Set(keys[i], i, time.Hour)
        }
}
*/

func SetupFreelru(cachesize int) {
        cache, _ := freelru.NewSharded[string, int](uint32(cachesize), func(s string) uint32 { return uint32(xxhash.Sum64String(s)) })
        for i := 0; i < cachesize; i++ {
                cache.AddWithLifetime(keys[i], i, time.Hour)
        }
}

func SetupOtter(cachesize int) {
        //cache, _ := otter.MustBuilder[string, int](cachesize).WithVariableTTL().Build()
        cache, _ := otter.MustBuilder[string, int](cachesize).Build()
        for i := 0; i < cachesize; i++ {
                cache.Set(keys[i], i)
        }
}

func SetupEcache(cachesize int) {
        cache := ecache.NewLRUCache(1024, uint16(cachesize/1024), time.Hour)
        for i := 0; i < cachesize; i++ {
                cache.Put(keys[i], i)
        }
}

func SetupRistretto(cachesize int) {
        cache, _ := ristretto.NewCache(&ristretto.Config{
                NumCounters: int64(10 * cachesize), // number of keys to track frequency of (10M).
                MaxCost:     int64(cachesize),      // maximum cost of cache (1M).
                BufferItems: 64,             // number of keys per Get buffer.
        })
        for i := 0; i < cachesize; i++ {
                cache.SetWithTTL(keys[i], i, 1, time.Hour)
        }
}

func SetupLxzan(cachesize int) {
        cache := lxzan.New[string, int](
                lxzan.WithBucketNum(128),
                lxzan.WithBucketSize(cachesize/128, cachesize/128),
                lxzan.WithInterval(time.Hour, time.Hour),
        )
        for i := 0; i < cachesize; i++ {
                cache.Set(keys[i], i, time.Hour)
        }
}

func SetupTheine(cachesize int) {
        cache, _ := theine.NewBuilder[string, int](int64(cachesize)).Build()
        for i := 0; i < cachesize; i++ {
                cache.SetWithTTL(keys[i], i, 1, time.Hour)
        }
}

func SetupCloudflare(cachesize int) {
        cache := cloudflare.NewMultiLRUCache(1024, uint(cachesize/1024))
        for i := 0; i < cachesize; i++ {
                cache.Set(keys[i], i, time.Now().Add(time.Hour))
        }
}

func SetupCcache(cachesize int) {
        cache := ccache.New(ccache.Configure[int]().MaxSize(int64(cachesize)).ItemsToPrune(100))
        for i := 0; i < cachesize; i++ {
                cache.Set(keys[i], i, time.Hour)
        }
}

func SetupHashicorp(cachesize int) {
        cache := hashicorp.NewLRU[string, int](cachesize, nil, time.Hour)
        for i := 0; i < cachesize; i++ {
                cache.Add(keys[i], i)
        }
}

Callback function for evict items

Clarification and motivation

This is awesome lib. But if have callback function for evict items or RemovalListener, it is very useful for maintain or free memory of item value.

Thanks.

Acceptance criteria

Add onheap caches to benchmarks

Clarification and motivation

1. Now offheap caches are included in the benchmarks: bigcache and fastcache. Comparison with which does not look fair.
2. I would like to add more different implementations to the comparisons in the throughput benchmarks, because very few implementations are used now.

Now we need to select those implementations.

Acceptance criteria

1. Added new implementations to comparisons and removed offheap caches.
2. Updated README.

Add save and restore to file

Clarification and motivation

Hello, i'm interested in testing the performance of your cache in my application. Currently I use theine. Only thing I can see that I am missing from the API is the ability to save and restore the cache from file, so warm restarts are possible (very important for my app).

Acceptance criteria

Important that the file object is e.g. a generic io.ReadCloser, as we use afs (abstract file system) to use any cloud object store.

Thanks!

support json unmarshal?

Clarification and motivation

i wanna unmarshal []byte from file to a *otter.Cache[string, any],but i get nothing in *otter.Cache[string, any]

Acceptance criteria

support json unmarshal or how can i unmarshal correctly

[BUG] The capacity of cache not work

Description

set capacity to 10, but the cache can store over 10 elements.

To Reproduce

Steps to reproduce the behavior:

cache, err := otter.MustBuilder[int, int](10). //set capacity to 10
		Cost(func(key int, value int) uint32 { return 1 }).
		WithTTL(time.Second).
		Build()
	if err != nil {
		panic(err)
	}

	// put 20(>10) items to cache
	for i := 0; i < 20; i++ {
		cache.Set(i, i)
	}

	cntCacheMiss := 0
	for i := 0; i < 20; i++ {
		value, ok := cache.Get(i)
		fmt.Println(value)
		if !ok {
			cntCacheMiss++
		}
	}
	assert.Equal(t, 10, cntCacheMiss) //fails

Expected behavior

test pass

Environment

• OS: windows 11
• go 1.20 amd64

Additional context

github.com/maypok86/otter v1.2.1

advised to add some methods

Clarification and motivation

i wanna add some method to otter to use more conveniently

Acceptance criteria

add three methods i'm thinking now

1. getKeys() get all keys as a slice of otter
2. getVals() get all vals as a slice of otter
3. setIfAbsent(key,val) with a bool as return result,this can set a new value when the key is not in otter,but not update valu when the key is not in otter

Shard

Clarification and motivation

We need a specific hash table that ignores hash collisions and supports node swapping

For the moment we will probably use a modification of the hash table from this repository, but in the future we should implement ConcurrentHashMap from the standard Java library or some other implementation.

Acceptance criteria

Added hash table implementation with desired characteristics and covered by tests

[BUG] About S3-FIFO evict

When the cache full , it should evict some nodes.
From the S3-FIFO paper, when evict node from small or main queue, it should try to evict the node from the tail, like:

But here , it choose the evicted node from head, is this more reasonable ?

Request Get Expiration With Get

Clarification and motivation

I have a script that updating some variables and users get them from cache. But I want to get expiration for showing them last update time.

Acceptance criteria

Get expiration time with get function or new function with expiration time

Add InitialCapacity function to builder

Clarification and motivation

Since otter supports cost-based eviction, we cannot pre-initialize internal structures, but in reality this feature is rarely used, and other users may not be happy because otter allocated more memory than it could. So we would like to add an InitialCapacity function with which users could preallocate otter's internal data structures.

The usage is envisioned like this

c, err := otter.MustBuilder[int, int](size).
    InitialCapacity(size).
    WithTTL(time.Second).
    Build()

Acceptance criteria

1. The InitialCapacity function is added to the builder.
2. Implementation tests are written

Benchmarks

Clarification and motivation

We need benchmarks to compare the performance of ours with the other caches available in go

Acceptance criteria

Implemented basic benchmarks for this project

adding more than the maximum capacity limit returns a Boolean value

#clarificationandmotivation

Use Set and SetWithTTL to return false if the Key setting exceeds the maximum capacity of NewCache and fails.

#Acceptance criteria

For example, if I use NewCache to set the maximum capacity to 3, and I set 4 Keys, the first three keys will return true if they do not exceed the maximum capacity limit, and the fourth key will return false if they exceed the maximum capacity limit.

Some potential bug and improvements

1. If my understanding is correct, the current implementation has a bug that may evict more objects than needed. I believe it should return at this line instead of continuing to evict because an object has already been evicted from the main queue https://github.com/maypok86/otter/blob/main/internal/s3fifo/small.go#L62

2. As you have mentioned (Yiling-J/theine-go#29), it is better to store a fingerprint instead of the full object in the ghost. Storing data in the ghost defeats the purpose of using a ghost queue...

3. the size of the ghost queue does not need to be accurate, so you can estimate using the total cache size divided by the mean object size. VMware found that the ghost queue size has little impact between 40% and 200% of the main queue size (in terms of #objects).

4. there are multiple if statement seems not necessary, e.g., https://github.com/maypok86/otter/blob/main/internal/s3fifo/small.go#L55

[BUG] Capacity overflowing uint32

Description

I am trying to use bytes as my cost function, but the policy is silently casting the capacity from an int to a uint32. This effectively makes the max cache size 4GB, where values over that overflow and get a ~random size

To Reproduce

Set the capacity to math.MaxUint32 + 1 every entry will be rejected.

Expected behavior

Normal caching (or at least an error)

Environment

• OS: macos

Additional context

About use module "github.com/dolthub/maphash"

This project use a module maphash, It use the golang map internal struct to generate the hash value, as:

a := any(make(map[K]struct{}))
i := (*mapiface)(unsafe.Pointer(&a))
h = i.typ.hasher

And there are some comments like:

//go:build go1.18 || go1.19
// +build go1.18 go1.19

If the map internal struct changed in the future, that convert will fail.

How about just use the map address as the hash value like:

m := any(make(map[int]struct{}))
p := uintptr(unsafe.Pointer(&m))
fmt.Println(p)

Or extract the hasher implement from goland(/src/runtime/alg.go)
func typehash(t *_type, p unsafe.Pointer, h uintptr) uintptr

is this lru cache? can u make it lru without ttl feature?

is this lru cache? can u make it lru without ttl feature?

Node generation

Clarification and motivation

Now otter static configuration of nodes, because of that in most cases, so it is worth trying with codegeneration to reduce memory consumption, and also it will allow using queue for expiration policy with constant ttl (queue) and timing wheel.

Acceptance criteria

1. Node generation added
2. Added expiration policies for constant ttl and variable ttl
3. The implementation is covered by tests

can you do a disk backed version of this?

is there any possibility for disk backed version? async or otherwise with in memory cache or without

Rethinking queues

Clarification and motivation

Currently, Otter uses ring buffer based queues, but this type of queue does not allow easy deletion of an arbitrary element, and we need it because of the expiration policy and arbitrary calls to the Delete function by the user.

Queues on a doubly-linked list can solve this problem, but instead require 16 bytes for each key-value pair (prev and next pointers), and put additional pressure on gc. So one would like to come up with a workaround for this.

And the workaround suggestion is pretty simple. Let's store in each key-value pair store an index to the location where it is stored in the buffer. For example, like this:

type Node[K comparable, V any] struct {
    ...
    policyIndex uint32
}

And when deleting it, just put nil in the right cell. Such a solution will additionally waste 12 bytes (pointer to node and index). The only problem is that we need to maintain indexes in nodes when expanding the buffer. Also, after deleting a node we still occupy space for the cell and when adding new nodes the buffer will have to be expanded twice, but we will not use half of these cells.

That's why we want to understand what type of queues suits us better and implement it

Acceptance criteria

1. The type of queues to be used in the future has been selected.
2. Implemented a new version of eviction policy on these queues.
3. Implementation tests are written

Set to invalid hashmap in concurrent situation

func (m *Map[K, V]) Delete(key K) node.Node[K, V] and func (m *Map[K, V]) Set(n node.Node[K, V]) node.Node[K, V] can run concurrently, when Delete run into this line and Set run into this loop, there is chance that Set node to an invalid(shrinked) table.
I think this may acceptable in this situation.

And here is another question, in hash map set, there is a judgment if m.newerTableExists(t), does it used to lower the probability set to an invalid table? Because when concurrently run multi set, there still may create a new table even after this judgment if m.newerTableExists(t)
Thanks!

Please update the link in the Readme

Hi @maypok86

Please update the link "BP-Wrapper: A Framework Making Any Replacement Algorithms (Almost) Lock Contention Free(http://web.cse.ohio-state.edu/hpcs/WWW/HTML/publications/papers/TR-09-1.pdf)".

I found another link from your reply Yiling-J/theine-go#29 (comment).

https://www.researchgate.net/publication/220966845_BP-Wrapper_A_System_Framework_Making_Any_Replacement_Algorithms_Almost_Lock_Contention_Free

And perhaps you can add the following links about s3fifo:
https://blog.jasony.me/system/cache/2023/08/01/s3fifo
https://blog.jasony.me/system/cache/2023/12/28/fifo

Thanks for the development of this project, I believe there will be a stable version for production use soon ;)

[BUG] 100% CPU usage with idle cache

Thank you for this library @maypok86

Description

Creating a cache leads to one core being utilized at 100%. Creating multiple caches bumps this by one core at a time.

To Reproduce

Steps to reproduce the behavior:

package main

import "github.com/maypok86/otter"

// We assign this so that we're sure it doesn't get garbage collected
var Cache *otter.Cache[string, string]

func main() {
	b, err := otter.NewBuilder[string, string](1_000)
	if err != nil {
		panic(err)
	}

	c, err := b.Build()
	if err != nil {
		panic(err)
	}

	Cache = c

	// Block forever
	select {}
}

Expected behavior

CPU usage is negligible.

Environment

• OS: Both in Linux/Ubuntu and MacOS

Additional context

I believe the issue lies in the goroutine spawned here https://github.com/maypok86/otter/blob/main/cache.go#L74

How the cost works?

Every node has a cost value, when add a node otter will compare the node cost with the policy.MaxAvailableCost(), if it is larger than later, otter will reject to add this node.
I don't fullt understand how the cost infect the s3-FIFO policy. As i know, in s3-FIFO, the length of the queues of s3-FIFO is limited by the cost, a node with large cost will occupy more queue space, but the cost won't affect the evict sequence.
So what the cost does to the otter, will it improve some performance or hit rate in some cases？

v2 contracts

Just random thoughts about new contracts...

Feature: Adding more rich stats

Clarification and motivation

I need stats for

1. how many set operation is rejected at time
2. cost metrics for adding/updating at time
3. how many keys are evicted at time

these stats give me a chance to tune cache configuration and to monitor cache operation is fine.

Acceptance criteria

[BUG] crash because of a bad use of unsafe.Pointer?

Description

I tried to use Otter in FrankenPHP and it made our test suite crash when using the race detector with the following error:

runtime: marked free object in span 0x7f1032cb5358, elemsize=48 freeindex=4 (bad use of unsafe.Pointer? try -d=checkptr)

It looks like it crashes when calling internal/core.NewCache().

To Reproduce

Here is the PR: dunglas/frankenphp#540
And the specific patch to switch to Otter (that creates the crash): dunglas/frankenphp@3dfc3e2 (#540)

Expected behavior

No crash :)

Environment

• OS: Ubuntu (GitHub Actions)

FrankenPHP relies a lot on cgo (but not where otter is used).

Additional context

Relevant logs

runtime: marked free object in span 0x7f1032cb5358, elemsize=48 freeindex=4 (bad use of unsafe.Pointer? try -d=checkptr)
0xc000210000 alloc marked  
0xc000210030 alloc marked  
0xc000210060 alloc unmarked
0xc000210090 alloc marked  
0xc0002100c0 alloc marked  
0xc0002100f0 alloc unmarked
0xc000210120 alloc marked  
0xc000210150 free  unmarked
0xc000210180 free  unmarked
0xc0002101b0 free  unmarked
0xc0002101e0 free  unmarked
0xc000210210 alloc unmarked
0xc000210240 free  unmarked
0xc000210270 free  unmarked
0xc0002102a0 free  marked   zombie
0x000000c0002102a0:  0x746163696c707061  0x77772d782f6e6f69 
0x000000c0002102b0:  0x752d6d726f662d77  0x65646f636e656c72 
0x000000c0002102c0:  0x0000000000000064  0x0000000000000000 
0xc0002102d0 alloc marked  
0xc000210300 alloc unmarked
0xc000210330 alloc unmarked
0xc000210360 free  unmarked
0xc000210390 alloc unmarked
0xc0002103c0 alloc marked  
0xc0002103f0 free  unmarked
0xc000210420 alloc unmarked
0xc000210450 alloc marked  
0xc000210480 alloc marked  
github.com/dunglas/frankenphp_test.runTest.func1({0xba4770, 0xc0002a4d40}, 0xc00067b200)
	/home/runner/work/frankenphp/frankenphp/frankenphp_test.go:72 +0x127 fp=0xc000249d18 sp=0xc000249c88 pc=0x9c2767
github.com/dunglas/frankenphp_test.TestPostSuperGlobals_worker.testPostSuperGlobals.func1(0xc0004ca3e0, 0x0?, 0x63)
	/home/runner/work/frankenphp/frankenphp/frankenphp_test.go:281 +0x44e fp=0xc000249f70 sp=0xc000249d18 pc=0x9c8eee
github.com/dunglas/frankenphp_test.runTest.func2(0x63)
	/home/runner/work/frankenphp/frankenphp/frankenphp_test.go:86 +0x6e fp=0xc000249fb8 sp=0xc000249f70 pc=0x9c25ae
github.com/dunglas/frankenphp_test.runTest.gowrap2()
	/home/runner/work/frankenphp/frankenphp/frankenphp_test.go:88 +0x42 fp=0xc000249fe0 sp=0xc000249fb8 pc=0x9c2502
runtime.goexit({})
	/opt/hostedtoolcache/go/1.22.0/x64/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000249fe8 sp=0xc000249fe0 pc=0x483441
created by github.com/dunglas/frankenphp_test.runTest in goroutine 793
	/home/runner/work/frankenphp/frankenphp/frankenphp_test.go:85 +0x9fb

goroutine 1716 gp=0xc000203340 m=nil [select]:
runtime.gopark(0xb0c940, 0x0, 0x9, 0x3, 0x1)
	/opt/hostedtoolcache/go/1.22.0/x64/src/runtime/proc.go:402 +0x136 fp=0xc00022d9d0 sp=0xc00022d9a0 pc=0x447776
runtime.selectgo(0xc00022dc30, 0xc00022dbb0, 0xc00022dc50?, 0x1, 0x12?, 0x1)
	/opt/hostedtoolcache/go/1.22.0/x64/src/runtime/select.go:327 +0xac6 fp=0xc00022db80 sp=0xc00022d9d0 pc=0x45b5c6
github.com/dunglas/frankenphp.ServeHTTP({0xba4770, 0xc0002a43c0}, 0xc00067a120)
	/home/runner/work/frankenphp/frankenphp/frankenphp.go:462 +0x325 fp=0xc00022dc88 sp=0xc00022db80 pc=0x926de5
github.com/dunglas/frankenphp_test.runTest.func1({0xba4770, 0xc0002a43c0}, 0xc00067a000)
	/home/runner/work/frankenphp/frankenphp/frankenphp_test.go:72 +0x127 fp=0xc00022dd18 sp=0xc00022dc88 pc=0x9c2767
github.com/dunglas/frankenphp_test.TestPostSuperGlobals_worker.testPostSuperGlobals.func1(0xc0004ca3e0, 0x4d45545f52454e4e?, 0x5b)
	/home/runner/work/frankenphp/frankenphp/frankenphp_test.go:281 +0x44e fp=0xc00022df70 sp=0xc00022dd18 pc=0x9c8eee
github.com/dunglas/frankenphp_test.runTest.func2(0x5b)
	/home/runner/work/frankenphp/frankenphp/frankenphp_test.go:86 +0x6e fp=0xc00022dfb8 sp=0xc00022df70 pc=0x9c25ae
github.com/dunglas/frankenphp_test.runTest.gowrap2()
	/home/runner/work/frankenphp/frankenphp/frankenphp_test.go:88 +0x42 fp=0xc00022dfe0 sp=0xc00022dfb8 pc=0x9c2502
runtime.goexit({})
	/opt/hostedtoolcache/go/1.22.0/x64/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00022dfe8 sp=0xc00022dfe0 pc=0x483441
created by github.com/dunglas/frankenphp_test.runTest in goroutine 793
	/home/runner/work/frankenphp/frankenphp/frankenphp_test.go:85 +0x9fb

Thundering herd avoiding LoaderFunc on misses

Clarification and motivation

My go-to cache library for Go so far has been bluele/gcache - probably much less performant than many other alternatives but it has a very handy helper function that handles coordinating loads to avoid thundering herd.
https://pkg.go.dev/github.com/bluele/gcache#readme-loading-cache

Acceptance criteria

Builder could have a new WithLoaderFunc(func(K) V).
Calling Get(key_not_loaded_yet) in multiple goroutines concurrently will call the optional LoaderFunc once and then return to all goroutines all at once.

Cache implementation

Clarification and motivation

We need to implement a basic version of the cache with ttl support in the api

Acceptance criteria

Basic version of the cache is implemented