How can I understand better why some time allocation fail about cachelib

[jaesoo-fb]

Hi @Chokoabigail,

allocate can fail if CacheAllocator cannot find items to be evicted at the end of eviction queue. The maximum number of retries (say N) is determined by evictionSearchTries and the eviction will be failed if none of the N items at the end of eviction queue for given allocation class are eligible to be evicted; i.e.,

To debug this, you can check how many handles are outstanding by checking CacheAllocator::getNumActiveHandles.

If the number is much higher than expected, you need to take a look at the client code for any leaking handles.

Otherwise if the number is expected, I think you can retry after some delay.

Thanks.

[Chokoabigail]

The client code is actually very simple, the most complex function is the PutKey and in it, there is a concern for memory leaks.

What should I do with this code:

auto chainedItemHandle = gCache_->allocateChainedItem(parentItemHandle, chunkSize);
if (!chainedItemHandle)
{
...
}

It potentially can be after several successful chainedItemHandle allocations (or none), should I call gCache_->remove(key); on the same key I used in auto parentItemHandle = gCache_->allocate(defaultPool_, key, parentItemSize);?
Should I call free on parentItemHandle ? Should I call free on every successful chainedItemHandle allocation before the loop?

This is the rest of the client code:

std::string GetCacheStatus()
{
	std::string res;
	std::string seperator = "[::]";

	res += "isNvmCacheEnabled: " + std::to_string(gCache_->isNvmCacheEnabled()) + seperator;

	// Get the statistics map
	auto statsMap = gCache_->getNvmCacheStatsMap();
	for (const auto& kv : statsMap.getCounts()) {
		std::string k = "statsMap.getCounts()" + kv.first + ":";
		res += k + std::to_string(gCache_->isNvmCacheEnabled()) + seperator;
	}

	// Get the rate map
	for (const auto& kv : statsMap.getRates()) {
		std::string k = "statsMap.getRates()" + kv.first + ":";
		res += k + std::to_string(gCache_->isNvmCacheEnabled()) + seperator;
	}

	return res;
}



bool RemoveKey(CacheKey key)
{
	auto rr = gCache_->remove(key);
	if (rr == Cache::RemoveRes::kSuccess) {
		return true;
	}
	return false;
}

std::string GetKey(CacheKey key) 
{
	std::string result;
	uint32_t fregmentNumber = 0;

	auto parent = gCache_->find(key);
	if (!parent)
	{
		return result;
	}

	auto iobuf_chainedItems = gCache_->convertToIOBuf(std::move(parent));
			
	for (const auto& item : iobuf_chainedItems) 
	{
		// First item is the parent item so we can continue over it
		if (fregmentNumber == 0)
		{
			fregmentNumber++;
			continue;
		}

		// Reconstruct the value
		folly::StringPiece sp2{item};
		result += sp2;
	}

	return result; 
}

[jaesoo-fb]

@Chokoabigail As long as the handle is disposed, you are good. Those have zero references, so should be eligible to be evicted.

I can see you allocated 100MB for RAM cache, meaning you would have only 24 slabs or so. Some allocation class might not have enough slabs allocated or even 0. I would suggest check the distribution of slabs to alloc classes and what was the failing allocation class.

[Chokoabigail]

@Chokoabigail As long as the handle is disposed, you are good. Those have zero references, so should be eligible to be evicted.

I can see you allocated 100MB for RAM cache, meaning you would have only 24 slabs or so. Some allocation class might not have enough slabs allocated or even 0. I would suggest check the distribution of slabs to alloc classes and what was the failing allocation class.

How can I do that? (p.s. In production, we have the same client with a 16GB of RAM, and it also fails there from time to time (once a month or so))

[jaesoo-fb]

@Chokoabigail You can refer to the ACStats which is part of the PoolStats returned by CacheAllocator::getPoolStats(...)

I noticed that there is no public support to print those stats. I think you can refer to the cachebench implementation here (-report_ac_memory_usage_stat option)

[jaesoo-fb]

Let me close this for now. Feel free to reopen or open a new one if needed