Malformed cache data about cache-manager

[francescosalvi]

I have a working repro, and the explanation for the behavior (and why it appears transient):

package.json (install latest version of cache-manager and cache-manager-ioredis-yet alongside the last former working versions, in my specific case 5.5.3/2.0.4)

{
  "name": "cache-manager-fiddle",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "cache-manager-prev": "npm:[email protected]",
    "cache-manager-ioredis-yet-prev": "npm:[email protected]",
    "cache-manager": "5.6.1",
    "cache-manager-ioredis-yet": "2.1.1"
  }
}

index.mjs

import { redisStore as createRedisStoreCurr } from 'cache-manager-ioredis-yet';
import { redisStore as createRedisStorePrev } from 'cache-manager-ioredis-yet-prev';
import { caching as createCacheCurr } from 'cache-manager';
import { caching as createCachePrev } from 'cache-manager-prev';

const connOptions = { host: 'localhost', port: 6379 };
const cachePrev = await createCachePrev(await createRedisStorePrev(connOptions));

const key = 'someKey';

await cachePrev.set(key, 'hello cache', 5000);
console.log(await cachePrev.get(key)); // outputs: hello cache

const cacheCurr = await createCacheCurr(await createRedisStoreCurr(connOptions));

console.log(await cacheCurr.get(key)); // outputs: { v: 'hello cache' }

process.exit(0);

Hence what we are abserving is a breaking change the internal expectations cache-manager-ioredis-yet makes about the schema it uses. In other words, all is fine if the cache store gets completely wiped an repopulated across package upgrade, but all data previously present becomes invalid.

[Kauhsa]

I suspect #681 has something to do with this.

[evenisse]

Yep. I think the guidance is to flush the cache when moving to this and that resolves it. Going to close this as there isnt a good way to handle it.

Hello,
It's not a solution. With 2 applications sharing the same cache data, if one application can't be updated, it's impossible to upgrade the other to the new version of cache-manager. Data storage must be independent of the library used.

[jaredwray]

@rivatove - we upgraded to telejson a while back which could store the objects that way but should not affect you retrieving the value. Can you give an example of how you see this happening? Do you have code example to reproduce?

[rivatove]

@rivatove - we upgraded to telejson a while back which could store the objects that way but should not affect you retrieving the value. Can you give an example of how you see this happening? Do you have code example to reproduce?

Unfortunately I don't have more detailed examples as it happened in production.

Most important info:

• rolling back to 5.5.3/2.0.4 solved the issue completely;
• the issue was localized to the retrieval logic, it did not corrupt the data in the store. 2 versions could operate against the same store, and only the newest version would include this bug.
• it did not occur 100% of the time. The version used to store the value in the cache could have played a part.

[jaredwray]

I have a working repro, and the explanation for the behavior (and why it appears transient):

package.json (install latest version of cache-manager and cache-manager-ioredis-yet alongside the last former working versions, in my specific case 5.5.3/2.0.4)

{
  "name": "cache-manager-fiddle",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "cache-manager-prev": "npm:[email protected]",
    "cache-manager-ioredis-yet-prev": "npm:[email protected]",
    "cache-manager": "5.6.1",
    "cache-manager-ioredis-yet": "2.1.1"
  }
}

index.mjs

import { redisStore as createRedisStoreCurr } from 'cache-manager-ioredis-yet';
import { redisStore as createRedisStorePrev } from 'cache-manager-ioredis-yet-prev';
import { caching as createCacheCurr } from 'cache-manager';
import { caching as createCachePrev } from 'cache-manager-prev';

const connOptions = { host: 'localhost', port: 6379 };
const cachePrev = await createCachePrev(await createRedisStorePrev(connOptions));

const key = 'someKey';

await cachePrev.set(key, 'hello cache', 5000);
console.log(await cachePrev.get(key)); // outputs: hello cache

const cacheCurr = await createCacheCurr(await createRedisStoreCurr(connOptions));

console.log(await cacheCurr.get(key)); // outputs: { v: 'hello cache' }

process.exit(0);

Hence what we are abserving is a breaking change the internal expectations cache-manager-ioredis-yet makes about the schema it uses. In other words, all is fine if the cache store gets completely wiped an repopulated across package upgrade, but all data previously present becomes invalid.

Thanks for doing this. Can you tell me what the schema looks like in previous as maybe we can just handle an conversion for it in the new code base?

[francescosalvi]

@jaredwray no problem!

Can you tell me what the schema looks like in previous

If I do a "raw" get using ioredis right after the set using 5.5.3/2.1.1 in my snippet above, I get back {"v":"hello cache"}, so it appears that the wrapping of the user/business payload with the v field was already in place in the previous version.

This is confirmed If I diff the implementations ("prev" -> "curr"):

What's changed is that previously stringify used to include the "wrapping" parts in redis, while in the new version they are stripped using slice.

I am thinking that even by implementing a heuristic that handles both the old and new format (like: "if it's an object with strictly one property named v, then it must come from version < xxx and we should unwrap it"), there probably isn't a way to 100% exclude false positives (users could have a v as part of their business payload ?).

[jaredwray]

Yep. I think the guidance is to flush the cache when moving to this and that resolves it. Going to close this as there isnt a good way to handle it.

[rvitaliy]

agree with @evenisse

another problem what is see is that two different applications can use different approach/libraries to store and get values, the contract of what i put and what i get at the level of the storage MUST be respected!
Libraries like this that have the purpose to be a wrapper that allow easier change the storage driver or just hide implementation detail MUST not alterate input and output datas at the storage level.

i mean:

NodeJSApplication: cacheManager.put('ping', 'pong');

storage: [{'ping': {'v': 'pong'}}]  // expected result storage: [{'ping': 'pong'}]

PHPApplication: redisDirectImplementation.get('ping') > result {'v': 'pong'} // ERROR!
GolangApplication: redisDirectImplementation.get('ping') > result {'v': 'pong'} // ERROR!

// and viceversa

GolangApplication|PHPApplication.put('key1', 'value1');
NodeJSApplication.cacheManager.get('key1') > result 'value1' expected {'v': 'value1'}

[rvitaliy]

@jaredwray Hi, given all these considerations, do you intend to investigate further to determine if we need to rollback?

IMHO: yes, we should.

[jaredwray]

@rvitaliy - I beleive it is not doing this anymore. What version are you on as what is the code example that does this?

[evenisse]

My problem is different. It's not between 2 versions of cache-manager but between 2 applications.
One application uses a redis library directly (ioredis) and the other uses cache-manager.
As cache-manager only accepts object with v field, it isn't possible to set data in the first app with a redis lib and read then in the other app with cache-manager. And reciprocally.

import { caching as createCacheCurr } from 'cache-manager';
import Redis from 'ioredis';
const redis = new Redis();

const connOptions = { host: 'localhost', port: 6379 };
const cacheCurr = await createCacheCurr(await createRedisStoreCurr(connOptions));

const key = 'someKey';

await redis.set(key, 'hello cache', 'PX', 5000);
console.log('From Redis: ', await redis.get(key)); // outputs: hello cache
console.log('From Cache: ', await cacheCurr.get(key)); // error

process.exit(0);

The output is

undefined:1
{"v":hello cache}
     ^

SyntaxError: Unexpected token 'h', "{"v":hello cache}" is not valid JSON
    at JSON.parse (<anonymous>)
    at Object.parse (/home/evenisse/test_redis_cache_manager/node_modules/telejson/dist/index.js:1565:24)
    at parse (/home/evenisse/test_redis_cache_manager/node_modules/cache-manager-ioredis-yet/dist/index.js:33:35)
    at Object.get (/home/evenisse/test_redis_cache_manager/node_modules/cache-manager-ioredis-yet/dist/index.js:61:24)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async file:///home/evenisse/test_redis_cache_manager/index.mjs:13:29

Why do you use an object with a v field and not directly th real value?