gemire / hibernate-memcached Goto Github PK

If you create a cacheable query that uses a projection (i.e. return only the 
id, not the whole entity, for example), then the memcached key generated for 
that query will not necessarily be the same each time it is run.  The hibernate 
QueryKey will include a non-null custom transformer.  Any KeyStrategy's which 
use toString (everything but HashCodeKeyStrategy) will invoke QueryKey.toString 
which will call the result transformrer's toString, which is the default 
Object.toString() method that includes the instance identifier.  This will be 
different for different JVMs, so multiple processes in a cluster will not share 
the cache entry, nor will the same process if it is restarted.

Unfortunately, I don't see a simple fix, only doing some digging in with 
introspection or hacky manipulation of the result of toString().  Anyone have a 
better idea?

See...
http://groups.google.com/group/spymemcached/browse_thread/thread/8b1445d120045cc
4#

http://dustin.github.com/java-memcached-client/apidocs/net/spy/memcached/Failure
Mode.html

- <exptime> is expiration time. If it's 0, the item never expires
  (although it may be deleted from the cache to make place for other
  items). If it's non-zero (either Unix time or offset in seconds from
  current time), it is guaranteed that clients will not be able to
  retrieve this item after the expiration time arrives (measured by
  server time).  

but you code is error
           Calendar calendar = Calendar.getInstance();
            calendar.setTime(new Date());
            calendar.add(Calendar.SECOND, cacheTimeSeconds);

            memcachedClient.set(key, o, calendar.getTime());

just send cacheTimeSeconds is ok

memcachedClient.set(key,o,new Date(cacheTimeSeconds*1l);

otherwise ,the object will not expire

http://www.whalin.com/memcached/

I attached the patch file for removing "as" statement when generating query 
cache key.

I got following sql statement when I request some from memcached or database.
    select
        this_.id as id4_0_,
        this_.version as version4_0_,
        this_.eng_name as eng3_4_0_,
        this_.name as name4_0_,
        this_.status as status4_0_ 
    from
        bbs this_

I got following sql statement when I restarted the web application without any 
data change.
    select
        this_.id as id3_0_,
        this_.version as version3_0_,
        this_.eng_name as eng4_3_0_,
        this_.name as name3_0_,
        this_.status as status3_0_ 
    from
        bbs this

Above 2 sqls have same results. But the query cache keys are different because 
of "as" statement.
So I tried remove "as" statement when generating query cache key and it works 
fine for me.

I catch some problems recently. Our website come very slowly in pick time
and i dump the jvm and find there have hundreds of Thread in
TIMED_WAITING(parking) status. the cpu and memory usaged is not very high.
Here is the detail information:

 "ActiveMQ Session Task" prio=10 tid=0x00002aabbd798800 nid=0x6f63 waiting
on condition [0x000000006aa60000..0x000000006aa60d90]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00002aaac5385f20> (a
java.util.concurrent.CountDownLatch$Sync)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(Abstr
actQueuedSynchronizer.java:947)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(Abst
ractQueuedSynchronizer.java:1239)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253)
        at
net.spy.memcached.MemcachedClient$OperationFuture.get(MemcachedClient.java:1486)
        at
net.spy.memcached.MemcachedClient$GetFuture.get(MemcachedClient.java:1539)
        at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:758)
        at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:778)
        at
com.googlecode.hibernate.memcached.spymemcached.SpyMemcache.get(SpyMemcache.java
:29)
        at
com.googlecode.hibernate.memcached.MemcachedCache.memcacheGet(MemcachedCache.jav
a:124)
        at
com.googlecode.hibernate.memcached.MemcachedCache.get(MemcachedCache.java:153)
        at
org.hibernate.cache.NonstrictReadWriteCache.get(NonstrictReadWriteCache.java:69)
        at
org.hibernate.cache.impl.bridge.EntityAccessStrategyAdapter.get(EntityAccessStra
tegyAdapter.java:55)
        at
org.hibernate.event.def.DefaultLoadEventListener.loadFromSecondLevelCache(Defaul
tLoadEventListener.java:524)
        at
org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener
.java:397)
        at
org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.j
ava:165)
        at
org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventLis
tener.java:223)
        at
org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener
.java:126)
        at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:905)
        at org.hibernate.impl.SessionImpl.get(SessionImpl.java:842)
        at org.hibernate.impl.SessionImpl.get(SessionImpl.java:835)
        at
com.crushorflush.service.notification.serializer.HibernateEntitySerializer.unSer
ialize(HibernateEntitySerializer.java:32)
        at
com.crushorflush.service.notification.impl.activemq.ActiveMQManager.notifyListen
ers(ActiveMQManager.java:156)
        at
com.crushorflush.service.notification.impl.activemq.ActiveMQLoginAgent$1.onMessa
ge(ActiveMQLoginAgent.java:70)
        at
org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.jav
a:1021)
        - locked <0x00002aab4a084a78> (a java.lang.Object)
        at
org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.jav
a:122)
        at
org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java
:192)
        at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
        at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:8
86)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

So we need to restart the web servers and then it works well. It happens
some times every several days.

  I use the Hibernate-memcached 1.1.0 and with memcached 1.2.6 in Red Hat
Enterprise Linux Server release 5.3 (Tikanga). The JDK is 1.6.0_11.
  I check the source code and have no idea what's happened. It worked well
for a long time.
  does anyone else catch the same problem? or have any comments on that?
  Thanks very much.

Hogan

Since clear disabled by default, how can one clear the cache after the DB 
changes outside the app ? 
Thanks

The group id of the dependency of spymemcached is spy instead of net.spy,
this causes maven not to find the dependency.
(I changed the group id value to net.spy locally here and it worked out fine)

From a conversation with JasonB

> > I believe that the dogpile protection could be helpful, but I see that
> > it requires 2 requests to the servers. Could this be sped up with a
> > get_multi? How much overhead does it tend to add?
>
> I'm not sure where I'd be able to use multi_get in there, did you have
> a suggestion there? I'm not totally convinced that the dogpile stuff
> is implemented in the best way. It works, but it could be better.

My thought was, in memcacheGet(), you could retrieve the dogpile token
and the actual object in parallel. If the dogpile token is null, just
ignore the object return. If the dogpile token is valid, you already
have your return value.

We've been using this library to enable memcached for the hibernate second 
level cache and query cache.  After some careful debugging, we found that the 
default HashCodeKeyStrategy was generating identical memcache keys for 
different query parameters.

For example, one query had parameters 60877 and "1.1.1" while another query had 
35891 and "2.2.1".  The HashCodeKeyStrategy uses the 4-byte instance hashCode 
for the given key, which in this case was the org.hibernate.cache.QueryKey.  
That implementation uses the common pattern of building a hash by taking the 
multiplying the current partial result by a prime (37) and adding the next 
hashCode.  This results in a collision in this case (and many others we found). 
 i.e. Long.valueOf(60877).hashCode() * 37 + "1.1.1".hashCode() == 
Long.valueOf(35891).hashCode() * 37 + "2.2.1".hashCode()

This 4-byte hashCode is fine in a situation like java.util.HashMap where an 
equals() check is performed for any keys with matching hash code, but this 
library does not store the original Key and is unable to perform such a check 
to detect a collision, so any collision results in the wrong value being 
returned.

The existing Md5KeyStrategy and Sha1KeyStrategy suffer from the same common 
collision problem because they take their MD5 of SHA1 hash after the 4-byte 
hashCode is computed, due to the fact that they inherit 
HashCodeKeyStrategy.transformKeyObject.

The StringKeyStrategy also doesn't work because the query strings are too long, 
have the parameters truncated and cause more collisions.

Thus none of the existing key strategies are suitable to be used for the 
hibernate query cache, unless your application can tolerate sometimes getting 
the wrong result for a hibernate query.

A key strategy that takes a longer hash (MD5, SHA) of the string version of the 
key should be sufficient.  We're currently evaluating whether to simply disable 
the query cache entirely ( See 
http://tech.puredanger.com/2009/07/10/hibernate-query-cache/ ).  If we decide 
to continue using it, I will probably submit a patch with such a strategy.

it may be add a new property to MemcachedCacheProvider like below:
private Cache cache;

and in the method buildCached , add in last :

cache = new SpyMemcache(client);

and add a new static method;

public static Cache getCache(){
   return Cache ;
} 

so everybody can reuse the memcached client in coding for other uses .

Patch to clean up Eclipse and Findbugs warnings.

Here's what was changed:
Removed wagon-svn plugin from POM; my maven cannot find this plugin (don't
recommend this as global change, it just got included in the file)

MemcachedCache:
Moved dogpile key generation to inside the if statement. This way, users
not using dogpile key will not try to generate it.
Removed extraneous else statement
Changed parseLong to valueOf to remove extraneous boxing

MemcachedCacheProvider:
Chained exceptions properly

DangaMemcacheClientFactory:
Exception chaining

StringUtils:
Removed extranous variable allocation (direct return instead)

Appreciate your consideration of the patch.

In Hibernate 3.3.0.GA there is new cache SPI (http://in.relation.to/9273.lace).
Will it be supported in hebernate-memcached?

What steps will reproduce the problem?
1. started memcached server with the confguration 
memcached -d -m 6144 -u root  -p 11211

2.Testing application with Jmeter



What is the expected output? What do you see instead?

improve the performance
instead degrading performance


What version of the product are you using? On what operating system?
1.3 & windows7


Please provide any additional information below.

waiting threads are increasing.

Steps will reproduce the problem:
1. Configure second-level cache via hibernate-memcached 1.0
2. Start AS
3. Restart memcached while performing some operation with EntityManager.

com.googlecode.hibernate.memcached.MemcachedCache.get() will throw an
ChatException: Exception waiting for value. Due to this exception
EntityManager.find() and Query.getSingleResult() fail and do not return
expected value. I suppose that in this case expected entity must be fetched
from database. I don't know whether this is an issue of hibernate or
hibernate-memcached or this is even an expected behavior.

What steps will reproduce the problem?
1. Network timeout issues during 'get' calls

See the discussion at
http://groups.google.com/group/hibernate-memcached/browse_thread/thread/62a4939a
46ed4bc1

Essentially 1.2 introduced getMulti into the Dogpile prevention feature.
The calls to getMulti were returning a null Map that was not being checked.
The null map generally only comes back when there is an error being
swallowed by one of the client impls (spymemcached in this case).

What is the CacheConcurrencyStrategy's supported by hibernate-memcached? The 
hibernate documentation outlines that cache provides may not fully support 
all strategies, analysis of the source code has not helped clear up the 
issue.

http://docs.jboss.org/hibernate/core/3.3/reference/en/html/performance.html#p
erformance-cache-compat-matrix

AbstractHibernateTestCase (on trunk) extends BaseTestCase, which is not
checked in.

Ive got a problem using this cached implementation. Ive already set the the 
following properties:

hibernate.cache.provider_class                                
hibernate.cache.use_query_cache                                
hibernate.cache.use_second_level_cache                              
hibernate.memcached.servers

Based on the logs, my application was successfully storing the objects in the 
cache.


<9 set org.hibernate.cache.UpdateTimestampsCache:0:952189583 768 300 5
>9 STORED
<9 set org.hibernate.cache.UpdateTimestampsCache:0:952189583 768 300 5
>9 STORED
<9 set org.hibernate.cache.UpdateTimestampsCache:0:-995484462 768 300 5
>9 STORED
<9 set org.hibernate.cache.UpdateTimestampsCache:0:952189583 768 300 5
>9 STORED
<9 set org.hibernate.cache.UpdateTimestampsCache:0:952189583 768 300 5
>9 STORED
<9 set org.hibernate.cache.UpdateTimestampsCache:0:-995484462 768 300 5
>9 STORED
<9 set org.hibernate.cache.UpdateTimestampsCache:0:952189583 768 300 5
>9 STORED
<9 set org.hibernate.cache.UpdateTimestampsCache:0:952189583 768 300 5
>9 STORED

But I didn't find any get method to signify that my objects are being retrieved 
in the cache.

Am i missing something?

As the summary states i am using hibernate-memcached in my web application 
which runs on tomcat. As soon as i active the cache it is not possible to stop 
tomcat the normal way but i have to kill it. I tracked the problem an i am 
quite sure that this comes from a thread that is started for the communication 
with the memcached server. This thread doesn't get stopped anymore and so 
tomcat cannot be stopped as well.

I am not sure which class the thread is of and if this is your fault or the 
fault of spymemcached client but could you please give me some feedback on this 
problem.

Thanks a lot!

It would be nice to be able to but the memcached threads in daemon mode so
that they don't prevent a JVM shutdown.

There is code in the spymemcached_2.3 branch that needs to get merged to
Master.

1. core lib is hibernate-memcached-1.2.2.jar,memcached-2.4.2.jar and 
hibernate-core-3.3.2.GA.jar

2. use hibernate query and setCacheable(true)
code:

String hql = " select menu from User user, Role role, Auth auth, Resource 
resource, Menu menu where user.id=:userId and user.roleId=role.id and 
role.id=auth.roleId and auth.resourceId=resource.id and resource.rtype=:rtype 
and resource.objectId=menu.id and menu.parentId=:parentId and 
menu.moduleId=:moduleId ";
        Map<String, Integer> filterMap = new HashMap<String, Integer>(2);
        filterMap.put("userId", userId);
        filterMap.put("moduleId", moduleId);
        filterMap.put("rtype", ResourceConstants.TYPE_MENU);
        filterMap.put("parentId", parentId);
        return super.find(hql, filterMap, true);

super.find(hql, filterMap, true) code is:

public <X> List<X> find(String hql, Map<String, ?> values, boolean useCache) {
    Query query = getSession().createQuery(queryString);
    query.setCacheable(useCache);
     if (values != null) {
       query.setProperties(values);
     }
     return query.list();
}

3. turn log4j level debug , i see this:
One:
DEBUG: com.googlecode.hibernate.memcached.HashCodeKeyStrategy - Transformed key 
[sql: select menu4_.id as id5_, menu4_.create_time as create2_5_, menu4_.is_dir 
as is3_5_, menu4_.last_modify_time as last4_5_, menu4_.module_id as module5_5_, 
menu4_.name as name5_, menu4_.parent_id as parent7_5_, menu4_.sort_index as 
sort8_5_, menu4_.url as url5_ from user user0_, role role1_, auth auth2_, 
resource resource3_, menu menu4_ where user0_.id=? and user0_.role_id=role1_.id 
and role1_.id=auth2_.role_id and auth2_.resource_id=resource3_.id and 
resource3_.rtype=? and resource3_.object_id=menu4_.id and menu4_.parent_id=? 
and menu4_.module_id=?; parameters: ; named parameters: {parentId=0, 
moduleId=2, userId=1, rtype=2}] to hashCode [2082154712]
DEBUG: com.googlecode.hibernate.memcached.HashCodeKeyStrategy - Final cache 
key: [org.hibernate.cache.StandardQueryCache:0:2082154712]
DEBUG: com.googlecode.hibernate.memcached.MemcachedCache - 
Memcache.get(org.hibernate.cache.StandardQueryCache:0:2082154712)
DEBUG: com.googlecode.hibernate.memcached.spymemcached.SpyMemcache - 
MemcachedClient.get(org.hibernate.cache.StandardQueryCache:0:2082154712)

Two:
DEBUG: com.googlecode.hibernate.memcached.HashCodeKeyStrategy - Transformed key 
[sql: select menu4_.id as id5_, menu4_.create_time as create2_5_, menu4_.is_dir 
as is3_5_, menu4_.last_modify_time as last4_5_, menu4_.module_id as module5_5_, 
menu4_.name as name5_, menu4_.parent_id as parent7_5_, menu4_.sort_index as 
sort8_5_, menu4_.url as url5_ from user user0_, role role1_, auth auth2_, 
resource resource3_, menu menu4_ where user0_.id=? and user0_.role_id=role1_.id 
and role1_.id=auth2_.role_id and auth2_.resource_id=resource3_.id and 
resource3_.rtype=? and resource3_.object_id=menu4_.id and menu4_.parent_id=? 
and menu4_.module_id=?; parameters: ; named parameters: {parentId=1, 
moduleId=1, userId=1, rtype=2}] to hashCode [2082154712]
DEBUG: com.googlecode.hibernate.memcached.HashCodeKeyStrategy - Final cache 
key: [org.hibernate.cache.StandardQueryCache:0:2082154712]
DEBUG: com.googlecode.hibernate.memcached.MemcachedCache - 
Memcache.get(org.hibernate.cache.StandardQueryCache:0:2082154712)

see the hashcode with two different hql out put in log, bug get same hashcode 
why? please help me. 

this problem make my application read wrong data from hibernate cache

What steps will reproduce the problem?
1. Configured the hibernate-memcached as suggested and working fine in all 
aspects when we are using that from individual applications.
2. One application is running on WAS and the other on Tomcat. it is working 
fine with either of them individually but if we are running both the 
applications at once it is failing to cast the object stored by one app in the 
other app.
3. We are getting the following exception

Caused by: java.lang.ClassCastException: org.hibernate.cache.ReadWriteCache$Item
        at org.hibernate.cache.entry.StructuredCacheEntry.destructure(StructuredCacheEntry.java:23)
        at org.hibernate.event.def.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:470)
        at org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:328)
        at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:123)
        at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:177)
        at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:87)
        at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:809)
        at org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:781)
        at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:266)
        at org.hibernate.type.EntityType.resolve(EntityType.java:303)
        at org.hibernate.engine.TwoPhaseLoad.initializeEntity(TwoPhaseLoad.java:113)
        at org.hibernate.loader.Loader.initializeEntitiesAndCollections(Loader.java:842)
        at org.hibernate.loader.Loader.doQuery(Loader.java:717)
        at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:223)
        at org.hibernate.loader.Loader.doList(Loader.java:2147)
        at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2026)
        at org.hibernate.loader.Loader.list(Loader.java:2021)
        at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:94)
       at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1483)
        at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:298)


What is the expected output? What do you see instead?
For the same query criteria executed in one application, has to cache the 
result and it should be able to serve the same from cache if I am using the 
same query from other application.
But it is failing to fetch that object stating a classcast Exception.

What version of the product are you using? On what operating system?
<dependency>
<groupId>com.googlecode</groupId>
  <artifactId>hibernate-memcached</artifactId>
  <version>1.3</version>
</dependency>
<dependency>
  <groupId>spy</groupId>
  <artifactId>spymemcached</artifactId>
  <version>2.7.3</version>
</dependency>

Unix

Please provide any additional information below.

Need to understand if there is any provider specific object that will be 
holding this data in the memcache server.

What steps will reproduce the problem?
1. my hibernate mapping files include following content
info.xml
...
<class name="Info" table="info">
...
<one-to-one class="Desc" name="desc"/>
</class>
...

desc.xml
...
<class name="Desc" table="desc">
...
<one-to-one class="Info" name="info"/>
</class>
...

2. use Info and Desc


got following exception stack trace:
java.lang.StackOverflowError
    at net.spy.SpyObject.<init>(SpyObject.java:21)
    at
net.spy.memcached.protocol.BaseOperationImpl.<init>(BaseOperationImpl.java:31)
    at
net.spy.memcached.protocol.ascii.OperationImpl.<init>(OperationImpl.java:34)
    at
net.spy.memcached.protocol.ascii.BaseGetOpImpl.<init>(BaseGetOpImpl.java:30)
    at
net.spy.memcached.protocol.ascii.GetOperationImpl.<init>(GetOperationImpl.java:1
9)
    at
net.spy.memcached.protocol.ascii.AsciiOperationFactory.get(AsciiOperationFactory
.java:37)
    at net.spy.memcached.MemcachedClient.asyncGet(MemcachedClient.java:644)
    at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:764)
    at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:784)
    at
com.googlecode.hibernate.memcached.spymemcached.SpyMemcache.get(SpyMemcache.java
:29)
    at
com.googlecode.hibernate.memcached.MemcachedCache.memcacheGet(MemcachedCache.jav
a:124)
    at
com.googlecode.hibernate.memcached.MemcachedCache.get(MemcachedCache.java:153)
    at org.hibernate.cache.ReadWriteCache.get(ReadWriteCache.java:80)
    at
org.hibernate.event.def.DefaultLoadEventListener.loadFromSecondLevelCache(Defaul
tLoadEventListener.java:486)
    at
org.hibernate.event.def.DefaultLoadEventListener.doLoad(DefaultLoadEventListener
.java:357)
    at
org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.j
ava:139)
    at
org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventLis
tener.java:195)
    at
org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener
.java:103)
    at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:878)

i'm using hibernate-memcached-1.1.0.jar, memcached-2.1.jar,spy-2.4.jar，and
memcached server is memcached-1.2.7.tar.gz
OS: x86_64 centos5

I trying to replace the Secondary cache in hibernate with Amazon ElastiCache 

Any guide on how to do this? 

Thank you.