Re: [eclipselink-users] Stale data with cache invalidation

[James Sutherland <jamesssss@xxxxxxxxx>]

My guess as to what is occurring is,

- Server 1 updates status s1, send out invalidation 1
- Server 2 receives invalidation 1
- Server 2 reads status s1 from db
- Server 1 updates status s2, send out invalidation 2
- Server 2 receives invalidation 2
- Server 2 refreshes status to s1, marks marks as valid as just read from db

Not sure on the best solution.  If something is so frequently updated maybe
caching it is not a good idea, you could disable the shared cache, or use a
weak cache for it and disable cache coordination for that class.

You could use optimistic locking, such that it will still be refreshed the
next time it is attempted to be updated.  Or a cache invalidation timeout,
to ensure it eventually gets refreshed.

It would probably be worth logging a bug, although I'm not sure there is
anything that can be done for it with asynchronous cache coordination.



Mark Howard wrote:
> 
> Hi,
> 
> We are seeing an occasional issue with our application that I suspect may
> be an eclipselink issue. In some cases, when an entity is invalidated it
> is never refreshed. 
> 
> Our application effectively sets a status flag on an entity and reads this
> flag. A single node is responsible for setting the status and multiple
> nodes may read the status, making it available through a web service
> interface. For the purposes of testing this issue, we have a test harness
> that continuously polls each node to check the status. When the status is
> changed, the other nodes may return different status values (as expected
> since the cache invalidation isn't transactional), but will eventually all
> agree on the new status. In our testing, they are typically synchronised
> by the second poll. 
> 
> In some cases, one node will never see the new status. I've added logging
> to check that the cache invalidation message is being sent and received
> correctly in the cases where the problems occur (it is) and that there
> aren't any exceptions when processing the invalidation message. The
> invalidation message contains two entities, which we would expect to both
> be re-read from the database. SQL logging shows that the one of these
> entities is refreshed from the database and also eagerly loaded entities
> referenced from the second are loaded, but the main entity is not re-read
> from the database. As a result, the status is not updated. 
> 
> This only shows up when the system is under heavy load (e.g. 30 threads
> continuously polling for status of around 200 active processes which will
> change state every 2 minutes). Most of the invalid entities will be
> refreshed correctly, with around 1 in 200 invalid entities per node never
> being updated. 
> 
> I haven't yet been able to create a simple test case for this due to the
> number of components included in the application. Initial attempts at
> reproducing it with simple test cases have all failed so far. 
> 
> My suspicions were that another transaction is overwriting the invalid
> status of the cache entry while it is being reloaded in another thread,
> even though it isn't modifying the entity. In terms of the code,
> CacheKey.setReadTime appears to blindly overwrite the invalidation state
> and is called from multiple locations, but I'm not familiar enough with
> the code to check if these are really all only related to database reads. 
> 
> I've tried this with eclipselink 1.1.3 and 2.1.1 with the same results.
> We're using cache invalidation with a simple gridgain based node discovery
> and message transport system. The entity model includes an inheritance
> structure with join table orm strategy. 
> 
> Has anyone seen anything similar to this? or have any tips for what we
> should be investigating?
> 
> Regards,
> 
> Mark Howard
> 


-----
http://wiki.eclipse.org/User:James.sutherland.oracle.com James Sutherland 
http://www.eclipse.org/eclipselink/
 EclipseLink ,  http://www.oracle.com/technology/products/ias/toplink/
TopLink 
Wiki:  http://wiki.eclipse.org/EclipseLink EclipseLink , 
http://wiki.oracle.com/page/TopLink TopLink 
Forums:  http://forums.oracle.com/forums/forum.jspa?forumID=48 TopLink , 
http://www.nabble.com/EclipseLink-f26430.html EclipseLink 
Book:  http://en.wikibooks.org/wiki/Java_Persistence Java Persistence 
-- 
View this message in context: http://old.nabble.com/Stale-data-with-cache-invalidation-tp29846503p30000417.html
Sent from the EclipseLink - Users mailing list archive at Nabble.com.