Eclipse Community Forums: EclipseLink » Eclipselink Performance test failed at 95 percentile

Home » Eclipse Projects » EclipseLink » Eclipselink Performance test failed at 95 percentile(Eclipselink is calling Millions of call to DB)

Show: Today's Messages :: Show Polls :: Message Navigator

Eclipselink Performance test failed at 95 percentile [message #1753476]

Mon, 06 February 2017 22:06

Prashant Saraf

Messages: 10
Registered: February 2015

Junior Member

Hi All,

I have CXF Jax-WS base webservices that is querying Oracle DB using Eclipselink JPA.
DB structure is Little hybride like use of Anchor table to maintain the history. I have close to 30 table out of which few are reference table like country name, type and 10 Data table 10 anchor table for data table and rest are relationship tables.

Webservices is for CRUD and I have
roperty name="eclipselink.persistence-context.flush-mode" value="commit" />

Now when good load on the server say 10-20 Transaction running in parallel for minute or so I saw at end of test(5 Min test) JPA(eclipselink) is pulling hell lots of data from DB with no relationship, all that is happening at the end of transaction when flush is happening. and it fails for transaction timeout.

when I re-run same request without load it get processed in 100MS.

Report message to a moderator

Re: Eclipselink Performance test failed at 95 percentile [message #1753572 is a reply to message #1753476]

Tue, 07 February 2017 18:49

Chris Delahunt

Messages: 1300
Registered: July 2009

Senior Member

There isn't enough information given on what you are doing in your test(s) to comment on anything.

Report message to a moderator

Re: Eclipselink Performance test failed at 95 percentile [message #1753661 is a reply to message #1753572]

Wed, 08 February 2017 15:33

Prashant Saraf

Messages: 10
Registered: February 2015

Junior Member

We are running a load test(500 Transaction in 5 min ) to insert data into 30 tables.
Each request is unique request for concurrent requests so no data overlap,
Each transaction can be either pure insert or update to existing data. the code process is read xml and store that into database using JPA, XML and DB are different model so translation classes are there.
For first few minutes test runs smooth, suddenly JPA started behaving weird with pulling all the records with no relation to transaction.
if I run same request again it process well with appropriate data behaviour.

In code all of my entity relation are cascade and I do persist at once that is end of transaction.
I have below property to avoid unnecessary db call when I am processing data.
<property name="eclipselink.persistence-context.flush-mode" value="commit" />
I also have
<shared-cache-mode>ENABLE_SELECTIVE</shared-cache-mode>

the help I am looking is what could cause eclipselink to get all the data from table without any relationship.

Report message to a moderator

Re: Eclipselink Performance test failed at 95 percentile [message #1753682 is a reply to message #1753661]

Wed, 08 February 2017 18:10

Chris Delahunt

Messages: 1300
Registered: July 2009

Senior Member

Nothing you've provided gives any indication as to why EclipseLink would start 'pulling all records'. EclipseLink wouldn't do that by itself - something within the application forces it to query data.

Check the process causing the issue, and ensure you have logging on. Check the data you are trying to use - you mention you update, so could you be calling merge on data that is accumulating from your load tests?

Report message to a moderator

Re: Eclipselink Performance test failed at 95 percentile [message #1753689 is a reply to message #1753682]

Wed, 08 February 2017 23:17

Prashant Saraf

Messages: 10
Registered: February 2015

Junior Member

At the end of transaction I am calling entitymanger.persist(rootEntity). EntityManger is managed by Spring Injection. all the data fetch is happening by persist operation only. we are not using merge anywhere. my understanding is persist should manage both new and update.
The strange part in whole test is if I run same data on without load it process flawlessly.

I am really not sure what detail I should be proving, I have all default configuration, whatever I had in persistnace.xml is shared.

Again thanks for help. it would be great if i get some pointer to go on.

Report message to a moderator

Re: Eclipselink Performance test failed at 95 percentile [message #1753764 is a reply to message #1753689]

Thu, 09 February 2017 21:41

Chris Delahunt

Messages: 1300
Registered: July 2009

Senior Member

Persist is for new entities and will only insert into the database -it doesn't do updates. If your code works and runs flawlessly, then it isn't something standard causing the issue - it'll be something buried, specific to your usage that isn't thread safe. If you can get an exception when it starts pulling data, you'd be able to see what part of the code is causing the query.

Report message to a moderator

Re: Eclipselink Performance test failed at 95 percentile [message #1753982 is a reply to message #1753764]

Mon, 13 February 2017 16:29

Prashant Saraf

Messages: 10
Registered: February 2015

Junior Member

I am doing Persist because of performance and at the end of services I am calling it so there will be no modification to it. with merge, EM will create a duplicate copy to work on which I don't want.
About the performance issue.
-- I am sending of unique request to a webservices that store data into oracle DB. I have close to 30 tables.
-- when I send data at 5 threads in concurrent for 5 min. first couple of minutes I do not see any issue.
-- issue stats when CPU goes to 100% and some how few request pulls whole table data for no reason when I do call Persist/Merge
-- if we take those Request out and re-run, it get processed in few millisec.
-- if it was programming error then re-run could have catched the issue.
Not sure why when I call persist/merge it

Report message to a moderator

Re: Eclipselink Performance test failed at 95 percentile [message #1754117 is a reply to message #1753982]

Tue, 14 February 2017 15:30

Chris Delahunt

Messages: 1300
Registered: July 2009

Senior Member

I am still sceptical it has anything to do with persist - persist cannot be used on existing ,unmanaged entities. So it is likely these entities are already managed and it is a no-op (other than to cascade the persist which happens on synchronization anyway), in which case Merge too would be a no-op.

Chances are good you have some shared objects or synchronization slowing things down. Without seeing what the application is doing in those threads, there isn't much we can help with. Grab threaddumps during the slow down to see what the threads might be getting stuck on, or causing a bottleneck. A few thread dumps might help show where the problem is in the code, or if some database access is locking, such as if you are using select for update.

Report message to a moderator

Re: Eclipselink Performance test failed at 95 percentile [message #1756068 is a reply to message #1754117]

Fri, 10 March 2017 14:53

Prashant Saraf

Messages: 10
Registered: February 2015

Junior Member

Hi Chris,

Thanks for all the help, I have new finding to the problem, we are able to isolate issue and right now if I do only create operation on those table then there is no issue I get consistant 400 to 500 MS response. but if I start update in between then suddenly issue occurs.
Below is sudo logic we have

check root entity is present in the DB
if No, create new, all data will be new
If yes, get the root entity, traverse through each entity and check if any data has changed with incoming data and update the root entity tree graph.

to avoid enitimanager DB sync call we have marked
<property name="eclipselink.persistence-context.flush-mode" value="commit" />

is this a issue as all entities are cascade and I am having close to 1M record on root table.

Report message to a moderator

Re: Eclipselink Performance test failed at 95 percentile [message #1757925 is a reply to message #1756068]

Tue, 21 March 2017 21:26

Prashant Saraf

Messages: 10
Registered: February 2015

Junior Member

Just update:

After having Static weaving we have reduced 200M DB calls to 10M DB call. The question still existing why dynamic weaving not working for 15-20 table classes.

Report message to a moderator

Previous Topic:	SOAP IllegalArgumentException for array parameter
Next Topic:	Adding dependency to EclipseLink (ant+maven+Tycho)

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Fri Jun 22 17:34:30 GMT 2018

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter