Hi Thomas,
Answers will be inline.
On 04/04/2011 01:12 PM, Thomas Becker wrote:
HI Matthieu,
do you have any more log information from before that crash? It
looks like it happened during a young generation garbage
collection. Your eden space (where all new objects are placed in
memory) was full and there was not much space (probably not
enough) left in the old generation for objects which survived the
young generation gc. The interesting part of your crash dump
indicating this, is:
These are the only logs related to Jetty that I am aware of. We
parsed /var/log but could not find anything relevant, besides our
main application log (the one that sends data to solr). More about
this below.
Heap
PSYoungGen total 417088K, used 417056K [0x00007f04eb080000,
0x00007f050a3f0000, 0x00007f0515b20000)
eden space 416960K, 100% used
[0x00007f04eb080000,0x00007f05047b0000,0x00007f05047b0000)
from space 128K, 75% used
[0x00007f05055d0000,0x00007f05055e8000,0x00007f05055f0000)
to space 47232K, 0% used
[0x00007f05075d0000,0x00007f05075e8000,0x00007f050a3f0000)
PSOldGen total 529920K, used 377402K [0x00007f0495b20000,
0x00007f04b60a0000, 0x00007f04eb080000)
object space 529920K, 71% used
[0x00007f0495b20000,0x00007f04acbae9b0,0x00007f04b60a0000)
PSPermGen total 21504K, used 20440K [0x00007f048b320000,
0x00007f048c820000, 0x00007f0495b20000)
object space 21504K, 95% used
[0x00007f048b320000,0x00007f048c7163a8,0x00007f048c820000)
As solr/lucene usage can be quite memory intensive it's likely
that your app simply ran out of memory. Maybe during indexing. So
your next steps will be to find out what the application was doing
before the crash and think about what might have caused it to run
oom. If you can't find the root cause this way, you should try to
get a heap dump and analyze what was consuming memory.
That makes sense : reading our app logs shows that malformed data
was sent to our solr server, which never answered back with any
acknowledgement of indexation. Now I guess that said sent data
builds up with each iteration, apparently leading to this crash.
Try
setting these JVM options (http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html)
in your startup script:
-XX:HeapDumpPath=./java_pid<pid>.hprof Path to directory or
filename for heap dump. Manageable. (Introduced in 1.4.2
update 12, 5.0 update 7.) -XX:-HeapDumpOnOutOfMemoryError Dump
heap to file when java.lang.OutOfMemoryError is thrown. Manageable.
(Introduced in 1.4.2 update 12, 5.0 update 7.)
-XX:-PrintGCDetails Print more details at garbage collection. Manageable.
(Introduced in 1.4.0.) -XX:-PrintGCTimeStamps Print timestamps at
garbage collection. Manageable (Introduced in 1.4.0.)
Again it might be normal and that your application simply needs a
bigger heap. Your current settings are: "jvm_args: -Xms512M
-Xmx2048M". If you don't have more memory available you might need
to rethink your indexing strategy (if the oom happened during
indexing). But it might as well be that you've got a memory leak
of some kind. You'll have to read your log files and try to get a
heap dump.
As indexing is memory intensive, proper tuning of your jvm memory
settings is crucial. Normally you want a big newsize and the
overall heap space as big as you have physical memory. But that
heavily depends on your indexing strategy.
We assign 50% of the available memory to solr, the rest is left for
our main application. This was a test appliance, with less memory
than what we expect to use in production.
Hope I did not confuse you too much. Please not that this is most
certainly NOT a jetty issue.
Cheers,
Thomas
You were very clear, thanks for your help ! One more question, but I
guess that'd be more of a Solr question : would it be okay to have
Jetty be restarted right after such a crash ? We can't have the
indexation service down for too long.
On
04/04/2011 12:20, Matthieu Huin wrote:
Greetings
all,
I am currently using solr as the backend behind a log
aggregation and
search system my team is developing. All was well and good until
I
noticed a test server crashing quite unexpectedly. We'd like to
dig more
into the incident but none of us has much experience with Jetty
crash
logs - not to mention that our Java is very rusty.
The crash log is joined as an attachment.
Could anyone help us with understanding what went wrong there ?
Also, would it be possible and/or wise to automatically restart
the
server in case of such a crash ?
Thanks for your help. If you need any extra info about that
case, do not
hesitate to ask !
Matthieu Huin
( Resending this message as I used to wrong e-mail to send it,
sorry for the inconvenience)
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
jetty-users@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/jetty-users
|
