Out Java team tried to debug this issue using Java tools. The problem we encountered is that this extra memory is not tracked by Java at all (off heap memory). We realised this using NMT -
.
There was not change in NMT tracked memory with and without annotations enabled. Please see the output of "jcmd <PID> VM.native_memory summary scale=MB" below:
Glassfish + annotations disabled (551 MB RES on startup):
Total: reserved=2397MB, committed=1159MB
- Java Heap (reserved=1024MB, committed=1024MB)
(mmap: reserved=1024MB, committed=1024MB)
- Class (reserved=1077MB, committed=60MB)
(classes #10470)
(malloc=1MB #15133)
(mmap: reserved=1076MB, committed=59MB)
- Thread (reserved=23MB, committed=23MB)
(thread #80)
(stack: reserved=23MB, committed=23MB)
- Code (reserved=249MB, committed=28MB)
(malloc=5MB #8188)
(mmap: reserved=244MB, committed=23MB)
- GC (reserved=3MB, committed=3MB)
(mmap: reserved=3MB, committed=3MB)
- Internal (reserved=3MB, committed=3MB)
(malloc=3MB #13793)
- Symbol (reserved=14MB, committed=14MB)
(malloc=10MB #111702)
(arena=4MB #1)
- Native Memory Tracking (reserved=2MB, committed=2MB)
(tracking overhead=2MB)
Glassfish + annotations (1909 MB RES on startup):
Total: reserved=2400MB, committed=1162MB
- Java Heap (reserved=1024MB, committed=1024MB)
(mmap: reserved=1024MB, committed=1024MB)
- Class (reserved=1079MB, committed=61MB)
(classes #10552)
(malloc=1MB #16204)
(mmap: reserved=1078MB, committed=59MB)
- Thread (reserved=22MB, committed=22MB)
(thread #74)
(stack: reserved=22MB, committed=22MB)
- Code (reserved=249MB, committed=30MB)
(malloc=6MB #8349)
(mmap: reserved=244MB, committed=25MB)
- GC (reserved=3MB, committed=3MB)
(mmap: reserved=3MB, committed=3MB)
- Internal (reserved=3MB, committed=3MB)
(malloc=3MB #13773)
- Symbol (reserved=14MB, committed=14MB)
(malloc=11MB #112818)
(arena=4MB #1)
- Native Memory Tracking (reserved=3MB, committed=3MB)
(tracking overhead=2MB)
- Arena Chunk (reserved=1MB, committed=1MB)
(malloc=1MB)
Apache + annotations (1900 MB RES on startup):
Total: reserved=2400MB, committed=1163MB
- Java Heap (reserved=1024MB, committed=1024MB)
(mmap: reserved=1024MB, committed=1024MB)
- Class (reserved=1079MB, committed=61MB)
(classes #10679)
(malloc=1MB #16766)
(mmap: reserved=1078MB, committed=60MB)
- Thread (reserved=23MB, committed=23MB)
(thread #80)
(stack: reserved=23MB, committed=23MB)
- Code (reserved=249MB, committed=30MB)
(malloc=6MB #8492)
(mmap: reserved=244MB, committed=25MB)
- GC (reserved=3MB, committed=3MB)
(mmap: reserved=3MB, committed=3MB)
- Internal (reserved=3MB, committed=3MB)
(malloc=3MB #14043)
- Symbol (reserved=14MB, committed=14MB)
(malloc=11MB #113587)
(arena=4MB #1)
- Native Memory Tracking (reserved=3MB, committed=3MB)
(tracking overhead=2MB)