| XMI Resource & "OutOfMemoryError: GC Overhead limit exceeded" [message #1490784] |
Fri, 28 November 2014 12:45  |
 crothy donald
Messages: 31
Registered: September 2013 |
Member |
|
|
Hello,
loading an 50 MB big XMI Resource consecutive lead to "OutOfMemoryError: GC Overhead limit exceeded" error.
I wrote a simple unit test and on my machine, Windows7 with jdk-7u55-windows-x64 emf is able to load the resource 6 times before the jvm throws the exception.
See attached stacktrace, it seams to me that the SAXParser or the XMLHandler does not release the Strings / Maps used during loading so the garbage collector doesn't recover enough heap space and therefore quits.
This matches with the explanation from Oracle: "...This exception is typically thrown because the amount of live data barely fits into the Java heap having little free space for new allocations."
I also tried the performance options on load (parser pool & name to feature map) described on page 495 in the emf bible with no effect.
Tweaking the head size delayed the OutOfMemoryError, so it is not even an workaround.
Any hints, suggestions?
Christian
//----------------------------- stacktrace ------------------------------------------//
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOfRange(Arrays.java:2694)
at java.lang.String.<init>(String.java:203)
at java.lang.StringBuffer.toString(StringBuffer.java:561)
at org.eclipse.emf.ecore.xmi.impl.XMLHandler.endElement(XMLHandler.java:1606)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:609)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1781)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2957)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:649)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:333)
at org.eclipse.emf.ecore.xmi.impl.XMLLoadImpl.load(XMLLoadImpl.java:175)
at org.eclipse.emf.ecore.xmi.impl.XMLResourceImpl.doLoad(XMLResourceImpl.java:261)
at org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(ResourceImpl.java:1518)
at org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(ResourceImpl.java:1297)
at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLoad(ResourceSetImpl.java:259)
at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLoadHelper(ResourceSetImpl.java:274)
at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResource(ResourceSetImpl.java:406)
at PerformanceTestIntegration.loadResource(PerformanceTestIntegration.java:60)
//----------------------------- test ------------------------------------------//
{
// resource is ~50 MB
Runtime runtime = Runtime.getRuntime();
int MB = 1024 * 1024;
for( i = 0; i < 30; i++ ) {
long startTime = System.currentTimeMillis();
UdmType udm = loadResource(nsr4);
assertTrue(udm != null);
long elapsedTime = System.currentTimeMillis() - startTime;
System.out.println(i + " elapsed time : " + formatInterval(elapsedTime) + " free mem : "
+ (runtime.freeMemory() / MB));
}
assertEquals(i, 10);
}
private EObject loadResource(URI uri) {
// Create a resource set to hold the resources.
//
ResourceSet resourceSet = new ResourceSetImpl();
// Register the appropriate resource factory to handle all file
// extensions.
//
resourceSet.getResourceFactoryRegistry().getExtensionToFactoryMap().put("*.xmi", new MyResourceFactoryImpl());
// Register the package to ensure it is available during loading.
//
resourceSet.getPackageRegistry().put(MyPackage.eNS_URI, MyPackage.eINSTANCE);
Resource resource = resourceSet.getResource(uri, true);
return resource.getContents().get(0);
}
//----------------------------- output ------------------------------------------//
0 elapsed time : 00:00:09.464 free mem : 102
1 elapsed time : 00:00:08.986 free mem : 150
2 elapsed time : 00:00:07.199 free mem : 297
3 elapsed time : 00:00:07.324 free mem : 241
4 elapsed time : 00:00:10.059 free mem : 141
5 elapsed time : 00:00:20.229 free mem : 115
|
|
|
| Re: XMI Resource & "OutOfMemoryError: GC Overhead limit exceeded" [message #1494163 is a reply to message #1490784] |
Mon, 01 December 2014 08:26  |
Ed Merks
Messages: 33141
Registered: July 2009 |
Senior Member |
|
|
Christian,
Comments below.
On 28/11/2014 1:45 PM, crothy donald wrote:
> Hello,
>
> loading an 50 MB big XMI Resource consecutive lead to
> "OutOfMemoryError: GC Overhead limit exceeded" error.
> I wrote a simple unit test and on my machine, Windows7 with
> jdk-7u55-windows-x64 emf is able to load the resource 6 times before
> the jvm throws the exception.
> See attached stacktrace, it seams to me that the SAXParser or the
> XMLHandler does not release the Strings / Maps used during loading so
> the garbage collector doesn't recover enough heap space and therefore
> quits.
The question is always of course, what holds onto the SAXParser or the
XMLHandler? Certainly resources loading doesn't generally leak or
someone would have noticed in the last 14 years. It's not clear how
quickly udm will be garbage collected. Certainly in principle that's
possible on each loop iteration but is that what happens in practice?
> This matches with the explanation from Oracle: "...This exception is
> typically thrown because the amount of live data barely fits into the
> Java heap having little free space for new allocations."
So unless it's actually a long-term leak (likely only if you're
retaining instances long term) rather than simply treading near the
threshold of memory exhaustion, increasing your heap capacity is necessary.
>
> I also tried the performance options on load (parser pool & name to
> feature map) described on page 495 in the emf bible with no effect.
> Tweaking the head size delayed the OutOfMemoryError, so it is not even
> an workaround.
If it's a long term leak, you'd need to find the source but as I said,
the pattern you show is a very general one and that does not generally
lead to leaks, although, as I suggested, it's not clear how quickly udm
will be garbage collected and certainly any reference to any object will
leak the entire resource set.
>
> Any hints, suggestions?
> Christian
>
>
>
> //----------------------------- stacktrace
> ------------------------------------------//
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.Arrays.copyOfRange(Arrays.java:2694)
> at java.lang.String.<init>(String.java:203)
> at java.lang.StringBuffer.toString(StringBuffer.java:561)
> at
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.endElement(XMLHandler.java:1606)
> at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:609)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1781)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2957)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
> at
> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
> at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
> at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:649)
> at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:333)
> at
> org.eclipse.emf.ecore.xmi.impl.XMLLoadImpl.load(XMLLoadImpl.java:175)
> at
> org.eclipse.emf.ecore.xmi.impl.XMLResourceImpl.doLoad(XMLResourceImpl.java:261)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(ResourceImpl.java:1518)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(ResourceImpl.java:1297)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLoad(ResourceSetImpl.java:259)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLoadHelper(ResourceSetImpl.java:274)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResource(ResourceSetImpl.java:406)
> at
> PerformanceTestIntegration.loadResource(PerformanceTestIntegration.java:60)
>
>
>
> //----------------------------- test
> ------------------------------------------//
> {
> // resource is ~50 MB
> Runtime runtime = Runtime.getRuntime();
> int MB = 1024 * 1024;
>
> for( i = 0; i < 30; i++ ) {
> long startTime = System.currentTimeMillis();
> UdmType udm = loadResource(nsr4);
> assertTrue(udm != null);
> long elapsedTime = System.currentTimeMillis() - startTime;
>
> System.out.println(i + " elapsed time : " +
> formatInterval(elapsedTime) + " free mem : "
> + (runtime.freeMemory() / MB));
> }
>
> assertEquals(i, 10);
> }
>
> private EObject loadResource(URI uri) {
> // Create a resource set to hold the resources.
> //
> ResourceSet resourceSet = new ResourceSetImpl();
>
> // Register the appropriate resource factory to handle all file
> // extensions.
> //
> resourceSet.getResourceFactoryRegistry().getExtensionToFactoryMap().put("*.xmi",
> new MyResourceFactoryImpl());
>
> // Register the package to ensure it is available during loading.
> //
> resourceSet.getPackageRegistry().put(MyPackage.eNS_URI,
> MyPackage.eINSTANCE);
>
> Resource resource = resourceSet.getResource(uri, true);
>
> return resource.getContents().get(0);
> }
>
>
> //----------------------------- output
> ------------------------------------------//
> 0 elapsed time : 00:00:09.464 free mem : 102
> 1 elapsed time : 00:00:08.986 free mem : 150
> 2 elapsed time : 00:00:07.199 free mem : 297
> 3 elapsed time : 00:00:07.324 free mem : 241
> 4 elapsed time : 00:00:10.059 free mem : 141
> 5 elapsed time : 00:00:20.229 free mem : 115
Ed Merks
Professional Support: https://www.macromodeling.com/
|
|
|
Powered by
FUDForum. Page generated in 0.02851 seconds
] 
Search
Help
Register
Login
Home
