Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF » XMI Resource & "OutOfMemoryError: GC Overhead limit exceeded"(Loading an 50 MB big XMI Resource consecutive lead to "OutOfMemoryError: GC Overhead limit exceeded" error.)
XMI Resource & "OutOfMemoryError: GC Overhead limit exceeded" [message #1490784] Fri, 28 November 2014 12:45 Go to next message
crothy donald is currently offline crothy donaldFriend
Messages: 31
Registered: September 2013
Member
Hello,

loading an 50 MB big XMI Resource consecutive lead to "OutOfMemoryError: GC Overhead limit exceeded" error.
I wrote a simple unit test and on my machine, Windows7 with jdk-7u55-windows-x64 emf is able to load the resource 6 times before the jvm throws the exception.
See attached stacktrace, it seams to me that the SAXParser or the XMLHandler does not release the Strings / Maps used during loading so the garbage collector doesn't recover enough heap space and therefore quits.
This matches with the explanation from Oracle: "...This exception is typically thrown because the amount of live data barely fits into the Java heap having little free space for new allocations."

I also tried the performance options on load (parser pool & name to feature map) described on page 495 in the emf bible with no effect.
Tweaking the head size delayed the OutOfMemoryError, so it is not even an workaround.

Any hints, suggestions?
Christian



//----------------------------- stacktrace ------------------------------------------//
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOfRange(Arrays.java:2694)
at java.lang.String.<init>(String.java:203)
at java.lang.StringBuffer.toString(StringBuffer.java:561)
at org.eclipse.emf.ecore.xmi.impl.XMLHandler.endElement(XMLHandler.java:1606)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:609)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1781)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2957)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:649)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:333)
at org.eclipse.emf.ecore.xmi.impl.XMLLoadImpl.load(XMLLoadImpl.java:175)
at org.eclipse.emf.ecore.xmi.impl.XMLResourceImpl.doLoad(XMLResourceImpl.java:261)
at org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(ResourceImpl.java:1518)
at org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(ResourceImpl.java:1297)
at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLoad(ResourceSetImpl.java:259)
at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLoadHelper(ResourceSetImpl.java:274)
at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResource(ResourceSetImpl.java:406)
at PerformanceTestIntegration.loadResource(PerformanceTestIntegration.java:60)



//----------------------------- test ------------------------------------------//
{
// resource is ~50 MB

Runtime runtime = Runtime.getRuntime();
int MB = 1024 * 1024;

for( i = 0; i < 30; i++ ) {
long startTime = System.currentTimeMillis();
UdmType udm = loadResource(nsr4);
assertTrue(udm != null);
long elapsedTime = System.currentTimeMillis() - startTime;

System.out.println(i + " elapsed time : " + formatInterval(elapsedTime) + " free mem : "
+ (runtime.freeMemory() / MB));
}

assertEquals(i, 10);
}

private EObject loadResource(URI uri) {
// Create a resource set to hold the resources.
//
ResourceSet resourceSet = new ResourceSetImpl();

// Register the appropriate resource factory to handle all file
// extensions.
//
resourceSet.getResourceFactoryRegistry().getExtensionToFactoryMap().put("*.xmi", new MyResourceFactoryImpl());

// Register the package to ensure it is available during loading.
//
resourceSet.getPackageRegistry().put(MyPackage.eNS_URI, MyPackage.eINSTANCE);

Resource resource = resourceSet.getResource(uri, true);

return resource.getContents().get(0);
}


//----------------------------- output ------------------------------------------//
0 elapsed time : 00:00:09.464 free mem : 102
1 elapsed time : 00:00:08.986 free mem : 150
2 elapsed time : 00:00:07.199 free mem : 297
3 elapsed time : 00:00:07.324 free mem : 241
4 elapsed time : 00:00:10.059 free mem : 141
5 elapsed time : 00:00:20.229 free mem : 115
Re: XMI Resource &amp; &quot;OutOfMemoryError: GC Overhead limit exceeded&quot; [message #1494163 is a reply to message #1490784] Mon, 01 December 2014 08:26 Go to previous message
Ed Merks is currently offline Ed MerksFriend
Messages: 30540
Registered: July 2009
Senior Member
Christian,

Comments below.

On 28/11/2014 1:45 PM, crothy donald wrote:
> Hello,
>
> loading an 50 MB big XMI Resource consecutive lead to
> "OutOfMemoryError: GC Overhead limit exceeded" error.
> I wrote a simple unit test and on my machine, Windows7 with
> jdk-7u55-windows-x64 emf is able to load the resource 6 times before
> the jvm throws the exception.
> See attached stacktrace, it seams to me that the SAXParser or the
> XMLHandler does not release the Strings / Maps used during loading so
> the garbage collector doesn't recover enough heap space and therefore
> quits.
The question is always of course, what holds onto the SAXParser or the
XMLHandler? Certainly resources loading doesn't generally leak or
someone would have noticed in the last 14 years. It's not clear how
quickly udm will be garbage collected. Certainly in principle that's
possible on each loop iteration but is that what happens in practice?
> This matches with the explanation from Oracle: "...This exception is
> typically thrown because the amount of live data barely fits into the
> Java heap having little free space for new allocations."
So unless it's actually a long-term leak (likely only if you're
retaining instances long term) rather than simply treading near the
threshold of memory exhaustion, increasing your heap capacity is necessary.
>
> I also tried the performance options on load (parser pool & name to
> feature map) described on page 495 in the emf bible with no effect.
> Tweaking the head size delayed the OutOfMemoryError, so it is not even
> an workaround.
If it's a long term leak, you'd need to find the source but as I said,
the pattern you show is a very general one and that does not generally
lead to leaks, although, as I suggested, it's not clear how quickly udm
will be garbage collected and certainly any reference to any object will
leak the entire resource set.
>
> Any hints, suggestions?
> Christian
>
>
>
> //----------------------------- stacktrace
> ------------------------------------------//
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.util.Arrays.copyOfRange(Arrays.java:2694)
> at java.lang.String.<init>(String.java:203)
> at java.lang.StringBuffer.toString(StringBuffer.java:561)
> at
> org.eclipse.emf.ecore.xmi.impl.XMLHandler.endElement(XMLHandler.java:1606)
> at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:609)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1781)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2957)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
> at
> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
> at
> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
> at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:649)
> at
> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:333)
> at
> org.eclipse.emf.ecore.xmi.impl.XMLLoadImpl.load(XMLLoadImpl.java:175)
> at
> org.eclipse.emf.ecore.xmi.impl.XMLResourceImpl.doLoad(XMLResourceImpl.java:261)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(ResourceImpl.java:1518)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(ResourceImpl.java:1297)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLoad(ResourceSetImpl.java:259)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLoadHelper(ResourceSetImpl.java:274)
> at
> org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResource(ResourceSetImpl.java:406)
> at
> PerformanceTestIntegration.loadResource(PerformanceTestIntegration.java:60)
>
>
>
> //----------------------------- test
> ------------------------------------------//
> {
> // resource is ~50 MB
> Runtime runtime = Runtime.getRuntime();
> int MB = 1024 * 1024;
>
> for( i = 0; i < 30; i++ ) {
> long startTime = System.currentTimeMillis();
> UdmType udm = loadResource(nsr4);
> assertTrue(udm != null);
> long elapsedTime = System.currentTimeMillis() - startTime;
>
> System.out.println(i + " elapsed time : " +
> formatInterval(elapsedTime) + " free mem : "
> + (runtime.freeMemory() / MB));
> }
>
> assertEquals(i, 10);
> }
>
> private EObject loadResource(URI uri) {
> // Create a resource set to hold the resources.
> //
> ResourceSet resourceSet = new ResourceSetImpl();
>
> // Register the appropriate resource factory to handle all file
> // extensions.
> //
> resourceSet.getResourceFactoryRegistry().getExtensionToFactoryMap().put("*.xmi",
> new MyResourceFactoryImpl());
>
> // Register the package to ensure it is available during loading.
> //
> resourceSet.getPackageRegistry().put(MyPackage.eNS_URI,
> MyPackage.eINSTANCE);
>
> Resource resource = resourceSet.getResource(uri, true);
>
> return resource.getContents().get(0);
> }
>
>
> //----------------------------- output
> ------------------------------------------//
> 0 elapsed time : 00:00:09.464 free mem : 102
> 1 elapsed time : 00:00:08.986 free mem : 150
> 2 elapsed time : 00:00:07.199 free mem : 297
> 3 elapsed time : 00:00:07.324 free mem : 241
> 4 elapsed time : 00:00:10.059 free mem : 141
> 5 elapsed time : 00:00:20.229 free mem : 115
Previous Topic:import an xmi file into EMF
Next Topic:Validation best practices
Goto Forum:
  


Current Time: Mon Oct 14 05:47:00 GMT 2019

Powered by FUDForum. Page generated in 0.01901 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top