Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Web Tools Project (WTP) » Large XML files loaded twice into memory
Large XML files loaded twice into memory [message #1219674] Thu, 05 December 2013 05:55 Go to next message
Philippe Marschall is currently offline Philippe Marschall
Messages: 120
Registered: July 2009
Senior Member
Hi

It seems that WTP loads each XML file twice into memory as a single,
huge String. Once for the task scanner and once for the XML markup
validator. If you have every large XML files (400 MB) this makes your
Eclipse crash even if you don't open them.

Making the task scanner and the XML markup validator not load the entire
file into memory is probably non-trivial. Would it be an option to skip
large files (say bigger than 10 MB) to at least avoid crashing Eclipse?

Cheers
Philippe
Re: Large XML files loaded twice into memory [message #1220253 is a reply to message #1219674] Tue, 10 December 2013 11:21 Go to previous messageGo to next message
Nick Sandonato is currently offline Nick Sandonato
Messages: 125
Registered: July 2009
Senior Member
Hi Philippe,

Thanks for your comments. There was a bug opened for the XML task scanner [1] that was just fixed. The Markup Validator does indeed work by loading the entire structured document into memory, including the model, to check the markup. We might be able to refactor a lot of the validation into a streaming fashion similar to the task scanner. I would have to do some investigation.

Your suggestion of skipping over files that are too large is definitely a possibility too. If we're going to crash anyway, there's no point in even trying to validate it. But it would definitely be cooler to get more of a streaming solution.

[1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=364880

[Updated on: Tue, 10 December 2013 11:22]

Report message to a moderator

Re: Large XML files loaded twice into memory [message #1220576 is a reply to message #1220253] Thu, 12 December 2013 10:22 Go to previous messageGo to next message
Philippe Marschall is currently offline Philippe Marschall
Messages: 120
Registered: July 2009
Senior Member
On 10.12.13 17:21, Nick Sandonato wrote:
> Hi Philippe,
>
> Thanks for your comments. There was a bug opened for the XML task
> scanner [1] that was just fixed. The Markup Validator does indeed work
> by loading the entire structured document into memory, including the
> model, to check the markup. We might be able to refactor a lot of the
> validation into a streaming fashion similar to the task scanner. I would
> have to do some investigation.

That is excellent news.

Cheers
Philippe
Re: Large XML files loaded twice into memory [message #1222601 is a reply to message #1220576] Fri, 20 December 2013 16:43 Go to previous messageGo to next message
Nick Sandonato is currently offline Nick Sandonato
Messages: 125
Registered: July 2009
Senior Member
Philippe Marschall wrote on Thu, 12 December 2013 10:22
On 10.12.13 17:21, Nick Sandonato wrote:
> Hi Philippe,
>
> Thanks for your comments. There was a bug opened for the XML task
> scanner [1] that was just fixed. The Markup Validator does indeed work
> by loading the entire structured document into memory, including the
> model, to check the markup. We might be able to refactor a lot of the
> validation into a streaming fashion similar to the task scanner. I would
> have to do some investigation.

That is excellent news.

Cheers
Philippe


Philippe,

I pushed some changes that should dramatically improve the performance of the XML validator for larger files. Check out https://bugs.eclipse.org/424552 for information.
Re: Large XML files loaded twice into memory [message #1277969 is a reply to message #1219674] Wed, 26 March 2014 14:26 Go to previous message
barrieu dominique is currently offline barrieu dominique
Messages: 2
Registered: July 2012
Junior Member
Well,

On my plateform 4.3.0.v20130911-1000 wuth java.runtime.version=1.7.0_04, when I try to open an xml file size 79 Mo an exception is thrown :

Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
at java.lang.AbstractStringBuilder.append(Unknown Source)
at java.lang.StringBuffer.append(Unknown Source)
at org.eclipse.core.internal.filebuffers.ResourceTextFileBuffer.setDocumentContent(ResourceTextFileBuffer.java:575)
at org.eclipse.core.internal.filebuffers.ResourceTextFileBuffer.initializeFileBufferContent(ResourceTextFileBuffer.java:288)
at org.eclipse.core.internal.filebuffers.ResourceFileBuffer.create(ResourceFileBuffer.java:245)

So it's possible to open a large XML file with Eclipse "XML Editor" ?
Previous Topic:XSL embedded editor
Next Topic:Simple question about deployment assembly
Goto Forum:
  


Current Time: Mon Apr 21 05:52:59 EDT 2014

Powered by FUDForum. Page generated in 0.01672 seconds