Eclipse Community Forums: ServerTools (WTP) » Large XML files loaded twice into memory

Home » Language IDEs » ServerTools (WTP) » Large XML files loaded twice into memory

Large XML files loaded twice into memory [message #1219674]

Thu, 05 December 2013 05:55

Eclipse User

Hi

It seems that WTP loads each XML file twice into memory as a single,
huge String. Once for the task scanner and once for the XML markup
validator. If you have every large XML files (400 MB) this makes your
Eclipse crash even if you don't open them.

Making the task scanner and the XML markup validator not load the entire
file into memory is probably non-trivial. Would it be an option to skip
large files (say bigger than 10 MB) to at least avoid crashing Eclipse?

Cheers
Philippe

Re: Large XML files loaded twice into memory [message #1220253 is a reply to message #1219674]

Tue, 10 December 2013 11:21

Eclipse User

Hi Philippe,

Thanks for your comments. There was a bug opened for the XML task scanner [1] that was just fixed. The Markup Validator does indeed work by loading the entire structured document into memory, including the model, to check the markup. We might be able to refactor a lot of the validation into a streaming fashion similar to the task scanner. I would have to do some investigation.

Your suggestion of skipping over files that are too large is definitely a possibility too. If we're going to crash anyway, there's no point in even trying to validate it. But it would definitely be cooler to get more of a streaming solution.

[1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=364880

[Updated on: Tue, 10 December 2013 11:22] by Moderator

Re: Large XML files loaded twice into memory [message #1220576 is a reply to message #1220253]

Thu, 12 December 2013 10:22

Eclipse User

On 10.12.13 17:21, Nick Sandonato wrote:
> Hi Philippe,
>
> Thanks for your comments. There was a bug opened for the XML task
> scanner [1] that was just fixed. The Markup Validator does indeed work
> by loading the entire structured document into memory, including the
> model, to check the markup. We might be able to refactor a lot of the
> validation into a streaming fashion similar to the task scanner. I would
> have to do some investigation.

That is excellent news.

Cheers
Philippe

Re: Large XML files loaded twice into memory [message #1222601 is a reply to message #1220576]

Fri, 20 December 2013 16:43

Eclipse User

Philippe Marschall wrote on Thu, 12 December 2013 10:22

On 10.12.13 17:21, Nick Sandonato wrote:
> Hi Philippe,
>
> Thanks for your comments. There was a bug opened for the XML task
> scanner [1] that was just fixed. The Markup Validator does indeed work
> by loading the entire structured document into memory, including the
> model, to check the markup. We might be able to refactor a lot of the
> validation into a streaming fashion similar to the task scanner. I would
> have to do some investigation.

That is excellent news.

Cheers
Philippe

Philippe,

I pushed some changes that should dramatically improve the performance of the XML validator for larger files. Check out https://bugs.eclipse.org/424552 for information.

Re: Large XML files loaded twice into memory [message #1277969 is a reply to message #1219674]

Wed, 26 March 2014 14:26

Eclipse User

Well,

On my plateform 4.3.0.v20130911-1000 wuth java.runtime.version=1.7.0_04, when I try to open an xml file size 79 Mo an exception is thrown :

Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
at java.lang.AbstractStringBuilder.append(Unknown Source)
at java.lang.StringBuffer.append(Unknown Source)
at org.eclipse.core.internal.filebuffers.ResourceTextFileBuffer.setDocumentContent(ResourceTextFileBuffer.java:575)
at org.eclipse.core.internal.filebuffers.ResourceTextFileBuffer.initializeFileBufferContent(ResourceTextFileBuffer.java:288)
at org.eclipse.core.internal.filebuffers.ResourceFileBuffer.create(ResourceFileBuffer.java:245)

So it's possible to open a large XML file with Eclipse "XML Editor" ?

Previous Topic:	XSL embedded editor
Next Topic:	Simple question about deployment assembly

Goto Forum:

-=] Back to Top [=-

Current Time: Wed Jul 23 05:05:15 EDT 2025

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter