Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Language IDEs » ServerTools (WTP) » Large XML files loaded twice into memory
Large XML files loaded twice into memory [message #1219674] Thu, 05 December 2013 10:55 Go to next message
Philippe Marschall is currently offline Philippe MarschallFriend
Messages: 121
Registered: July 2009
Senior Member
Hi

It seems that WTP loads each XML file twice into memory as a single,
huge String. Once for the task scanner and once for the XML markup
validator. If you have every large XML files (400 MB) this makes your
Eclipse crash even if you don't open them.

Making the task scanner and the XML markup validator not load the entire
file into memory is probably non-trivial. Would it be an option to skip
large files (say bigger than 10 MB) to at least avoid crashing Eclipse?

Cheers
Philippe

Report message to a moderator

Re: Large XML files loaded twice into memory [message #1220253 is a reply to message #1219674] Tue, 10 December 2013 16:21 Go to previous messageGo to next message
Nick Sandonato is currently offline Nick SandonatoFriend
Messages: 126
Registered: July 2009
Senior Member
Hi Philippe,

Thanks for your comments. There was a bug opened for the XML task scanner [1] that was just fixed. The Markup Validator does indeed work by loading the entire structured document into memory, including the model, to check the markup. We might be able to refactor a lot of the validation into a streaming fashion similar to the task scanner. I would have to do some investigation.

Your suggestion of skipping over files that are too large is definitely a possibility too. If we're going to crash anyway, there's no point in even trying to validate it. But it would definitely be cooler to get more of a streaming solution.

[1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=364880

[Updated on: Tue, 10 December 2013 16:22]

Report message to a moderator

Re: Large XML files loaded twice into memory [message #1220576 is a reply to message #1220253] Thu, 12 December 2013 15:22 Go to previous messageGo to next message
Philippe Marschall is currently offline Philippe MarschallFriend
Messages: 121
Registered: July 2009
Senior Member
On 10.12.13 17:21, Nick Sandonato wrote:
> Hi Philippe,
>
> Thanks for your comments. There was a bug opened for the XML task
> scanner [1] that was just fixed. The Markup Validator does indeed work
> by loading the entire structured document into memory, including the
> model, to check the markup. We might be able to refactor a lot of the
> validation into a streaming fashion similar to the task scanner. I would
> have to do some investigation.

That is excellent news.

Cheers
Philippe

Report message to a moderator

Re: Large XML files loaded twice into memory [message #1222601 is a reply to message #1220576] Fri, 20 December 2013 21:43 Go to previous messageGo to next message
Nick Sandonato is currently offline Nick SandonatoFriend
Messages: 126
Registered: July 2009
Senior Member
Philippe Marschall wrote on Thu, 12 December 2013 10:22
On 10.12.13 17:21, Nick Sandonato wrote:
> Hi Philippe,
>
> Thanks for your comments. There was a bug opened for the XML task
> scanner [1] that was just fixed. The Markup Validator does indeed work
> by loading the entire structured document into memory, including the
> model, to check the markup. We might be able to refactor a lot of the
> validation into a streaming fashion similar to the task scanner. I would
> have to do some investigation.

That is excellent news.

Cheers
Philippe


Philippe,

I pushed some changes that should dramatically improve the performance of the XML validator for larger files. Check out https://bugs.eclipse.org/424552 for information.

Report message to a moderator

Re: Large XML files loaded twice into memory [message #1277969 is a reply to message #1219674] Wed, 26 March 2014 18:26 Go to previous message
barrieu dominique is currently offline barrieu dominiqueFriend
Messages: 2
Registered: July 2012
Junior Member
Well,

On my plateform 4.3.0.v20130911-1000 wuth java.runtime.version=1.7.0_04, when I try to open an xml file size 79 Mo an exception is thrown :

Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
at java.lang.AbstractStringBuilder.append(Unknown Source)
at java.lang.StringBuffer.append(Unknown Source)
at org.eclipse.core.internal.filebuffers.ResourceTextFileBuffer.setDocumentContent(ResourceTextFileBuffer.java:575)
at org.eclipse.core.internal.filebuffers.ResourceTextFileBuffer.initializeFileBufferContent(ResourceTextFileBuffer.java:288)
at org.eclipse.core.internal.filebuffers.ResourceFileBuffer.create(ResourceFileBuffer.java:245)

So it's possible to open a large XML file with Eclipse "XML Editor" ?

Report message to a moderator

Previous Topic:XSL embedded editor
Next Topic:Simple question about deployment assembly
Goto Forum:
  

[ Syndicate this forum (XML) ] [ RSS ]

Current Time: Fri Apr 19 17:07:43 GMT 2024

Powered by FUDForum. Page generated in 0.02780 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top

Sign up to our Newsletter

A fresh new issue delivered monthly