Eclipse Community Forums: SeMantic Information Logistics Architecture (SMILA) » Serialized Java objects remains on disk after deserializing

Home » Eclipse Projects » SeMantic Information Logistics Architecture (SMILA) » Serialized Java objects remains on disk after deserializing

Show: Today's Messages :: Show Polls :: Message Navigator

Serialized Java objects remains on disk after deserializing [message #655711]

Tue, 22 February 2011 12:05

SMILANewBee

Messages: 42
Registered: August 2010

Member

The web crawler consists of two threads. The first one downloads the content and post processes it. The results is written to disk as Java serialized object into the directory workspace/.metadata/.plugins/org.eclipse.smila.connectivity. framework.crawler.web.

The second thread gets the serialized objects and processes them further to add them to the blackboard. After the second thread deserialized the object it doesn't remove the file. So the directory is swellow with unnecessary data.

To prevent swelling in the class WebCrawler one line can be added to the deserializeIndexDocument method. After the statement

IOUtils.closeQuietly(objstream);

a simple

file.delete();

can be written delete the file after deserializing it.

Report message to a moderator

Re: Serialized Java objects remains on disk after deserializing [message #655897 is a reply to message #655711]

Wed, 23 February 2011 08:22

Andrej Rosenheinrich

Messages: 22
Registered: August 2010

Junior Member

Can someone confirm if those thoughts are right? Or would it cause sidefeffects to delete those files?

Report message to a moderator

Re: Serialized Java objects remains on disk after deserializing [message #656247 is a reply to message #655897]

Thu, 24 February 2011 16:44

Eclipse User

Originally posted by: juergen.schumacher.attensity.com

Am 23.02.2011, 09:22 Uhr, schrieb Andrej Rosenheinrich =

<andrej.rosenheinrich@unister-gmbh.de>:

> Can someone confirm if those thoughts are right? Or would it cause =

> sidefeffects to delete those files?

Sorry, I'm not very accustomed to the Web Crawer, so I'm not sure. From =
=

reading the code,
I could imagine situations where a URL is visited twice during crawling =
=

and the second visit
happens before the first visit is completely processed, so the file woul=
d =

not be recreated
and the processing of the first visit would delete it before the second =
=

visit can read it.
Or something like this. On the other hand this sounds like a dubious =

behaviour anyway (:

Daniel, Tom, do you know anything about this?

Regards,
J=C3=BCrgen

Report message to a moderator

Re: Serialized Java objects remains on disk after deserializing [message #657635 is a reply to message #656247]

Thu, 03 March 2011 14:18

Andrej Rosenheinrich

Messages: 22
Registered: August 2010

Junior Member

To correct the first post here, the serialized files are deleted. This is done (if we understand the code right) in the close() method of the WebCrawler class. So only when the crawl is finished. If the crawl runs for a long time (assume days or weeks) and downloads a significant amount of pages this will be a problem. So maybe it would be the better option to delete these files instantly after processing is finished.

Report message to a moderator

Re: Serialized Java objects remains on disk after deserializing [message #657795 is a reply to message #657635]

Fri, 04 March 2011 05:45

SMILANewBee

Messages: 42
Registered: August 2010

Member

Hello,

we have also noticed, that deleting the files instantly a performance decline was detected. We have too many hard drives accesses. So it would be better if the files would be deleted blockwise.

Report message to a moderator

Re: Serialized Java objects remains on disk after deserializing [message #658164 is a reply to message #657795]

Mon, 07 March 2011 08:55

Eclipse User

Originally posted by: juergen.schumacher.attensity.com

Hi,

Am 04.03.2011, 06:45 Uhr, schrieb SMILANewBee <nils.thieme@unister.de>:
> Hello,
>
> we have also noticed, that deleting the files instantly a performance =
=

> decline was detected. We have too many hard drives accesses. So it wou=
ld =

> be better if the files would be deleted blockwise.

Could you create a bugzilla issue for this, so we are able to track it?
Further discussion should be done in bugzilla then.
Thank you!

Regards,
J=C3=BCrgen.

Report message to a moderator

Re: Serialized Java objects remains on disk after deserializing [message #658371 is a reply to message #658164]

Tue, 08 March 2011 09:05

thomas menzel

Messages: 81
Registered: July 2009

Member

to give my 2 cents:

On 24.02.2011 17:44, Jürgen Schumacher wrote:
>
> Daniel, Tom, do you know anything about this?
>

nope, sorry

On 07.03.2011 09:55, Jürgen Schumacher wrote:
>
> Could you create a bugzilla issue for this, so we are able to track it?
> Further discussion should be done in bugzilla then.
> Thank you!
>

+1 and plz post the issue link here, so people know where to go

--
thomas menzel aka tom

thomas menzel aka tom

Report message to a moderator