Eclipse Community Forums: EclipseLink » Persisting thousands of files

Help

Home

Home » Eclipse Projects » EclipseLink » Persisting thousands of files

Show: Today's Messages :: Show Polls :: Message Navigator

Persisting thousands of files [message #385455]

Wed, 11 February 2009 13:53

Andre Ribeiro

Messages: 58
Registered: July 2009

Member

Hi all,

I'm implementing an use case that requires me to persist, update and
read thousands of database records (containing files in this case).
So far I've been using Spring + EclipseLink successfully with some
hundreds of rows per transaction, but once i get to the magnitude of
thousands i get OutOfMemory errors of Java heap space exceptions, etc ...

So, i would like to know if there's any implemented (unit-) test in
EclipseLink to persist, update and grab some large amount (thousands or
hundreds of thousands) of rows.

Currently my code is roughly as follows ...

@Service
public class FileImportingService {

...

@Transactional(rollbackFor = Exception.class)
public void insertFilesFromDisk(Map<String, MyFile> discPaths_files,
IProgressMonitor monitor) {
for (String discPathString : discPaths_files.keySet()) {
myFile = read_file_from_disc(discPathString);
getJpaTemplate().persist(myFile);
monitor.worked(1);
}
}

....

}

Thanks in advance for the ideas, suggestions ...

Cheers,
André Ribeiro

Report message to a moderator

Re: Persisting thousands of files [message #385456 is a reply to message #385455]

Wed, 11 February 2009 15:38

Doug Clarke

Messages: 155
Registered: July 2009

Senior Member

Andre,

I am working on some large scale example applications as well. My data
loading was running out of memory so I increased the heap size on the Java
VM startup but that only got me a bit further. It turned out that I was
simply requiring more memory for my data load scenario then was available.

To get around this I modified my data load to flush the data I am loading
in batches. This allowed me to still have a single transaction for the
entire data set but EclipseLink would not need to manage all of the
instances in memory until after commit. After each batch of objects that I
added using entityManager.persist I then issued:

entityManager.flush();
entityManager.clear();

Using this approach I have been able to continually increase the volume of
objects I load and I simply tested with various batch sizes to find the
optimal size.

DOug

Report message to a moderator

Re: Persisting thousands of files [message #385482 is a reply to message #385456]

Thu, 12 February 2009 13:36

Andre Ribeiro

Messages: 58
Registered: July 2009

Member

Hi Doug,

Thank you very much for your tips :)

Are you're example applications available somewhere with public access?
I'd find it very useful to see some sample implementations to get some
ideas.

Cheers,
Andre

Doug Clarke wrote:
> Andre,
>
> I am working on some large scale example applications as well. My data
> loading was running out of memory so I increased the heap size on the
> Java VM startup but that only got me a bit further. It turned out that
> I was simply requiring more memory for my data load scenario then was
> available.
>
> To get around this I modified my data load to flush the data I am
> loading in batches. This allowed me to still have a single transaction
> for the entire data set but EclipseLink would not need to manage all
> of the instances in memory until after commit. After each batch of
> objects that I added using entityManager.persist I then issued:
>
> entityManager.flush();
> entityManager.clear();
>
> Using this approach I have been able to continually increase the
> volume of objects I load and I simply tested with various batch sizes
> to find the optimal size.
>
> DOug
>

Report message to a moderator

Previous Topic:	SQL statement that createNamedQuery(...) method sent to the database?
Next Topic:	EntityManager life cycle in OSGi application

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Thu Apr 18 16:21:02 GMT 2024

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter