Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » EclipseLink » Persisting thousands of files
Persisting thousands of files [message #385455] Wed, 11 February 2009 13:53 Go to next message
Andre Ribeiro is currently offline Andre Ribeiro
Messages: 58
Registered: July 2009
Member
Hi all,

I'm implementing an use case that requires me to persist, update and
read thousands of database records (containing files in this case).
So far I've been using Spring + EclipseLink successfully with some
hundreds of rows per transaction, but once i get to the magnitude of
thousands i get OutOfMemory errors of Java heap space exceptions, etc ...

So, i would like to know if there's any implemented (unit-) test in
EclipseLink to persist, update and grab some large amount (thousands or
hundreds of thousands) of rows.

Currently my code is roughly as follows ...

@Service
public class FileImportingService {

...

@Transactional(rollbackFor = Exception.class)
public void insertFilesFromDisk(Map<String, MyFile> discPaths_files,
IProgressMonitor monitor) {
for (String discPathString : discPaths_files.keySet()) {
myFile = read_file_from_disc(discPathString);
getJpaTemplate().persist(myFile);
monitor.worked(1);
}
}

....

}

Thanks in advance for the ideas, suggestions ...

Cheers,
André Ribeiro
Re: Persisting thousands of files [message #385456 is a reply to message #385455] Wed, 11 February 2009 15:38 Go to previous messageGo to next message
Doug Clarke is currently offline Doug Clarke
Messages: 155
Registered: July 2009
Senior Member
Andre,

I am working on some large scale example applications as well. My data
loading was running out of memory so I increased the heap size on the Java
VM startup but that only got me a bit further. It turned out that I was
simply requiring more memory for my data load scenario then was available.

To get around this I modified my data load to flush the data I am loading
in batches. This allowed me to still have a single transaction for the
entire data set but EclipseLink would not need to manage all of the
instances in memory until after commit. After each batch of objects that I
added using entityManager.persist I then issued:

entityManager.flush();
entityManager.clear();

Using this approach I have been able to continually increase the volume of
objects I load and I simply tested with various batch sizes to find the
optimal size.

DOug
Re: Persisting thousands of files [message #385482 is a reply to message #385456] Thu, 12 February 2009 13:36 Go to previous message
Andre Ribeiro is currently offline Andre Ribeiro
Messages: 58
Registered: July 2009
Member
Hi Doug,

Thank you very much for your tips :)

Are you're example applications available somewhere with public access?
I'd find it very useful to see some sample implementations to get some
ideas.

Cheers,
Andre

Doug Clarke wrote:
> Andre,
>
> I am working on some large scale example applications as well. My data
> loading was running out of memory so I increased the heap size on the
> Java VM startup but that only got me a bit further. It turned out that
> I was simply requiring more memory for my data load scenario then was
> available.
>
> To get around this I modified my data load to flush the data I am
> loading in batches. This allowed me to still have a single transaction
> for the entire data set but EclipseLink would not need to manage all
> of the instances in memory until after commit. After each batch of
> objects that I added using entityManager.persist I then issued:
>
> entityManager.flush();
> entityManager.clear();
>
> Using this approach I have been able to continually increase the
> volume of objects I load and I simply tested with various batch sizes
> to find the optimal size.
>
> DOug
>
Previous Topic:SQL statement that createNamedQuery(...) method sent to the database?
Next Topic:EntityManager life cycle in OSGi application
Goto Forum:
  


Current Time: Tue Sep 30 10:17:18 GMT 2014

Powered by FUDForum. Page generated in 0.02201 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software