[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
[geogig-dev] pg export on large tables causes OutOfMemoryError
|
Hi all,
I'm trying to export a large table of point features from geogig to PostGIS. The table contains about 11 million rows.
Pg import fetches all these records into a geogig repository without failing.
But pg export fails and runs into a OutOfMemoryError.
In the default configuration (2 GB memory) I'm able to export about 35%, that is 3.9 million rows.
If I give the JVM 4 GB of memory, it fails later (at about 63%, 7 million rows), but it still fails.
Before failing, exporting gets extremely slow because the JVM is occupied with garbage collection nearly all of the time. In the 4GB example the OutOfMemory came after about 8 hours.
I'm using a Oracle JRE 1.7., 64bit on a windows machine. My Geogig binary is a recent build (September 2015) from the current master branch.
I get a
Exception in thread "main" com.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199)
at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4736)
at org.locationtech.geogig.di.caching.ObjectDatabaseCacheInterceptor$CacheHelper.get(ObjectDatabaseCacheInterceptor.java:211)
at org.locationtech.geogig.di.caching.ObjectDatabaseCacheInterceptor$CachingObjectDatabase.get(ObjectDatabaseCacheInterceptor.java:104)
at org.locationtech.geogig.di.caching.ObjectDatabaseCacheInterceptor$CachingObjectDatabase.getTree(ObjectDatabaseCacheInterceptor.java:109)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$FeatureBuckets.resolveBucketEntries(DepthTreeIterator.java:363)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:306)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:289)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:303)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:289)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:303)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:289)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Features.computeNext(DepthTreeIterator.java:254)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Features.computeNext(DepthTreeIterator.java:238)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:43)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator.computeNext(DepthTreeIterator.java:150)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator.computeNext(DepthTreeIterator.java:37)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:43)
at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:43)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:43)
at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:43)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1144)
at org.geotools.feature.collection.DelegateFeatureIterator.hasNext(DelegateFeatureIterator.java:67)
at org.geotools.data.store.ContentFeatureStore.addFeatures(ContentFeatureStore.java:261)
at org.locationtech.geogig.geotools.plumbing.ExportOp._call(ExportOp.java:200)
at org.locationtech.geogig.geotools.plumbing.ExportOp._call(ExportOp.java:74)
at org.locationtech.geogig.api.AbstractGeoGigOp.call(AbstractGeoGigOp.java:132)
at org.locationtech.geogig.geotools.cli.DataStoreExport.runInternal(DataStoreExport.java:149)
at org.locationtech.geogig.cli.AbstractCommand.run(AbstractCommand.java:70)
at org.locationtech.geogig.cli.GeogigCLI.executeInternal(GeogigCLI.java:559)
at org.locationtech.geogig.cli.GeogigCLI.execute(GeogigCLI.java:405)
at org.locationtech.geogig.cli.GeogigCLI.main(GeogigCLI.java:350)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.locationtech.geogig.api.Node.create(Node.java:135)
at org.locationtech.geogig.storage.datastream.FormatCommonV2.readNode(FormatCommonV2.java:532)
at org.locationtech.geogig.storage.datastream.FormatCommonV2.readTree(FormatCommonV2.java:227)
at org.locationtech.geogig.storage.datastream.DataStreamSerializationFactoryV2$TreeSerializer.readBody(DataStreamSerializationFactoryV2.java:207)
at org.locationtech.geogig.storage.datastream.DataStreamSerializationFactoryV2$TreeSerializer.readBody(DataStreamSerializationFactoryV2.java:199)
at org.locationtech.geogig.storage.datastream.DataStreamSerializationFactoryV2.read(DataStreamSerializationFactoryV2.java:72)
at org.locationtech.geogig.storage.AbstractObjectDatabase.get(AbstractObjectDatabase.java:141)
at org.locationtech.geogig.storage.AbstractObjectDatabase.get(AbstractObjectDatabase.java:95)
at org.locationtech.geogig.storage.ForwardingObjectDatabase.get(ForwardingObjectDatabase.java:76)
at org.locationtech.geogig.di.caching.ObjectDatabaseCacheInterceptor$ValueLoader.call(ObjectDatabaseCacheInterceptor.java:171)
at org.locationtech.geogig.di.caching.ObjectDatabaseCacheInterceptor$ValueLoader.call(ObjectDatabaseCacheInterceptor.java:158)
at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4739)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)
at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4736)
at org.locationtech.geogig.di.caching.ObjectDatabaseCacheInterceptor$CacheHelper.get(ObjectDatabaseCacheInterceptor.java:211)
at org.locationtech.geogig.di.caching.ObjectDatabaseCacheInterceptor$CachingObjectDatabase.get(ObjectDatabaseCacheInterceptor.java:104)
at org.locationtech.geogig.di.caching.ObjectDatabaseCacheInterceptor$CachingObjectDatabase.getTree(ObjectDatabaseCacheInterceptor.java:109)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$FeatureBuckets.resolveBucketEntries(DepthTreeIterator.java:363)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:306)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:289)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:303)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:289)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:303)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:289)
Any ideas how to cope with this (apart from giving even more memory)? I guess we have a memory leak here...
Is it possible to export only a certain region from a geogig repository into a PostGIS database? That would speed up processes a lot.
I'd prefer to have a complete Repository and doing a sparse export.
Kind regards,
Sebastian
