Re: [geogig-dev] pg export on large tables causes OutOfMemoryError

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]

Re: [geogig-dev] pg export on large tables causes OutOfMemoryError

From: Gabriel Roldan <groldan@xxxxxxxxxxxxxxxx>
Date: Wed, 4 Nov 2015 17:50:22 -0300
Delivered-to: geogig-dev@xxxxxxxxxxxxxxxx
List-archive: <http://www.locationtech.org/mailman/private/geogig-dev>
List-help: <mailto:geogig-dev-request@locationtech.org?subject=help>
List-subscribe: <http://www.locationtech.org/mailman/listinfo/geogig-dev>, <mailto:geogig-dev-request@locationtech.org?subject=subscribe>
List-unsubscribe: <http://www.locationtech.org/mailman/options/geogig-dev>, <mailto:geogig-dev-request@locationtech.org?subject=unsubscribe>

Hmmm, awful.

I'll give it a try and come back with an answer/patch asap. Definitely looks like a mem leak.

Re the sparse export, it's definitely doable but I think there's no bbox argument for the export command right now.

We can work on it.

On Tue, Nov 3, 2015 at 5:02 AM, <Schmidt.Sebastian2@xxxxxx> wrote:

Hi all,

I'm trying to export a large table of point features from geogig to PostGIS. The table contains about 11 million rows.

Pg import fetches all these records into a geogig repository without failing.

But pg export fails and runs into a OutOfMemoryError.
In the default configuration (2 GB memory) I'm able to export about 35%, that is 3.9 million rows.
If I give the JVM 4 GB of memory, it fails later (at about 63%, 7 million rows), but it still fails.
Before failing, exporting gets extremely slow because the JVM is occupied with garbage collection nearly all of the time. In the 4GB example the OutOfMemory came after about 8 hours.

I'm using a Oracle JRE 1.7., 64bit on a windows machine. My Geogig binary is a recent build (September 2015) from the current master branch.

I get a
Exception in thread "main" com.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199)
at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4736)
at org.locationtech.geogig.di.caching.ObjectDatabaseCacheInterceptor$CacheHelper.get(ObjectDatabaseCacheInterceptor.java:211)
at org.locationtech.geogig.di.caching.ObjectDatabaseCacheInterceptor$CachingObjectDatabase.get(ObjectDatabaseCacheInterceptor.java:104)
at org.locationtech.geogig.di.caching.ObjectDatabaseCacheInterceptor$CachingObjectDatabase.getTree(ObjectDatabaseCacheInterceptor.java:109)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$FeatureBuckets.resolveBucketEntries(DepthTreeIterator.java:363)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:306)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:289)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:303)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:289)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:303)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:289)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Features.computeNext(DepthTreeIterator.java:254)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Features.computeNext(DepthTreeIterator.java:238)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:43)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator.computeNext(DepthTreeIterator.java:150)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator.computeNext(DepthTreeIterator.java:37)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:43)
at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:43)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:43)
at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:43)
at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1144)
at org.geotools.feature.collection.DelegateFeatureIterator.hasNext(DelegateFeatureIterator.java:67)
at org.geotools.data.store.ContentFeatureStore.addFeatures(ContentFeatureStore.java:261)
at org.locationtech.geogig.geotools.plumbing.ExportOp._call(ExportOp.java:200)
at org.locationtech.geogig.geotools.plumbing.ExportOp._call(ExportOp.java:74)
at org.locationtech.geogig.api.AbstractGeoGigOp.call(AbstractGeoGigOp.java:132)
at org.locationtech.geogig.geotools.cli.DataStoreExport.runInternal(DataStoreExport.java:149)
at org.locationtech.geogig.cli.AbstractCommand.run(AbstractCommand.java:70)
at org.locationtech.geogig.cli.GeogigCLI.executeInternal(GeogigCLI.java:559)
at org.locationtech.geogig.cli.GeogigCLI.execute(GeogigCLI.java:405)
at org.locationtech.geogig.cli.GeogigCLI.main(GeogigCLI.java:350)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.locationtech.geogig.api.Node.create(Node.java:135)
at org.locationtech.geogig.storage.datastream.FormatCommonV2.readNode(FormatCommonV2.java:532)
at org.locationtech.geogig.storage.datastream.FormatCommonV2.readTree(FormatCommonV2.java:227)
at org.locationtech.geogig.storage.datastream.DataStreamSerializationFactoryV2$TreeSerializer.readBody(DataStreamSerializationFactoryV2.java:207)
at org.locationtech.geogig.storage.datastream.DataStreamSerializationFactoryV2$TreeSerializer.readBody(DataStreamSerializationFactoryV2.java:199)
at org.locationtech.geogig.storage.datastream.DataStreamSerializationFactoryV2.read(DataStreamSerializationFactoryV2.java:72)
at org.locationtech.geogig.storage.AbstractObjectDatabase.get(AbstractObjectDatabase.java:141)
at org.locationtech.geogig.storage.AbstractObjectDatabase.get(AbstractObjectDatabase.java:95)
at org.locationtech.geogig.storage.ForwardingObjectDatabase.get(ForwardingObjectDatabase.java:76)
at org.locationtech.geogig.di.caching.ObjectDatabaseCacheInterceptor$ValueLoader.call(ObjectDatabaseCacheInterceptor.java:171)
at org.locationtech.geogig.di.caching.ObjectDatabaseCacheInterceptor$ValueLoader.call(ObjectDatabaseCacheInterceptor.java:158)
at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4739)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)
at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4736)
at org.locationtech.geogig.di.caching.ObjectDatabaseCacheInterceptor$CacheHelper.get(ObjectDatabaseCacheInterceptor.java:211)
at org.locationtech.geogig.di.caching.ObjectDatabaseCacheInterceptor$CachingObjectDatabase.get(ObjectDatabaseCacheInterceptor.java:104)
at org.locationtech.geogig.di.caching.ObjectDatabaseCacheInterceptor$CachingObjectDatabase.getTree(ObjectDatabaseCacheInterceptor.java:109)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$FeatureBuckets.resolveBucketEntries(DepthTreeIterator.java:363)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:306)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:289)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:303)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:289)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:303)
at org.locationtech.geogig.api.plumbing.diff.DepthTreeIterator$Buckets.computeNext(DepthTreeIterator.java:289)

Any ideas how to cope with this (apart from giving even more memory)? I guess we have a memory leak here...

Is it possible to export only a certain region from a geogig repository into a PostGIS database? That would speed up processes a lot.
I'd prefer to have a complete Repository and doing a sparse export.

Kind regards,
Sebastian
_______________________________________________
geogig-dev mailing list
geogig-dev@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geogig-dev

Gabriel Roldán
Software Developer | Boundless
groldan@xxxxxxxxxxxxxxxx
@boundlessgeo

References:
- [geogig-dev] pg export on large tables causes OutOfMemoryError
  - From: Schmidt.Sebastian2

Prev by Date: [geogig-dev] Some questions about data modeling
Next by Date: Re: [geogig-dev] GeoGig support for WKT
Previous by thread: [geogig-dev] pg export on large tables causes OutOfMemoryError
Next by thread: [geogig-dev] Some questions about data modeling
Index(es):
- Date
- Thread

Breadcrumbs