Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geowave-dev] Multiband geotiff problem

Rich, I was testing with the original geotiff https://www.sendspace.com/file/zkde1q and i'm getting an exception in the index write


java.util.NoSuchElementException
    at java.util.TreeMap.key(TreeMap.java:1221)
    at java.util.TreeMap$NavigableSubMap.firstKey(TreeMap.java:1452)
    at mil.nga.giat.geowave.adapter.raster.adapter.RasterDataAdapter.convertToIndex(RasterDataAdapter.java:474)
    at mil.nga.giat.geowave.adapter.raster.adapter.RasterDataAdapter.convertToIndex(RasterDataAdapter.java:1)
    at mil.nga.giat.geowave.core.store.IndependentAdapterIndexWriter.write(IndependentAdapterIndexWriter.java:57)
    at mil.nga.giat.geowave.core.store.IndexCompositeWriter.write(IndexCompositeWriter.java:36)

This is the updated ingest code using the last geowave

       Hints hints = new Hints();
       hints.put(Hints.DEFAULT_COORDINATE_REFERENCE_SYSTEM, CRS.decode("EPSG:4326"));

       GridCoverage2d coverage = new GeoTiffReader(new File("x.tiff"), hints).read(null);

        geowaveDataStore = new AccumuloDataStore(new AccumuloIndexStore(instance),
                new AccumuloAdapterStore(instance),
                new AccumuloDataStatisticsStore(instance),
                new AccumuloSecondaryIndexDataStore(instance),
                new AccumuloAdapterIndexMappingStore(instance),
                instance);

        PrimaryIndex index = new SpatialIndexBuilder().setAllTiers(true).createIndex();
        final Map<String, String> metadata = new HashMap<String, String>();

        RasterDataAdapter adapter = new RasterDataAdapter(coverageName, metadata, coverage, 256, true, new NoDataMergeStrategy());
        try (IndexWriter indexWriter = geowaveDataStore.createWriter(adapter, index)) {
             indexWriter.write(coverage);
        } catch (IOException e) {}


I'm forcing the CRS as the geotiff shows an unknown one (imported from grib), but was told it was in fact EPSG:4326.


Maybe is this some kind of bug or my ingest code is missing something?

Thanks,

De: geowave-dev-bounces@xxxxxxxxxxxxxxxx <geowave-dev-bounces@xxxxxxxxxxxxxxxx> en nombre de Jose M <soloninguno@xxxxxxxxxxx>
Enviado: martes, 10 de mayo de 2016 05:04 p.m.
Para: geowave-dev@xxxxxxxxxxxxxxxx
Asunto: Re: [geowave-dev] Multiband geotiff problem
 

Rich,

After starting from scratch again i got the same issue.  But by some kind of accident i think i discover the problem here.


When I update the geowave code to the latest, I saw there were some changes in the way to ingest the code and how to create the accumulo data store. And I also saw some issues with my project dependencies as it seems the geotiff reader from geotools wasn't included anymore (i suppose i had it auto included with some old geowave dependency). So I start to look more in depth in my dependencies and found out that I had the gt-epsg-hsql database included because of the imageio ext dependency. Although it was the only epsg database i had in the classpath, it wasn't the same database that geowave has as dependency, so I decided to switch it, excluding gt-epsg-hsql and adding gt-epsg-wtk as a standalone dependency.  After doing that and ingesting again in geowave, i got the entire geotiff rendering ok in geoserver!


I'm not trully sure why it fails before because i didn't have both wtk and hsql databases at the same time. But at least i have now the simple case working with the usgs sample, and I can start trying with the my real geotiffs (like the one I post in my first message).


Thanks and hope you don't lost much time looking at this.


De: geowave-dev-bounces@xxxxxxxxxxxxxxxx <geowave-dev-bounces@xxxxxxxxxxxxxxxx> en nombre de Rich Fecher <rfecher@xxxxxxxxx>
Enviado: lunes, 09 de mayo de 2016 03:46 p.m.
Para: geowave-dev@xxxxxxxxxxxxxxxx
Asunto: Re: [geowave-dev] Multiband geotiff problem
 
Jose,
I downloaded geoserver 2.8.2 without any special extensions, built the latest version of geowave for geoserver by enabling building the shaded jar from the geowave-deploy module `cd deploy; mvn package -P geotools-container-singlejar`  Then I copied the shaded jar from the target directory to geoserver's WEB-INF/lib.  I can't imagine I did much differently, and for ingest I simply ran you code snippet on the same geotiff (o41078a5.tif).  As mentioned, I was hoping to recreate the problem, but geoserver's openlayers preview of that image worked well unless I zoomed to a scale that wasn't within the image pyramid (in which case it just wouldn't update tiles).  Using GeoServer 2.7.x worked with zooming in and out, but the fix in the PR linked below will deprecate support of GeoServer 2.7.x for future releases of GeoWave in favor of supporting GeoServer 2.8.x and future releases.  I am very interested in trying to understand what is happening in your situation and helping as best I can.  I haven't seen an image broken in pieces as you experienced, but would like to get to the bottom of it if there is an underlying issue.

Thanks!
Rich


On Mon, May 9, 2016 at 1:32 PM, Jose M <soloninguno@xxxxxxxxxxx> wrote:

Hi Rich,


Thanks a lot for the big analysis you made.


So with that usgs geotiff (o41078a5.tif) you don't see in geoserver the tiff broken in pieces as I do? That's really strange. Can I ask you what geoserver version are you using (and maybe the datastore/layer configuration)? Do you have any of the geoserver extensions or maybe tweak any of the gdal/crs data?


You say the code seems from an older version, how an updated ingest code should look? I don't think it's related but who knows. I'm also working with a 0.9.1-SNAPSHOT although i download it on february (the geowave-adapter-raster jar version I see is 0.9.1-20160201.220751-4, so maybe not the last one).


Meanwhile I while update geowave code to the latest and check with the PR, but want to be sure I test again with the same configuration you have. I was also thinking about issues in the JAI installation but I suppose if that was the case the geotiff would directly not work.


About the multiband yes, I understand how it works now in geowave, but I think having both cases working out of the box will be great. But i have no rush, If i need the bands separately I can split them before ingesting with something like this i think.

        GridCoverage2D []bands = new GridCoverage2D[coverage.getNumSampleDimensions()];
        for(int i=0; i<bands.length; i++)
            bands[i] = (GridCoverage2D) Operations.DEFAULT.selectSampleDimension(coverage, new int[]{i});

Kind regards,




De: geowave-dev-bounces@xxxxxxxxxxxxxxxx <geowave-dev-bounces@xxxxxxxxxxxxxxxx> en nombre de Rich Fecher <rfecher@xxxxxxxxx>
Enviado: lunes, 09 de mayo de 2016 01:39 p.m.

Para: geowave-dev@xxxxxxxxxxxxxxxx
Asunto: Re: [geowave-dev] Multiband geotiff problem
 
Jose,
In digging into this a little, I was disappointed that I could not recreate the issue you are seeing.  Your ingest code snippet looks great, although it does appear to be based on an older version of geowave than I had tested with. That being said I don't think the underlying raster implementation likely changed so again I am disappointed that I can't recreate it.  We have been working 0.9.1 on a '0.9.1' branch and are closing in on our release (we merged the '0.9.1' dev branch in with master around the middle of last week, look out for the 0.9.1 release in the next couple days, we'll send an announcement to the list).

So the behavior that I see which seems a bit different than the png's that you provided, is that it just works when there is a matching resolution tile set with the zoom level in open layers preview.  Then when I zoom in too tight, beyond the original resolution image, or when I zoom out to the point that it will be scaling the lowest resolution pyramid, there is an issue with scaling where the underlying color model of the scaled images do not match the original color model.  I suspected the issue could be traced to significant changes that have been undergone in geotools 14.x integrating JAI-EXT into geotools for raster operations: https://osgeo-org.atlassian.net/browse/GEOT-5061

GeoWave stores the original image as tiles (optionally building an image pyramid on ingest). The appropriate tiles are quickly retrieved from GeoWave and the resultant WMS request is composed using a mosaic'ing approach that is very closely related to the 'imagemosaic' geotools extension.  So having discovered this change:
https://github.com/geotools/geotools/blame/42c4cc0c3d850b1dea50c07108f171aace35a798/modules/plugin/imagemosaic-jdbc/src/main/java/org/geotools/gce/imagemosaic/jdbc/AbstractThread.java#L101

And applying it to GeoWave, the scaling now works.  We strive to maintain support for the latest release and the previous one for all of our dependencies which is constantly a challenge.  We test those variations by applying a very large build matrix on our own jenkins CI but also a smaller subset in travis CI (https://github.com/ngageoint/geowave/blob/master/.travis.yml).  This made it very difficult to support raster operations for geotools 13.x and 14.x simultaneously given the scale of the JAI-EXT changes.  For 0.9.1 we'd like to update that support to 15.x and 14.x so we can move forward with the implications of these changes.

Here's the PR that will fix the scaling issue (as well as add a method RasterUtils.forceRenderingHints() that will give a user an opportunity to override the rendering hints that we apply by default in the case as you were running into that you may want to apply a lenient datum shift or some other less common hint).

https://github.com/ngageoint/geowave/compare/GEOWAVE-760?expand=1

Regarding multi-band, keep in mind the current capability is to handle the multi-band exactly as it is ingested.  So if you have many bands, it will retain the bands exactly and not subset bands.  The question that you had as I understood it is that you wanted to ingest a mutli-band image and have it automatically separate the bands out into completely different coverages.  I see that as a reasonable use case - and I also see as a reasonable use case the inverse operation of combining bands into the same coverage when the ingested data is separated by band.  I think there will be new hooks within geowave in the near future to allow many of these operations.  We will be picking this issue back up again in our next release cycle and will be continuing to flesh out any common use cases: https://github.com/ngageoint/geowave/issues/346 (there is a branch that has most of the code completed for this issue, but needs to be brought up to speed with the latest GeoWave).  Anyways, I did just want to clarify what is meant by multi-band support.  I think the most basic use case is that exactly the bands ingested are how GeoWave presents the data, and an extension to that which we will work is splitting and merging bands between coverages.

Please try the latest with this pull request and let us know if you have any issue.

Rich

 

On Fri, May 6, 2016 at 12:13 AM, Jose M <soloninguno@xxxxxxxxxxx> wrote:

Hi Rich and guys


Continuing testing this problem, i made another try using this usgs sample geotiff http://download.osgeo.org/geotiff/samples/usgs/o41078a5.tif


If i directly publish the geotiff file in geoserver using Geotiff Raster Data Source i get this preview (good)

https://i.imgsafe.org/5024bce.png


But if i ingest this geotiff in geowave and then try to publish the layer using GeoWaveRasterFormat, i get this preview (bad), an error similar to the pictures in my previous email.

https://i.imgsafe.org/51b12c0.png


Given this sample case (geotiff seems to be already in EPSG:4326 so i suppose there is no transform) i almost sure i misusing something in geowave but don't know what.
I'm ingesting the data with this code


        String coverageName = "o41078a5.tif";

        GridCoverage2D coverage = new GeoTiffReader(new File(coverageName)).read(null);

        final Map<String, String> metadata = new HashMap<String, String>();
        RasterDataAdapter adapter = new RasterDataAdapter(coverageName, metadata, coverage, 256, true, new NoDataMergeStrategy());
        PrimaryIndex index = new SpatialIndexBuilder().setAllTiers(true).createIndex();
        try (IndexWriter indexWriter = geowaveDataStore.createIndexWriter(index, DataStoreUtils.DEFAULT_VISIBILITY)) {
          indexWriter.write(adapter, coverage);
        } catch (IOException e) {}

Is this wrong in anyway? 


Regards,


PS: Great to know that there will be multiband support in the future Rich!



De: geowave-dev-bounces@xxxxxxxxxxxxxxxx <geowave-dev-bounces@xxxxxxxxxxxxxxxx> en nombre de Jose M <soloninguno@xxxxxxxxxxx>
Enviado: jueves, 05 de mayo de 2016 02:12 p.m.

Para: geowave-dev@xxxxxxxxxxxxxxxx
Asunto: Re: [geowave-dev] Multiband geotiff problem
 

Hi Rich,


Thanks for looking at the problem having so little time.


I understand why you need the same CRS, and before showing what i'm seeing with your lenient suggestion, i need to give some more info.


Yesterday i was trying to load this geotiff directly in geoserver and when trying to publish the layer discover that the crs was unknown (Native CRS UNKNOWN). But looking at the gdalinfo i saw that it was very similar to the EPSG:45555 (WRF_Lambert_Conformal_Conic 2), so i set this CRS in the "Declared SRS" to EPSG:4555. That make the layer work in geoserver http://postimg.org/image/lonzjf2bl/


So i thought that maybe i can use this crs to load the geotiff before saving it it in geowave, and in fact doing that made the ingest work, the "bursa wolf" issue disappear. This is how I load the geotiff
Hints hints = new Hints();
hints.put(Hints.DEFAULT_COORDINATE_REFERENCE_SYSTEM, CRS.decode("EPSG:45555"));
GridCoverage2D coverage = new GeoTiffReader(new File(rasterFileName),  hint ).read(null);

Nevetherless when publishing the layer through the GeoWaveRasterFormat (both native and declared srs are EPSG:4326 now) this is what i see
http://postimg.org/image/axvrjzzb5/
So it's not looking good.

Now, returning to  your lenient suggestion, when adding this in RasterDataAdapter
   resampleHints.put(Hints.LENIENT_DATUM_SHIFT, Boolean.valueOf(true));
and loading the geotiff without the Hints.DEFAULT_COORDINATE_REFERENCE_SYSTEM
   GridCoverage2D coverage = new GeoTiffReader(new File(rasterFileName),  null).read(null);

It doesn't throw any "bursa wolf" error, but the image seems wrong too (but different than before)
http://postimg.org/image/w1aa5hvup/

So i'm not sure looking at this pictures if geowave is ingesting right or not. Maybe is an issue of the GeoWaveRasterFormat adapter for geoserver?


Thanks,



De: geowave-dev-bounces@xxxxxxxxxxxxxxxx <geowave-dev-bounces@xxxxxxxxxxxxxxxx> en nombre de Rich Fecher <rfecher@xxxxxxxxx>
Enviado: jueves, 05 de mayo de 2016 12:08 p.m.
Para: geowave-dev@xxxxxxxxxxxxxxxx
Asunto: Re: [geowave-dev] Multiband geotiff problem
 
Hi Jose,
I'm at a conference will be traveling back soon, but I checked out your geotiff ingest issue.  I ran gdalinfo to see what CRS is picked up and it appears there is an "unknown" datum.  Constructing a gridcoverage directly through geotools works without any resampling.  Without a well-defined datum, resampling is challenging.  There is a hint you can provide to the resampling operation to be lenient, but essentially you have to use at your own risk.  We need to index all data within the same table using a common coordinate reference system - we choose EPSG:4326, but we will expose a parameter in 0.9.2 to make the CRS of the index configurable.  In your case because the datum is unknown, we'd have to set the transformation to be lenient:
adding this line:
resampleHints.put(Hints.LENIENT_DATUM_SHIFT, Boolean.valueOf(true));

to this method:
https://github.com/ngageoint/geowave/blob/master/extensions/adapters/raster/src/main/java/mil/nga/giat/geowave/adapter/raster/adapter/RasterDataAdapter.java#L772

will properly resample this file, although given that it is just being lenient with an unknown datum here is the bbox of that geotiff that it re-projected to:
min longitude=-41.12983544958792, max longitude=-22.92163541303102, min latitude=-67.82543644308618, max latitude=-48.178035892105775

While I think defaulting to lenient datum shift may or may not be what a user of our library expects, I think there should be some way to programmatically change the default resample hints so that you have a reasonable ability to force operations to do what you want such as this case.

Rich
On Wed, May 4, 2016 at 7:11 PM, Jose M <soloninguno@xxxxxxxxxxx> wrote:

Hi mate!


Didn't know that, thanks!



De: geowave-dev-bounces@xxxxxxxxxxxxxxxx <geowave-dev-bounces@xxxxxxxxxxxxxxxx> en nombre de data sedai <datasedai@xxxxxxxxx>
Enviado: miércoles, 04 de mayo de 2016 07:16 p.m.
Para: geowave-dev@xxxxxxxxxxxxxxxx
Asunto: Re: [geowave-dev] Multiband geotiff problem
 

Hi Jose, most of the team is at a conference.  I think we should have someone back who can answer this in a day or two.

On May 4, 2016 6:12 PM, "Jose M" <soloninguno@xxxxxxxxxxx> wrote:

Hi sorry, anyone could give a hint on this?


Regards,


De: geowave-dev-bounces@xxxxxxxxxxxxxxxx <geowave-dev-bounces@xxxxxxxxxxxxxxxx> en nombre de Jose M <soloninguno@xxxxxxxxxxx>
Enviado: martes, 03 de mayo de 2016 05:49 p.m.
Para: geowave-dev@xxxxxxxxxxxxxxxx
Asunto: [geowave-dev] Multiband geotiff problem
 

Hi,


I need to ingest some multi band geotiffs with geowave, but I'm getting this exception when using RasterDataAdapter.


org.geotools.coverage.processing.CannotReprojectException: Impossible to reproject layer "x".
    at org.geotools.coverage.processing.operation.Resample.doOperation(Resample.java:261)
    at org.geotools.coverage.processing.CoverageProcessor.doOperation(CoverageProcessor.java:605)
    at org.geotools.coverage.processing.CoverageProcessor.doOperation(CoverageProcessor.java:627)
    at org.geotools.coverage.processing.Operations.doOperation(Operations.java:887)
    at org.geotools.coverage.processing.Operations.resample(Operations.java:486)
    at mil.nga.giat.geowave.adapter.raster.adapter.RasterDataAdapter$MosaicPerPyramidLevelBuilder$1.next(RasterDataAdapter.java:676)
    at mil.nga.giat.geowave.adapter.raster.adapter.RasterDataAdapter$MosaicPerPyramidLevelBuilder$1.next(RasterDataAdapter.java:1)
    at mil.nga.giat.geowave.core.store.IteratorWrapper.next(IteratorWrapper.java:75)
    at mil.nga.giat.geowave.datastore.accumulo.AccumuloIndexWriter.write(AccumuloIndexWriter.java:188)
    at mil.nga.giat.geowave.datastore.accumulo.AccumuloIndexWriter.write(AccumuloIndexWriter.java:168)


Caused by: org.opengis.referencing.operation.OperationNotFoundException: Bursa Wolf Parameters are required.
    at org.geotools.referencing.operation.DefaultCoordinateOperationFactory.createOperationStep(DefaultCoordinateOperationFactory.java:1082)
    at org.geotools.referencing.operation.DefaultCoordinateOperationFactory.createOperationStep(DefaultCoordinateOperationFactory.java:1146)
    at org.geotools.referencing.operation.DefaultCoordinateOperationFactory.createOperationStep(DefaultCoordinateOperationFactory.java:891)
    at org.geotools.referencing.operation.DefaultCoordinateOperationFactory.createOperationStep(DefaultCoordinateOperationFactory.java:1008)
    at org.geotools.referencing.operation.DefaultCoordinateOperationFactory.createOperation(DefaultCoordinateOperationFactory.java:286)
    at org.geotools.referencing.operation.BufferedCoordinateOperationFactory.createOperation(BufferedCoordinateOperationFactory.java:256)
    at org.geotools.coverage.processing.operation.Resampler2D.reproject(Resampler2D.java:475)
    at org.geotools.coverage.processing.operation.Resample.doOperation(Resample.java:258)
    ... 12 more


I was checking the accumulo logs and don't see nothing there, so it seems it's a client problem, but in the same project i have a standalone geotools main that open the image ok (just to note, it needs the imageio gdal extensions).


Anyone has seen this error before? Is in fact because of the imagio gdal extension missing in accumulo, or I need to reproject the geotiff before ingesting with geowave?


Here is the geotiff file if anyone wants to try https://www.sendspace.com/file/zkde1q


By the way, i was thinking on creating one coverage per band. Is this a good idea? I don't know how I can write each band separately to the index, is there any utility to do that with geowave?


Regards,


_______________________________________________
geowave-dev mailing list
geowave-dev@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geowave-dev


_______________________________________________
geowave-dev mailing list
geowave-dev@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geowave-dev



_______________________________________________
geowave-dev mailing list
geowave-dev@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geowave-dev



_______________________________________________
geowave-dev mailing list
geowave-dev@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geowave-dev


Back to the top