Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geowave-dev] Multiband geotiff problem
Jose,
In digging into this a little, I was disappointed that I could not recreate the issue you are seeing.  Your ingest code snippet looks great, although it does appear to be based on an older version of geowave than I had tested with. That being said I don't think the underlying raster implementation likely changed so again I am disappointed that I can't recreate it.  We have been working 0.9.1 on a '0.9.1' branch and are closing in on our release (we merged the '0.9.1' dev branch in with master around the middle of last week, look out for the 0.9.1 release in the next couple days, we'll send an announcement to the list).

So the behavior that I see which seems a bit different than the png's that you provided, is that it just works when there is a matching resolution tile set with the zoom level in open layers preview.  Then when I zoom in too tight, beyond the original resolution image, or when I zoom out to the point that it will be scaling the lowest resolution pyramid, there is an issue with scaling where the underlying color model of the scaled images do not match the original color model.  I suspected the issue could be traced to significant changes that have been undergone in geotools 14.x integrating JAI-EXT into geotools for raster operations: https://osgeo-org.atlassian.net/browse/GEOT-5061

GeoWave stores the original image as tiles (optionally building an image pyramid on ingest). The appropriate tiles are quickly retrieved from GeoWave and the resultant WMS request is composed using a mosaic'ing approach that is very closely related to the 'imagemosaic' geotools extension.  So having discovered this change:
https://github.com/geotools/geotools/blame/42c4cc0c3d850b1dea50c07108f171aace35a798/modules/plugin/imagemosaic-jdbc/src/main/java/org/geotools/gce/imagemosaic/jdbc/AbstractThread.java#L101

And applying it to GeoWave, the scaling now works.  We strive to maintain support for the latest release and the previous one for all of our dependencies which is constantly a challenge.  We test those variations by applying a very large build matrix on our own jenkins CI but also a smaller subset in travis CI (https://github.com/ngageoint/geowave/blob/master/.travis.yml).  This made it very difficult to support raster operations for geotools 13.x and 14.x simultaneously given the scale of the JAI-EXT changes.  For 0.9.1 we'd like to update that support to 15.x and 14.x so we can move forward with the implications of these changes.

Here's the PR that will fix the scaling issue (as well as add a method RasterUtils.forceRenderingHints() that will give a user an opportunity to override the rendering hints that we apply by default in the case as you were running into that you may want to apply a lenient datum shift or some other less common hint).

https://github.com/ngageoint/geowave/compare/GEOWAVE-760?expand=1

Regarding multi-band, keep in mind the current capability is to handle the multi-band exactly as it is ingested.  So if you have many bands, it will retain the bands exactly and not subset bands.  The question that you had as I understood it is that you wanted to ingest a mutli-band image and have it automatically separate the bands out into completely different coverages.  I see that as a reasonable use case - and I also see as a reasonable use case the inverse operation of combining bands into the same coverage when the ingested data is separated by band.  I think there will be new hooks within geowave in the near future to allow many of these operations.  We will be picking this issue back up again in our next release cycle and will be continuing to flesh out any common use cases: https://github.com/ngageoint/geowave/issues/346 (there is a branch that has most of the code completed for this issue, but needs to be brought up to speed with the latest GeoWave).  Anyways, I did just want to clarify what is meant by multi-band support.  I think the most basic use case is that exactly the bands ingested are how GeoWave presents the data, and an extension to that which we will work is splitting and merging bands between coverages.

Please try the latest with this pull request and let us know if you have any issue.

Rich

 

On Fri, May 6, 2016 at 12:13 AM, Jose M <soloninguno@xxxxxxxxxxx> wrote:

Hi Rich and guys


Continuing testing this problem, i made another try using this usgs sample geotiff http://download.osgeo.org/geotiff/samples/usgs/o41078a5.tif


If i directly publish the geotiff file in geoserver using Geotiff Raster Data Source i get this preview (good)

https://i.imgsafe.org/5024bce.png


But if i ingest this geotiff in geowave and then try to publish the layer using GeoWaveRasterFormat, i get this preview (bad), an error similar to the pictures in my previous email.

https://i.imgsafe.org/51b12c0.png


Given this sample case (geotiff seems to be already in EPSG:4326 so i suppose there is no transform) i almost sure i misusing something in geowave but don't know what.
I'm ingesting the data with this code


        String coverageName = "o41078a5.tif";

        GridCoverage2D coverage = new GeoTiffReader(new File(coverageName)).read(null);

        final Map<String, String> metadata = new HashMap<String, String>();
        RasterDataAdapter adapter = new RasterDataAdapter(coverageName, metadata, coverage, 256, true, new NoDataMergeStrategy());
        PrimaryIndex index = new SpatialIndexBuilder().setAllTiers(true).createIndex();
        try (IndexWriter indexWriter = geowaveDataStore.createIndexWriter(index, DataStoreUtils.DEFAULT_VISIBILITY)) {
          indexWriter.write(adapter, coverage);
        } catch (IOException e) {}

Is this wrong in anyway? 


Regards,


PS: Great to know that there will be multiband support in the future Rich!



De: geowave-dev-bounces@xxxxxxxxxxxxxxxx <geowave-dev-bounces@xxxxxxxxxxxxxxxx> en nombre de Jose M <soloninguno@xxxxxxxxxxx>
Enviado: jueves, 05 de mayo de 2016 02:12 p.m.

Para: geowave-dev@xxxxxxxxxxxxxxxx
Asunto: Re: [geowave-dev] Multiband geotiff problem
 

Hi Rich,


Thanks for looking at the problem having so little time.


I understand why you need the same CRS, and before showing what i'm seeing with your lenient suggestion, i need to give some more info.


Yesterday i was trying to load this geotiff directly in geoserver and when trying to publish the layer discover that the crs was unknown (Native CRS UNKNOWN). But looking at the gdalinfo i saw that it was very similar to the EPSG:45555 (WRF_Lambert_Conformal_Conic 2), so i set this CRS in the "Declared SRS" to EPSG:4555. That make the layer work in geoserver http://postimg.org/image/lonzjf2bl/


So i thought that maybe i can use this crs to load the geotiff before saving it it in geowave, and in fact doing that made the ingest work, the "bursa wolf" issue disappear. This is how I load the geotiff
Hints hints = new Hints();
hints.put(Hints.DEFAULT_COORDINATE_REFERENCE_SYSTEM, CRS.decode("EPSG:45555"));
GridCoverage2D coverage = new GeoTiffReader(new File(rasterFileName),  hint ).read(null);

Nevetherless when publishing the layer through the GeoWaveRasterFormat (both native and declared srs are EPSG:4326 now) this is what i see
http://postimg.org/image/axvrjzzb5/
So it's not looking good.

Now, returning to  your lenient suggestion, when adding this in RasterDataAdapter
   resampleHints.put(Hints.LENIENT_DATUM_SHIFT, Boolean.valueOf(true));
and loading the geotiff without the Hints.DEFAULT_COORDINATE_REFERENCE_SYSTEM
   GridCoverage2D coverage = new GeoTiffReader(new File(rasterFileName),  null).read(null);

It doesn't throw any "bursa wolf" error, but the image seems wrong too (but different than before)
http://postimg.org/image/w1aa5hvup/

So i'm not sure looking at this pictures if geowave is ingesting right or not. Maybe is an issue of the GeoWaveRasterFormat adapter for geoserver?


Thanks,



De: geowave-dev-bounces@xxxxxxxxxxxxxxxx <geowave-dev-bounces@xxxxxxxxxxxxxxxx> en nombre de Rich Fecher <rfecher@xxxxxxxxx>
Enviado: jueves, 05 de mayo de 2016 12:08 p.m.
Para: geowave-dev@xxxxxxxxxxxxxxxx
Asunto: Re: [geowave-dev] Multiband geotiff problem
 
Hi Jose,
I'm at a conference will be traveling back soon, but I checked out your geotiff ingest issue.  I ran gdalinfo to see what CRS is picked up and it appears there is an "unknown" datum.  Constructing a gridcoverage directly through geotools works without any resampling.  Without a well-defined datum, resampling is challenging.  There is a hint you can provide to the resampling operation to be lenient, but essentially you have to use at your own risk.  We need to index all data within the same table using a common coordinate reference system - we choose EPSG:4326, but we will expose a parameter in 0.9.2 to make the CRS of the index configurable.  In your case because the datum is unknown, we'd have to set the transformation to be lenient:
adding this line:
resampleHints.put(Hints.LENIENT_DATUM_SHIFT, Boolean.valueOf(true));

to this method:
https://github.com/ngageoint/geowave/blob/master/extensions/adapters/raster/src/main/java/mil/nga/giat/geowave/adapter/raster/adapter/RasterDataAdapter.java#L772

will properly resample this file, although given that it is just being lenient with an unknown datum here is the bbox of that geotiff that it re-projected to:
min longitude=-41.12983544958792, max longitude=-22.92163541303102, min latitude=-67.82543644308618, max latitude=-48.178035892105775

While I think defaulting to lenient datum shift may or may not be what a user of our library expects, I think there should be some way to programmatically change the default resample hints so that you have a reasonable ability to force operations to do what you want such as this case.

Rich
On Wed, May 4, 2016 at 7:11 PM, Jose M <soloninguno@xxxxxxxxxxx> wrote:

Hi mate!


Didn't know that, thanks!



De: geowave-dev-bounces@xxxxxxxxxxxxxxxx <geowave-dev-bounces@xxxxxxxxxxxxxxxx> en nombre de data sedai <datasedai@xxxxxxxxx>
Enviado: miércoles, 04 de mayo de 2016 07:16 p.m.
Para: geowave-dev@xxxxxxxxxxxxxxxx
Asunto: Re: [geowave-dev] Multiband geotiff problem
 

Hi Jose, most of the team is at a conference.  I think we should have someone back who can answer this in a day or two.

On May 4, 2016 6:12 PM, "Jose M" <soloninguno@xxxxxxxxxxx> wrote:

Hi sorry, anyone could give a hint on this?


Regards,


De: geowave-dev-bounces@xxxxxxxxxxxxxxxx <geowave-dev-bounces@xxxxxxxxxxxxxxxx> en nombre de Jose M <soloninguno@xxxxxxxxxxx>
Enviado: martes, 03 de mayo de 2016 05:49 p.m.
Para: geowave-dev@xxxxxxxxxxxxxxxx
Asunto: [geowave-dev] Multiband geotiff problem
 

Hi,


I need to ingest some multi band geotiffs with geowave, but I'm getting this exception when using RasterDataAdapter.


org.geotools.coverage.processing.CannotReprojectException: Impossible to reproject layer "x".
    at org.geotools.coverage.processing.operation.Resample.doOperation(Resample.java:261)
    at org.geotools.coverage.processing.CoverageProcessor.doOperation(CoverageProcessor.java:605)
    at org.geotools.coverage.processing.CoverageProcessor.doOperation(CoverageProcessor.java:627)
    at org.geotools.coverage.processing.Operations.doOperation(Operations.java:887)
    at org.geotools.coverage.processing.Operations.resample(Operations.java:486)
    at mil.nga.giat.geowave.adapter.raster.adapter.RasterDataAdapter$MosaicPerPyramidLevelBuilder$1.next(RasterDataAdapter.java:676)
    at mil.nga.giat.geowave.adapter.raster.adapter.RasterDataAdapter$MosaicPerPyramidLevelBuilder$1.next(RasterDataAdapter.java:1)
    at mil.nga.giat.geowave.core.store.IteratorWrapper.next(IteratorWrapper.java:75)
    at mil.nga.giat.geowave.datastore.accumulo.AccumuloIndexWriter.write(AccumuloIndexWriter.java:188)
    at mil.nga.giat.geowave.datastore.accumulo.AccumuloIndexWriter.write(AccumuloIndexWriter.java:168)


Caused by: org.opengis.referencing.operation.OperationNotFoundException: Bursa Wolf Parameters are required.
    at org.geotools.referencing.operation.DefaultCoordinateOperationFactory.createOperationStep(DefaultCoordinateOperationFactory.java:1082)
    at org.geotools.referencing.operation.DefaultCoordinateOperationFactory.createOperationStep(DefaultCoordinateOperationFactory.java:1146)
    at org.geotools.referencing.operation.DefaultCoordinateOperationFactory.createOperationStep(DefaultCoordinateOperationFactory.java:891)
    at org.geotools.referencing.operation.DefaultCoordinateOperationFactory.createOperationStep(DefaultCoordinateOperationFactory.java:1008)
    at org.geotools.referencing.operation.DefaultCoordinateOperationFactory.createOperation(DefaultCoordinateOperationFactory.java:286)
    at org.geotools.referencing.operation.BufferedCoordinateOperationFactory.createOperation(BufferedCoordinateOperationFactory.java:256)
    at org.geotools.coverage.processing.operation.Resampler2D.reproject(Resampler2D.java:475)
    at org.geotools.coverage.processing.operation.Resample.doOperation(Resample.java:258)
    ... 12 more


I was checking the accumulo logs and don't see nothing there, so it seems it's a client problem, but in the same project i have a standalone geotools main that open the image ok (just to note, it needs the imageio gdal extensions).


Anyone has seen this error before? Is in fact because of the imagio gdal extension missing in accumulo, or I need to reproject the geotiff before ingesting with geowave?


Here is the geotiff file if anyone wants to try https://www.sendspace.com/file/zkde1q


By the way, i was thinking on creating one coverage per band. Is this a good idea? I don't know how I can write each band separately to the index, is there any utility to do that with geowave?


Regards,


_______________________________________________
geowave-dev mailing list
geowave-dev@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geowave-dev


_______________________________________________
geowave-dev mailing list
geowave-dev@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geowave-dev



_______________________________________________
geowave-dev mailing list
geowave-dev@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geowave-dev


Back to the top