Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geotrellis-user] S3COGLayerWriter - update

I don’t know.. I’m not catching it (I’m not a scala developer as you already know I guess ;)) I think you’re right about the metadata error. Do you see something wrong in our code concerning this specific line?

 

I tried to call other layer that don’t use this metadata value, but I still got an error:

 

[my-system-akka.actor.default-dispatcher-114] [akka.actor.ActorSystemImpl(my-system)] Error during processing of request: '2'. Completing with 500 Internal Server Error response.

 

I will try to better catch my errors to have more precise information.

 

I can’t connect to the geotrellis gitter.. are you still using it?

 

Thanks again for your help.

 

Josée-Anne

 

 

De : geotrellis-user-bounces@xxxxxxxxxxxxxxxx <geotrellis-user-bounces@xxxxxxxxxxxxxxxx> De la part de Grigory Pomadchin
Envoyé : 26 novembre 2018 16:12
À : geotrellis-user@xxxxxxxxxxxxxxxx
Objet : Re: [geotrellis-user] S3COGLayerWriter - update

 

Let me guess, do you have this error on line 82? val metadata = valueReader.attributeStore.readMetadata[TileLayerMetadata[SpatialKey]](layer)

 

On Mon, Nov 26, 2018 at 11:33 PM Langlois, Josée-Anne (NRCan/RNCan) <josee-anne.langlois@xxxxxxxxx> wrote:

 

 

De : geotrellis-user-bounces@xxxxxxxxxxxxxxxx <geotrellis-user-bounces@xxxxxxxxxxxxxxxx> De la part de Grigory Pomadchin
Envoyé : 26 novembre 2018 15:31
À : geotrellis-user@xxxxxxxxxxxxxxxx
Objet : Re: [geotrellis-user] S3COGLayerWriter - update

 

How do you define your reader? (the full code)

 

On Mon, Nov 26, 2018 at 11:09 PM Langlois, Josée-Anne (NRCan/RNCan) <josee-anne.langlois@xxxxxxxxx> wrote:

Thanks again Grigory for your quick and comprehensive answer!

 

I must have a mistake in my script.. I keep having the following errors :

 

When I request a tile for the zoom level 0 I got that error :

 

[my-system-akka.actor.default-dispatcher-19] [akka.actor.ActorSystemImpl(my-system)] Error during processing of request: 'TileLayerMetadata expected'. Completing with 500 Internal Server Error response.

 

And when I request a tile for all other zoom level (from 1 to 14) I got that error :

[my-system-akka.actor.default-dispatcher-11] [akka.actor.ActorSystemImpl(my-system)] Error during processing of request: 'Attribute metadata not found for layer Layer(name = "arcticdemCOG", zoom = 9)'. Completing with 500 Internal Server Error response.

 

Attached is the metadata file of my S3 COG layer. Do you see something wrong in it?

 

 

 

Regards.

Josée-Anne

 

De : geotrellis-user-bounces@xxxxxxxxxxxxxxxx <geotrellis-user-bounces@xxxxxxxxxxxxxxxx> De la part de Grigory Pomadchin
Envoyé : 26 novembre 2018 12:54
À : geotrellis-user@xxxxxxxxxxxxxxxx
Objet : Re: [geotrellis-user] S3COGLayerWriter - update

 

Just use a common zoom level (integer) as usual, reader would manage itself where to take data from. this n_m structure is just a description of how the catalog looks like. Lets go through a small example. 
For instance there is a catalog of the following zoom levels: 0_8, 9_11, 12_13

0_8 means that this folder contains tiles with the base TIFF IFD at zoom level 8 and the smalles ovr is at zoomlvl 0
9_11 means that the base TIFF IFD is at zoom level 11

And you want to read key from zoom level 7. You need just to use a common reader API:

S3COGValueReader(...).reader[K, V](LayerId(name, 7)).read(key) //> would return you a tile at zoom level 7

 

Reader itself would figure out that it has to go into 0_8 folder and that it needs to pick up the first overview of the tiff that corresponds to the key you passed into the function argument.

There is also a small (but probably a bad) example of how we can use COGs API in chatta demo: https://github.com/pomadchin/geotrellis-chatta-demo/commit/f7e1d88168f611d6192c1335371a5514c9882ee1

The idea is that COGs API looks very much like a common Avro API. The only significant difference is in the ingest process.

Let me know if you want to know smth else,

Best, 

Grigory
 

 

On Mon, Nov 26, 2018 at 8:06 PM Langlois, Josée-Anne (NRCan/RNCan) <josee-anne.langlois@xxxxxxxxx> wrote:

Thanks Grigory! Another question ;) : When I want to read the COG layer using the S3COGValueReader  in order to serve the data as a Tile service (xyz) (our script is inspired by the Geotrellis Landsat Tutorial ), what zoom value should we use? n_m or n ?!
 
Thanks!
Josée-Anne

 

De : geotrellis-user-bounces@xxxxxxxxxxxxxxxx <geotrellis-user-bounces@xxxxxxxxxxxxxxxx> De la part de Grigory Pomadchin
Envoyé : 23 novembre 2018 04:20
À : geotrellis-user@xxxxxxxxxxxxxxxx
Objet : Re: [geotrellis-user] S3COGLayerWriter - update

 

Hey! 

Yeah, it's normal. COGs would be stored in partial pyramids, that partial pyramid will contain zoom levels n_m (from n (base level for the tiff in that folder) to m (min sized overview)). That's how we resolve an issue of too many files (the case when you need to observe your map on the 0 zoom level, obviously in this case fetching thousands of tiles to perform merge won't be efficient). 

In attribute store we have a single json file per layername; that's also normal. Actually most of the data can be calculated on the fly, so there are no needs to store everything for each zoom level.

Best, 

Grigory

 

On Fri, Nov 23, 2018 at 12:36 AM Langlois, Josée-Anne (NRCan/RNCan) <josee-anne.langlois@xxxxxxxxx> wrote:

Thanks! Look like it will work. I might have other questions concerning the COG. For example, in the catalog, the zoom level is not in the same format than in native geotrellis format (I have 0_0 instead of 0). Also, is it normal to have only one json file in the _attribute folder for this layer even if I have created 14 levels of the pyramids?

Thanks again for your help!

Josée-Anne

 

De : geotrellis-user-bounces@xxxxxxxxxxxxxxxx <geotrellis-user-bounces@xxxxxxxxxxxxxxxx> De la part de Grigory Pomadchin
Envoyé : 22 novembre 2018 12:07
À : geotrellis-user@xxxxxxxxxxxxxxxx
Objet : Re: [geotrellis-user] S3COGLayerWriter - update

 

Hey, there!

It is quite near to the thing you are inserting into the arguments - you need to put there an Option of a function, try this:

writer.update(layernane, rdd, 0, Some((v1: GeoTiff[Tile], v2: GeoTiff[Tile]) => {

    val extent = v1.raster.extent

    val tile = v1.raster.tile.combineDouble(v2.raster.tile){ (z1, z2) => {

      if(isData(z1) && isData(z2)) (z1 + z2) / 2

      else if (isData(z1)) z1

      else if (isData(z2)) z2

      else{

        Double.NaN // TODO: investigate why returning NODATA is not producing the expected output

      }

    }

    }

    GeoTiff(tile,extent,WebMercator)

  }))

Best,

Grigory

 

On Thu, Nov 22, 2018 at 7:52 PM Langlois, Josée-Anne (NRCan/RNCan) <josee-anne.langlois@xxxxxxxxx> wrote:

Hello all!

 

I’m trying to use the new S3COGLayerWriter to write DEM in an S3 catalog. I managed to use the write to properly add dem data in my COG S3 catalog. However, I’m not able to use the update on the writer. Can someone share an example of a scala script used to update data in a S3 COG catalog?

 

Here is the script I’m trying to implement, but I keep having an error :

 

found   : (geotrellis.raster.io.geotiff.SinglebandGeoTiff, geotrellis.raster.io.geotiff.SinglebandGeoTiff) => geotrellis.raster.io.geotiff.SinglebandGeoTiff

[error]  required: Option[(geotrellis.raster.io.geotiff.GeoTiff[?], geotrellis.raster.io.geotiff.GeoTiff[?]) => geotrellis.raster.io.geotiff.GeoTiff[?]]

 

I tried to use GeoTiff type for v1 and v2, but there is no combineDouble method for this type.

 

val store: S3AttributeStore = S3AttributeStore(bucket, catalog)

val writer = S3COGLayerWriter(store)

 

writer.update(layername, rdd, z, (v1:SinglebandGeoTiff, v2:SinglebandGeoTiff) => {
  val extent = v1.raster.extent
 
  val tile = v1.raster.tile.combineDouble(v2.raster.tile){ (z1, z2) => {
              if(isData(z1) && isData(z2)) (z1 + z2) / 2
              else if (isData(z1)) z1
              else if (isData(z2)) z2
              else{
                Double.NaN // TODO: investigate why returning NODATA is not producing the expected output
              }
            }
          }
  GeoTiff(tile,extent,WebMercator)
})

 

Thanks!

 

 

Josée-Anne Langlois

 

_______________________________________________
geotrellis-user mailing list
geotrellis-user@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geotrellis-user


 

--

С уважением,
Помадчин Григорий.
Skype: daunnc
ICQ: 133662

_______________________________________________
geotrellis-user mailing list
geotrellis-user@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geotrellis-user


 

--

С уважением,
Помадчин Григорий.
Skype: daunnc
ICQ: 133662

_______________________________________________
geotrellis-user mailing list
geotrellis-user@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geotrellis-user


 

--

С уважением,
Помадчин Григорий.
Skype: daunnc
ICQ: 133662

_______________________________________________
geotrellis-user mailing list
geotrellis-user@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geotrellis-user


 

--

С уважением,
Помадчин Григорий.
Skype: daunnc
ICQ: 133662

_______________________________________________
geotrellis-user mailing list
geotrellis-user@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geotrellis-user


 

--

С уважением,
Помадчин Григорий.
Skype: daunnc
ICQ: 133662

Back to the top