De : geotrellis-user-bounces@xxxxxxxxxxxxxxxx <geotrellis-user-bounces@xxxxxxxxxxxxxxxx>
De la part de Grigory Pomadchin
Envoyé : 26 novembre 2018 15:31
À : geotrellis-user@xxxxxxxxxxxxxxxx
Objet : Re: [geotrellis-user] S3COGLayerWriter - update
How do you define your reader? (the full code)
Thanks again Grigory for your quick and comprehensive answer!
I must have a mistake in my script.. I keep having the following errors :
When I request a tile for the zoom level 0 I got that error :
[my-system-akka.actor.default-dispatcher-19] [akka.actor.ActorSystemImpl(my-system)] Error during processing
of request: 'TileLayerMetadata expected'. Completing with 500 Internal Server Error response.
And when I request a tile for all other zoom level (from 1 to 14) I got that error :
[my-system-akka.actor.default-dispatcher-11] [akka.actor.ActorSystemImpl(my-system)] Error during processing
of request: 'Attribute metadata not found for layer Layer(name = "arcticdemCOG", zoom = 9)'. Completing with 500 Internal Server Error response.
Attached is the metadata file of my S3 COG layer. Do you see something wrong in it?
![]()
Regards.
Josée-Anne
De :
geotrellis-user-bounces@xxxxxxxxxxxxxxxx <geotrellis-user-bounces@xxxxxxxxxxxxxxxx>
De la part de Grigory Pomadchin
Envoyé : 26 novembre 2018 12:54
À : geotrellis-user@xxxxxxxxxxxxxxxx
Objet : Re: [geotrellis-user] S3COGLayerWriter - update
Just use a common zoom level (integer) as usual, reader would manage itself where to take data from. this n_m structure is just a description of how the catalog looks like. Lets
go through a small example.
For instance there is a catalog of the following zoom levels: 0_8, 9_11, 12_13
0_8 means that this folder contains tiles with the base TIFF IFD at zoom level 8 and the smalles ovr is at zoomlvl 0
9_11 means that the base TIFF IFD is at zoom level 11
And you want to read key from zoom level 7. You need just to use a common reader API:
S3COGValueReader(...).reader[K,
V](LayerId(name,
7)).read(key) //> would return you a tile at zoom level 7
Reader itself would figure out that it has to go into 0_8 folder and that it needs to pick up the first overview of the tiff that corresponds to the key you passed into the function
argument.
There is also a small (but probably a bad) example of how we can use COGs API in chatta demo: https://github.com/pomadchin/geotrellis-chatta-demo/commit/f7e1d88168f611d6192c1335371a5514c9882ee1
The idea is that COGs API looks very much like a common Avro API. The only significant difference is in the ingest process.
Let me know if you want to know smth else,
Best,
Grigory
Thanks Grigory! Another question ;) : When I want to read the COG layer using the S3COGValueReader in order to serve the data as a Tile service (xyz) (our script is inspired by the Geotrellis Landsat Tutorial ), what zoom value should we use? n_m or n ?!
Thanks!
Josée-Anne
De :
geotrellis-user-bounces@xxxxxxxxxxxxxxxx
<geotrellis-user-bounces@xxxxxxxxxxxxxxxx>
De la part de Grigory Pomadchin
Envoyé : 23 novembre 2018 04:20
À : geotrellis-user@xxxxxxxxxxxxxxxx
Objet : Re: [geotrellis-user] S3COGLayerWriter - update
Hey!
Yeah, it's normal. COGs would be stored in partial pyramids, that partial pyramid will contain zoom levels n_m (from n (base level for the tiff in that folder) to m (min sized overview)). That's how we resolve an issue of too many files (the case when you need
to observe your map on the 0 zoom level, obviously in this case fetching thousands of tiles to perform merge won't be efficient).
In attribute store we have a single json file per layername; that's also normal. Actually most of the data can be calculated on the fly, so there are no needs to store everything for each zoom level.
Best,
Grigory
Thanks! Look like it will work. I might have other questions concerning the COG. For example, in the
catalog, the zoom level is not in the same format than in native geotrellis format (I have 0_0 instead of 0). Also, is it normal to have only one json file in the _attribute folder for this layer even if I have created 14 levels of the pyramids?
Thanks again for your help!
Josée-Anne
De :
geotrellis-user-bounces@xxxxxxxxxxxxxxxx
<geotrellis-user-bounces@xxxxxxxxxxxxxxxx>
De la part de Grigory Pomadchin
Envoyé : 22 novembre 2018 12:07
À : geotrellis-user@xxxxxxxxxxxxxxxx
Objet : Re: [geotrellis-user] S3COGLayerWriter - update
Hey, there!
It is quite near to the thing you are inserting into the arguments - you need to put there an Option of a function, try this:
writer.update(layernane, rdd, 0, Some((v1: GeoTiff[Tile], v2: GeoTiff[Tile]) => {
val extent = v1.raster.extent
val tile = v1.raster.tile.combineDouble(v2.raster.tile){ (z1, z2) => {
if(isData(z1) && isData(z2)) (z1 + z2) / 2
Double.NaN // TODO: investigate why returning NODATA is not producing the expected output
GeoTiff(tile,extent,WebMercator)
Hello all!
I’m trying to use the new S3COGLayerWriter to write DEM in an S3 catalog. I managed to use the write to properly add dem data in my COG S3 catalog. However, I’m not able to use
the update on the writer. Can someone share an example of a scala script used to update data in a S3 COG catalog?
Here is the script I’m trying to implement, but I keep having an error :
found : (geotrellis.raster.io.geotiff.SinglebandGeoTiff, geotrellis.raster.io.geotiff.SinglebandGeoTiff) => geotrellis.raster.io.geotiff.SinglebandGeoTiff
[error] required: Option[(geotrellis.raster.io.geotiff.GeoTiff[?], geotrellis.raster.io.geotiff.GeoTiff[?]) => geotrellis.raster.io.geotiff.GeoTiff[?]]
I tried to use GeoTiff type for v1 and v2, but there is no combineDouble method for this type.
val store: S3AttributeStore = S3AttributeStore(bucket, catalog)
val writer = S3COGLayerWriter(store)
writer.update(layername, rdd, z, (v1:SinglebandGeoTiff, v2:SinglebandGeoTiff) => {
val
extent = v1.raster.extent
val
tile = v1.raster.tile.combineDouble(v2.raster.tile){ (z1, z2) => {
if(isData(z1) && isData(z2)) (z1 + z2) /
2
else if
(isData(z1)) z1
else if
(isData(z2)) z2
else{
Double.NaN // TODO: investigate why returning NODATA is not producing the expected output
}
}
}
GeoTiff(tile,extent,WebMercator)
})
Thanks!
Josée-Anne Langlois
_______________________________________________
geotrellis-user mailing list
geotrellis-user@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geotrellis-user
--
С уважением,
Помадчин Григорий.
Skype: daunnc
ICQ: 133662
_______________________________________________
geotrellis-user mailing list
geotrellis-user@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geotrellis-user
--
С уважением,
Помадчин Григорий.
Skype: daunnc
ICQ: 133662
_______________________________________________
geotrellis-user mailing list
geotrellis-user@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geotrellis-user
--
С уважением,
Помадчин Григорий.
Skype: daunnc
ICQ: 133662
_______________________________________________
geotrellis-user mailing list
geotrellis-user@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geotrellis-user
--
С уважением,
Помадчин Григорий.
Skype: daunnc
ICQ: 133662
|
