Re: [geotrellis-user] Questions about GeoTrellis tiles

[Antonio Rodriges <antonio.rrz@xxxxxxxxx>]

Thanks Ross!

ср, 20 февр. 2019 г. в 22:24, Ross Bernet <rbernet@xxxxxxxxxx>:
>
> Antonio,
>
> Apologies for the slow response and thank you for the encouraging words.
>
> Our gitter, as you've seen, is a much more active venue for GeoTrellis discussion.
>
> It looks like you've had your questions answered there, but for posterity, I'll link to the responses:
>
>
> 1.
>
> >>>
> As per:  https://gitter.im/geotrellis/geotrellis?at=5c657441126af75deba5e2ed
>
> TileLayerMetadata
> https://github.com/locationtech/geotrellis/blob/master/spark/src/main/scala/geotrellis/spark/TileLayerMetadata.scala
> Talking more generic, probably it’s smth you want to know as well; we store metadata in a single json file in a set of json files.
> The entire metadata stored in attribute store consists of Header (contains information about where and what type of layer is stored), Metadata (tile layer metadata), KeyIndex (KeyIndex that is used to index layer), Schema (Avro schema used to prevent avro tiles corruption)
>
>
> 2
>
> >>>
> As per: https://gitter.im/geotrellis/geotrellis?at=5c657218126af75deba5d24b
>
> hey @antonio-rodriges we’re using TIFF to read data
> GeoTiff allows to read images smartly and to load the consistently on a Spark cluster
> internaly we operate with Arrays of data (:
> the output format is Avro (in case of GeoTrellis avro layers) or geotiffs (COGS, some custom exports / etc)
>
>
> 3.
>
> >>>
> As per: https://gitter.im/geotrellis/geotrellis?at=5c657218126af75deba5d24b
> it’s not possible to perform operations on a compressed data in RDDs; otherwise for each operation you’ll have to uncompress chunks to apply function to compress back; which is very expensive.
>
> on the avro side it uses avro deflate compression
>
> if you want to persist everything as COGs / TIFFs you can choose a deflate compression level using java deflate compressor
>
>
> 4.
>
> >>>
> As per (question changed slightly): https://gitter.im/geotrellis/geotrellis?at=5c6aaed7adf6cb0b2cdbd35d
>
> Antonio Rodriges @antonio-rodriges Feb 18 08:10
> Hi, when GeoTrellis reads a layer from HDFS, does it read MapFile by MapFile sequenstially or GeoTrellis reads all MapFiles in parallel?
>
> Grigory @pomadchin Feb 18 08:31
> @antonio-rodriges it uses SequenceFileInputFormat or FilterMapFileInputFormat depending on the request
> @antonio-rodriges https://github.com/locationtech/geotrellis/blob/master/spark/src/main/scala/geotrellis/spark/io/hadoop/HadoopRDDReader.scala
>
> Were you able to get all of your questions answered? Don't hesitate to follow up if you have additional questions. Although, As you've seen, you're likely to get a faster response on Gitter.
>
> Best,
> Ross
>
>
> On Wed, Feb 13, 2019 at 2:29 PM Antonio Rodriges <antonio.rrz@xxxxxxxxx> wrote:
>>
>> Hi,
>>
>> First of all, thank you for developing GeoTrellis.
>>
>> Please, consider several questions:
>>
>> 1.
>> I found "layer.metadata" as a container for layer information:
>> https://github.com/locationtech/geotrellis/blob/master/docs/guide/examples.rst
>> Is there any other information/metadata about the layer in addition to
>> "layer.metadata"?
>> Is there a function that will format and print "layer.metadata"?
>>
>> 2.
>> There is a tile hierarchy diagram at
>> https://geotrellis.readthedocs.io/en/latest/architecture/high-performance-scala.html
>> The diagram has GeoTiffTile for many cell types (int16, byte, etc.).
>> Do you use GeoTiffTiles during data processing or only during data ingestion?
>>
>> 3.
>> Do you have options to compress GeoTrellis tiles that you use in RDD?
>>
>> 4.
>> You use Hadoop MapFiles to persist GeoTrellis collections in Avro format
>> https://geotrellis.readthedocs.io/en/latest/architecture/002-hdfs-raster-layers.html
>> Avro supports deflate, Snappy compression codecs
>> https://avro.apache.org/docs/1.8.1/spec.html
>> Do you compress tiles when you write them to MapFiles?
>>
>> Thanks,
>> Antonio
>> _______________________________________________
>> geotrellis-user mailing list
>> geotrellis-user@xxxxxxxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>> https://dev.locationtech.org/mailman/listinfo/geotrellis-user
>
>
>
> --
> Ross Bernet, Project Manager, GeoTrellis
> Azavea | 990 Spring Garden Street, 5th Floor, Philadelphia, PA
> rbernet@xxxxxxxxxx | office: 215.701.7717  | mobile: 858.692.1872
>
> Azavea is a B Corporation - we apply geospatial technology for civic and social impact
> while advancing the state-of-the-art through research. Join us.
> _______________________________________________
> geotrellis-user mailing list
> geotrellis-user@xxxxxxxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.locationtech.org/mailman/listinfo/geotrellis-user