Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geowave-dev] Geowave and Hbase
The sizes of those jars will change with the actual release.  We are working details of how we want each packaged and deployed now in a branch.  For now, that accumulo jar is bloated by hbase dependencies (won't be the case in the release), and the hbase jar is very minimal.  The HBase jar will work for server-side spatial and spatial-temporal filtering, but not for arbitrary CQL.  We will be more inclusive with the release to enable all things for both datastores.

Yes, the "examples" is a good place to look as a very basic starting point.  Also, you may be able to find a lot of interesting examples in the "test" project.  We use annotations to test all of our underlying datastores (and permutations of versions to support the latest release and the previous minor version release) with the same set of integration tests so that we can be sure we maintain support for all of our advertised capabilities on all of the underlying data store.  So for more advanced examples, you can look through our integration tests.  Here's an example of how the annotation is used to test basic ingest and query: https://github.com/ngageoint/geowave/blob/master/test/src/test/java/mil/nga/giat/geowave/test/basic/GeoWaveBasicSpatialVectorIT.java#L39

The HBase annotation just tells the test runner to make sure to setup a mini-HBase cluster and to connect to it (https://github.com/ngageoint/geowave/blob/master/test/src/main/java/mil/nga/giat/geowave/test/HBaseStoreTestEnvironment.java). The advantage of following these integration test examples for your code is that none of them directly depend on Accumulo or HBase - so as we continue to add data stores such as Amazon's DynamoDB or Apache Cassandra you will be able to seamlessly switch which one you connect to without modifying your code if you choose.  Of course, using general code is never as straightforward as the specific, so the "examples" project shows explicit implementations to ingest and query each data store.


On Tue, Dec 6, 2016 at 1:10 PM, Carl M <soloninguno@xxxxxxxxxxx> wrote:

Hi Rich,


I think I manage to compile the hbase ok (although something strange, the geowave-deploy-0.9.3-SNAPSHOT-accumulo-singlejar.jar has a size of 173MB, and the geowave-deploy-0.9.3-SNAPSHOT-hbase-singlejar.jar only has 1.5MB).


As I'm using geowave from java code (not command line) is any java test available to check? Is SimpleHBaseIngestTest the right place to look at? Or do you think is better to wait for the final release?


Best,



De: geowave-dev-bounces@locationtech.org <geowave-dev-bounces@locationtech.org> en nombre de Rich Fecher <rfecher@xxxxxxxxx>
Enviado: lunes, 05 de diciembre de 2016 04:57 p.m.

Para: geowave-dev@xxxxxxxxxxxxxxxx
Asunto: Re: [geowave-dev] Geowave and Hbase
 
The build args should be particular to the versions on your cluster (those are for having travis-ci run automated tests on a particular version of cloudera).  If you don't override with your own versions, it will use HBase 1.2.3 with Hadoop 2.7.3, and GeoServer/GeoTools 2.10.0/16.0.
At the root level of the project this will build a single jar that you can drop into geoserver:
'mvn package -am -pl deploy -P geotools-container-singlejar'

This will build a jar that gives you all of geowave's commandline tools:
'mvn package -am -pl deploy -P geowave-tools-singlejar'

And if you choose to use HBase custom filters and coprocessors (ie. server-side processing, it will be faster), this is how you build a jar you can put on HBase's classpath.  We'll have an rpm that finds HBase's home directory on HDFS and puts it in a lib directory.
'mvn package -am -pl deploy -P hbase-container-singlejar'

We'll also have a HBase centric quick-start guide so it'll be pretty easy to walk through a basic use case.  We'll be shooting to release by the end of this week.

Rich

On Mon, Dec 5, 2016 at 2:15 PM, Carl M <soloninguno@xxxxxxxxxxx> wrote:

Thank you very much Rich.


Great to know that Hbase support is almost ready. Do you have an estimated time for the 0.9.3 release?


In the meantime maybe i could play with it and build it myself, could you tell me the right BUILD_ARGS i need to use? I think I found one line in travis.yml in repo code, but I'm not sure if they are ok as they have a reference to accumulo


"ACCUMULO_VERSION='1.6.0-cdh5.1.4' ACCUMULO_API='1.6' HBASE_VERSION='1.2.0-cdh5.7.1' HADOOP_VERSION='2.6.0-cdh5.4.0' GEOTOOLS_VERSION='16.0' GEOSERVER_VERSION='2.10.0' PLATFORM_VERSION='cloudera' BUILD_DOCS=false"


As I understand, after building the client I would only need to change the guava version in geoserver, is that right?


Best regards,



De: geowave-dev-bounces@locationtech.org <geowave-dev-bounces@locationtech.org> en nombre de Rich Fecher <rfecher@xxxxxxxxx>
Enviado: lunes, 05 de diciembre de 2016 03:40 p.m.
Para: geowave-dev@xxxxxxxxxxxxxxxx
Asunto: Re: [geowave-dev] Geowave and Hbase
 
Yes, we have been working hard to ensure *everything* is supported in HBase that is supported in Accumulo, to include the GeoServer plugin.  Our upcoming 0.9.3 release is the first release that will have full HBase support with comparable performance so one could reasonably choose between Accumulo and HBase based on your needs.  0.9.3 will officially be released very soon, currently we are in final stages of ensuring complete Google BigTable support in this release as well.  In the meantime, if you were to try to build the latest yourself, the most significant catch to having an HBase client within the same webapp as GeoServer is differing guava versions and in particular usage of conflicting Stopwatch calls between GeoServer and HBase's client (https://issues.apache.org/jira/browse/HBASE-15737).  We have applied this patch to remedy our distribution - so our packaged jars and rpms that we distribute will work seamlessly without worrying about this conflict.

But I'm bringing it to your attention ahead of time in case you decide to build it yourself.  Alternatively Guava 16.0 has proven to work with both HBase and GeoServer from our testing (GeoServer comes with Guava 17.0 however so you'd have to remove that and add guava 16.0 to work around the problem this way). 

On Mon, Dec 5, 2016 at 1:13 PM, Carl M <soloninguno@xxxxxxxxxxx> wrote:

Hi guys,


I see that geowave now supports Hbase as store. Does the geoserver plugin also works with Hbase, or it only works with Accumulo?


Thanks in advance,


_______________________________________________
geowave-dev mailing list
geowave-dev@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geowave-dev



_______________________________________________
geowave-dev mailing list
geowave-dev@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geowave-dev



_______________________________________________
geowave-dev mailing list
geowave-dev@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geowave-dev


Back to the top