Re: [geomesa-users] Tips on ingesting a lot of data into GeoMesa

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]

Re: [geomesa-users] Tips on ingesting a lot of data into GeoMesa

From: Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx>
Date: Tue, 24 Jan 2017 10:35:42 -0500
Delivered-to: geomesa-users@xxxxxxxxxxxxxxxx
List-archive: <https://dev.locationtech.org/mhonarc/lists/geomesa-users>
List-help: <mailto:geomesa-users-request@locationtech.org?subject=help>
List-subscribe: <https://dev.locationtech.org/mailman/listinfo/geomesa-users>, <mailto:geomesa-users-request@locationtech.org?subject=subscribe>
List-unsubscribe: <https://dev.locationtech.org/mailman/options/geomesa-users>, <mailto:geomesa-users-request@locationtech.org?subject=unsubscribe>
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1

Hmm, ok I'm not sure about the guava jar... but I believe that error means that you can't connect to the BigTable instance you're trying to. Do you have the google-cloud-sdk installed? Can you hit your bigtable instance from there?

Thanks,

Emilio

On 01/24/2017 10:29 AM, Damiano Albani wrote:

Compared my own lib/ directory, there's only 2 files which are missing:

hbase-client-1.2.3.jar

hbase-common-1.2.3.jar

In addition to those 2 JARs, I have meticulously copied the necessary JARs over to fix the error messages.

But now I'm back again at the "StopWatch" issue I mentioned previously.

Yet, after some googling, it looks like it's due to the version of Guava.

There originally was a guava-17.0.jar in lib/, whereas hadoop-common-2.5.2 depends on guava-11.0.2.

After replacing the Guava JAR, the error disappeared.

And I run into this now:

ERROR Can't get the locations

org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations

at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:312)

at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:151)

at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)

at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)

at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)

at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)

at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)

at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)

at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)

at org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)

at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)

at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:303)

at org.locationtech.geomesa.hbase.data.HBaseMetadataAdapter$class.checkIfTableExists(HBaseBackedMetadata.scala:33)

at org.locationtech.geomesa.hbase.data.HBaseBackedMetadata.checkIfTableExists(HBaseBackedMetadata.scala:19)

at org.locationtech.geomesa.index.metadata.CachedLazyMetadata$class.$init$(CachedLazyMetadata.scala:36)

at org.locationtech.geomesa.hbase.data.HBaseBackedMetadata.<init>(HBaseBackedMetadata.scala:20)

at org.locationtech.geomesa.hbase.data.HBaseDataStore.<init>(HBaseDataStore.scala:27)

at org.locationtech.geomesa.hbase.data.HBaseDataStoreFactory.createDataStore(HBaseDataStoreFactory.scala:50)

at org.locationtech.geomesa.hbase.data.HBaseDataStoreFactory.createDataStore(HBaseDataStoreFactory.scala:23)

at org.geotools.data.DataAccessFinder.getDataStore(DataAccessFinder.java:130)

at org.geotools.data.DataStoreFinder.getDataStore(DataStoreFinder.java:89)

at org.locationtech.geomesa.tools.ingest.AbstractIngest.<init>(AbstractIngest.scala:71)

at org.locationtech.geomesa.tools.ingest.AutoIngest.<init>(AutoIngest.scala:33)

at org.locationtech.geomesa.tools.ingest.IngestCommand$class.execute(IngestCommand.scala:64)

at org.locationtech.geomesa.bigtable.tools.BigtableIngestCommand.execute(BigtableIngestCommand.scala:20)

at org.locationtech.geomesa.tools.Runner$class.main(Runner.scala:23)

at org.locationtech.geomesa.bigtable.tools.BigtableRunner$.main(BigtableRunner.scala:21)

at org.locationtech.geomesa.bigtable.tools.BigtableRunner.main(BigtableRunner.scala)

That happens after a couple of minutes of geomesa-bigtable sitting idle!?
On Tue, Jan 24, 2017 at 3:50 PM, Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:
You should be able to ignore the HADOOP_HOME warning - if you have that defined, it will load hadoop jars from there. But instead you can just copy them into the lib directory. You should keep the guava-17 jar - the HBase distro shades and relocates guava into a geomesa jar, which was causing your problem. We don't have to do that with BigTable though.

The tools might be hanging trying to resolve your BigTable instance - do you have the appropriate hbase-site.xml in the conf folder? Have you also set up the Google default authentication so that you can communicate with your instance? (I thought we had docs on that, but now I don't see them...) See here for instructions: https://cloud.google.com/docs/authentication

Here's the output of my tools lib folder (slightly older GeoMesa snapshot version but everything else should match):

accessors-smart-1.1.jar
activation-1.1.jar
aopalliance-1.0.jar
apacheds-i18n-2.0.0-M15.jar
apacheds-kerberos-codec-2.0.0-M15.jar
api-asn1-api-1.0.0-M20.jar
api-util-1.0.0-M20.jar
asm-3.1.jar
asm-4.0.jar
avro-1.7.5.jar
bigtable-hbase-1.1-0.9.4.jar
bigtable-hbase-1.2-0.9.4.jar
bigtable-hbase-mapreduce-0.9.4.jar
caffeine-2.3.3.jar
common-2.6.0.jar
commons-cli-1.2.jar
commons-codec-1.9.jar
commons-collections-3.2.2.jar
commons-compress-1.4.1.jar
commons-csv-1.0.jar
commons-daemon-1.0.13.jar
commons-el-1.0.jar
commons-httpclient-3.1.jar
commons-io-2.1.jar
commons-lang-2.6.jar
commons-logging-1.2.jar
commons-math-2.2.jar
commons-net-3.3.jar
commons-pool-1.6.jar
commons-pool2-2.2.jar
config-1.2.1.jar
core-0.26.jar
disruptor-3.3.0.jar
ecore-2.6.1.jar
findbugs-annotations-1.3.9-1.jar
GeographicLib-Java-1.44.jar
geomesa-bigtable-datastore_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-bigtable-tools_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-convert-all_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-convert-avro_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-convert-common_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-convert-fixedwidth_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-convert-json_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-convert-scripting_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-convert-text_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-convert-xml_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-feature-all_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-feature-avro_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-feature-common_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-feature-kryo_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-feature-nio_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-filter_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-hbase-datastore_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-hbase-tools_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-index-api_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-jobs_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-logger_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-tools_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-utils_2.11-1.3.0-m3-SNAPSHOT.jar
geomesa-z3_2.11-1.3.0-m3-SNAPSHOT.jar
gson-2.3.1.jar
gt-api-15.1.jar
gt-coverage-15.1.jar
gt-cql-15.1.jar
gt-data-15.1.jar
gt-epsg-hsql-15.1.jar
gt-geojson-15.1.jar
gt-graph-15.1.jar
gt-grid-15.1.jar
gt-main-15.1.jar
gt-metadata-15.1.jar
gt-opengis-15.1.jar
gt-process-15.1.jar
gt-process-feature-15.1.jar
gt-referencing-15.1.jar
gt-shapefile-15.1.jar
gt-transform-15.1.jar
gt-xml-15.1.jar
gt-xsd-core-15.1.jar
gt-xsd-fes-15.1.jar
gt-xsd-filter-15.1.jar
gt-xsd-gml2-15.1.jar
gt-xsd-gml3-15.1.jar
gt-xsd-ows-15.1.jar
gt-xsd-wfs-15.1.jar
guava-17.0.jar
guice-3.0.jar
guice-servlet-3.0.jar
hadoop-annotations-2.4.1.jar
hadoop-auth-2.5.1.jar
hadoop-hdfs-2.5.1.jar
hadoop-mapreduce-client-core-2.4.1.jar
hadoop-yarn-api-2.4.1.jar
hadoop-yarn-common-2.4.1.jar
hbase-annotations-1.1.2.jar
hbase-client-1.2.3.jar
hbase-common-1.1.2-tests.jar
hbase-common-1.2.3.jar
hbase-hadoop2-compat-1.1.2.jar
hbase-hadoop-compat-1.1.2.jar
hbase-prefix-tree-1.1.2.jar
hbase-procedure-1.1.2.jar
hbase-server-1.1.2.jar
hsqldb-2.3.0.jar
htrace-core-3.1.0-incubating.jar
httpclient-4.2.5.jar
httpcore-4.2.4.jar
jackson-core-asl-1.9.3.jar
jackson-jaxrs-1.9.3.jar
jackson-mapper-asl-1.9.3.jar
jackson-xc-1.9.3.jar
jamon-runtime-2.3.1.jar
jasper-compiler-5.5.23.jar
jasper-runtime-5.5.23.jar
javax.inject-1.jar
jaxb-api-2.2.2.jar
jaxb-impl-2.2.3-1.jar
jcl-over-slf4j-1.7.5.jar
jcommander-1.48.jar
jdom-1.1.3.jar
jersey-core-1.9.jar
jersey-guice-1.9.jar
jersey-json-1.9.jar
jersey-server-1.9.jar
jettison-1.1.jar
jetty-6.1.26.jar
jetty-sslengine-6.1.26.jar
jetty-util-6.1.26.jar
joda-convert-1.6.jar
joda-time-2.3.jar
json4s-ast_2.11-3.2.10.jar
json4s-core_2.11-3.2.10.jar
json4s-native_2.11-3.2.10.jar
json-path-2.2.0.jar
json-simple-1.1.jar
json-smart-2.2.1.jar
jsp-2.1-6.1.14.jar
jsp-api-2.1-6.1.14.jar
jsr-275-1.0-beta-2.jar
jt-affine-1.0.9.jar
jt-algebra-1.0.9.jar
jt-bandcombine-1.0.9.jar
jt-bandmerge-1.0.9.jar
jt-bandselect-1.0.9.jar
jt-binarize-1.0.9.jar
jt-border-1.0.9.jar
jt-buffer-1.0.9.jar
jt-classifier-1.0.9.jar
jt-colorconvert-1.0.9.jar
jt-colorindexer-1.0.9.jar
jt-crop-1.0.9.jar
jt-errordiffusion-1.0.9.jar
jt-format-1.0.9.jar
jt-imagefunction-1.0.9.jar
jt-iterators-1.0.9.jar
jt-lookup-1.0.9.jar
jt-mosaic-1.0.9.jar
jt-nullop-1.0.9.jar
jt-orderdither-1.0.9.jar
jt-piecewise-1.0.9.jar
jt-rescale-1.0.9.jar
jt-rlookup-1.0.9.jar
jts-1.13.jar
jt-scale-1.0.9.jar
jt-stats-1.0.9.jar
jt-translate-1.0.9.jar
jt-utilities-1.0.9.jar
jt-vectorbin-1.0.9.jar
jt-warp-1.0.9.jar
jt-zonal-1.0.9.jar
jul-to-slf4j-1.7.5.jar
kryo-2.21.jar
log4j-1.2.17.jar
mango-core-1.2.0.jar
metrics-core-2.2.0.jar
minlog-1.2.jar
net.opengis.fes-15.1.jar
net.opengis.ows-15.1.jar
net.opengis.wfs-15.1.jar
netty-tcnative-boringssl-static-1.1.33.Fork19.jar
objenesis-1.0.jar
org.w3.xlink-15.1.jar
paranamer-2.6.jar
parboiled-core-1.1.7.jar
parboiled-scala_2.11-1.1.7.jar
picocontainer-1.2.jar
protobuf-java-2.5.0.jar
reflectasm-1.07-shaded.jar
scala-compiler-2.11.7.jar
scala-library-2.11.7.jar
scala-logging_2.11-3.1.0.jar
scalap-2.11.7.jar
scala-parser-combinators_2.11-1.0.4.jar
scala-reflect-2.11.7.jar
scala-xml_2.11-1.0.5.jar
servlet-api-2.5-6.1.14.jar
servlet-api-2.5.jar
sfcurve-api_2.11-0.2.0.jar
sfcurve-zorder_2.11-0.2.0.jar
slf4j-api-1.7.5.jar
slf4j-log4j12-1.7.5.jar
spatial4j-0.4.1.jar
stax-api-1.0-2.jar
stream-2.9.2.jar
xml-commons-resolver-1.2.jar
xmlenc-0.52.jar
xsd-2.6.0.jar
xz-1.0.jar
On 01/24/2017 09:36 AM, Damiano Albani wrote:
Hum, I'm not sure I understand what I precisely need to copy/remove from hbase or hadoop distributions into the lib directory 😕
In fact, the geomesa-bigtable Bash script requires a HADOOP_HOME environment variable to be defined!?

But I have followed your advice and removed lib/guava-17.0.jar -- I suppose that clashes with the repackaged code in lib/bigtable-hbase-1.{1,2}-0.9.4.jar.

Alas, it hasn't completely solved my issue: once started, geomesa-bigtable seems to just sit idle.

Well, in fact, it appears to be doing Zookeeper-related work:

Any idea what's going on?
On Tue, Jan 24, 2017 at 2:50 PM, Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:
The tools don't ship with the hadoop jars - you need to copy them into the lib folder. In general we do that to support different client hadoop installs - for BigTable that might not make sense. Looks like we don't mention that in our docs, though... I will make sure we get that updated. I think the list of jars under geoserver should also work for the tools: http://www.geomesa.org/documentation/user/bigtable/install.html#installing-geomesa-bigtable-in-geoserver

HBase uses a different version of guava, which is why you are getting that second exception. You should remove any jars you copied from the HBase distro.

Thanks,

Emilio

On 01/24/2017 06:10 AM, Damiano Albani wrote:
Hi Emilio,

On Mon, Jan 23, 2017 at 9:26 PM, Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:

Because we're an Eclipse project, anything we host has to be blessed by Eclipse for provenance and license. As we haven't gotten this sign-off on all the BigTable dependencies yet, we unfortunately can't bundle it - we can still use it as a plugin, but you have to build it yourself. Hopefully we will be able to get it approved soon.

OK, I understand the reasoning.

So, I've built the BigTable tools and tried to play with the "geomesa-bigtable ingest" utility.

And I'm probably doing something wrong because I get an ClassNotFoundException for org.apache.hadoop.hbase.client.Connection.

Am I supposed to copy the JARs from the HBase distribution, say version 1.1.2?

Because even if I do that, I get a cryptic error when calling "geomesa-bigtable ingest":

java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator

Thanks,

--

Damiano Albani
Geodan
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________ geomesa-users mailing list geomesa-users@xxxxxxxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.locationtech.org/mailman/listinfo/geomesa-users
--
Damiano Albani Geodan
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________ geomesa-users mailing list geomesa-users@xxxxxxxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.locationtech.org/mailman/listinfo/geomesa-users
--
Damiano Albani Geodan
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users

Follow-Ups:
- Re: [geomesa-users] Tips on ingesting a lot of data into GeoMesa
  - From: Damiano Albani

References:
- [geomesa-users] Tips on ingesting a lot of data into GeoMesa
  - From: Damiano Albani
- Re: [geomesa-users] Tips on ingesting a lot of data into GeoMesa
  - From: Emilio Lahr-Vivaz
- Re: [geomesa-users] Tips on ingesting a lot of data into GeoMesa
  - From: Damiano Albani
- Re: [geomesa-users] Tips on ingesting a lot of data into GeoMesa
  - From: Emilio Lahr-Vivaz
- Re: [geomesa-users] Tips on ingesting a lot of data into GeoMesa
  - From: Damiano Albani
- Re: [geomesa-users] Tips on ingesting a lot of data into GeoMesa
  - From: Emilio Lahr-Vivaz
- Re: [geomesa-users] Tips on ingesting a lot of data into GeoMesa
  - From: Damiano Albani
- Re: [geomesa-users] Tips on ingesting a lot of data into GeoMesa
  - From: Emilio Lahr-Vivaz
- Re: [geomesa-users] Tips on ingesting a lot of data into GeoMesa
  - From: Damiano Albani

Prev by Date: Re: [geomesa-users] Tips on ingesting a lot of data into GeoMesa
Next by Date: Re: [geomesa-users] Tips on ingesting a lot of data into GeoMesa
Previous by thread: Re: [geomesa-users] Tips on ingesting a lot of data into GeoMesa
Next by thread: Re: [geomesa-users] Tips on ingesting a lot of data into GeoMesa
Index(es):
- Date
- Thread

Breadcrumbs