Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geomesa-users] Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable
Paul,

If you are using the current master branch there are also some examples in the geomesa-tools/examples directory

For example:
https://github.com/locationtech/geomesa/blob/master/geomesa-tools/examples/ingest/csv/ingest_csv.sh

Andrew

On 02/02/2016 04:26 PM, Jim Hughes wrote:
Hi Paul,

The GeoMesa command tool example/test script doesn't have any real data to ingest.  It merely creates a SimpleFeatureType (which you could register in GeoServer*) so that list/describe/delete/etc can be exercised. 

To see 'real' demo data, check our Quick Start here: http://www.geomesa.org/geomesa-quickstart/.  It has the advantage of providing a Java class which highlights how to use the GeoTools DataStore API.  After some data is written and read from Accumulo, the tutorial describes how to register a layer in GeoServer.  Once that happens, it shows the GeoServer OpenLayers preview page and the red dots you'll be rewarded with!

Cheers,

Jim

*  If you register an empty layer in GeoServer, most reasonable requests will work out just fine.  It'll be a little boring without data...

On 02/02/2016 04:19 PM, Paul Levy wrote:
OK, I'll give that a try in a bit.  Same with delete, it is now deletecatalog but I saw that one.

Assuming that runs correctly, it will place some data in a table?  Is that table accesbile in Geoserver to view the data?  I didn't get too far into reading about it yet, but I saw Add New Data Store, but I wasn't sure you can access an exsiting one using the same mechanism.

Thanks again for all your help


On Tue, Feb 2, 2016 at 2:27 PM, Jim Hughes <jnh5y@xxxxxxxx> wrote:
Hi Paul,

Ah!  Looks like the CREATE_CATALOG is different from CATALOG (which is the one being queried by list and describe). 

That should be the issue; shout if that isn't it.

Cheers,

Jim


On 02/02/2016 11:51 AM, Paul Levy wrote:
Jim, 

Thanks, that was the problem.  Now I just wanted to run through test-geomesa to make sure everything is working and the tables with features are being created but maybe the some of the commands or options have changed with 1.2 or my attributes aren't quite right.  I wasn't sure if CREATE_FEATURENAME and FEATURENAME need to be different.

I see the tables are created, but I assumed list and describe would have shown some feature attributes if the features were created.


I removed the export features right now so the test-geomesa is just:
USER=root
PASS=password
CREATE_CATALOG=test_catalog2
CREATE_FEATURENAME=test_feature2
SPEC=fid:String:index=true,dtg:Date,geom:Point:srid=4326
CATALOG=geomesa_catalog
FEATURENAME=test_feature2
MAXFEATURES=100

# Helper opts for accumulo. These can optionally be supplied from the 
# ACCUMULO_HOME configuration by excluding the arguments from the geomesa command
INST=geotrellis-accumulo-cluster
ZOO=localhost,localhost,localhost
ACC_OPTS="-u $USER -p $PASS -i $INST -z $ZOO"

geomesa create $ACC_OPTS -c ${CREATE_CATALOG} -f ${CREATE_FEATURENAME} -s ${SPEC} --dtg dtg
geomesa list $ACC_OPTS -c ${CATALOG}
geomesa describe $ACC_OPTS -c ${CATALOG} -f ${FEATURENAME}
geomesa explain $ACC_OPTS -c ${CATALOG} -f ${FEATURENAME} -q include

# clean up previous temp feature
geomesa deletecatalog $ACC_OPTS --force -c ${CREATE_CATALOG} -f ${CREATE_FEATURENAME}
geomesa list $ACC_OPTS -c ${CREATE_CATALOG}








Using GEOMESA_HOME = /home/vagrant/geomesa/geomesa-dist/target/geomesa-1.2.0-SNAPSHOT/dist/tools/geomesa-tools-1.2.0-SNAPSHOT
Error in registry file at line number #31
A descriptor is already registered against the name "org.geotools.ColorReduction" under registry mode "rendered"
Error in registry file at line number #32
A descriptor is already registered against the name "org.geotools.ColorInversion" under registry mode "rendered"
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Creating 'test_feature2' on catalog table 'test_catalog2' with spec 'fid:String:index=join,dtg:Date,*geom:Point:srid=4326:index=full:index-value=true'. Just a few moments...
Creating GeoMesa tables...
Feature 'test_feature2' on catalog table 'test_catalog2' with spec 'fid:String:index=join,dtg:Date,*geom:Point:srid=4326:index=full:index-value=true' successfully created.
Created feature test_feature2



Using GEOMESA_HOME = /home/vagrant/geomesa/geomesa-dist/target/geomesa-1.2.0-SNAPSHOT/dist/tools/geomesa-tools-1.2.0-SNAPSHOT
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Running List Features on catalog geomesa_catalog
Error in registry file at line number #31
A descriptor is already registered against the name "org.geotools.ColorReduction" under registry mode "rendered"
Error in registry file at line number #32
A descriptor is already registered against the name "org.geotools.ColorInversion" under registry mode "rendered"



Using GEOMESA_HOME = /home/vagrant/geomesa/geomesa-dist/target/geomesa-1.2.0-SNAPSHOT/dist/tools/geomesa-tools-1.2.0-SNAPSHOT
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Describing attributes of feature 'test_feature2' from catalog table 'geomesa_catalog'...
Error in registry file at line number #31
A descriptor is already registered against the name "org.geotools.ColorReduction" under registry mode "rendered"
Error in registry file at line number #32
A descriptor is already registered against the name "org.geotools.ColorInversion" under registry mode "rendered"
Error: feature 'test_feature2' not found. Check arguments...
java.lang.NullPointerException
at org.locationtech.geomesa.tools.commands.DescribeCommand.execute(DescribeCommand.scala:29)
at org.locationtech.geomesa.tools.Runner$.main(Runner.scala:23)
at org.locationtech.geomesa.tools.Runner.main(Runner.scala)





Using GEOMESA_HOME = /home/vagrant/geomesa/geomesa-dist/target/geomesa-1.2.0-SNAPSHOT/dist/tools/geomesa-tools-1.2.0-SNAPSHOT
Error in registry file at line number #31
A descriptor is already registered against the name "org.geotools.ColorReduction" under registry mode "rendered"
Error in registry file at line number #32
A descriptor is already registered against the name "org.geotools.ColorInversion" under registry mode "rendered"
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Error: Could not explain the query (include): Feature 'test_feature2' has not been initialized. Please call 'createSchema' first.
java.io.IOException: Feature 'test_feature2' has not been initialized. Please call 'createSchema' first.
at org.locationtech.geomesa.accumulo.data.AccumuloDataStore$$anonfun$validateMetadata$1.apply(AccumuloDataStore.scala:380)
at org.locationtech.geomesa.accumulo.data.AccumuloDataStore$$anonfun$validateMetadata$1.apply(AccumuloDataStore.scala:380)
at scala.Option.getOrElse(Option.scala:121)
at org.locationtech.geomesa.accumulo.data.AccumuloDataStore.validateMetadata(AccumuloDataStore.scala:380)
at org.locationtech.geomesa.accumulo.data.AccumuloDataStore.getQueryPlanner(AccumuloDataStore.scala:723)
at org.locationtech.geomesa.accumulo.data.AccumuloDataStore.planQuery(AccumuloDataStore.scala:717)
at org.locationtech.geomesa.accumulo.data.AccumuloDataStore.explainQuery(AccumuloDataStore.scala:707)
at org.locationtech.geomesa.tools.commands.ExplainCommand.execute(ExplainCommand.scala:24)
at org.locationtech.geomesa.tools.Runner$.main(Runner.scala:23)
at org.locationtech.geomesa.tools.Runner.main(Runner.scala)



Using GEOMESA_HOME = /home/vagrant/geomesa/geomesa-dist/target/geomesa-1.2.0-SNAPSHOT/dist/tools/geomesa-tools-1.2.0-SNAPSHOT
Error parsing arguments: Unknown option: -f
Usage: geomesa [command] [command options]
  Commands:
    create              Create a feature definition in a GeoMesa catalog
    deletecatalog       Delete a GeoMesa catalog completely (and all features in it)
    deleteraster        Delete a GeoMesa Raster Table
    describe            Describe the attributes of a given feature in GeoMesa
    env                 Examine the current GeoMesa environment
    explain             Explain how a GeoMesa query will be executed
    export              Export a GeoMesa feature
    getsft              Get the SimpleFeatureType of a feature
    help                Show help
    ingest              Ingest/convert various file formats into GeoMesa
    ingestraster        Ingest a raster file or raster files in a directory into GeoMesa
    list                List GeoMesa features for a given catalog
    queryrasterstats    Export queries and statistics about the last X number of queries to a CSV file.
    removeschema        Remove a schema and associated features from a GeoMesa catalog
    tableconf           Perform table configuration operations
    version             GeoMesa Version

Using GEOMESA_HOME = /home/vagrant/geomesa/geomesa-dist/target/geomesa-1.2.0-SNAPSHOT/dist/tools/geomesa-tools-1.2.0-SNAPSHOT
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Running List Features on catalog test_catalog2
Error in registry file at line number #31
A descriptor is already registered against the name "org.geotools.ColorReduction" under registry mode "rendered"
Error in registry file at line number #32
A descriptor is already registered against the name "org.geotools.ColorInversion" under registry mode "rendered"
test_feature2






--
Paul Levy : Pelican Mapping : 703-994-4125

On Mon, Feb 1, 2016 at 9:44 PM, Jim Hughes <jnh5y@xxxxxxxx> wrote:
Hi Paul,

Ah, I think I've seen this before.  $HADOOP_HOME/lib is being added to the Java classpath, but $HADOOP_HOME is not.  Since hadoop-common.jar is there, it isn't being put on the classpath. 

This might not be the perfect/permanent fix, but on line 174 of bin/geomesa, I'd change it from

hadoopDirs=("$HADOOP_LIB" "$HADOOP_COMMON" "$HADOOP_MAPREDUCE" "$HADOOP_YARN" "$HADOOP_HDFS")

to
hadoopDirs=("$HADOOP_LIB" "$HADOOP_COMMON" "$HADOOP_MAPREDUCE" "$HADOOP_YARN" "$HADOOP_HDFS" "$HADOOP_HOME")

That'd pick up the jars hanging out in the $HADOOP_HOME.  Other alternatives would include linking to them in $HADOOP_HOME/lib, etc.

Cheers,

Jim


On 2/1/2016 5:30 PM, Paul Levy wrote:
And geomesa classpath is picking up the hadoop jars:

/usr/lib/hadoop/lib/jetty-util-6.1.26.cloudera.4.jar
/usr/lib/hadoop/lib/api-util-1.0.0-M20.jar
/usr/lib/hadoop/lib/commons-lang-2.6.jar
/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar
/usr/lib/hadoop/lib/curator-framework-2.7.1.jar
/usr/lib/hadoop/lib/commons-math3-3.1.1.jar
/usr/lib/hadoop/lib/gson-2.2.4.jar
/usr/lib/hadoop/lib/commons-collections-3.2.2.jar
/usr/lib/hadoop/lib/jersey-server-1.9.jar
/usr/lib/hadoop/lib/commons-cli-1.2.jar
/usr/lib/hadoop/lib/jersey-json-1.9.jar
/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar
/usr/lib/hadoop/lib/commons-logging-1.1.3.jar
/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar
/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar
/usr/lib/hadoop/lib/commons-compress-1.4.1.jar
/usr/lib/hadoop/lib/jsch-0.1.42.jar
/usr/lib/hadoop/lib/commons-io-2.4.jar
/usr/lib/hadoop/lib/jsr305-3.0.0.jar
/usr/lib/hadoop/lib/mockito-all-1.8.5.jar
/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar
/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar
/usr/lib/hadoop/lib/zookeeper.jar
/usr/lib/hadoop/lib/jetty-6.1.26.cloudera.4.jar
/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar
/usr/lib/hadoop/lib/jersey-core-1.9.jar
/usr/lib/hadoop/lib/htrace-core-3.0.4.jar
/usr/lib/hadoop/lib/hamcrest-core-1.3.jar
/usr/lib/hadoop/lib/netty-3.6.2.Final.jar
/usr/lib/hadoop/lib/log4j-1.2.17.jar
/usr/lib/hadoop/lib/paranamer-2.3.jar
/usr/lib/hadoop/lib/java-xmlbuilder-0.4.jar
/usr/lib/hadoop/lib/commons-configuration-1.6.jar
/usr/lib/hadoop/lib/asm-3.2.jar
/usr/lib/hadoop/lib/commons-codec-1.4.jar
/usr/lib/hadoop/lib/api-asn1-api-1.0.0-M20.jar
/usr/lib/hadoop/lib/jets3t-0.9.0.jar
/usr/lib/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar
/usr/lib/hadoop/lib/avro.jar
/usr/lib/hadoop/lib/xmlenc-0.52.jar
/usr/lib/hadoop/lib/commons-digester-1.8.jar
/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar
/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar
/usr/lib/hadoop/lib/httpclient-4.2.5.jar
/usr/lib/hadoop/lib/commons-httpclient-3.1.jar
/usr/lib/hadoop/lib/guava-11.0.2.jar
/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar
/usr/lib/hadoop/lib/activation-1.1.jar
/usr/lib/hadoop/lib/stax-api-1.0-2.jar
/usr/lib/hadoop/lib/jettison-1.1.jar
/usr/lib/hadoop/lib/httpcore-4.2.5.jar
/usr/lib/hadoop/lib/xz-1.0.jar
/usr/lib/hadoop/lib/aws-java-sdk-1.7.14.jar
/usr/lib/hadoop/lib/jsp-api-2.1.jar
/usr/lib/hadoop/lib/junit-4.11.jar
/usr/lib/hadoop/lib/curator-client-2.7.1.jar
/usr/lib/hadoop/lib/servlet-api-2.5.jar
/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar
/usr/lib/hadoop/lib/logredactor-1.0.3.jar
/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar
/usr/lib/hadoop/lib/curator-recipes-2.7.1.jar
/usr/lib/hadoop/lib/commons-net-3.1.jar
/usr/lib/hadoop/lib/commons-el-1.0.jar
/usr/lib/hadoop/lib/apacheds-i18n-2.0.0-M15.jar

--
Paul Levy : Pelican Mapping : 703-994-4125

On Mon, Feb 1, 2016 at 5:29 PM, Paul Levy <plevy@xxxxxxxxxxxxxxxxxx> wrote:
Yes, this is Cloudera.

For 1.2, it specifies hadoop 2,2 or better, does it need to be 2.2?  This happens to be 2.6.

HADOOP_HOME is set to /usr/lib/hadoop which does have a common:

bin                                     parquet-avro.jar
client                                  parquet-cascading.jar
client-0.20                             parquet-column.jar
cloudera                                parquet-common.jar
etc                                     parquet-encoding.jar
hadoop-annotations-2.6.0-cdh5.4.9.jar   parquet-format.jar
hadoop-annotations.jar                  parquet-format-javadoc.jar
hadoop-auth-2.6.0-cdh5.4.9.jar          parquet-format-sources.jar
hadoop-auth.jar                         parquet-generator.jar
hadoop-aws-2.6.0-cdh5.4.9.jar           parquet-hadoop-bundle.jar
hadoop-aws.jar                          parquet-hadoop.jar
hadoop-common-2.6.0-cdh5.4.9.jar        parquet-jackson.jar
hadoop-common-2.6.0-cdh5.4.9-tests.jar  parquet-pig-bundle.jar
hadoop-common.jar                       parquet-pig.jar
hadoop-common-tests.jar                 parquet-protobuf.jar
hadoop-nfs-2.6.0-cdh5.4.9.jar           parquet-scala_2.10.jar
hadoop-nfs.jar                          parquet-scrooge_2.10.jar
lib                                     parquet-test-hadoop2.jar
libexec                                 parquet-thrift.jar
LICENSE.txt                             parquet-tools.jar
NOTICE.txt                              sbin

Thank you


--
Paul Levy : Pelican Mapping : 703-994-4125
On Mon, Feb 1, 2016 at 5:23 PM, Jim Hughes <jnh5y@xxxxxxxx> wrote:
Paul,

A +1 to Jeff's suggestion.  There's also the 'geomesa classpath' command which will show the jars which are on the classpath.  If you don't see something like /opt/hadoop/share/hadoop/common/hadoop-common-2.2.0.jar, that'd be the problem.

Cheers,

Jim


On 02/01/2016 05:20 PM, Jeff Gold wrote:
Paul-
  I believe that is because your HADOOP_HOME isn't set correctly and
the Hadoop jars cannot be found (I experienced this recently).  Do you
happen to be using a Cloudera distribution of Hadoop?


-Jeff

On Mon, Feb 1, 2016 at 5:07 PM, Paul Levy <paul.levy@xxxxxxxxx> wrote:
Just finished building the 1.2 snapshot and wanted to test it out.  Any
ideas on this error running test-geomesa after editing it for my cluster
config:

Using GEOMESA_HOME =
/home/vagrant/geomesa/geomesa-dist/target/geomesa-1.2.0-SNAPSHOT/dist/tools/geomesa-tools-1.2.0-SNAPSHOT
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/io/Writable
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at
org.locationtech.geomesa.accumulo.data.AccumuloDataStoreFactory.getDisplayName(AccumuloDataStoreFactory.scala:101)
at
org.geotools.data.DataAccessFinder.getDataStore(DataAccessFinder.java:111)
at org.geotools.data.DataStoreFinder.getDataStore(DataStoreFinder.java:89)
at
org.locationtech.geomesa.tools.DataStoreHelper.getDataStore(DataStoreHelper.scala:36)
at
org.locationtech.geomesa.tools.FeatureCreator$.createFeature(FeatureCreator.scala:22)
at
org.locationtech.geomesa.tools.commands.CreateCommand.execute(CreateCommand.scala:19)
at org.locationtech.geomesa.tools.Runner$.main(Runner.scala:23)
at org.locationtech.geomesa.tools.Runner.main(Runner.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.Writable
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 20 more

Thanks for your time

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from
this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users




_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users


_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users



_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users


_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users



_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users



_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users

Back to the top