Yeah I will try it soon. Would you suggest to try it with
"spark-submit" or do you know another possibility to run it on YARN?
Marcel.
Am 27.07.2015 20:11, schrieb Jim
Hughes:
Marcel,
That makes sense. Does your cluster have YARN available? I
wonder if you can try out Spark 1.3.1 running on YARN.
Alternatively, you could try installing Spark 1.3.1 on your own.
Thanks for keeping with it.
Cheers,
Jim
On 07/27/2015 01:56 PM, Marcel wrote:
You´re right the problem is fixed in version 1.3.1. But when
executing, a problem with different serialVersionUID appears due
to different spark versions (1.3.1 as maven dependency and 1.3.0
using the cloudera manager). In local mode everthing is fine :/
Marcel Jacob.
Am 27.07.2015 16:36, schrieb Jim
Hughes:
Hi Marcel,
Spark is a 'provided' dependency for GeoMesa, so you should be
able to use slightly different versions of Spark and still see
things work.
To that end, I'd suggest trying out Spark 1.3.1. It should
have the fix for the issues you linked to, and it will not
require you to rebuild Spark and work through issues like
that.
Thanks for the great questions. Feel free to post more
complete stack traces. They can help us figure out issues
more quickly and also other users can find threads like this
as well.
Cheers,
Jim
On 07/27/2015 07:16 AM, Marcel
wrote:
Hello,
I´ve found the problem. The featurename 'gdelt' was wrong.
But unfortunately I´ve got another problem using Spark with
Geomesa. I´m not quite sure where the error comes from, but
I assume its problem with Spark.
A ClassNotFoundException is thrown with following content:
"Failed to register classes with Kryo".
Please have a look at https://github.com/apache/spark/pull/4258
A solution is described there, but I´m not sure how to use
this "patch".
I´m using Spark version 1.3.0 and it´s not possible for me
to update my version, because I use GeoMesa.
Thanks in advance,
Marcel Jacob.
Am 22.07.2015 18:38, schrieb
Jim Hughes:
Hi Marcel,
From a quick look, I'm guessing that your DataStore is
null. I'd suggest adding a quick check to see if 'ds' is
null. You don't need to specify the 'featureName' to get
a datastore. I don't know if that would hurt anything,
but I'd suggest removing it.
Other than that, you can double-check the settings you are
passing by using the GeoMesa tools (http://www.geomesa.org/geomesa-tools-features/)
like 'list' and 'describe'. Other than that, you can use
the Accumulo shell to scan the 'gdelt' table to make sure
that sensible metadata is present in that table.
Let us know how getting a DataStore in this code works out
for you. I'll add the idea of a Java GeoMesaSpark
tutorial/example project to our list of additions to make.
Cheers,
Jim
On 07/22/2015 09:13 AM,
Marcel wrote:
Hey,
I´m trying to retrieve a RDD using the GeomesaSpark
class. Unfortunately a NullPointerException is thrown
during execution of this method:
GeoMesaSpark.rdd(conf,
sparkContext, ds, query1);
It says SimpleFeatureType.encodeType throws this
exception. Is something wrong with my datatore or my
arguments? Here is my code:
Map<String,
String> map = new
HashMap<String,
String>();
map.put("instanceId", "accumulo");
map.put("zookeepers",
"node1-scads02:2181");
map.put("user", "user");
map.put("password", "password");
map.put("tableName", "gdelt");
map.put("featureName", "event");
AccumuloDataStore ds = (AccumuloDataStore)
DataStoreFinder.getDataStore(map);
SparkConf sc = new SparkConf(true);
sc.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
sc.set("spark.kryo.serializer.buffer.mb", "24");
Configuration conf = new Configuration();
SparkConf sc2 = GeoMesaSpark.init(sc, ds);
SparkContext sparkContext = new SparkContext("spark://node1-scads02:7077", "countryWithMostEvent", sc2);
Filter f = Filter.INCLUDE;
Query query1 = new Query("gdelt", f, new
String[]{"Actor1CountryCode", "Actor2CountryCode"});
RDD<SimpleFeature> actorResultRDD =
GeoMesaSpark.rdd(conf, sparkContext, ds, query1);
Thanks again.
ps: It would be great when anybody could post a
working GeomesaSpark example in Java including a RDD
transformation.
Best regards
Marcel Jacob.
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
http://www.locationtech.org/mailman/listinfo/geomesa-users
|
