Re: [geomesa-users] Geomesa Spark Java API

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]

Re: [geomesa-users] Geomesa Spark Java API

From: Jose Bujalance <joseab56@xxxxxxxxx>
Date: Fri, 17 Mar 2017 09:42:51 +0100
Delivered-to: geomesa-users@xxxxxxxxxxxxxxxx
List-archive: <https://dev.locationtech.org/mhonarc/lists/geomesa-users>
List-help: <mailto:geomesa-users-request@locationtech.org?subject=help>
List-subscribe: <https://dev.locationtech.org/mailman/listinfo/geomesa-users>, <mailto:geomesa-users-request@locationtech.org?subject=subscribe>
List-unsubscribe: <https://dev.locationtech.org/mailman/options/geomesa-users>, <mailto:geomesa-users-request@locationtech.org?subject=unsubscribe>

Hi Tom,

I am using Windows 7, Java 1.8.0_111-b14, and Apache Maven 3.3.9.

Now, concerning the shade plugin version, I think it could be related with the issue I had. At the beggining I was using version 1.3.1. Then when Jim told me you were using version 2.4.3 at GeoMesa, I switched to that version, but the issue persisted. After a little bit of research, I realized a lot of poms doesn't specify any version for the shade plugin. I guess this uses the latest version by default. So I deleted the version anchor on the shade plugin and rebuilt the jar. Everything seems to work perfectly now.

This is how my shade plugin looks like now:

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-shade-plugin</artifactId>

<phase>package</phase>

<goals>

<goal>shade</goal>

</goals>

<createDependencyReducedPom>false</createDependencyReducedPom>

<exclude>org/objectweb/asm/**</exclude>

</excludes>

</filter>

</filters>

<pattern>com.clearspring.analytics</pattern>

<shadedPattern>org.locationtech.geomesa.shaded.com.clearspring.analytics</shadedPattern>

</relocation>

<pattern>com.google.common</pattern>

<shadedPattern>org.locationtech.geomesa.shaded.com.google.common</shadedPattern>

</relocation>

<pattern>org.apache.commons.codec</pattern>

<shadedPattern>org.locationtech.geomesa.shaded.org.apache.commons.codec</shadedPattern>

</relocation>

<!-- shade thrift because spark depends on 0.9.2 and accumulo 1.8

depends on 0.9.3 -->

<pattern>org.apache.thrift</pattern>

<shadedPattern>org.locationtech.geomesa.shaded.org.apache.thrift</shadedPattern>

</relocation>

</relocations>

<transformer

implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">

<Main-Class>com.praxedo.geomesa.geomesa_spark.Test</Main-Class>

</manifestEntries>

</transformer>

<transformer

implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />

</transformers>

</configuration>

</execution>

</executions>

</plugin>

Have a nice day,

José.

2017-03-16 19:34 GMT+01:00 Tom Kunicki <tkunicki@xxxxxxxx>:

Hi, José.

It appears some of our META-INF/services files don’t terminate w/ newlines. We haven’t had an issue with this in the past but it appears this is causing you an issue.

Would you mind sharing your development environment?

OS?
JDK?
maven version?
maven shade plugin version?

Thanks,

Tom
On Mar 16, 2017, at 11:41 AM, Jose Bujalance <joseab56@xxxxxxxxx> wrote:
Actually I already had that block on my shade plugin, but I am still having those long lines in my services. The good part is that when I edit them manually once the jar has been generated, everything works perfectly, getting the same result as with the scala code.
2017-03-16 16:33 GMT+01:00 Jose Bujalance <joseab56@xxxxxxxxx>:
Hi Jim,

Thanks for your answer. You are right ! This is what I found in the generated META-INF/services/org.geotools.filter._expression_.PropertyAccessorFactory:

org.locationtech.geomesa.convert.cql.ArrayPropertyAccessorFactoryorg.locationtech.geomesa.features.kryo.json.JsonPropertyAccessorFactory
org.geotools.filter._expression_.SimpleFeaturePropertyAccessorFactory
org.geotools.filter._expression_.ThisPropertyAccessorFactory
org.geotools.filter._expression_.DirectPropertyAccessorFactory

And now that you mention it, I had the exact same problem with the META-INF/org.locationtech.geomesa.spark.SpatialRDDProvider, which I modify manually every time I generate the jar (you can guess I am not very good with Maven ^^)

So, is this solved 'automatically' by adding the following block to the shade plugin in my pom ?

<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
</transformers>
Thank you,
José
2017-03-16 15:46 GMT+01:00 Jim Hughes <jnh5y@xxxxxxxx>:
Hi José,

Does project bundle a fat jar for use in Spark? You may need to add a block to the maven-shade-plugin (or whichever plugin you are using) to sort the META-INF services entry for org.geotools.filter._expression_.PropertyAccessorFactory.

You can check out your jar with something like...

> jar xvf ./org/locationtech/geomesa/geomesa-convert-common_2.11/1.3.1/geomesa-convert-common_2.11-1.3.1.jar META-INF/services/org.geotools.filter._expression_.PropertyAccessorFactory
> more META-INF/services/org.geotools.filter._expression_.PropertyAccessorFactory
org.locationtech.geomesa.convert.cql.ArrayPropertyAccessorFactory

My guess is that your jar has one long line with two factories on it...

As another note, GeoMesa 1.3.x depends on GeoTools 15.1. It might work with later versions of GeoTools.

Cheers,

Jim

1. https://github.com/locationtech/geomesa/blob/master/geomesa-accumulo/geomesa-accumulo-spark-runtime/pom.xml#L138-L140

On 03/16/2017 10:10 AM, Jose Bujalance wrote:
Hi again,

I am trying the new Java API for Geomesa-Spark provided in the 1.3.1 version, but I am having some troubles.

First of all, I have tested that everything works fine querying my Accumulo datastore through the Spark shell using the geomesa-accumulo-spark-runtime_2.11-1.3.1.jar. This is how my scala code looks like:

import org.apache.hadoop.conf.Configuration

import org.apache.spark.rdd.RDD

import org.apache.spark.{SparkConf, SparkContext}

import org.locationtech.geomesa.spark._

import org.geotools.data.{DataStoreFinder, Query}

import org.geotools.factory.CommonFactoryFinder

import org.geotools.filter.text.ecql.ECQL

import scala.collection.JavaConversions._

// Accumulo datastore params

val params = Map(

"instanceId" -> "hdp-accumulo-instance",

"zookeepers" -> "hdf-sb-a.praxedo.net:2181,hdf-sb-b.praxedo.net:2181",

"user" -> "root",

"password" -> "praxedo",

"tableName" -> "Geoloc_Praxedo"

)

// set the configuration to the existant SparkContext

val conf = sc.getConf

conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")

conf.set("spark.kryo.registrator", classOf[GeoMesaSparkKryoRegistrator].getName)

val sc = SparkContext.getOrCreate(conf)

// create RDD with a geospatial query using Geomesa functions

val spatialRDDProvider = GeoMesaSpark(params)

val filter = ECQL.toFilter("BBOX(coords, 2.249294, 48.815215, 2.419337, 48.904295)")

val query = new Query("history_1M", filter)

val resultRDD = spatialRDDProvider.rdd(new Configuration, sc, params, query)

resultRDD.count

This code works fine, giving the expected result.

Now I am trying to do the same thing on Java. This is how my code looks like:

package com.praxedo.geomesa.geomesa_spark;

import org.apache.hadoop.conf.Configuration;

import org.apache.spark.SparkConf;

import org.apache.spark.api.java.JavaSparkContext;

import org.geotools.data.Query;

import org.geotools.filter.text.cql2.CQLException;

import org.geotools.filter.text.ecql.ECQL;

import org.locationtech.geomesa.spark.api.java.*;

import java.io.IOException;

import java.util.HashMap;

import java.util.Map;

public class Test {

private final static String ACCUMULO_INSTANCE = "hdp-accumulo-instance";

private final static String ACCUMULO_ZOOKEEPERS = "hdf-sb-a.praxedo.net:2181,hdf-sb-b.praxedo.net:2181";

private final static String ACCUMULO_USER = "root";

private final static String ACCUMULO_PASSWORD = "password";

private final static String GEOMESA_CATALOG = "Geoloc_Praxedo";

private final static String GEOMESA_FEATURE = "history_1M";

public static void main(String[] args) throws IOException, CQLException {

//Spark configuration

SparkConf conf = new SparkConf().setAppName("MyAppName").setMaster("local[*]");

conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");

conf.set("spark.kryo.registrator", "org.locationtech.geomesa.spark.GeoMesaSparkKryoRegistrator");

JavaSparkContext jsc = new JavaSparkContext(conf);

//Datastore configuration

Map<String, String> parameters = new HashMap<>();

parameters.put("instanceId", ACCUMULO_INSTANCE);

parameters.put("zookeepers", ACCUMULO_ZOOKEEPERS);

parameters.put("user", ACCUMULO_USER);

parameters.put("password", ACCUMULO_PASSWORD);

parameters.put("tableName", GEOMESA_CATALOG);

JavaSpatialRDDProvider provider = JavaGeoMesaSpark.apply(parameters);

String predicate = "BBOX(coords, 2.249294, 48.815215, 2.419337, 48.904295)";

Query query = new Query(GEOMESA_FEATURE, ECQL.toFilter(predicate));

JavaSpatialRDD resultRDD = provider.rdd(new Configuration(), jsc, parameters, query);

System.out.println("Number of RDDs: " + resultRDD.count());

System.out.println("First RDD: " + resultRDD.first());

}

}

And here are the dependencies I am importing with Maven:

<dependency>

<groupId>junit</groupId>

<artifactId>junit</artifactId>

<version>3.8.1</version>

<scope>test</scope>

</dependency>

<dependency>

<groupId>org.apache.accumulo</groupId>

<artifactId>accumulo-core</artifactId>

<version>1.7.0</version>

</dependency>

<dependency>

<groupId>org.apache.accumulo</groupId>

<artifactId>accumulo-fate</artifactId>

<version>1.7.0</version>

</dependency>

<dependency>

<groupId>org.apache.spark</groupId>

<artifactId>spark-core_2.11</artifactId>

<version>2.0.0</version>

</dependency>

<dependency>

<groupId>org.locationtech.geomesa</groupId>

<artifactId>geomesa-spark-core_2.11</artifactId>

<version>1.3.1</version>

</dependency>

<dependency>

<groupId>org.locationtech.geomesa</groupId>

<artifactId>geomesa-spark-converter_2.11</artifactId>

<version>1.3.1</version>

</dependency>

<dependency>

<groupId>org.locationtech.geomesa</groupId>

<artifactId>geomesa-security_2.11</artifactId>

<version>1.3.1</version>

</dependency>

<dependency>

<groupId>org.locationtech.geomesa</groupId>

<artifactId>geomesa-spark-geotools_2.11</artifactId>

<version>1.3.1</version>

</dependency>

<dependency>

<groupId>org.locationtech.geomesa</groupId>

<artifactId>geomesa-spark-sql_2.11</artifactId>

<version>1.3.1</version>

</dependency>

<dependency>

<groupId>org.locationtech.geomesa</groupId>

<artifactId>geomesa-accumulo-datastore_2.11</artifactId>

<version>1.3.1</version>

</dependency>

<dependency>

<groupId>org.locationtech.geomesa</groupId>

<artifactId>geomesa-accumulo-spark_2.11</artifactId>

<version>1.3.1</version>

</dependency>

<dependency>

<groupId>org.locationtech.geomesa</groupId>

<artifactId>geomesa-utils_2.11</artifactId>

<version>1.3.1</version>

</dependency>

<dependency>

<groupId>org.locationtech.geomesa</groupId>

<artifactId>geomesa-index-api_2.11</artifactId>

<version>1.3.1</version>

</dependency>

<dependency>

<groupId>org.geotools</groupId>

<artifactId>gt-main</artifactId>

<version>16.1</version>

</dependency>

I have succesfully built the jar, but when I launch it on my cluster I am getting the following error:

Exception in thread "main" java.util.ServiceConfigurationError: org.geotools.filter._expression_.PropertyAccessorFactory: Provider org.locationtech.geomesa.convert.cql.ArrayPropertyAccessorFactoryorg.locationtech.geomesa.features.kryo.json.JsonPropertyAccessorFactory not found

at java.util.ServiceLoader.fail(ServiceLoader.java:239)

at java.util.ServiceLoader.access$300(ServiceLoader.java:185)

at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:372)

at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)

at java.util.ServiceLoader$1.next(ServiceLoader.java:480)

at org.geotools.filter._expression_.PropertyAccessors.<clinit>(PropertyAccessors.java:51)

at org.geotools.filter.AttributeExpressionImpl.evaluate(AttributeExpressionImpl.java:213)

at org.geotools.filter.AttributeExpressionImpl.evaluate(AttributeExpressionImpl.java:189)

at org.geotools.filter.FilterAttributeExtractor.visit(FilterAttributeExtractor.java:130)

at org.geotools.filter.AttributeExpressionImpl.accept(AttributeExpressionImpl.java:340)

at org.geotools.filter.visitor.DefaultFilterVisitor.visit(DefaultFilterVisitor.java:214)

at org.geotools.filter.spatial.BBOXImpl.accept(BBOXImpl.java:224)

at org.geotools.data.DataUtilities.propertyNames(DataUtilities.java:413)

at org.locationtech.geomesa.filter.FilterHelper$.propertyNames(FilterHelper.scala:469)

at org.locationtech.geomesa.filter.visitor.FilterExtractingVisitor.keep(FilterExtractingVisitor.scala:44)

at org.locationtech.geomesa.filter.visitor.FilterExtractingVisitor.visit(FilterExtractingVisitor.scala:133)

at org.geotools.filter.spatial.BBOXImpl.accept(BBOXImpl.java:224)

at org.locationtech.geomesa.filter.visitor.FilterExtractingVisitor$.apply(FilterExtractingVisitor.scala:28)

at org.locationtech.geomesa.index.strategies.SpatioTemporalFilterStrategy$class.getFilterStrategy(SpatioTemporalFilterStrategy.scala:37)

at org.locationtech.geomesa.accumulo.index.Z3Index$.getFilterStrategy(Z3Index.scala:21)

at org.locationtech.geomesa.index.api.FilterSplitter$$anonfun$5.apply(FilterSplitter.scala:122)

at org.locationtech.geomesa.index.api.FilterSplitter$$anonfun$5.apply(FilterSplitter.scala:122)

at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)

at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)

at scala.collection.immutable.List.foreach(List.scala:381)

at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)

at scala.collection.immutable.List.flatMap(List.scala:344)

at org.locationtech.geomesa.index.api.FilterSplitter.org$locationtech$geomesa$index$api$FilterSplitter$$getSimpleQueryOptions(FilterSplitter.scala:122)

at org.locationtech.geomesa.index.api.FilterSplitter.getQueryOptions(FilterSplitter.scala:104)

at org.locationtech.geomesa.index.api.StrategyDecider$$anonfun$1.apply(StrategyDecider.scala:52)

at org.locationtech.geomesa.index.api.StrategyDecider$$anonfun$1.apply(StrategyDecider.scala:52)

at org.locationtech.geomesa.utils.stats.MethodProfiling$class.profile(MethodProfiling.scala:26)

at org.locationtech.geomesa.index.api.StrategyDecider.profile(StrategyDecider.scala:18)

at org.locationtech.geomesa.index.api.StrategyDecider.getFilterPlan(StrategyDecider.scala:52)

at org.locationtech.geomesa.index.api.QueryPlanner$$anonfun$4.apply(QueryPlanner.scala:135)

at org.locationtech.geomesa.index.api.QueryPlanner$$anonfun$4.apply(QueryPlanner.scala:114)

at org.locationtech.geomesa.utils.stats.MethodProfiling$class.profile(MethodProfiling.scala:26)

at org.locationtech.geomesa.index.api.QueryPlanner.profile(QueryPlanner.scala:43)

at org.locationtech.geomesa.index.api.QueryPlanner.getQueryPlans(QueryPlanner.scala:114)

at org.locationtech.geomesa.index.api.QueryPlanner.planQuery(QueryPlanner.scala:61)

at org.locationtech.geomesa.index.geotools.GeoMesaDataStore.getQueryPlan(GeoMesaDataStore.scala:464)

at org.locationtech.geomesa.accumulo.data.AccumuloDataStore.getQueryPlan(AccumuloDataStore.scala:108)

at org.locationtech.geomesa.jobs.accumulo.AccumuloJobUtils$.getMultipleQueryPlan(AccumuloJobUtils.scala:117)

at org.locationtech.geomesa.spark.accumulo.AccumuloSpatialRDDProvider.rdd(AccumuloSpatialRDDProvider.scala:107)

at org.locationtech.geomesa.spark.api.java.JavaSpatialRDDProvider.rdd(JavaGeoMesaSpark.scala:37)

at com.praxedo.geomesa.geomesa_spark.Test.main(Test.java:43)

Maybe a missing dependency? Any idea on this problem?

Thanks for your time.

José.
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.locationtech.org/mailman/listinfo/geomesa-users

References:
- [geomesa-users] Geomesa Spark Java API
  - From: Jose Bujalance
- Re: [geomesa-users] Geomesa Spark Java API
  - From: Jim Hughes
- Re: [geomesa-users] Geomesa Spark Java API
  - From: Jose Bujalance
- Re: [geomesa-users] Geomesa Spark Java API
  - From: Jose Bujalance
- Re: [geomesa-users] Geomesa Spark Java API
  - From: Tom Kunicki

Prev by Date: Re: [geomesa-users] Geomesa Spark Java API
Next by Date: Re: [geomesa-users] How to use a Stament Toner layer as a base layer in geoserver
Previous by thread: Re: [geomesa-users] Geomesa Spark Java API
Next by thread: [geomesa-users] How to read hbase(geomesa) data without geoserver and Stealth
Index(es):
- Date
- Thread

Breadcrumbs