[geomesa-dev] [JIRA] (GEOMESA-3103) Problem with geomesa_pyspark in vers

["Dinko (JIRA)" <jira@xxxxxxxxxxxxxxxxxxxxx>]

Title: Message Title

Dinko created an issue

GeoMesa / GEOMESA-3103

Problem with geomesa_pyspark in version 3.2.0

Issue Type:	Bug
Affects Versions:	3.2.0
Assignee:	Unassigned
Created:	16/Jul/21 6:57 AM
Environment:	Spark 3.1.2 (requires geomesa build with scala 2.12) Geomesa 2.12_3.2.0 Accumulo 2.0.1 Hadoop 3.3.0 Python 3.7 ( i saw 3.9 cause problems)
Labels:	geomesa_pyspark
Priority:	Major
Reporter:	Dinko

I successfully ingested example csv data from examples folder.

But when i try to run pyspark sql query it wont show results (error bellow code, looks like some versions issue).
I use version setup explained in envirionment and for geomesa_pyspark i use this code.

import geomesa_pyspark

conf = geomesa_pyspark.configure(
    jars=['/spark/jars/geomesa-accumulo-spark-runtime-accumulo2_2.12-3.2.0.jar'],
    packages=['geomesa_pyspark','pytz'],
    spark_home='/spark').\
    setAppName('HelloGeomesaApp').setMaster('spark://spark-master:7077')

conf.set('spark.driver.host','172.21.0.12')

from pyspark.sql import SparkSession
spark = SparkSession.builder.config(conf=conf).getOrCreate()
spark.sparkContext.pythonExec = "/usr/local/bin/python3"

params = {
    "accumulo.instance.id": "accumulo",
    "accumulo.zookeepers": "zookeeper",
    "accumulo.user": "root",
    "accumulo.password": "GisPwd",
    "accumulo.catalog": "geomesa.example"
}

feature = "example-csv"

df = ( spark
    .read
    .format("geomesa")
    .options(**params)
    .option("geomesa.feature", feature) 
    .load()
)

df.createOrReplaceTempView("tbl")
spark.sql("show tables").show()

spark.sql("""
select count(*)
from tbl
""").show()

Py4JJavaError: An error occurred while calling o75.showString.
: java.lang.ArrayIndexOutOfBoundsException: 28499
    at com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.accept(BytecodeReadingParanamer.java:563)
    at com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.access$200(BytecodeReadingParanamer.java:338)
    at com.thoughtworks.paranamer.BytecodeReadingParanamer.lookupParameterNames(BytecodeReadingParanamer.java:103)
    at com.thoughtworks.paranamer.CachingParanamer.lookupParameterNames(CachingParanamer.java:79)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.getCtorParams(BeanIntrospector.scala:45)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1(BeanIntrospector.scala:59)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1$adapted(BeanIntrospector.scala:59)
    at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
    at scala.collection.Iterator.foreach(Iterator.scala:941)
    at scala.collection.Iterator.foreach$(Iterator.scala:941)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
    at scala.collection.IterableLike.foreach(IterableLike.scala:74)
    at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
    at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
    at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
    at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.findConstructorParam$1(BeanIntrospector.scala:59)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$19(BeanIntrospector.scala:181)
    at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
    at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
    at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
    at scala.collection.TraversableLike.map(TraversableLike.scala:238)
    at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
    at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$14(BeanIntrospector.scala:175)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$14$adapted(BeanIntrospector.scala:174)
    at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
    at scala.collection.immutable.List.foreach(List.scala:392)
    at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
    at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
    at scala.collection.immutable.List.flatMap(List.scala:355)
    at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.apply(BeanIntrospector.scala:174)

Add Comment

Get Jira notifications on your phone! Download the Jira Cloud app for Android or iOS
This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100168-sha1:7af8a0b)