[geomesa-users] Apply Geomesa function to standard Spark Dataset

[Diethard Steiner <diethard.steiner@xxxxxxxxx>]

Hi,

So I am sourcing some data with Spark SQL and now I want to use the Geomesa function `st_makePoint`:

    val ingestedData = (
      spark
        .read
        .option("header", "false")
        .option("delimiter","\\t")
        .option("ignoreLeadingWhiteSpace","true")
        .option("ignoreTrailingWhiteSpace","true")
        .option("treatEmptyValuesAsNulls","true")
        .option("dateFormat","yyyyMMdd")
        .schema(gdeltSchema)
        .csv(ingestFile)
      )

    ingestedData.show(10)
    println(ingestedData.getClass)

    ingestedData.createOrReplaceTempView("ingested_data")
    val prepedData = spark.sql("""SELECT *, st_makePoint(Actor1Geo_Lat, Actor1Geo_Long) as geom FROM ingested_data""")

I get following error:

 Exception in thread "main" org.apache.spark.sql.AnalysisException: Undefined function: 'st_makePoint'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.;

How do I resolve this?

Note: The other example on sourcing data directly from Accumulo via Spark SQL and using functions on it is working in my environment. So I assume I just need a way to convert a normal Spark Dataset into one that can use the GeoMesa functions.

Best regards,

Diethard