| [geomesa-users] Spark lost task java.lang.IllegalStateException: unread block data |
|
Hello everybody, I would like to ask. I am getting Lost task 1.0 in stage 0.0 (TID 1, 147.175.3.163): java.lang.IllegalStateException: unread block data error when trying to compile and run application from eclipse. I have no problem running the script when compiled as geomesa-compute. I am clearly missing some jars in the project buildpath. I am trying to run this code: package com.aimaps.iot;import java.text.SimpleDateFormat import org.apache.hadoop.conf.Configuration import org.apache.spark.{SparkConf, SparkContext} import org.geotools.data.{DataStoreFinder, Query} import org.geotools.factory.CommonFactoryFinder import org.geotools.filter.text.ecql.ECQL import org.locationtech.geomesa.accumulo.data.AccumuloDataStore import org.locationtech.geomesa.compute.spark.GeoMesaSpark import scala.collection.JavaConversions._ object GeomesaQuerry { val dbParams = Map( "instanceId" -> "accumulo", "zookeepers" -> "localhost:2181", "user" -> "root", "password" -> "pass1234", "tableName" -> "gdelt_ukraine") val feature = "event" val geom = "geom" val date = "SQLDATE" val bbox = "22.128811, 44.390411, 40.218079, 52.375359" val during = "2014-04-01T00:00:00.000Z/2014-04-31T23:59:59.999Z" val filter = s"bbox($geom, $bbox) AND $date during $during" def main (args: Array[String]) { val ds = DataStoreFinder.getDataStore(dbParams).asInstanceOf[AccumuloDataStore] val q = new Query(feature, ECQL.toFilter(filter)) val sparkConf = new SparkConf(true) sparkConf.setMaster("spark://munko-acer:7077") sparkConf.setAppName("GeomesaQuerry") //sparkConf.setExecutorEnv("SPARK_EXECUTOR_MEMORY", "4G") val sc = new SparkContext(GeoMesaSpark.init(sparkConf, ds)) val queryRDD = GeoMesaSpark.rdd(new Configuration, sc, dbParams, q, None) val dayAndFeature = queryRDD.mapPartitions { iter => val df = new SimpleDateFormat("yyyyMMdd") val ff = CommonFactoryFinder.getFilterFactory2 val exp = ff.property(date) iter.map { f => (df.format(exp.evaluate(f).asInstanceOf[java.util.Date]), f) } } val countByDay = dayAndFeature.map( x => (x._1, 1)).reduceByKey(_ + _) countByDay.collect().foreach(println) println("\n") ds.dispose() } }
And I get this error: Error while parsing JAI registry file "file:/home/milan/Tools/geomesa-1.2.6/dist/spark/geomesa-compute-1.2.6-shaded.jar!/META-INF/registryFile.jai" :Error in registry file at line number #31 A descriptor is already registered against the name "org.geotools.ColorReduction" under registry mode "rendered" Error in registry file at line number #32 A descriptor is already registered against the name "org.geotools.ColorInversion" under registry mode "rendered" Running Spark version 2.0.2 Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Your hostname, munko-acer resolves to a loopback address: 127.0.1.1; using 147.175.3.163 instead (on interface enp2s0) Set SPARK_LOCAL_IP if you need to bind to another address Changing view acls to: milan Changing modify acls to: milan Changing view acls groups to: Changing modify acls groups to: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(milan); groups with view permissions: Set(); users with modify permissions: Set(milan); groups with modify permissions: Set() Successfully started service 'sparkDriver' on port 40519. Registering MapOutputTracker Registering BlockManagerMaster Created local directory at /tmp/blockmgr-4814c20a-c5c5-4c74-b535-c5c44c5dd6cf MemoryStore started with capacity 1949.1 MB Registering OutputCommitCoordinator Logging initialized @2810ms jetty-9.2.z-SNAPSHOT Started o.s.j.s.ServletContextHandler@288ca5f0{/jobs,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@4068102e{/jobs/json,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@44bd4b0a{/jobs/job,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@6c008c24{/jobs/job/json,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@216e0771{/stages,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@21079a12{/stages/json,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@fcc6023{/stages/stage,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@67c5ac52{/stages/stage/json,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@36417a54{/stages/pool,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@2b8bb184{/stages/pool/json,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@472a11ae{/storage,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@dc79225{/storage/json,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@30e9ca13{/storage/rdd,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@46185a1b{/storage/rdd/json,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@51288417{/environment,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@60cf62ad{/environment/json,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@1e0895f5{/executors,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@1ac4ccad{/executors/json,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@fd9ebde{/executors/threadDump,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@14982a82{/executors/threadDump/json,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@4ee5b2d9{/static,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@72f8ae0c{/,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@323f3c96{/api,null,AVAILABLE} Started o.s.j.s.ServletContextHandler@6726cc69{/stages/stage/kill,null,AVAILABLE} Started ServerConnector@723d9a10{HTTP/1.1}{0.0.0.0:4040} Started @3024ms Successfully started service 'SparkUI' on port 4040. Bound SparkUI to 0.0.0.0, and started at http://147.175.3.163:4040 Connecting to master spark://munko-acer:7077... Successfully created connection to munko-acer/127.0.1.1:7077 after 26 ms (0 ms spent in bootstraps) Connected to Spark cluster with app ID app-20161201125709-0005 Executor added: app-20161201125709-0005/0 on worker-20161201102135-147.175.3.163-44853 (147.175.3.163:44853) with 4 cores Granted executor ID app-20161201125709-0005/0 on hostPort 147.175.3.163:44853 with 4 cores, 1024.0 MB RAM Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 34069. Server created on 147.175.3.163:34069 Registering BlockManager BlockManagerId(driver, 147.175.3.163, 34069) Registering block manager 147.175.3.163:34069 with 1949.1 MB RAM, BlockManagerId(driver, 147.175.3.163, 34069) Registered BlockManager BlockManagerId(driver, 147.175.3.163, 34069) Executor updated: app-20161201125709-0005/0 is now RUNNING Started o.s.j.s.ServletContextHandler@f1868c9{/metrics/json,null,AVAILABLE} SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 Desired query plan requires multiple scans - falling back to full table scan The query being executed requires multiple scans, which is not currently supported by geomesa. Your result set will be partially incomplete. Query: BBOX(geom, 22.128811,44.390411,40.218079,52.375359) AND SQLDATE DURING 2014-04-01T00:00:00+00:00/2014-05-01T23:59:59.999+00:00 Block broadcast_0 stored as values in memory (estimated size 2.9 MB, free 1946.2 MB) Block broadcast_0_piece0 stored as bytes in memory (estimated size 237.7 KB, free 1946.0 MB) Added broadcast_0_piece0 in memory on 147.175.3.163:34069 (size: 237.7 KB, free: 1948.9 MB) Created broadcast 0 from newAPIHadoopRDD at GeoMesaSpark.scala:142 Registered executor NettyRpcEndpointRef(null) (147.175.3.163:46958) with ID 0 Registering block manager 147.175.3.163:37089 with 366.3 MB RAM, BlockManagerId(0, 147.175.3.163, 37089) Starting job: collect at GeomesaQuerry.scala:55 Registering RDD 3 (map at GeomesaQuerry.scala:53) Got job 0 (collect at GeomesaQuerry.scala:55) with 2 output partitions Final stage: ResultStage 1 (collect at GeomesaQuerry.scala:55) Parents of final stage: List(ShuffleMapStage 0) Missing parents: List(ShuffleMapStage 0) Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at map at GeomesaQuerry.scala:53), which has no missing parents Block broadcast_1 stored as values in memory (estimated size 4.1 KB, free 1946.0 MB) Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.4 KB, free 1946.0 MB) Added broadcast_1_piece0 in memory on 147.175.3.163:34069 (size: 2.4 KB, free: 1948.9 MB) Created broadcast 1 from broadcast at DAGScheduler.scala:1012 Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at map at GeomesaQuerry.scala:53) Adding task set 0.0 with 2 tasks Stage 0 contains a task of very large size (4581 KB). The maximum recommended task size is 100 KB. Starting task 0.0 in stage 0.0 (TID 0, 147.175.3.163, partition 0, PROCESS_LOCAL, 4691322 bytes) Starting task 1.0 in stage 0.0 (TID 1, 147.175.3.163, partition 1, PROCESS_LOCAL, 4691421 bytes) Launching task 0 on executor id: 0 hostname: 147.175.3.163. Launching task 1 on executor id: 0 hostname: 147.175.3.163. Lost task 1.0 in stage 0.0 (TID 1, 147.175.3.163): java.lang.IllegalStateException: unread block data at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2449) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1385) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75) at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:253) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Starting task 1.1 in stage 0.0 (TID 2, 147.175.3.163, partition 1, PROCESS_LOCAL, 4691421 bytes) Lost task 0.0 in stage 0.0 (TID 0) on executor 147.175.3.163: java.lang.IllegalStateException (unread block data) [duplicate 1] Launching task 2 on executor id: 0 hostname: 147.175.3.163. Starting task 0.1 in stage 0.0 (TID 3, 147.175.3.163, partition 0, PROCESS_LOCAL, 4691322 bytes) Launching task 3 on executor id: 0 hostname: 147.175.3.163. Lost task 1.1 in stage 0.0 (TID 2) on executor 147.175.3.163: java.lang.IllegalStateException (unread block data) [duplicate 2] Starting task 1.2 in stage 0.0 (TID 4, 147.175.3.163, partition 1, PROCESS_LOCAL, 4691421 bytes) Launching task 4 on executor id: 0 hostname: 147.175.3.163. Lost task 0.1 in stage 0.0 (TID 3) on executor 147.175.3.163: java.lang.IllegalStateException (unread block data) [duplicate 3] Starting task 0.2 in stage 0.0 (TID 5, 147.175.3.163, partition 0, PROCESS_LOCAL, 4691322 bytes) Launching task 5 on executor id: 0 hostname: 147.175.3.163. Lost task 1.2 in stage 0.0 (TID 4) on executor 147.175.3.163: java.lang.IllegalStateException (unread block data) [duplicate 4] Starting task 1.3 in stage 0.0 (TID 6, 147.175.3.163, partition 1, PROCESS_LOCAL, 4691421 bytes) Launching task 6 on executor id: 0 hostname: 147.175.3.163. Lost task 0.2 in stage 0.0 (TID 5) on executor 147.175.3.163: java.lang.IllegalStateException (unread block data) [duplicate 5] Starting task 0.3 in stage 0.0 (TID 7, 147.175.3.163, partition 0, PROCESS_LOCAL, 4691322 bytes) Launching task 7 on executor id: 0 hostname: 147.175.3.163. Lost task 1.3 in stage 0.0 (TID 6) on executor 147.175.3.163: java.lang.IllegalStateException (unread block data) [duplicate 6] Task 1 in stage 0.0 failed 4 times; aborting job Cancelling stage 0 Stage 0 was cancelled ShuffleMapStage 0 (map at GeomesaQuerry.scala:53) failed in 1.704 s Job 0 failed: collect at GeomesaQuerry.scala:55, took 1.798401 s Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 6, 147.175.3.163): java.lang.IllegalStateException: unread block data at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2449) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1385) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75) at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:253) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1454) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1442) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1441) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1441) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811) at scala.Option.foreach(Option.scala:257) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1667) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1622) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1611) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:632) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1873) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1886) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1899) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1913) at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:912) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:358) at org.apache.spark.rdd.RDD.collect(RDD.scala:911) at com.aimaps.iot.GeomesaQuerry$.main(GeomesaQuerry.scala:55) at com.aimaps.iot.GeomesaQuerry.main(GeomesaQuerry.scala) Caused by: java.lang.IllegalStateException: unread block data at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2449) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1385) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373) at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75) at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:253) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Invoking stop() from shutdown hook Lost task 0.3 in stage 0.0 (TID 7) on executor 147.175.3.163: java.lang.IllegalStateException (unread block data) [duplicate 7] Stopped ServerConnector@723d9a10{HTTP/1.1}{0.0.0.0:4040} Removed TaskSet 0.0, whose tasks have all completed, from pool Stopped o.s.j.s.ServletContextHandler@6726cc69{/stages/stage/kill,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@323f3c96{/api,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@72f8ae0c{/,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@4ee5b2d9{/static,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@14982a82{/executors/threadDump/json,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@fd9ebde{/executors/threadDump,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@1ac4ccad{/executors/json,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@1e0895f5{/executors,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@60cf62ad{/environment/json,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@51288417{/environment,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@46185a1b{/storage/rdd/json,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@30e9ca13{/storage/rdd,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@dc79225{/storage/json,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@472a11ae{/storage,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@2b8bb184{/stages/pool/json,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@36417a54{/stages/pool,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@67c5ac52{/stages/stage/json,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@fcc6023{/stages/stage,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@21079a12{/stages/json,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@216e0771{/stages,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@6c008c24{/jobs/job/json,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@44bd4b0a{/jobs/job,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@4068102e{/jobs/json,null,UNAVAILABLE} Stopped o.s.j.s.ServletContextHandler@288ca5f0{/jobs,null,UNAVAILABLE} Stopped Spark web UI at http://147.175.3.163:4040 Shutting down all executors Asking each executor to shut down MapOutputTrackerMasterEndpoint stopped! MemoryStore cleared BlockManager stopped BlockManagerMaster stopped OutputCommitCoordinator stopped! Successfully stopped SparkContext Shutdown hook called Deleting directory /tmp/spark-bc960a3b-309c-4acb-bfe2-60969598c699 Thank you for your help. Milan --
|
