Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geomesa-users] Code adapted from geomesa-gdelt-master for different data doesn't update Accumulo
Bob,

I can't see anything wrong with your code yet. You mentioned not seeing any errors, did you also check the tablet server logs?  If you haven't, that might help diagnose the issue - the tserver logs are in opt/accumulo/logs.
We will keep looking and let you know what we find.

Hunter

On 05/20/2014 01:29 PM, Barnhart, Bob M. wrote:

After getting the geomesa-gdelt-master code to work with one of data files at http://data.gdeltproject.org/events/index.html (http://data.gdeltproject.org/events/20140507.export.CSV.zip) I decided to adapt the GDELTIngest.java, GDELTIngestMapper.java and GDELTQuery.java classes to work with a different data-set (see sensor.csv).

 

I believe I have made only minimal changes to the “GDELT*.java” files producing SensorIngest.java, SensorIngestMapper.java (enclosed) and SensorQuery.java (not included). The pom.xml and core-site.xml files are unchanged.

 

My data (see sensor.csv) is defined in SensorIngest.java as:

 

static List<String> attributes = Lists.newArrayList(

                                "EventID:Integer",

                                "Sensor:Integer",

                                "Time:Date",                                      // Date => a "Unix" time (e.g., millis since 1970.01.01)

                                "SensorGeo_Lat:Float",

                                "SensorGeo_Long:Float",

                                "Altitude:Integer",

                                "MinAxis:Float",

                                "MajAxis:Float",

                                "Orientation:Float",

                                "*geom:Point:srid=4326"           // SRID 4326 => standard coordinate system

                );

 

The critical fields are “Time:Date” (which is a Unix-like time), and the event lat/long (“SensorGeo_Lat/Long:Float”). I note that the GDELT data uses an “SQLDATE” field of the form “YYYYMMDD”. However, the lat/long fields are similar. I even renamed them to “SensorGeo_Lat” and “SensorGeo_Long” on the theory that “…Geo_...” is a significant naming convention.

 

I am running on an RHEL 6.5 server (VMWare Workstation VM), using Accumulo 1.5.1 and Hadoop 2.4.0.

 

Using the enclosed “./bin/data” Bash script, I run the command “data upload” to load the data into Hadoop, and then “data ingest” to ingest it into Accumulo (using Hadoop Map-Reduce).

 

I see no errors, but when I use the “accumulo shell” commands “table sensor” and “scan”, I only see 4 (metadata) entries in the target “sensor” table, rather than 124 “SimpleFeature” entries:

 

~METADATA_sensor attributes: []    EventID:Integer,Sensor:Integer,Time:Date,Latitude:Float,Longitude:Float,Altitude:Integer,MinAxis:Float,MajAxis:Float,Orientation:Float,geom:Point:srid=4326

~METADATA_sensor dtgfield: []    Time

~METADATA_sensor featureEncoding: []    avro

~METADATA_sensor schema: []    %~#s%99#r%sensor#cstr%0,3#gh%yyyyMMdd#d::%~#s%3,2#gh::%~#s%#id

 

I’m perplexed as to what I have not done or done wrong here.

 

Any thoughts or suggestions would be greatly appreciated!

 

Bob Barnhart

Chief Systems Engineer | 858 826 5596 (Office) | 619 972 9489 (Mobile) | Robert.M.Barnhart@xxxxxxxxxx

 

 



_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
http://www.locationtech.org/mailman/listinfo/geomesa-users

Back to the top