Hi Chris,
We recommend the steps below for ingesting a non-shapefile csv or
tsv:
1. in java code, get a handle on a DataStore using
DataStoreFinder.getDataStore()
2. create a SimpleFeatureType for GDELT using DataUtilities
3. call ds.createSchema(schemaType)
4. run map reduce job with that schema
I'm working on an example project in Java that I will send you
when complete.
Below is an example of using DataUtilities to create a
SimpleFeatureType for GDELT. You may need to double check some
of the types in the sftSpec String, I referred to the GDELT
online documentation:
String name = "gdelt";
String sftSpec =
"GLOBALEVENTID:Integer,SQLDATE:Date,MonthYear:Integer,Year:Integer,FractionDate:Float,Actor1Code:String,Actor1Name:String,Actor1CountryCode:String,Actor1KnownGroupCode:String,Actor1EthnicCode:String,Actor1Religion1Code:String,Actor1Religion2Code:String,Actor1Type1Code:String,Actor1Type2Code:String,Actor1Type3Code:String,Actor2Code:String,Actor2Name:String,Actor2CountryCode:String,Actor2KnownGroupCode:String,Actor2EthnicCode:String,Actor2Religion1Code:String,Actor2Religion2Code:String,Actor2Type1Code:String,Actor2Type2Code:String,Actor2Type3Code:String,IsRootEvent:Integer,EventCode:String,EventBaseCode:String,EventRootCode:String,QuadClass:Integer,GoldsteinScale:Float,NumMentions:Integer,NumSources:Integer,NumArticles:Integer,AvgTone:Float,Actor1Geo_Type:Integer,Actor1Geo_FullName:String,Actor1Geo_CountryCode:String,Actor1Geo_ADM1Code:String,Actor1Geo_Lat:Float,Actor1Geo_Long:Float,Actor1Geo_FeatureID:Integer,Actor2Geo_Type:Integer,Actor2Geo_FullName:String,Actor2Geo_CountryCode:String,Actor2Geo_ADM1Code:String,Actor2Geo_Lat:Float,Actor2Geo_Long:Float,Actor2Geo_FeatureID:Integer,ActionGeo_Type:Integer,ActionGeo_FullName:String,ActionGeo_CountryCode:String,ActionGeo_ADM1Code:String,ActionGeo_Lat:Float,ActionGeo_Long:Float,ActionGeo_FeatureID:Integer,DATEADDED:Integer";
SimpleFeatureType
featureType = DataUtilities.createType(name, sftSpec);
dataStore.createSchema(featureType);
Hunter
On 04/11/2014 01:22 PM, Chris Snider wrote:
Hi,
I saw some of the Geomesa YouTube videos
referencing loading data as well as the “Spatio-temporal
Indexing in Non-relational Distributed Databases” paper
referencing loading the GDELT dataset. Are there any
documented steps on how to load the GDELT dataset?
Additionally, I have been able to load
features to a feature type using the WFS-T endpoint. Is there
a better/faster/more efficient method of loading even modest
amounts of data? Example, I have a Natural Earth Country
Polygon set that I extracted the geometry, name and admin
columns from to push into Geomesa through the WFS-T endpoint.
I can only push between 5 and 10 rows without hitting a
timeout.
Thanks,
Chris Snider
Senior Software
Engineer
Intelligent
Software Solutions, Inc.
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
http://www.locationtech.org/mailman/listinfo/geomesa-users
|
