Hi
Emilio,
I’m
trying to create hbase importer which can import Polygons
stored as WKT in CSV files.
Everything
is building fine but when I try to import file I can see
Exception like my geometry was taken as Point :
Exception
in thread “main” java.lang.ClassCastException:
com.vividsolutions.jts.geom.Polygon cannot be cast to
com.vividsolutions.jts.geom.Point
I
can’t find solution for this problem.
Application
code:
/**
* Copyright 2016 Commonwealth Computer
Research, Inc.
*
* Licensed under the Apache License,
Version 2.0 (the License);
* you may not use this file except in
compliance with the License.
* You may obtain a copy of the License at
*
*
http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or
agreed to in writing, software
* distributed under the License is
distributed on an AS IS BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.
* See the License for the specific language
governing permissions and
* limitations under the License.
*/
//
package com.example.geomesa.hbase;
import com.beust.jcommander.internal.Lists;
import com.google.common.base.Joiner;
import
com.vividsolutions.jts.geom.Geometry;
import com.vividsolutions.jts.geom.Polygon;
import com.vividsolutions.jts.geom.Point;
import
com.vividsolutions.jts.geom.GeometryFactory;
import com.vividsolutions.jts.io.WKTReader;
import org.apache.commons.cli.*;
import org.geotools.data.*;
import org.geotools.factory.Hints;
import
org.geotools.feature.DefaultFeatureCollection;
import
org.geotools.feature.FeatureCollection;
import
org.geotools.feature.FeatureIterator;
import
org.geotools.feature.SchemaException;
import
org.geotools.feature.simple.SimpleFeatureBuilder;
import org.geotools.filter.text.cql2.CQL;
import
org.geotools.filter.text.cql2.CQLException;
import
org.geotools.geometry.jts.JTSFactoryFinder;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
import
org.locationtech.geomesa.utils.text.WKTUtils$;
import
org.geotools.swing.data.JFileDataStoreChooser;
import org.opengis.feature.Feature;
import
org.opengis.feature.simple.SimpleFeature;
import
org.opengis.feature.simple.SimpleFeatureType;
import org.opengis.filter.Filter;
import java.io.IOException;
import java.io.Serializable;
import java.io.File;
import java.io.BufferedReader;
import java.io.BufferedReader.*;
import java.io.FileReader;
import java.io.FileReader.*;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Random;
public class HBaseImporter {
static String TABLE_NAME =
"bigtable.table.name".replace(".", "_");
// sub-set of parameters
that are used to create the HBase DataStore
static String[]
HBASE_CONNECTION_PARAMS = new String[]{
TABLE_NAME
};
/**
* Creates a common set of
command-line options for the parser. Each option
* is described separately.
*/
static Options
getCommonRequiredOptions() {
Options options = new
Options();
Option tableNameOpt =
OptionBuilder.withArgName(TABLE_NAME)
.hasArg()
.isRequired()
.withDescription("table
name")
.create(TABLE_NAME);
options.addOption(tableNameOpt);
return options;
}
static Map<String,
Serializable> getHBaseDataStoreConf(CommandLine cmd)
{
Map<String ,
Serializable> dsConf = new HashMap<>();
for (String param :
HBASE_CONNECTION_PARAMS) {
dsConf.put(param.replace("_", "."),
cmd.getOptionValue(param));
}
System.out.println(
"DataStore Config: " + dsConf);
return dsConf;
}
static SimpleFeatureType
createSimpleFeatureType(String simpleFeatureTypeName)
throws SchemaException {
// list the attributes that
constitute the feature type
List<String>
attributes = Lists.newArrayList(
"ID:Integer",
"*WKT:Polygon:srid=4326",
// the "*" denotes the default geometry (used for
indexing)
//"What:java.lang.Long", //
some types require full qualification (see DataUtilities
docs)
"date:Date",
// a date-time field is
optional, but can be indexed
"osm_id:String",
"name:String",
"type:String"
);
// create the bare
simple-feature type
String
simpleFeatureTypeSchema =
Joiner.on(",").join(attributes);
SimpleFeatureType
simpleFeatureType =
DataUtilities.createType(simpleFeatureTypeName,
simpleFeatureTypeSchema);
System.out.println(
"SimpleFeatureType: " + simpleFeatureType);
return simpleFeatureType;
}
static FeatureCollection
createNewFeatures(SimpleFeatureType simpleFeatureType,
File file) throws IOException {
/*
* We create a
FeatureCollection into which we will put each Feature
created from a record
* in the input csv data
file
*/
DefaultFeatureCollection
featureCollection = new DefaultFeatureCollection();
/*
* GeometryFactory will be
used to create the geometry attribute of each feature (a
Point
* object for the location)
*/
GeometryFactory
geometryFactory =
JTSFactoryFinder.getGeometryFactory(null);
SimpleFeatureBuilder
featureBuilder = new
SimpleFeatureBuilder(simpleFeatureType);
WKTReader georeader = new
WKTReader( geometryFactory );
java.io.FileReader infile =
new java.io.FileReader(file);
java.io.BufferedReader
reader = new java.io.BufferedReader(infile);
try {
/* First line of the data
file is the header */
String line =
reader.readLine();
System.out.println("Header:
" + line);
System.out.println( "Feature Collection creation
start");
int lines =
0;
for (line =
reader.readLine(); line != null; line =
reader.readLine()) {
if (line.trim().length()
> 0) { // skip blank lines
String tokens[] =
line.split("\\;");
int ID =
Integer.parseInt(tokens[0].trim());
//Geometry WKT = WKTUtils$.MODULE$.read(tokens[1]);
Polygon WKT = (Polygon)
georeader.read(tokens[1]);
String osm_id =
tokens[2].trim();
String name =
tokens[3].trim();
String type =
tokens[4].trim();
featureBuilder.add(ID);
featureBuilder.add(WKT);
featureBuilder.add(osm_id);
featureBuilder.add(name);
featureBuilder.add(type);
SimpleFeature feature =
featureBuilder.buildFeature(null);
featureCollection.add(feature);
/* System.out.println( ". " +
feature.getProperty("ID").getValue() + "|" +
feature.getProperty("WKT").getValue() + "|" +
feature.getProperty("osm_id").getValue() + "|" +
feature.getProperty("name").getValue() + "|" +
feature.getProperty("type").getValue());*/
lines++;
}
}
System.out.println( "Feature Count: " + lines);
} finally {
reader.close();
System.out.println( "Feature Collection creation
finished");
return
featureCollection;
}
}
static void
insertFeatures(String simpleFeatureTypeName,
DataStore
dataStore,
FeatureCollection featureCollection)
throws IOException {
FeatureStore featureStore =
(FeatureStore)dataStore.getFeatureSource(simpleFeatureTypeName);
System.out.println("getFeatureSource: " + featureStore);
System.out.println("featureCollection: " +
featureCollection);
featureStore.addFeatures(featureCollection);
System.out.println("Insert
finished");
}
public
static void main(String[] args) throws Exception {
// find out where -- in
HBase -- the user wants to store data
CommandLineParser parser =
new BasicParser();
Options options =
getCommonRequiredOptions();
CommandLine cmd =
parser.parse( options, args);
// verify that we can see
this HBase destination in a GeoTools manner
Map<String,
Serializable> dsConf = getHBaseDataStoreConf(cmd);
DataStore dataStore =
DataStoreFinder.getDataStore(dsConf);
assert dataStore != null;
// establish specifics
concerning the SimpleFeatureType to store
String
simpleFeatureTypeName = "ImporterDev2";
SimpleFeatureType
simpleFeatureType =
createSimpleFeatureType(simpleFeatureTypeName);
// write Feature-specific
metadata to the destination table in HBase
// (first creating the
table if it does not already exist); you only need
// to create the
FeatureType schema the *first* time you write any
Features
// of this type to the
table
System.out.println("Creating feature-type (schema): " +
simpleFeatureTypeName);
dataStore.createSchema(simpleFeatureType);
File file =
JFileDataStoreChooser.showOpenFile("csv", null);
if (file == null) {
//System.out.println("Chosen CSV File: " + file);
//.getSelectedFile().getName()
return;
}
System.out.println("Chosen
CSV File: " + file);
// create new features
locally, and add them to this table
System.out.println("Creating new features");
FeatureCollection
featureCollection = createNewFeatures(simpleFeatureType,
file);
System.out.println("Inserting new features");
insertFeatures(simpleFeatureTypeName,
dataStore, featureCollection);
}
}
Hi Marcin,
The command line tools currently support accumulo and (to a
lesser degree) kafka. For ingesting data into hbase, you
have to use the geotools API for creating simple features.
We have several tutorials on how to do this using accumulo -
those would be a good place to start. You should be able to
adapt them to hbase by replacing the accumulo data store
with an hbase data store. Because they are both geotools
DataStore implementations, almost all of the code would
remain the same.
Single process ingest/query:
http://www.geomesa.org/documentation/tutorials/geomesa-quickstart-accumulo.html
Map/reduce ingest:
http://www.geomesa.org/documentation/tutorials/geomesa-examples-gdelt.html
Streaming ingest using storm:
http://www.geomesa.org/documentation/tutorials/geomesa-quickstart-storm.html
The hbase documentation describes how to get an hbase data
store:
http://www.geomesa.org/documentation/user/hbase_datastore.html
Thanks,
Emilio
On 07/04/2016 04:34 AM, Marjasiewicz,
Marcin wrote:
Line
Tools and HBase
Good morning,
I would like to ask
a question about using Command Line Tools with HBase. Is
it supported or it is only tool for Accumulo?
If not what is
proper format to storage geometry in HBase? For example
if I will import CSV with geometry as WKT will it work
with Geomesa Hbase Geoserver plugin?
I have already
configured Geoserver plugin to work with my HBase. Now I
am looking for some solution to load geospatial data
into it.
My sample data row
in HBase table looks that:
Best Regards
_______________________________
Marcin Marjasiewicz
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geomesa-users
_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geomesa-users
