| Re: [geowave-dev] geowave-tools -hdfsingest |
|
When reading these supported formats from the geotools-vector plugin it seems that CSV is supported.
That´s why I first tried the following: java -cp deploy/target/geowave-deploy-0.8.9-SNAPSHOT-tools.jar:extensions/formats/geotools-vector/target/geowave-format-vector-0.8.9-SNAPSHOT.jar mil.nga.giat.geowave.core.cli.GeoWaveMain -hdfsingest -b /home/marcel/testData/verysmall.csv -hdfs node1-scads02:8020 -i accumulo -n geowave -p secret -resourceman node1-scads02:8032 -u root -z node1-scads02:2181 -hdfsbase node1-scads02:8020/ WARN [main] hdfs.StageToHdfsDriver (StageToHdfsDriver.java:runInternal(95)) - Plugin provider 'geotools-vector' does not support staging to HDFS java.lang.UnsupportedOperationException: GeoTools vector files cannot be ingested using intermediate avro files at mil.nga.giat.geowave.format.geotools.vector.GeoToolsVectorDataStoreIngestFormat.getAvroFormatPlugin(GeoToolsVectorDataStoreIngestFormat.java:29)... As a second try I wrote a mapreduce job which ingests my csv data using the GeomesaOutputFormat. The job runs successfully in local mode. But when starting the command with yarn the GeomesaOutputFormat was not found. I already added geowave-datastore-accumulo-0.8.9-SNAPSHOT.jar to my path. Thanks, Marcel Jacob. Date: Mon, 14 Sep 2015 14:19:28 -0400 From: andy@xxxxxxxxxxxxx To: geowave-dev@xxxxxxxxxxxxxxxx Subject: Re: [geowave-dev] geowave-tools -hdfsingest Since v0.8.7 GeoWave has a command line tools framework that relies on Java SPI discoverable libraries to provide functionality. When we package GeoWave for installation via RPM the file structure looks as follows: [andy@c1-app-02 ~]$ tree /usr/local/geowave/tools /usr/local/geowave/tools ├── geowave-tools-build.properties ├── geowave-tools.jar ├── geowave-tools.sh └── plugins ├── geowave-format-geolife-0.8.8.1.jar ├── geowave-format-gpx-0.8.8.1.jar ├── geowave-format-raster-0.8.8.1.jar ├── geowave-format-tdrive-0.8.8.1.jar └── geowave-format-vector-0.8.8.1.jar The /usr/local/geowave/tools/geowave-tools.sh script is then soft-linked into the path of all users using the name "geowave" and takes care of setting the classpath to include anything in the plugins directory Snippet from the geowave-tools.sh script CLASSPATH="/usr/local/geowave/tools/geowave-tools.jar:/usr/local/geowave/tools/plugins/*" exec $JAVA $GEOWAVE_TOOL_JAVA_OPT -cp $CLASSPATH mil.nga.giat.geowave.core.cli.GeoWaveMain "$@" This setup allows us to reference a "geowave" command to perform any of the configured actions. An example of an ingest operation on an RPM installed system is shown at http://ngageoint.github.io/geowave/documentation.html#ingest-example If you want to run tools commands from your local repo in development you'd build the application and then set up the classpath to include the various jars yourself. Set your desired dependencies --------------------------------------------------------------------------- export BUILD_ARGS="-Daccumulo.version=1.6.0-cdh5.1.4 -Dhadoop.version=2.6.0-cdh5.4.0 -Dgeotools.version=13.2 -Dgeoserver.version=2.7.2 -Dvendor.version=cdh5 -P cloudera" export EXTRA_ARGS="-Dfindbugs.skip=true -DskipFormat=true -DskipITs=true -DskipTests=true" Build the project and the tools.jar --------------------------------------------------------------------------- cd YOUR_GEOWAVE_SOURCE_DIR mvn clean install $BUILD_ARGS $EXTRA_ARGS mvn package -P geowave-tools-singlejar $BUILD_ARGS $EXTRA_ARGS Ensure all the plugins you want are on the classpath along with the tools.jar --------------------------------------------------------------------------- java -cp deploy/target/geowave-deploy-0.8.9-SNAPSHOT-tools.jar:extensions/formats/geotools-vector/target/geowave-format-vector-0.8.9-SNAPSHOT.jar mil.nga.giat.geowave.core.cli.GeoWaveMain -localingest -l Once you've verified that the plugins can be discovered by the tools.jar you'd use the remaining arguments "-b, -i etc." the same as the instructions show for a packaged installation. Let’s download the Admin 0 - Countries shapefile: http://naciscdn.org/naturalearth/50m/cultural/ne_50m_admin_0_countries.zip $ mkdir ingest $ mv ne_50m_admin_0_countries.zip ingest/ $ cd ingest $ unzip ne_50m_admin_0_countries.zip $ rm ne_50m_admin_0_countries.zip $ cd .. $ geowave -localingest \ -b ./ingest \ -i ACCUMULO_INSTANCE_NAME \ -n geowave.50m_admin_0_countries \ -f geotools-vector \ -u USERNAME \ -p PASSWORD \ -z ZOOKEEPER_HOST_NAME:2181 One last note is that there is no csv plugin. The examples I've shown use the geotools-vector plugin which supports a number of formats http://docs.geotools.org/stable/userguide/geotools.html#supported-formats Andy On Mon, Sep 14, 2015 at 11:22 AM, Marcel <m.jacob@xxxxxxxxxxx> wrote: Hello, _______________________________________________ geowave-dev mailing list geowave-dev@xxxxxxxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://www.locationtech.org/mailman/listinfo/geowave-dev |