Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geowave-dev] geowave-tools -hdfsingest

Since v0.8.7 GeoWave has a command line tools framework that relies on Java SPI discoverable libraries to provide functionality. 

When we package GeoWave for installation via RPM the file structure looks as follows:

[andy@c1-app-02 ~]$ tree /usr/local/geowave/tools
/usr/local/geowave/tools
├── geowave-tools-build.properties
├── geowave-tools.jar
├── geowave-tools.sh
└── plugins
    ├── geowave-format-geolife-0.8.8.1.jar
    ├── geowave-format-gpx-0.8.8.1.jar
    ├── geowave-format-raster-0.8.8.1.jar
    ├── geowave-format-tdrive-0.8.8.1.jar
    └── geowave-format-vector-0.8.8.1.jar

The /usr/local/geowave/tools/geowave-tools.sh script is then soft-linked into the path of all users using the name "geowave" and takes care of setting the classpath to include anything in the plugins directory

Snippet from the geowave-tools.sh script
CLASSPATH="/usr/local/geowave/tools/geowave-tools.jar:/usr/local/geowave/tools/plugins/*"
exec $JAVA $GEOWAVE_TOOL_JAVA_OPT -cp $CLASSPATH mil.nga.giat.geowave.core.cli.GeoWaveMain "$@"

This setup allows us to reference a "geowave" command to perform any of the configured actions. An example of an ingest operation on an RPM installed system is shown at http://ngageoint.github.io/geowave/documentation.html#ingest-example

If you want to run tools commands from your local repo in development you'd build the application and then set up the classpath to include the various jars yourself.

Set your desired dependencies
---------------------------------------------------------------------------
export BUILD_ARGS="-Daccumulo.version=1.6.0-cdh5.1.4 -Dhadoop.version=2.6.0-cdh5.4.0 -Dgeotools.version=13.2 -Dgeoserver.version=2.7.2 -Dvendor.version=cdh5 -P cloudera"
export EXTRA_ARGS="-Dfindbugs.skip=true -DskipFormat=true -DskipITs=true -DskipTests=true"

Build the project and the tools.jar
---------------------------------------------------------------------------
cd YOUR_GEOWAVE_SOURCE_DIR
mvn clean install $BUILD_ARGS $EXTRA_ARGS 
mvn package -P geowave-tools-singlejar $BUILD_ARGS $EXTRA_ARGS

Ensure all the plugins you want are on the classpath along with the tools.jar
---------------------------------------------------------------------------
java -cp deploy/target/geowave-deploy-0.8.9-SNAPSHOT-tools.jar:extensions/formats/geotools-vector/target/geowave-format-vector-0.8.9-SNAPSHOT.jar mil.nga.giat.geowave.core.cli.GeoWaveMain -localingest -l

Once you've verified that the plugins can be discovered by the tools.jar you'd use the remaining arguments "-b, -i etc." the same as the instructions show for a packaged installation.

From: http://ngageoint.github.io/geowave/documentation.html#ingest-example

Let’s download the Admin 0 - Countries shapefile: http://naciscdn.org/naturalearth/50m/cultural/ne_50m_admin_0_countries.zip

$ mkdir ingest
$ mv ne_50m_admin_0_countries.zip ingest/
$ cd ingest
$ unzip ne_50m_admin_0_countries.zip
$ rm ne_50m_admin_0_countries.zip
$ cd ..
$ geowave -localingest \
      -b ./ingest \
      -i ACCUMULO_INSTANCE_NAME \
      -n geowave.50m_admin_0_countries \ 
      -f geotools-vector \ 
      -u USERNAME \
      -p PASSWORD \
      -z ZOOKEEPER_HOST_NAME:2181

One last note is that there is no csv plugin. The examples I've shown use the geotools-vector plugin which supports a number of formats http://docs.geotools.org/stable/userguide/geotools.html#supported-formats

Andy


On Mon, Sep 14, 2015 at 11:22 AM, Marcel <m.jacob@xxxxxxxxxxx> wrote:
Hello,
I want to ingest some data via the geowave-tools operation hdfsingest using the following command:

java -cp geowave-deploy-0.8.9-SNAPSHOT-tools.jar mil.nga.giat.geowave.core.cli.GeoWaveMain -hdfsingest \-b /home/marcel/testData/verysmall.csv \-hdfs node1-scads02:8020 \-i accumulo \-n geowave \-p secret \-resourceman node1-scads02:8032 \-u root \-z node1-scads02:2181

FATAL [main] ingest.AbstractIngestCommandLineDriver (AbstractIngestCommandLineDriver.java:applyArguments(154)) - There were no ingest format plugin providers found

In your documentation it says that there were plugin jars needed. What do you mean by that? Where can I find them?

Thanks,
Marcel Jacob.

_______________________________________________
geowave-dev mailing list
geowave-dev@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geowave-dev

Back to the top