Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geowave-dev] t-drive data ingest
The script to install in the HDP sandbox used the release GeoWave RPM repo so you've got 0.8.7 installed and master is currently at 0.8.8-SNAPSHOT but I don't think anything should break. The plugins are loaded dynamically using SPI so yes you can just drop it into the /usr/local/geowave/tools/plugins directory and given the correct ownership and permissions it will get picked up.

I've uploaded a patched version of that jar to S3 which you could download and replace if you just want it to work as advertised.

https://s3.amazonaws.com/geowave/0.8.7-tdrive-plugin/geowave-format-tdrive-0.8.7.jar

Andy 

On Fri, Jun 26, 2015 at 12:56 PM, Scott <sctevl@xxxxxxxxx> wrote:
Thanks Andy,

  I did run without the .txt once (using txt) and I did see that error. I guess that's what I should have focused on. Makes much more sense now. 

  I cloned from the repo again and am using maven to build the jars (mvn install). For the original setup, I used the rpms. Should I start on a clean accumulo or is it possible to swap out the old jars for the new ones?

Scott

On Sat, Jun 20, 2015 at 10:08 AM, Andy Spohn <andy@xxxxxxxxxxxxx> wrote:
Scott,

The quick exit to the ingest command with no logging is due to the FileVisitor finding no files to ingest. The default extension is csv which you've correctly used the -x to override but the regex adds the "." already so .txt finds nothing as it needs txt to work as expected. I'm thinking we should probably have some debug logging in there to explain what it's searching for to aid in debugging this sort of problem.

Fix: Use -x txt instead of -x .txt


Once you get past that it will unfortunately still fail due to a typo in a property name within the plugin. I just submitted a pull request to fix, you'd have to replace the existing plugin before you'll be able to ingest tdrive files, sorry.

Fix: The tdrive format plugin needs this update - https://github.com/ngageoint/geowave/pull/433

/usr/local/geowave/tools/plugins/geowave-format-tdrive-0.8.7.jar


Andy


On Fri, Jun 19, 2015 at 4:01 PM, Scott <sctevl@xxxxxxxxx> wrote:
Hello,

  I'm trying to use the T-Drive data ingest tool on GeoWave with the latest Hortonworks Sandbox. The shapefile ingest works - for both the natural earth file in the example and for my own shapefile. 

  When I run the t-drive ingest I don't get any errors but I don't get a table either. I have the txt files unzipped into the ingest directory. Here's the command I'm using:

# geowave -localingest -b ~/ingest -dim spatial-temporal -f tdrive -i hdp-accumulo-instance -n geowave.tdrive -u geowave -p geowave -x .txt -z sandbox.hortonworks.com:2181

  Am I using the t-drive ingest correctly by trying to have it grab all of the txt files in a directory?

  I'm not seeing any errors but there is a warning about the application context that I'm also seeing for the shapefile sample:

Jun 19, 2015 7:41:59 PM org.geoserver.platform.GeoServerExtensions checkContext

WARNING: Extension lookup 'GeoServerResourceLoader', but ApplicationContext is unset.

Jun 19, 2015 7:41:59 PM org.geoserver.platform.GeoServerExtensions checkContext

WARNING: Extension lookup 'GeoServerResourceLoader', but ApplicationContext is unset.

Jun 19, 2015 7:41:59 PM org.geoserver.platform.GeoServerExtensions checkContext

WARNING: Extension lookup 'GeoServerResourceLoader', but ApplicationContext is unset.

Jun 19, 2015 7:41:59 PM org.geoserver.platform.GeoServerExtensions checkContext

WARNING: Extension lookup 'GeoServerResourceLoader', but ApplicationContext is unset.

2015-06-19 19:41:59,816 INFO  [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT

2015-06-19 19:41:59,820 INFO  [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:host.name=sandbox.hortonworks.com

2015-06-19 19:41:59,820 INFO  [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.version=1.7.0_79

2015-06-19 19:41:59,821 INFO  [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.vendor=Oracle Corporation

2015-06-19 19:41:59,821 INFO  [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79.x86_64/jre

2015-06-19 19:41:59,821 INFO  [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.class.path=/usr/local/geowave/tools/geowave-tools.jar:/usr/local/geowave/tools/plugins/geowave-format-geolife-0.8.7.jar:/usr/local/geowave/tools/plugins/geowave-format-tdrive-0.8.7.jar:/usr/local/geowave/tools/plugins/geowave-format-gpx-0.8.7.jar:/usr/local/geowave/tools/plugins/geowave-format-raster-0.8.7.jar:/usr/local/geowave/tools/plugins/geowave-format-vector-0.8.7.jar

2015-06-19 19:41:59,821 INFO  [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

2015-06-19 19:41:59,822 INFO  [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.io.tmpdir=/tmp

2015-06-19 19:41:59,822 INFO  [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:java.compiler=<NA>

2015-06-19 19:41:59,822 INFO  [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:os.name=Linux

2015-06-19 19:41:59,822 INFO  [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:os.arch=amd64

2015-06-19 19:41:59,822 INFO  [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:os.version=2.6.32-504.23.4.el6.x86_64

2015-06-19 19:41:59,823 INFO  [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.name=root

2015-06-19 19:41:59,823 INFO  [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.home=/root

2015-06-19 19:41:59,823 INFO  [main] zookeeper.ZooKeeper (Environment.java:logEnv(100)) - Client environment:user.dir=/root

2015-06-19 19:41:59,825 INFO  [main] zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=sandbox.hortonworks.com:2181 sessionTimeout=30000 watcher=org.apache.accumulo.fate.zookeeper.ZooSession$ZooWatcher@7865657e

2015-06-19 19:41:59,863 INFO  [main-SendThread(sandbox.hortonworks.com:2181)] zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server sandbox.hortonworks.com/192.168.99.90:2181. Will not attempt to authenticate using SASL (unknown error)

2015-06-19 19:41:59,875 INFO  [main-SendThread(sandbox.hortonworks.com:2181)] zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(852)) - Socket connection established to sandbox.hortonworks.com/192.168.99.90:2181, initiating session

2015-06-19 19:41:59,889 INFO  [main-SendThread(sandbox.hortonworks.com:2181)] zookeeper.ClientCnxn (ClientCnxn.java:onConnected(1235)) - Session establishment complete on server sandbox.hortonworks.com/192.168.99.90:2181, sessionid = 0x14e0d1c50aa001f, negotiated timeout = 30000


_______________________________________________
geowave-dev mailing list
geowave-dev@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geowave-dev



_______________________________________________
geowave-dev mailing list
geowave-dev@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geowave-dev



_______________________________________________
geowave-dev mailing list
geowave-dev@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.locationtech.org/mailman/listinfo/geowave-dev


Back to the top