Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geomesa-users] Error while generating heatmap using Density process

Thanks, that's good to know. Generally we used to work with most geoserver versions, but they recently make some breaking package changes so specific versions of GeoMesa were tied to specific versions of geoserver. We jumped straight from geoserver 2.15 to 2.17, so we probably never tested 2.16. Good to hear that it's working for you so far - you may run into issues with specific bits of functionality later on, since we compiled against 2.17, but it's also possible everything will work. If you do hit any issues, let us know!

Thanks,

Emilio

On 10/6/20 9:32 AM, Sangeetha S wrote:
Hi Emilio,

That's good to know :)
An observation: In the documentation, it is given that GeoServer 2.17.x is required for GeoServer 3.x. We use GeoServer 2.16.0. As we cannot upgrade it at the moment, due to other connected applications, we simply tried to run GeoMesa 3.0.0 with it and it seems to work without any errors.

Thanks,
Sangeetha

On Tue, Oct 6, 2020 at 3:26 PM Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:
Hello,

I don't think you need to change it if there isn't an explicit jar path. Without the path, it should just look for it on the default hbase classpath. I believe the number is either a timestamp or hash, but it's an internal hbase thing so I'm not entirely sure. Since things are working, it must be ok :)

Thanks,

Emilio

On 10/6/20 4:41 AM, Sangeetha S wrote:
Hi Emilio,

thanks for the reply. Yes, after using geomesa-hbase-distributed-runtime-hbase1_2.11-3.0.0.jar in all nodes, there are no errors while making WMS requests. I did not alter the existing HBase tables though. The coprocessor attribute is defined as:

{coprocessor$1 => '|org.locationtech.geomesa.hbase.coprocessor.GeoMesaCoprocessor|1073741823|'}

Does this have to be changed? What does 1073741823 stand for?

Thanks and regards,
Sangeetha


On Mon, Sep 21, 2020 at 5:57 PM Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:
Hi Sangeetha,

If you're using hbase 1.4, you should use the `geomesa-hbase-distributed-runtime-hbase1_2.11-3.0.0.jar`, not the `hbase2` jar. You may also need to use the hbase shell to 'alter table' and point to the new jar. If you ingest the example csv or run the quickstart into a new table, you should be able to see the correct coprocessor configs on that table, which you can then copy to your existing tables. Also make sure the geomesa-hbase-distributed-runtime jar is on each region server and the master node (you can install it in hdfs to avoid having to copy it around the cluster).

Thanks,

Emilio

On 9/21/20 11:25 AM, Sangeetha S wrote:
Hi Emilio,

In the last few days, I've been testing out the same process with GeoMesa 3.0.0. This time, I encountered a different error at a previous step. I'll start with the steps I had done during installation and provide the error details at the end of this message. That might perhaps give a clue on why this is happening.
  • I downloaded this file and unpacked it - https://github.com/locationtech/geomesa/releases/download/geomesa_2.11-3.0.0/geomesa-hbase-dist_2.11-3.0.0-bin.tar.gz
  • I stopped HBase and copied the geomesa-hbase-distributed-runtime-hbase2_2.11-3.0.0.jar to the lib directory of hbase. Restarted HBase
  • In a fresh installation of GeoServer 2.17.0, I installed the WPS extension.
  • Then, I copied all the jars in geomesa-hbase-gs-plugin_2.11-3.0.0-install.tar.gz to WEB-INF/lib folder of GeoServer.
  • I copied the geomesa-process-wps_2.11-3.0.0.jar to the same folder.
  • I copied the following jars into the same folder:
    • commons-cli-1.2.jar
    • commons-configuration-1.6.jar
    • commons-io-2.6.jar
    • commons-logging-1.1.1.jar
    • hadoop-auth-2.7.1.jar
    • hadoop-client-2.7.4.jar
    • hadoop-common-2.7.1.jar
    • hadoop-hdfs-2.7.1.jar
    • hadoop-mapreduce-client-core-2.7.1.jar
    • hbase-client-1.4.12.jar
    • hbase-common-1.4.12.jar
    • hbase-hadoop-compat-1.4.12.jar
    • hbase-protocol-1.4.12.jar
    • htrace-core-3.1.0-incubating.jar
    • metrics-core-2.2.0.jar
    • netty-3.6.2.Final.jar
    • netty-all-4.0.23.Final.jar
    • protobuf-java-2.5.0.jar
    • zookeeper-3.4.10.jar
  • Restarted GeoServer
  • Created HBase GeoMesa datastore and a layer
  • When I preview the layer with Openlayers option, I get an error: Could not obtain the next feature:org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.ClassNotFoundException: org.locationtech.geomesa.hbase.rpc.filter.CqlTransformFilter
The entire stack trace is attached with this email. WFS requests work fine, but WMS does not work. I tried to apply the heatmap style from here (https://www.geomesa.org/documentation/stable/tutorials/geomesa-examples-gdelt.html), however, that results in the same error. The missing class seems to be defined in geomesa-hbase-rpc_2.11-3.0.0.jar and this jar is already in the WEB-INF/lib folder of the GeoServer. I tested with GeoServer 2.16.0 also and the results are the same.

Could you please help me solve this error?

Thanks,
Sangeetha


On Mon, Jun 29, 2020 at 8:36 PM Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:
Hi Sangeetha,

In the newly released 3.0 we've upgraded hbase client version to 1.4.12. Also, the hbase client jars are no longer bundled in a shaded jar, so you should be able to swap them out to your specific version in a geoserver instance without rebuilding the plugin.

Thanks,

Emilio

On 6/25/20 4:50 AM, Sangeetha S wrote:
Hi Emilio,

I haven't yet tried to change the HBase version and rebuild the plugin jar. However, I installed the newly released GeoMesa 2.4.1 and I hit the same error.

Thanks,
Sangeetha

On Sat, May 23, 2020 at 12:15 AM Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:
Hi Sangeetha,

I hit the same error recently. I think something changed in a recent bug-fix version of HBase 1.4. For instance, it works with 1.4.4 but doesn't seem to work with 1.4.12. At this point, I'd recommend checking out the geomesa source code and modifying the pom.xml <hbase.version> to 1.4.12, then re-building the geomesa-hbase-gs-plugin shaded jar.

Thanks,

Emilio

On 5/22/20 10:18 AM, Sangeetha S wrote:
Hi Emilio,

Thanks a lot for the detailed steps. I followed the same, however there is an error while creating a datastore in the GeoServer. I have attached the error with this email. Could you please look at it?

Thanks,
Sangeetha

On Tue, May 19, 2020 at 8:14 PM Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:
Hi Sangeetha,

I was able to run heatmaps ok. I did the following, starting with a fresh geoserver 2.15 with the wps plugin:

1. Install the geomesa-hbase-gs-plugin shaded jar from https://search.maven.org/remotecontent?filepath=org/locationtech/geomesa/geomesa-hbase-gs-plugin_2.11/2.4.0/geomesa-hbase-gs-plugin_2.11-2.4.0-shaded.jar into geoserver/WEB-INF/lib
2. Run the 'install-hadoop.sh' script from the binary distribution pointing at geoserver/WEB-INF/lib, after fixing the curl command in common-functions.sh to add an -L for handling maven redirects (see https://geomesa.atlassian.net/browse/GEOMESA-2768 for details)
3. Copied the wps jar from geomesa-hbase_2.11-2.4.0/dist/gs-plugins/geomesa-process-wps_2.11-2.4.0.jar into geoserver/WEB-INF/lib
4. Copied the hbase-site.xml into geoserver/WEB-INF/classes
5. Started geoserver and installed the 'heatmap' style from https://www.geomesa.org/documentation/tutorials/geomesa-examples-gdelt.html#heatmaps
6. Installed the geomesa-hbase-distributed-runtime jar into $HBASE_HOME/lib
7. Ingested the example-csv data bundled with the tools: ./geomesa-hbase ingest -c 240.test -C example-csv -s example-csv ../examples/ingest/csv/example.csv
8. Registered the store and layer in geoserver, displayed the layer preview, and appended '&styles=heatmap' to the URL

Hope that helps,

Emilio

On 5/15/20 1:32 AM, Sangeetha S wrote:
Hi Emilio,

I tried with both 2.15.5 and 2.16.0. I get the same error in both cases.

Thanks,
Sangeetha

On Thu, May 14, 2020 at 5:34 PM Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:
Hello,

What version of GeoServer are you using? I believe it has to be 2.15[1]. If that's what you're using, I'll try to re-create the issue.

Thanks,

Emilio

[1]: https://www.geomesa.org/documentation/user/geoserver.html#geoserver-versions

On 5/14/20 3:13 AM, Sangeetha S wrote:
Hi Emilio,

In a newly installed GeoServer, I installed the WPS extension and then copied the geomesa-hbase-gs-plugin_2.11-2.4.0-shaded.jar and geomesa-process-wps_2.11-2.4.0.jar to WEB-INF/lib folder of GeoServer. However, the GeoServer does not start anymore.

Thanks,
Sangeetha



On Wed, May 13, 2020 at 4:01 PM Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:
Hi Sangeetha,

It shouldn't be a problem to run on a different machine, as long as you have appropriate network connectivity to the cluster.

You can try using the geomesa shaded jar, which would replace the files in the geomesa-gs-plugin.tgz and all the hbase/hadoop jars:

https://search.maven.org/remotecontent?filepath=org/locationtech/geomesa/geomesa-hbase-gs-plugin_2.11/2.4.0/geomesa-hbase-gs-plugin_2.11-2.4.0-shaded.jar

So from a vanilla geoserver, you would install just that jar and the geomesa-process-wps jar.

Thanks,

Emilio

On 5/13/20 9:29 AM, Sangeetha S wrote:
Hi Emilio,

An update: The GeoServer does not start if I replace hbase-shaded-client with hbase-client. There are too many errors.
Could you please guide me on how to proceed?

One more question, the GeoServer runs on a different machine from where Hadoop, HBase and GeoMesa have been installed. Does that make a difference?

Thanks,
Sangeetha

On Tue, May 12, 2020 at 5:30 PM Sangeetha S <sanggeoceg@xxxxxxxxx> wrote:
Hi Emilio,

Thanks for your reply. I have added to the GeoServer all jars present in $GEOMESA_HBASE_HOME/dist/gs-plugins/geomesa-hbase-gs-plugin_2.11-2.4.0-install.tar.gz as well as the hbase-shaded-client-1.4.12.jar and htrace-core-1.3.0-incubating.jar.

I'll try replacing hbase-shaded-client with hbase-client and let you know how it goes.

Thanks,
Sangeetha

On Mon, May 11, 2020 at 7:23 PM Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:
Hello,

I think the issue comes from using the hbase-shaded client jar. Could you try using the non-shaded hbase client jar, and/or provide the list of jars you've added to geoserver?

Thanks,

Emilio

On 5/11/20 4:17 AM, Sangeetha S wrote:
Dear GeoMesa devs and users,

I am trying to generate a heatmap of line strings stored in an HBase table using GeoMesa density process. HBase is running on top of Hadoop HDFS. I have created a HBase datastore and published the layer in the GeoServer. When I use the standard line style, I am able to make WMS requests. However, when I execute the geomesa density process, I get the error the following error:

java.lang.NoSuchMethodException: org.locationtech.geomesa.hbase.proto.GeoMesaProto$GeoMesaCoprocessorService.newStub(org.apache.hadoop.hbase.shaded.com.google.protobuf.RpcChannel)

The full error log is attached with this email. I have installed Hadoop 2.7.1, HBase 1.4.12 and GeoMesa 2.4.0.
Could you please help me in finding what is going wrong and how to fix it? Please let me know if you need further information.

Thanks,
Sangeetha.

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/geomesa-users

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/geomesa-users


--
Sangeetha Shankar.


--
Sangeetha Shankar.

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/geomesa-users

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/geomesa-users


--
Sangeetha Shankar.

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/geomesa-users

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/geomesa-users


--
Sangeetha Shankar.

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/geomesa-users

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/geomesa-users


--
Sangeetha Shankar.

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/geomesa-users

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/geomesa-users


--
Sangeetha Shankar.

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/geomesa-users

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/geomesa-users


--
Sangeetha Shankar.

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/geomesa-users

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/geomesa-users


--
Sangeetha Shankar.

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/geomesa-users

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/geomesa-users


--
Sangeetha Shankar.

_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxx
To unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/geomesa-users


Back to the top