Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geomesa-users] Geomesa Connection Issue with HBase RO cluster
Correcting above email.

Added the log to ensure org.locationtech.geomesa.hbase.data.HBaseConnectionPool is getting right hbase.config.xml. As per logs Geomesa is setting up right org.apache.hadoop.conf.Configuration [1], but still HBase client is not applying it. The value is getting honored only if we are adding hbase-site.xml in the class path.



On Fri, Mar 27, 2020 at 11:43 PM Amit Srivastava <amit.bit96@xxxxxxxxx> wrote:
Thanks Emilio, I updated the EMR Hbase jar, but I see an issue because the parameter "hbase.meta.table.suffix" is not getting override via hbase.config.xml [1] if I am using below jars in the classpath. As per HBase document, "hbase.meta.table.suffix" should be override because the metadata table name in RO cluster is hbase:meta_clusterId

Class Paths:
<classpath path="jars/hbase-common-1.3.1.jar" />
<classpath path="jars/hbase-client-1.3.1.jar" />
<classpath path="jars/hbase-server-1.3.1.jar" />
<classpath path="jars/hbase-site.zip" />
<classpath path="jars/geomesa-hbase-datastore_2.11-2.4.0.jar" />
<classpath path="jars/geomesa-hbase-spark-runtime_2.11-2.4.0.jar" />

Code: Where I am creating DataStore

    public DataStore newInstance(final ExecutionContext executionContext, final String catalog) throws IOException {
        final Map<String, String> hbaseDataStoreParameters = ImmutableMap.of(
                "hbase.catalog", catalog,
                "hbase.zookeepers", executionContext.getClusterDetails().getMasterPublicDns(),
                "hbase.config.xml", getXml(executionContext.getClusterDetails().getClusterId()),
                "hbase.remote.filtering", "false");
            return DataStoreFinder.getDataStore(hbaseDataStoreParameters);
    }

    private String getXml(final String clusterId) {
        return String.format("<configuration>\n" +
                "   <property>\n" +
                "      <name>hbase.meta.table.suffix</name>\n" +
                "      <value>%s</value>\n" +
                "      <final>true</final>\n" +
                "   </property>\n" +
                "   <property>\n" +
                "      <name>hbase.global.readonly.enabled</name>\n" +
                "      <value>true</value>\n" +
                "      <final>true</final>\n" +
                "   </property>\n" +
                "   <property>\n" +
                "      <name>hbase.meta.startup.refresh</name>\n" +
                "      <value>true</value>\n" +
                "      <final>true</final>\n" +
                "   </property>\n" +
                "</configuration>", clusterId);
    }



On Fri, Mar 27, 2020 at 11:49 AM Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:
I think so, I haven't actually done it before. Another possibility would be to put the jars in your local .m2 cache. If you get it working, please circle back and we can add it to the docs!

Thanks,

Emilio

On 3/27/20 12:06 PM, Amit Srivastava wrote:
Thanks Emilio,

I am using geomesa-hbase-spark-runtime_2.11-2.4.0.jar. Do I need to add below repository tag in this pom.xml (https://github.com/locationtech/geomesa/blob/master/pom.xml) to get the appropriate EMR version for client?
<repository>
    <id>emr-${aws-emr.version}-artifacts</id>
    <name>EMR ${aws-emr.version} Releases Repository</name>
    <releases>
        <enabled>true</enabled>
    </releases>
    <snapshots>
        <enabled>false</enabled>
    </snapshots>
    <url>https://s3.us-west-1.amazonaws.com/us-west-1-emr-artifacts/emr-${aws-emr.version}/repos/maven/</url>
</repository>


--Amit


On Fri, Mar 27, 2020 at 8:16 AM Emilio Lahr-Vivaz <elahrvivaz@xxxxxxxx> wrote:
Hi Amit,

You should just need to use the AWS HBase jars instead of the regular HBase jars everywhere in the install guide: https://www.geomesa.org/documentation/user/hbase/install.html

If you're using the geomesa-hbase-spark-runtime jar, that bundles the HBase jars inside it, so you'll need to rebuild it from source, using the repository Austin linked to get the AWS HBase jars.

Thanks,

Emilio

On 3/27/20 11:05 AM, Amit Srivastava wrote:
Hi Emilio and Austin,

I am facing a connection issue [1] with the HBase RO cluster. I want to update the HBase jar in the Geomesa.

I need a suggestion from you. Can you please tell me where do I need to make this change in Geomesa (version 2.4.0)? I see many places where I need to make changes. Can you point me in the right direction?


--

Regards,

Amit Kumar Srivastava




--

Regards,

Amit Kumar Srivastava




--

Regards,

Amit Kumar Srivastava



--

Regards,

Amit Kumar Srivastava

Back to the top