Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[geomesa-users] 回复: -EXT- problem with query data by spark
  • From: zhou lihuang <zlh_0923@xxxxxxxxxxx>
  • Date: Fri, 3 Jan 2025 02:00:36 +0000
  • Accept-language: zh-CN, en-US
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4wKGguVxGQcTzoPxWrM2EDhx2nQ7v0dpZ0YJsKnW7s8=; b=A0S60vs1V4169SfVDNVXPgZwkt1IdPGuf5kLyDqYsqq+AU7Eyjcr7gRVxiSZdEvaGxoUzI01vrkvrHE44o7eN2JqhGf7r9IHNo0iQzSELv/HS5XjYeLa5SPby0YD/6xNGrmvvwvbI9kBeVFtY5ovZ7vvN9W8Rye0qFIVLqh06mVw9qKReX8GLHec4KVoCbOoND4Zp9EMSG4jylIjbzLLnFAdzGjC8/A3y1W2szlIgnIv/zIlXS29PXwy2lxdp5ZCn6d2BpuzNd0D9puHhiPOHvrO3tEv1a38MzmA26LX6sgUgnfpYdgJHjDC3Af5cG2FgGx5mEtlCYCYNru03hx2gg==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ItHxGHyuFekFOjT/P26DVbC7oqLJOFjj4kk1gOm+xtERAEuDQpKahuf1XWvUGohM3VocR4MslvTvgPmOjxGPiJC6Zc4Idx+ph42T125+TCDacGPsBy16lJKtGs4ptbl6LdckdiLGRHmqJNUJ/x/npLeQztBeOWEP2NCBWQ+BNxf4y+2+Lh2cO2bNjhdUs5wY8tfYP8TGgPsTAcleDTioceakJXpmq6+EI9xvn8D194TODORMKTHxmdLoryhmwgy9J4i4TL+OFYxjGeCZWmXx/l0b8cS7Vks24AVQo5UrN+VcLgriudC6yWByGxHX7heq8/ldrYS/v/uou3jwi84FQQ==
  • Delivered-to: geomesa-users@xxxxxxxxxxx
  • List-archive: <https://dev.eclipse.org/mailman/private/geomesa-users/>
  • List-help: <mailto:geomesa-users-request@eclipse.org?subject=help>
  • List-subscribe: <https://dev.eclipse.org/mailman/listinfo/geomesa-users>, <mailto:geomesa-users-request@eclipse.org?subject=subscribe>
  • List-unsubscribe: <https://dev.eclipse.org/mailman/options/geomesa-users>, <mailto:geomesa-users-request@eclipse.org?subject=unsubscribe>
  • Msip_labels:
  • Thread-index: AQHbVq7JTz7d2Lbquku0N+XqMCCq+bMDh1cegADN+MA=
  • Thread-topic: [geomesa-users] -EXT- problem with query data by spark
First of all, thannk you for your reply.

I've found the possible reason after multiple tests：
When HBase stores only a small amount of data, the data exists only in the MemStore and hasn't been flushed to the StoreFile yet. In this case, when using Spark to query through spatialRDDProvider.rdd, no data will be obtained. However, once the data has been flushed to the StoreFile, the query results will be normal.

I'm not sure whether this is a bug of GeoMesa or not.

Best Regards,
Mike

发件人: geomesa-users <geomesa-users-bounces@xxxxxxxxxxx> 代表 Lahr-Vivaz, Emilio via geomesa-users <geomesa-users@xxxxxxxxxxx>
发送时间: 2025年1月2日 21:33
收件人: Geomesa project user mailing list <geomesa-users@xxxxxxxxxxx>
抄送: Lahr-Vivaz, Emilio <Emilio.Lahr-Vivaz@xxxxxxxxxxx>
主题: Re: [geomesa-users] -EXT- problem with query data by spark
 
I think ".count" will short-circuit by default. Can you try ".show" instead and see if that works? If you really just want a count, try setting the system property "geomesa.force.count" to true: https://www.geomesa.org/documentation/stable/user/datastores/runtime_config.html#geomesa-force-count 

Thanks,

Emilio Lahr-Vivaz
General Atomics, CCRi
From: geomesa-users <geomesa-users-bounces@xxxxxxxxxxx> on behalf of zhou lihuang via geomesa-users <geomesa-users@xxxxxxxxxxx>
Sent: Wednesday, December 25, 2024 4:31 AM
To: geomesa-users@xxxxxxxxxxx <geomesa-users@xxxxxxxxxxx>
Cc: zhou lihuang <zlh_0923@xxxxxxxxxxx>
Subject: -EXT-[geomesa-users] problem with query data by spark
 

WARNING:  This message is from an external source.  Evaluate the message carefully BEFORE clicking on links or opening attachments.

hello everyone:

I used RDD Provider to query data, but retrieve 0 data (there are 2 features).
And I used DataStore created by DataStoreFinder.getDataStore，It’s successfully get 2 features.
code is as follows:



The env is :
geomesa: 4.0.5
spark: 3.3.0
hbase: 2.2.0

I've tried modify geomesa version and dependencies version, but it didn't work.

How can I fix this problem now? 

Thank you everyone.

Best Regards, 
Mike 

Back to the top