[geomesa-dev] [JIRA] (GEOMESA-2333) Implement Spark (+ SQL) count to use

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]

[geomesa-dev] [JIRA] (GEOMESA-2333) Implement Spark (+ SQL) count to use datastore count methods

From: "Emilio Lahr-Vivaz (JIRA)" <jira@xxxxxxxxxxxxxxxxxxxxx>
Date: Fri, 20 Jul 2018 14:07:48 +0000 (UTC)
Auto-submitted: auto-generated
Delivered-to: geomesa-dev@xxxxxxxxxxxxxxxx
List-archive: <https://dev.locationtech.org/mhonarc/lists/geomesa-dev>
List-help: <mailto:geomesa-dev-request@locationtech.org?subject=help>
List-subscribe: <https://dev.locationtech.org/mailman/listinfo/geomesa-dev>, <mailto:geomesa-dev-request@locationtech.org?subject=subscribe>
List-unsubscribe: <https://dev.locationtech.org/mailman/options/geomesa-dev>, <mailto:geomesa-dev-request@locationtech.org?subject=unsubscribe>

Title: Message Title

Issue Type:	Improvement
Assignee:	Unassigned
Created:	20/Jul/18 10:07 AM
Priority:	Major
Reporter:	Emilio Lahr-Vivaz

It appears that right now, df.count() is bringing back the results and counting them in spark. We should override this and use our optimized counting (e.g. stats iterator), through the data store API. We should make sure to set the hint to use exact counts.

Add Comment

Get Jira notifications on your phone! Download the Jira Cloud app for Android or iOS

This message was sent by Atlassian Jira

Prev by Date: [geomesa-dev] [JIRA] (GEOMESA-2332) geomesa_pyspark should support Python 3
Next by Date: [geomesa-dev] [JIRA] (GEOMESA-2334) Docs - add example of setting custom FID when saving data frame
Previous by thread: [geomesa-dev] [JIRA] (GEOMESA-2332) geomesa_pyspark should support Python 3
Next by thread: [geomesa-dev] [JIRA] (GEOMESA-2334) Docs - add example of setting custom FID when saving data frame
Index(es):
- Date
- Thread

Breadcrumbs