Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[geomesa-users] Error importing csv into geomesa-redis with nifi

Hi,

I have troubles using the NiFi (1.16.1) processor PutGeoMesaRedisRecord 3.2.0 to ingest csv data to geomesa-redis.

The data are in a comma separated csv with a geometry field and a datetime field, for example (the first 5 lines, my csv contains hundreds rows):
dtg,domain,owner,poly_cid,mean_of_transport,counter,name,geom,length
2022-07-01 10:00:00,co-mil,nifi,436,bike,0,Via_Anselmo_Ronchetti,"LINESTRING (9.2001050000000000 45.4657210000000000, 9.2001460000000000 45.4658880000000000, 9.2001620000000000 45.4661670000000000, 9.2001620000000000 45.4663500000000000, 9.2001650000000000 45.4663540000000000, 9.2001790000000000 45.4666470000000000, 9.2001530000000000 45.4669260000000000)",134.27992849449834
2022-07-01 10:00:00,co-mil,nifi,436,bus,0,Via_Anselmo_Ronchetti,"LINESTRING (9.2001050000000000 45.4657210000000000, 9.2001460000000000 45.4658880000000000, 9.2001620000000000 45.4661670000000000, 9.2001620000000000 45.4663500000000000, 9.2001650000000000 45.4663540000000000, 9.2001790000000000 45.4666470000000000, 9.2001530000000000 45.4669260000000000)",134.27992849449834
2022-07-01 10:00:00,co-mil,nifi,436,car,0,Via_Anselmo_Ronchetti,"LINESTRING (9.2001050000000000 45.4657210000000000, 9.2001460000000000 45.4658880000000000, 9.2001620000000000 45.4661670000000000, 9.2001620000000000 45.4663500000000000, 9.2001650000000000 45.4663540000000000, 9.2001790000000000 45.4666470000000000, 9.2001530000000000 45.4669260000000000)",134.27992849449834
2022-07-01 10:00:00,co-mil,nifi,436,metro,0,Via_Anselmo_Ronchetti,"LINESTRING (9.2001050000000000 45.4657210000000000, 9.2001460000000000 45.4658880000000000, 9.2001620000000000 45.4661670000000000, 9.2001620000000000 45.4663500000000000, 9.2001650000000000 45.4663540000000000, 9.2001790000000000 45.4666470000000000, 9.2001530000000000 45.4669260000000000)",134.27992849449834
2022-07-01 10:00:00,co-mil,nifi,436,pawns,1,Via_Anselmo_Ronchetti,"LINESTRING (9.2001050000000000 45.4657210000000000, 9.2001460000000000 45.4658880000000000, 9.2001620000000000 45.4661670000000000, 9.2001620000000000 45.4663500000000000, 9.2001650000000000 45.4663540000000000, 9.2001790000000000 45.4666470000000000, 9.2001530000000000 45.4669260000000000)",134.27992849449834

Before all, I created a schema on Redis through geomesa-redis command line tool:
./geomesa-redis_2.11-3.2.0/bin/geomesa-redis create-schema -u redis://myuser:mypwd@redis-geomesa:11637 -c simulator -f sim -s dtg:Date,domain:String,owner:String,poly_cid:Integer,mean_of_transport:String,counter:Integer,name:String,geom:LineString:srid=4326,length:Double

After that I have configured the NiFi processor PutGeoMesaRedisRecord 3.2.0:
<properties>
<entry><key>record-reader</key><value>CSVReaderGeomesa</value></entry>
<entry><key>feature-type-name</key><value>sim</value></entry>
<entry><key>feature-id-col</key><value>${geomesa.id.col}</value></entry>
<entry><key>geometry-cols</key><value>geom:LineString:srid=4326</value></entry>
<entry><key>geometry-serialization</key><value>WKT</value></entry>
<entry><key>json-cols</key><value>${geomesa.json.cols}</value></entry>
<entry><key>default-date-col</key><value>${geomesa.default.dtg.col}</value></entry>
<entry><key>visibilities-col</key><value>${geomesa.visibilities.col}</value></entry>
<entry><key>schema-compatibility</key><value><name>existing</name></value></entry>
<entry><key>write-mode</key><value><name>append</name></value></entry>

<entry><key>redis.url</key><value>redis://myuser:mypwd@redis-geomesa:11637</value></entry>
<entry><key>redis.catalog</key><value>simulator</value></entry>
...
</properties>

And the CSVReaderGeomesa controller service:
<properties>
...
<entry><key>schema-access-strategy</key><value>infer-schema</value></entry>
<entry><key>csv-reader-csv-parser</key><value>commons-csv</value></entry>
<entry><key>Date Format</key><value>yyyy-MM-dd HH:mm:ss</value></entry>
<entry><key>Time Format</key><value>HH:mm:ss</value></entry>
<entry><key>Timestamp Format</key><value>yyyy-MM-dd HH:mm:ss</value></entry>
<entry><key>CSV Format</key><value>custom</value></entry>
<entry><key>Value Separator</key><value>,</value></entry>
<entry><key>Record Separator</key><value>\n</value></entry>
<entry><key>Skip Header Line</key><value>true</value></entry>
<entry><key>ignore-csv-header</key><value>true</value></entry>
<entry><key>Quote Character</key><value>"</value></entry>
<entry><key>Escape Character</key><value>\</value></entry>
...
</properties>

So, when NiFi loads and processes the csv the PutGeoMesaRedisRecord give me an error:
java.lang.IllegalArgumentException: requirement failed: Z shards must be between 1 and 127

If I modify the csv with only 1 line, it works.

I don't understand if the error was in the schema creation on Redis or in the NiFi config processor or in the csv data?


--
Michele Andreoli


Back to the top