Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [geomesa-users] Fwd: Re: Batchscanner - error closing output stream because stream is closed 
Andrew:
Yeah, there are some Batch Scans intermingled while the BatchWriter is open. 



On 7/11/17 4:00 PM, Andrew Hulbert wrote:
Are you doing any reading with a BatchScanner during ingest for some reason or only doing BatchWriter action? 


On 07/11/2017 01:46 PM, David Boyd wrote:

I get them really frequently during data ingestion.
I have not seen any issues I can specifically identify but
they are annoying.

I guess I can set my log4j props to not log them.



On 7/11/17 1:25 PM, Anthony Fox wrote:

I think I've seen these exceptions every now and then but there were no
adverse effects on the client side.  I think I chalked it up to
transient network connection issues and timing.


David Boyd <dboyd@xxxxxxxxxxxxxxxxx> writes:


All:
I had sent the original of this to the accumulo list back in April. 
I finally got a chance to look and enable debug logging.

I am curious if anyone on this list has seen the error?




-------- Forwarded Message --------
Subject: Re: Batchscanner - error closing output stream because stream 
is closed
Date:     Mon, 10 Jul 2017 09:59:03 -0400
From:     David Boyd <dboyd@xxxxxxxxxxxxxxxxx>
To:     user@xxxxxxxxxxxxxxxxxxx



All:
Finally, got some time to look into this further. I enabled debug 
logging on accumulo.core.client.   I really
only got one interesting new error which is
java.nio.channels.ClosedByInterruptException - see below:


2017-07-10 13:34:56,106 | DEBUG | [main] |
(AccumuloPersistor.java:1556) - Feature Update Time: 594, Updated: true 
2017-07-10 13:34:56,118 | DEBUG | [main] |
(AccumuloPersistor.java:1441) - Linkage Update Time: 12 Updated: true
2017-07-10 13:34:56,120 |  WARN | [batch scanner 181- 1 looking up 1
ranges at bdpnode6.bdpdev.incadencecorp.com:9997] |
(TIOStreamTransport.java:112) - Error closing output stream.
java.io.IOException: The stream is closed
         at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118) 
         at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) 
         at
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.FilterOutputStream.close(FilterOutputStream.java:158) 
         at
org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110) 
         at
org.apache.thrift.transport.TFramedTransport.close(TFramedTransport.java:89) 
         at
org.apache.accumulo.core.client.impl.ThriftTransportPool$CachedTTransport.close(ThriftTransportPool.java:309) 
         at
org.apache.accumulo.core.client.impl.ThriftTransportPool.returnTransport(ThriftTransportPool.java:571) 
         at
org.apache.accumulo.core.rpc.ThriftUtil.returnClient(ThriftUtil.java:151) 
         at
org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:686) 
         at
org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:349) 
         at
org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
         at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
         at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
         at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35) 
         at java.lang.Thread.run(Thread.java:748)
2017-07-10 13:34:56,121 | DEBUG | [batch scanner 181- 1 looking up 1
ranges at bdpnode6.bdpdev.incadencecorp.com:9997] |
(TabletServerBatchReaderIterator.java:689) - Server :
bdpnode6.bdpdev.incadencecorp.com:9997 msg :
java.nio.channels.ClosedByInterruptException
2017-07-10 13:34:56,121 | DEBUG | [batch scanner 181- 1 looking up 1
ranges at bdpnode6.bdpdev.incadencecorp.com:9997] |
(TabletServerBatchReaderIterator.java:366) -
org.apache.thrift.transport.TTransportException:
java.nio.channels.ClosedByInterruptException
java.io.IOException: org.apache.thrift.transport.TTransportException:
java.nio.channels.ClosedByInterruptException
         at
org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:691) 
         at
org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:349) 
         at
org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
         at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
         at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
         at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35) 
         at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TTransportException:
java.nio.channels.ClosedByInterruptException
         at
org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161) 
         at
org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:158) 
         at
org.apache.accumulo.core.client.impl.ThriftTransportPool$CachedTTransport.flush(ThriftTransportPool.java:320) 
         at
org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)
         at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.send_closeMultiScan(TabletClientService.java:365) 
         at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Client.closeMultiScan(TabletClientService.java:356) 
         at
org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:683) 
         ... 6 more
Caused by: java.nio.channels.ClosedByInterruptException
         at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:478) 
         at
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63) 
         at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) 
         at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159) 
         at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117) 
         at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) 
         at
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
         at
org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159) 
         ... 12 more
2017-07-10 13:34:56,140 | DEBUG | [main] |
(AccumuloPersistor.java:1441) - Linkage Update Time: 22 Updated: true
2017-07-10 13:34:56,152 | DEBUG | [main] |
(AccumuloPersistor.java:1441) - Linkage Update Time: 11 Updated: true
2017-07-10 13:34:56,152 | DEBUG | [batch scanner 183- 1 looking up 1
ranges at bdpnode8.bdpdev.incadencecorp.com:9997] |
(TabletServerBatchReaderIterator.java:689) - Server :
bdpnode8.bdpdev.incadencecorp.com:9997 msg :
java.nio.channels.ClosedByInterruptException
I attempted enabling TRACE level output but got a divide/0 exception in 
one of the TRACE calls (bug was fixed in 1.7.3)

As close as I can tell from the logs and my code, the failure is
happening after I have done all my updates.
No exceptions seem to percolate back to my code.




On 4/19/17 2:56 PM, Keith Turner wrote:
On Wed, Apr 19, 2017 at 12:38 PM, David Boyd <dboyd@xxxxxxxxxxxxxxxxx> wrote: 
All:
I am getting this stack trace periodically based on no pattern I can 
determine from my application.

If you are seeing this regularly, try to enable Accumulo debug
logging.  That may allow you see messages like [1] which may help
understand the cause.
https://github.com/apache/accumulo/blob/rel/1.7.2/core/src/main/java/org/apache/accumulo/core/client/impl/TabletServerBatchReaderIterator.java#L689 


Is this a message I should be worried about?

What is a technique to trace this back to my code and the cause?

Obviously something is closing things before the thread closes it.
2017-04-19 12:33:53,423 | WARN | [batch scanner 19824- 8 looking up 1 ranges at accumulodev:9997] | (TIOStreamTransport.java:112) - Error closing 
output stream.
java.io.IOException: The stream is closed
      at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118) 
      at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at java.io.FilterOutputStream.close(FilterOutputStream.java:158) 
      at
org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:110) 
      at
org.apache.thrift.transport.TFramedTransport.close(TFramedTransport.java:89) 
      at
org.apache.accumulo.core.client.impl.ThriftTransportPool$CachedTTransport.close(ThriftTransportPool.java:309) 
      at
org.apache.accumulo.core.client.impl.ThriftTransportPool.returnTransport(ThriftTransportPool.java:571) 
      at
org.apache.accumulo.core.rpc.ThriftUtil.returnClient(ThriftUtil.java:151) 
      at
org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:686) 
      at
org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:349) at org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57) 
      at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
      at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
      at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35) 
      at java.lang.Thread.run(Thread.java:745)


--
========= mailto:dboyd@xxxxxxxxxxxxxxxxx ============
David W. Boyd
VP,  Data Solutions
10432 Balls Ford, Suite 240
Manassas, VA 20109
office:   +1-703-552-2862
cell:     +1-703-402-7908
============== http://www.incadencecorp.com/ ============
ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture 
Chair ANSI/INCITS TC Big Data
Co-chair NIST Big Data Public Working Group Reference Architecture
First Robotic Mentor - FRC, FTC - www.iliterobotics.org
Board Member- USSTEM Foundation - www.usstem.org

The information contained in this message may be privileged
and/or confidential and protected from disclosure.
If the reader of this message is not the intended recipient
or an employee or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication
is strictly prohibited.  If you have received this communication
in error, please notify the sender immediately by replying to
this message and deleting the material from any computer.



_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit 
https://dev.locationtech.org/mailman/listinfo/geomesa-users




_______________________________________________
geomesa-users mailing list
geomesa-users@xxxxxxxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit 
https://dev.locationtech.org/mailman/listinfo/geomesa-users


--
========= mailto:dboyd@xxxxxxxxxxxxxxxxx ============
David W. Boyd
VP,  Data Solutions
10432 Balls Ford, Suite 240
Manassas, VA 20109
office:   +1-703-552-2862
cell:     +1-703-402-7908
============== http://www.incadencecorp.com/ ============
ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture
Chair ANSI/INCITS TC Big Data
Co-chair NIST Big Data Public Working Group Reference Architecture
First Robotic Mentor - FRC, FTC - www.iliterobotics.org
Board Member- USSTEM Foundation - www.usstem.org

The information contained in this message may be privileged
and/or confidential and protected from disclosure.
If the reader of this message is not the intended recipient
or an employee or agent responsible for delivering this message
to the intended recipient, you are hereby notified that any
dissemination, distribution or copying of this communication
is strictly prohibited.  If you have received this communication
in error, please notify the sender immediately by replying to
this message and deleting the material from any computer.

Back to the top