Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Eclipse Communications Framework (ECF) » Sending a big file with FileShare
Sending a big file with FileShare [message #607474] Wed, 26 July 2006 13:49 Go to next message
Jaap Reitsma is currently offline Jaap ReitsmaFriend
Messages: 69
Registered: July 2009
Member
Hi Scott,

We didn't realize it immediately but the combination of sending blocks
asynchronously and priming the next send in the send method itself means
that the message queue is very quickly filled with many many receive
requests (including the byte block to send) queued for sending to the remote
party. At least I didn't see any built-in limitations. For a big file it
means that after sometime the sending stops with a connection exception from
org.eclipse.ecf.provider.comm.tcp.Client.queueObject. The reason of the
exception we couldn't find, it looks like the receiver on the Client side
gets an EOFException, we did not understand this behaviour. The receiver
triggers a disconnect and in turn that will cause a ConnectionException in
the queueObject, that's what we think. With trace on the error did not
happen, presumable because the client is then too busy printing all trace
information, so that it cannot fill-up the buffers fast enough.

Is there some way we can manage the number of blocks pending for
transmission? Or are we overlooking something?

Any ideas about the best way to proceed?

Kind regards,

Jaap
Re: Sending a big file with FileShare [message #607477 is a reply to message #607474] Wed, 26 July 2006 14:09 Go to previous messageGo to next message
Scott Lewis is currently offline Scott LewisFriend
Messages: 1038
Registered: July 2009
Senior Member
Hi Jaap,

Jaap Reitsma wrote:
> Hi Scott,
>
> We didn't realize it immediately but the combination of sending blocks
> asynchronously and priming the next send in the send method itself means
> that the message queue is very quickly filled with many many receive
> requests (including the byte block to send) queued for sending to the remote
> party. At least I didn't see any built-in limitations. For a big file it
> means that after sometime the sending stops with a connection exception from
> org.eclipse.ecf.provider.comm.tcp.Client.queueObject. The reason of the
> exception we couldn't find, it looks like the receiver on the Client side
> gets an EOFException, we did not understand this behaviour. The receiver
> triggers a disconnect and in turn that will cause a ConnectionException in
> the queueObject, that's what we think. With trace on the error did not
> happen, presumable because the client is then too busy printing all trace
> information, so that it cannot fill-up the buffers fast enough.
>
> Is there some way we can manage the number of blocks pending for
> transmission? Or are we overlooking something?


No this is just a limitation of this relatively simple minded file
sending implementation approach.


>
> Any ideas about the best way to proceed?


I think what would be best would be to introduce a simple flow control
to the protocol (e.g. xon/xoff)...to allow the receiver to indicate when
the sender should pause sending...and when to continue to send.

AND we should simply cap/slow down the sending rate to something more
reasonable for network performance (especially if you are expecting WAN
rates).

How large of file sizes are we talking about, incidently?

Scott


>
> Kind regards,
>
> Jaap
>
>
Re: Sending a big file with FileShare [message #607478 is a reply to message #607477] Wed, 26 July 2006 14:27 Go to previous messageGo to next message
Jaap Reitsma is currently offline Jaap ReitsmaFriend
Messages: 69
Registered: July 2009
Member
Hi Scott,

Looking at the code it seems pretty advanced to me though. The files we want
to transfer can have a size of several megabytes (we tested with 14Mb) . Is
sending that kind of files in a peer to peer situation non-intentional use
of the FileShare functionality? We didn't want to introduce another
protocol, but now I am thinking to switch to http for the file transfer.

The fileshare must offer protection for a message producer to prevent
internal sendbuffer overflow as wel as for the message consumer to prevent
receive buffer overflow. I don't know how e.g. JMS implementation handle
this, it might be interesting to take a look.

Kind regards,

Jaap



"Scott Lewis" <slewis@composent.com> wrote in message
news:44C777B2.4070901@composent.com...
> Hi Jaap,
>
> Jaap Reitsma wrote:
>> Hi Scott,
>>
>> We didn't realize it immediately but the combination of sending blocks
>> asynchronously and priming the next send in the send method itself means
>> that the message queue is very quickly filled with many many receive
>> requests (including the byte block to send) queued for sending to the
>> remote party. At least I didn't see any built-in limitations. For a big
>> file it means that after sometime the sending stops with a connection
>> exception from org.eclipse.ecf.provider.comm.tcp.Client.queueObject. The
>> reason of the exception we couldn't find, it looks like the receiver on
>> the Client side gets an EOFException, we did not understand this
>> behaviour. The receiver triggers a disconnect and in turn that will cause
>> a ConnectionException in the queueObject, that's what we think. With
>> trace on the error did not happen, presumable because the client is then
>> too busy printing all trace information, so that it cannot fill-up the
>> buffers fast enough.
>>
>> Is there some way we can manage the number of blocks pending for
>> transmission? Or are we overlooking something?
>
>
> No this is just a limitation of this relatively simple minded file sending
> implementation approach.
>
>
>>
>> Any ideas about the best way to proceed?
>
>
> I think what would be best would be to introduce a simple flow control to
> the protocol (e.g. xon/xoff)...to allow the receiver to indicate when the
> sender should pause sending...and when to continue to send.
>
> AND we should simply cap/slow down the sending rate to something more
> reasonable for network performance (especially if you are expecting WAN
> rates).
>
> How large of file sizes are we talking about, incidently?
>
> Scott
>
>
>>
>> Kind regards,
>>
>> Jaap
>>
Re: Sending a big file with FileShare [message #607480 is a reply to message #607478] Wed, 26 July 2006 15:02 Go to previous messageGo to next message
Scott Lewis is currently offline Scott LewisFriend
Messages: 1038
Registered: July 2009
Senior Member
Hi Jaap,

Jaap Reitsma wrote:
> Hi Scott,
>
> Looking at the code it seems pretty advanced to me though. The files we want
> to transfer can have a size of several megabytes (we tested with 14Mb) . Is
> sending that kind of files in a peer to peer situation non-intentional use
> of the FileShare functionality? We didn't want to introduce another
> protocol, but now I am thinking to switch to http for the file transfer.


You might want to just implement the fileshare sender API using http
PUT. I'll work with you on this if you like. For testing, I built a
file share retriever that uses http GET for retrieving files. Obviously
http has flow control built in, and if you want just point-to-point file
delivery it's a natural protocol, of course. Of course, it means
running an http server but that's not a problem these days.

Also...FYI there is work going on implementing fileshare retriever using
Bitorrent. You may be able to use that as well (don't know).


>
> The fileshare must offer protection for a message producer to prevent
> internal sendbuffer overflow as wel as for the message consumer to prevent
> receive buffer overflow. I don't know how e.g. JMS implementation handle
> this, it might be interesting to take a look.


I don't think JMS has any built in flow control...although it's
possible. I'll try to have a look.


Scott


>
> Kind regards,
>
> Jaap
>
>
>
> "Scott Lewis" <slewis@composent.com> wrote in message
> news:44C777B2.4070901@composent.com...
>> Hi Jaap,
>>
>> Jaap Reitsma wrote:
>>> Hi Scott,
>>>
>>> We didn't realize it immediately but the combination of sending blocks
>>> asynchronously and priming the next send in the send method itself means
>>> that the message queue is very quickly filled with many many receive
>>> requests (including the byte block to send) queued for sending to the
>>> remote party. At least I didn't see any built-in limitations. For a big
>>> file it means that after sometime the sending stops with a connection
>>> exception from org.eclipse.ecf.provider.comm.tcp.Client.queueObject. The
>>> reason of the exception we couldn't find, it looks like the receiver on
>>> the Client side gets an EOFException, we did not understand this
>>> behaviour. The receiver triggers a disconnect and in turn that will cause
>>> a ConnectionException in the queueObject, that's what we think. With
>>> trace on the error did not happen, presumable because the client is then
>>> too busy printing all trace information, so that it cannot fill-up the
>>> buffers fast enough.
>>>
>>> Is there some way we can manage the number of blocks pending for
>>> transmission? Or are we overlooking something?
>>
>> No this is just a limitation of this relatively simple minded file sending
>> implementation approach.
>>
>>
>>> Any ideas about the best way to proceed?
>>
>> I think what would be best would be to introduce a simple flow control to
>> the protocol (e.g. xon/xoff)...to allow the receiver to indicate when the
>> sender should pause sending...and when to continue to send.
>>
>> AND we should simply cap/slow down the sending rate to something more
>> reasonable for network performance (especially if you are expecting WAN
>> rates).
>>
>> How large of file sizes are we talking about, incidently?
>>
>> Scott
>>
>>
>>> Kind regards,
>>>
>>> Jaap
>>>
>
Re: Sending a big file with FileShare [message #607489 is a reply to message #607480] Thu, 10 August 2006 07:14 Go to previous message
Jaap Reitsma is currently offline Jaap ReitsmaFriend
Messages: 69
Registered: July 2009
Member
Hi Scott,

For the time being we use now http put/get for file transfer. Due to project
deadlines we did not use the ECF abstractions. At least it works now....

Kind regards,

Jaap

"Scott Lewis" <slewis@composent.com> wrote in message
news:44C7841E.2010505@composent.com...
> Hi Jaap,
>
> Jaap Reitsma wrote:
>> Hi Scott,
>>
>> Looking at the code it seems pretty advanced to me though. The files we
>> want to transfer can have a size of several megabytes (we tested with
>> 14Mb) . Is sending that kind of files in a peer to peer situation
>> non-intentional use of the FileShare functionality? We didn't want to
>> introduce another protocol, but now I am thinking to switch to http for
>> the file transfer.
>
>
> You might want to just implement the fileshare sender API using http PUT.
> I'll work with you on this if you like. For testing, I built a file share
> retriever that uses http GET for retrieving files. Obviously http has
> flow control built in, and if you want just point-to-point file delivery
> it's a natural protocol, of course. Of course, it means running an http
> server but that's not a problem these days.
>
> Also...FYI there is work going on implementing fileshare retriever using
> Bitorrent. You may be able to use that as well (don't know).
>
>
>>
>> The fileshare must offer protection for a message producer to prevent
>> internal sendbuffer overflow as wel as for the message consumer to
>> prevent receive buffer overflow. I don't know how e.g. JMS implementation
>> handle this, it might be interesting to take a look.
>
>
> I don't think JMS has any built in flow control...although it's possible.
> I'll try to have a look.
>
>
> Scott
>
>
>>
>> Kind regards,
>>
>> Jaap
>>
>>
>>
>> "Scott Lewis" <slewis@composent.com> wrote in message
>> news:44C777B2.4070901@composent.com...
>>> Hi Jaap,
>>>
>>> Jaap Reitsma wrote:
>>>> Hi Scott,
>>>>
>>>> We didn't realize it immediately but the combination of sending blocks
>>>> asynchronously and priming the next send in the send method itself
>>>> means that the message queue is very quickly filled with many many
>>>> receive requests (including the byte block to send) queued for sending
>>>> to the remote party. At least I didn't see any built-in limitations.
>>>> For a big file it means that after sometime the sending stops with a
>>>> connection exception from
>>>> org.eclipse.ecf.provider.comm.tcp.Client.queueObject. The reason of the
>>>> exception we couldn't find, it looks like the receiver on the Client
>>>> side gets an EOFException, we did not understand this behaviour. The
>>>> receiver triggers a disconnect and in turn that will cause a
>>>> ConnectionException in the queueObject, that's what we think. With
>>>> trace on the error did not happen, presumable because the client is
>>>> then too busy printing all trace information, so that it cannot fill-up
>>>> the buffers fast enough.
>>>>
>>>> Is there some way we can manage the number of blocks pending for
>>>> transmission? Or are we overlooking something?
>>>
>>> No this is just a limitation of this relatively simple minded file
>>> sending implementation approach.
>>>
>>>
>>>> Any ideas about the best way to proceed?
>>>
>>> I think what would be best would be to introduce a simple flow control
>>> to the protocol (e.g. xon/xoff)...to allow the receiver to indicate when
>>> the sender should pause sending...and when to continue to send.
>>>
>>> AND we should simply cap/slow down the sending rate to something more
>>> reasonable for network performance (especially if you are expecting WAN
>>> rates).
>>>
>>> How large of file sizes are we talking about, incidently?
>>>
>>> Scott
>>>
>>>
>>>> Kind regards,
>>>>
>>>> Jaap
>>>>
>>
Previous Topic:How to get XMPPConnection from XMPPClientSOContainer
Next Topic:Query adapters without creating a container?
Goto Forum:
  


Current Time: Wed Aug 12 20:56:25 GMT 2020

Powered by FUDForum. Page generated in 0.02482 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top