Home » Archived » Eclipse Communications Framework (ECF) » Sending a big file with FileShare
|
| Re: Sending a big file with FileShare [message #607477 is a reply to message #607474] |
Wed, 26 July 2006 14:09   |
 Scott Lewis
Messages: 1038
Registered: July 2009 |
Senior Member |
|
|
Hi Jaap,
Jaap Reitsma wrote:
> Hi Scott,
>
> We didn't realize it immediately but the combination of sending blocks
> asynchronously and priming the next send in the send method itself means
> that the message queue is very quickly filled with many many receive
> requests (including the byte block to send) queued for sending to the remote
> party. At least I didn't see any built-in limitations. For a big file it
> means that after sometime the sending stops with a connection exception from
> org.eclipse.ecf.provider.comm.tcp.Client.queueObject. The reason of the
> exception we couldn't find, it looks like the receiver on the Client side
> gets an EOFException, we did not understand this behaviour. The receiver
> triggers a disconnect and in turn that will cause a ConnectionException in
> the queueObject, that's what we think. With trace on the error did not
> happen, presumable because the client is then too busy printing all trace
> information, so that it cannot fill-up the buffers fast enough.
>
> Is there some way we can manage the number of blocks pending for
> transmission? Or are we overlooking something?
No this is just a limitation of this relatively simple minded file
sending implementation approach.
>
> Any ideas about the best way to proceed?
I think what would be best would be to introduce a simple flow control
to the protocol (e.g. xon/xoff)...to allow the receiver to indicate when
the sender should pause sending...and when to continue to send.
AND we should simply cap/slow down the sending rate to something more
reasonable for network performance (especially if you are expecting WAN
rates).
How large of file sizes are we talking about, incidently?
Scott
>
> Kind regards,
>
> Jaap
>
>
|
|
|
| Re: Sending a big file with FileShare [message #607478 is a reply to message #607477] |
Wed, 26 July 2006 14:27 |
Jaap Reitsma
Messages: 69
Registered: July 2009 |
Member |
|
|
Hi Scott,
Looking at the code it seems pretty advanced to me though. The files we want
to transfer can have a size of several megabytes (we tested with 14Mb) . Is
sending that kind of files in a peer to peer situation non-intentional use
of the FileShare functionality? We didn't want to introduce another
protocol, but now I am thinking to switch to http for the file transfer.
The fileshare must offer protection for a message producer to prevent
internal sendbuffer overflow as wel as for the message consumer to prevent
receive buffer overflow. I don't know how e.g. JMS implementation handle
this, it might be interesting to take a look.
Kind regards,
Jaap
"Scott Lewis" <slewis@composent.com> wrote in message
news:44C777B2.4070901@composent.com...
> Hi Jaap,
>
> Jaap Reitsma wrote:
>> Hi Scott,
>>
>> We didn't realize it immediately but the combination of sending blocks
>> asynchronously and priming the next send in the send method itself means
>> that the message queue is very quickly filled with many many receive
>> requests (including the byte block to send) queued for sending to the
>> remote party. At least I didn't see any built-in limitations. For a big
>> file it means that after sometime the sending stops with a connection
>> exception from org.eclipse.ecf.provider.comm.tcp.Client.queueObject. The
>> reason of the exception we couldn't find, it looks like the receiver on
>> the Client side gets an EOFException, we did not understand this
>> behaviour. The receiver triggers a disconnect and in turn that will cause
>> a ConnectionException in the queueObject, that's what we think. With
>> trace on the error did not happen, presumable because the client is then
>> too busy printing all trace information, so that it cannot fill-up the
>> buffers fast enough.
>>
>> Is there some way we can manage the number of blocks pending for
>> transmission? Or are we overlooking something?
>
>
> No this is just a limitation of this relatively simple minded file sending
> implementation approach.
>
>
>>
>> Any ideas about the best way to proceed?
>
>
> I think what would be best would be to introduce a simple flow control to
> the protocol (e.g. xon/xoff)...to allow the receiver to indicate when the
> sender should pause sending...and when to continue to send.
>
> AND we should simply cap/slow down the sending rate to something more
> reasonable for network performance (especially if you are expecting WAN
> rates).
>
> How large of file sizes are we talking about, incidently?
>
> Scott
>
>
>>
>> Kind regards,
>>
>> Jaap
>>
|
|
|
| Re: Sending a big file with FileShare [message #607480 is a reply to message #607478] |
Wed, 26 July 2006 15:02 |
Scott Lewis
Messages: 1038
Registered: July 2009 |
Senior Member |
|
|
Hi Jaap,
Jaap Reitsma wrote:
> Hi Scott,
>
> Looking at the code it seems pretty advanced to me though. The files we want
> to transfer can have a size of several megabytes (we tested with 14Mb) . Is
> sending that kind of files in a peer to peer situation non-intentional use
> of the FileShare functionality? We didn't want to introduce another
> protocol, but now I am thinking to switch to http for the file transfer.
You might want to just implement the fileshare sender API using http
PUT. I'll work with you on this if you like. For testing, I built a
file share retriever that uses http GET for retrieving files. Obviously
http has flow control built in, and if you want just point-to-point file
delivery it's a natural protocol, of course. Of course, it means
running an http server but that's not a problem these days.
Also...FYI there is work going on implementing fileshare retriever using
Bitorrent. You may be able to use that as well (don't know).
>
> The fileshare must offer protection for a message producer to prevent
> internal sendbuffer overflow as wel as for the message consumer to prevent
> receive buffer overflow. I don't know how e.g. JMS implementation handle
> this, it might be interesting to take a look.
I don't think JMS has any built in flow control...although it's
possible. I'll try to have a look.
Scott
>
> Kind regards,
>
> Jaap
>
>
>
> "Scott Lewis" <slewis@composent.com> wrote in message
> news:44C777B2.4070901@composent.com...
>> Hi Jaap,
>>
>> Jaap Reitsma wrote:
>>> Hi Scott,
>>>
>>> We didn't realize it immediately but the combination of sending blocks
>>> asynchronously and priming the next send in the send method itself means
>>> that the message queue is very quickly filled with many many receive
>>> requests (including the byte block to send) queued for sending to the
>>> remote party. At least I didn't see any built-in limitations. For a big
>>> file it means that after sometime the sending stops with a connection
>>> exception from org.eclipse.ecf.provider.comm.tcp.Client.queueObject. The
>>> reason of the exception we couldn't find, it looks like the receiver on
>>> the Client side gets an EOFException, we did not understand this
>>> behaviour. The receiver triggers a disconnect and in turn that will cause
>>> a ConnectionException in the queueObject, that's what we think. With
>>> trace on the error did not happen, presumable because the client is then
>>> too busy printing all trace information, so that it cannot fill-up the
>>> buffers fast enough.
>>>
>>> Is there some way we can manage the number of blocks pending for
>>> transmission? Or are we overlooking something?
>>
>> No this is just a limitation of this relatively simple minded file sending
>> implementation approach.
>>
>>
>>> Any ideas about the best way to proceed?
>>
>> I think what would be best would be to introduce a simple flow control to
>> the protocol (e.g. xon/xoff)...to allow the receiver to indicate when the
>> sender should pause sending...and when to continue to send.
>>
>> AND we should simply cap/slow down the sending rate to something more
>> reasonable for network performance (especially if you are expecting WAN
>> rates).
>>
>> How large of file sizes are we talking about, incidently?
>>
>> Scott
>>
>>
>>> Kind regards,
>>>
>>> Jaap
>>>
>
|
|
|
| Re: Sending a big file with FileShare [message #607489 is a reply to message #607480] |
Thu, 10 August 2006 07:14 |
Jaap Reitsma
Messages: 69
Registered: July 2009 |
Member |
|
|
Hi Scott,
For the time being we use now http put/get for file transfer. Due to project
deadlines we did not use the ECF abstractions. At least it works now....
Kind regards,
Jaap
"Scott Lewis" <slewis@composent.com> wrote in message
news:44C7841E.2010505@composent.com...
> Hi Jaap,
>
> Jaap Reitsma wrote:
>> Hi Scott,
>>
>> Looking at the code it seems pretty advanced to me though. The files we
>> want to transfer can have a size of several megabytes (we tested with
>> 14Mb) . Is sending that kind of files in a peer to peer situation
>> non-intentional use of the FileShare functionality? We didn't want to
>> introduce another protocol, but now I am thinking to switch to http for
>> the file transfer.
>
>
> You might want to just implement the fileshare sender API using http PUT.
> I'll work with you on this if you like. For testing, I built a file share
> retriever that uses http GET for retrieving files. Obviously http has
> flow control built in, and if you want just point-to-point file delivery
> it's a natural protocol, of course. Of course, it means running an http
> server but that's not a problem these days.
>
> Also...FYI there is work going on implementing fileshare retriever using
> Bitorrent. You may be able to use that as well (don't know).
>
>
>>
>> The fileshare must offer protection for a message producer to prevent
>> internal sendbuffer overflow as wel as for the message consumer to
>> prevent receive buffer overflow. I don't know how e.g. JMS implementation
>> handle this, it might be interesting to take a look.
>
>
> I don't think JMS has any built in flow control...although it's possible.
> I'll try to have a look.
>
>
> Scott
>
>
>>
>> Kind regards,
>>
>> Jaap
>>
>>
>>
>> "Scott Lewis" <slewis@composent.com> wrote in message
>> news:44C777B2.4070901@composent.com...
>>> Hi Jaap,
>>>
>>> Jaap Reitsma wrote:
>>>> Hi Scott,
>>>>
>>>> We didn't realize it immediately but the combination of sending blocks
>>>> asynchronously and priming the next send in the send method itself
>>>> means that the message queue is very quickly filled with many many
>>>> receive requests (including the byte block to send) queued for sending
>>>> to the remote party. At least I didn't see any built-in limitations.
>>>> For a big file it means that after sometime the sending stops with a
>>>> connection exception from
>>>> org.eclipse.ecf.provider.comm.tcp.Client.queueObject. The reason of the
>>>> exception we couldn't find, it looks like the receiver on the Client
>>>> side gets an EOFException, we did not understand this behaviour. The
>>>> receiver triggers a disconnect and in turn that will cause a
>>>> ConnectionException in the queueObject, that's what we think. With
>>>> trace on the error did not happen, presumable because the client is
>>>> then too busy printing all trace information, so that it cannot fill-up
>>>> the buffers fast enough.
>>>>
>>>> Is there some way we can manage the number of blocks pending for
>>>> transmission? Or are we overlooking something?
>>>
>>> No this is just a limitation of this relatively simple minded file
>>> sending implementation approach.
>>>
>>>
>>>> Any ideas about the best way to proceed?
>>>
>>> I think what would be best would be to introduce a simple flow control
>>> to the protocol (e.g. xon/xoff)...to allow the receiver to indicate when
>>> the sender should pause sending...and when to continue to send.
>>>
>>> AND we should simply cap/slow down the sending rate to something more
>>> reasonable for network performance (especially if you are expecting WAN
>>> rates).
>>>
>>> How large of file sizes are we talking about, incidently?
>>>
>>> Scott
>>>
>>>
>>>> Kind regards,
>>>>
>>>> Jaap
>>>>
>>
|
|
|
| |
  |
Goto Forum:
Current Time: Tue Apr 23 14:01:36 GMT 2024
Powered by FUDForum. Page generated in 0.02994 seconds |
] 
Search
Help
Register
Login
Home
