This project has moved and is read-only. For the latest updates, please go here.

Nito Async Library can support large file transfer

Jun 26, 2010 at 8:37 AM

1) i want to transfer large files upto 1 GB, can Nito Sync Library support to transfer large files from client to server and vise versa,

if not, then i used message framing to break large files into small frames and the send from client to server (i suggest) .

2) also in some scenerion,i want to send message to server, the network link between client and server is down for a interval, client request a connection server and it fails to connect, now i want that client can request upo 5 times to connect to server, is it possible using Nito Async Library

3) Now Client to connected to server, clinet request the server, but the request can not received by server completely for some reason, i want that server send a response that request not properly received and client send this request again.

 

 

Jun 26, 2010 at 5:25 PM

Please be aware that Nito.Async.Sockets are thread-safe wrappers for the .NET Socket class. There is nothing in this library that handles "syncing" of files directly.

1) Nito.Async has the same capabilities and limitations of regular TCP/IP sockets. You can send any size of file, but you'll need to implement message framing into your protocol. The "simple" sockets define a protocol with message framing that allows messages up to 2 GB.

2) You can retry a connection attempt as many times as you'd like. There is no built-in retry functionality in Nito.Async.

3) If a client sends a message to the server, then that message will either be received or it will fail. Failed messages will result in the socket being closed. You can connect again if you wish; again, there is no built-in functionality for this in Nito.Async.

       -Steve

Jun 29, 2010 at 8:19 AM
thanks for your reply Steve. now i have two options 1) break large file in small chunk, and transfer using Nito.Async library, and on ther server side receive chunk and write every chunk to a seperate file on the serve, when all chunk received, merge all chunk into a single file which is actual file to be transfered from client. 2) write thread safe wrapper for .net socket function BeginSendFile() now i have a question, which of the above approach is better, in my scenerion my clients may on be on GPRS connection this situation may be considerable as compare to the clients that are on braodband link or some high speed link. waiting for your reply thanks.
Jun 29, 2010 at 4:53 PM

BeginSendFile does have a speed advantage; it is able to send the file data directly from the file system driver to the network driver without copying it into a user-mode buffer in between. It is up to you: sending the file (in chunks) may work better with your protocol, and it would allow the injection of tracing later. So, sending it through your protocol may have some usability benefits, while BeginSendFile has a speed benefit.

      -Steve

Jul 2, 2010 at 8:39 AM
Thanks Steve for your replies. as you replied BeginSendFile have a speed advantage, i wrote test client and server application for file transfer mechanism. i follow the same approach that is mention in Nito.Async Library i.e., send file data followed by data size, it can transfer file upto 24MB either client and server running on same machine or separately, but problem occured when i am trying to send file larger than 24MB, an exception raised in server application with this message "An operation on a socket could not be performed because the system lacked suffic ient buffer space or because a queue was full" as Nito.Async library claimed that it can transfer data upto 2GB, i think this problem may occur when using Nito.Asyn to send large string data. what is the cause of this error and hoe resolve error. waiting for your reply. Thanks
Jul 2, 2010 at 12:33 PM
Edited Jul 6, 2010 at 1:52 PM

The protocol used by the Nito.Async library allows message sizes of up to 2 GB. However, operating system support for messages this large varies. I described some of the OS limitations on my blog a few months ago: http://nitoprograms.blogspot.com/2010/03/io-limitation-in-windows.html

The short answer is that large files should be split into chunks. This is certainly required if you need to support pre-Vista operating systems. As a general rule, it's a good idea to do so anyway.

       -Steve

Jul 5, 2010 at 1:26 PM

Thanks Steve for your reply to queries.

 

 

now i decide to transfer large file in chunks, but there are some questions

 

to transfer large file

 

1) opne file using some I/O classes like BinaryReader, establish connection to server, read desired amount of data from opend file,  send chunk data to server until all data transfered to server and close connection to server, while server create or open file, received chunks data, write data to file. (in this case a single connection to server used to data transfer)

 

2) open file, read desired chunk size data, then create connection to server, send chunk data with some additonal information like chunk index, etc., on server side, server received chunk data write chunk data to temp file with indexing (abc.exe.0,abc.exe.1 ,  abc.exe.2.E) when server received last chunk (chunk index 'E'), it read all temp file and merge all temp file.

 

for example client read file

 

 

 

 

int iBufferSize = 20480; 

objBinReader = new BinaryReader(File.Open(sCurrentFileName, FileMode.Open));

if ((iBufferSize * iTotalSentDataSlot) < ObjBinReader.BaseStream.Length)
{
    sAppendRequest = "";
    if (iBufferSize * (iTotalSentDataSlot + 1) < ObjBinReader.BaseStream.Length)
    {
 ObjBinReader.Read(bSentData, 0, iBufferSize);
 
    }
    else
    {
 int iRemLen = (int)ObjBinReader.BaseStream.Length - iBufferSize * iTotalSentDataSlot;
 bSentData = new byte[iRemLen];
 ObjBinReader.Read(bSentData, 0, iRemLen);
    }
   
    here call nito.async library function to send data
    iTotalSentDataSlot++;

 

 

 

 

 

 

 

which of the two approach is better

 

Jul 5, 2010 at 10:36 PM

A lot of protocol design decisions don't have a clear "better" answer. In this case, the first approach has the benefit of simplicity, while the second approach could allow a failed transfer to retry, only sending the data that wasn't correctly received.

I would lean towards the first approach unless you really need the partial transfer recovery that the second approach allows.

       -Steve

Jul 6, 2010 at 8:18 AM

Thanks Steve,

Now i decide to send file in chunks with second approach.

i've another question, is it good approach to write same server application that hanlde both normal messages and file or both are handled by separate applications

we have our component written in c# for message transfering, our own framework take string and build string request for the server, convert string request to byte[] array and transfer to server,  if i used same application for file transfer and message transfer, first read file data in byte array, convert it to Base64String, then append it to request and then convert request to byte[] for transfering to server

as i think both message receiving server and  file receiving server are separate application.

i need your suggession.

thanks

Jul 6, 2010 at 1:50 PM
Edited Jul 6, 2010 at 1:51 PM

I've written a few things concerning application protocol design. The important parts are in my TCP/IP FAQ here:
  http://nitoprograms.blogspot.com/2009/04/tcpip-net-sockets-faq.html

I also presented a talk on application protocol design last year at a local BarCamp; the slides are here:
  http://barcampgr.org/wiki/BarCampGrandRapids4Presentations

Of course, any particular decisions such as what you're asking depend on your specific application requirements; those decisions are really up to you.

       -Steve

Jul 12, 2010 at 11:47 AM

hi Steve,

now i've some more question about nito.async library, as client/server  sample applications provided with library, all are windows applications, in which human interaction involved for connection establishment, sending messages to server. i want to write an application using nito.async  library in which i want to establish a coonection to server using  ClinetTcpSocket.ConnectAsync() function and also pass the message that will be send to server when connection established successfully in connect_completed callback function (Connect_Completed will call the SocketPacketProtocol.WritePacketAsync). ConnectAsync with oveload does not support to pass the message that will be send to server. how i accomplish this task, i'm only developer in my office to accomplish this task, please help me.

 

 

 

 

 

Jul 12, 2010 at 1:53 PM

In order to send an initial message, handle the ConnectCompleted event and send the message then.

       -Steve

Jul 12, 2010 at 2:19 PM

thanks for your reply.

i want to send messages without human interference. let say in a single click or when application starts, application connect to server, connection established i.e., in connect_comlplete send message to server, and wait for response in packet_arrived . all is done in a single cycle. now when connect_completed fire, its don't know what to send, how i inform connect_completed that a particular message is to be send to the server which is set before connection to server.

thanks.

Jul 12, 2010 at 4:49 PM

You send the message from within your ConnectCompleted handler. If you need to store the message before connecting to the server, use a variable.

        -Steve

Jul 16, 2010 at 10:55 AM
Edited Jul 16, 2010 at 1:30 PM

hi steve,
thanks for your cooperation, but i think i can't expalin my problen in detail. now i explain in detail.
we have currently used message based architecture, and have a library for this purpose and this library work as follows. we call any process a transaction and its parameter is written in a xml file with its data type, length and start index.
1. our front end web application / windows application or windows service initiates a process. a process may have multiple transactions.
2. to initiates a process, application pass request to library class1 function class1.ProcessTransaction(string) in the format like query string in web application e.g. FristName=FirstName&LastName=LastName&Designation=Designation&Company=Company
3. this request is passed to class2. class2.somefunction() has switch case for different type of transaction nature either local database transaction , remote server transaction etc.
 class2.somefunction calls appropriate function depending upon transaction nature and generate XmlDocument based on result of the transaction
 in case of remote server transaction class2.somefunction(XmlDocument , string) call class3.somefunction(XmlDocument , string) .
4. class3 is responsible for generating formatted request of fixed length as defined in a XML configuration file let say formatted request based on XML file of lenght 100 is "FirstName  LastName  Designation  Company  ".
 class3.somefunction call class4.ProcessRemoteTransaction(string), this function establish connection to remote server, send request, and receive response from server,
 and return back response to class3.someFunction(XmlDocument , string). now class3.somefunction utilize response and create XmlNodes in XmlDocument with our specific pattern and return back to class2.somefunction.
 if more transaction are be performed in this process class2.somefunction() call their respected function based on transaction nature to complete a single process.
 when all transaction all completed and a single XmlDocument generated depending upon all transaction results, control back to application and application display the result to user GUI. 


as client/server sample project provided with async library they are all windows form application and all event handler ClientSocket_ConnectCompleted, ClientSocket_WriteCompleted, ClientSocket_PacketArrived are defined at form level.
now i want to embed Nito.Async in my existing library, and my library function will call Nito.Async function. how i embed this, where event handler ClientSocket_ConnectCompleted, ClientSocket_WriteCompleted, ClientSocket_PacketArrived defined either at application level or at our Library level.

application call --> our Existing Library function call --> Nito.Async Library

please guide me how i accomplish this task. if you can help by some flow chart it can help me. also tell me how to use this library in web applications like lowerlevelwebclient and server
thanks
 

 

 

Jul 16, 2010 at 2:36 PM

It sounds like your architecture structure is assuming that communication with the server will be synchronous. Nito.Async (as the name implies) is all about asynchronous systems.

To be more specific, this part won't work well: "class4.ProcessRemoteTransaction(string), this function establish connection to remote server, send request, and receive response from server"

There is no built-in support in Nito.Async for waiting for a response.

Nito.Async would work better in an architecture where class2 would set up a sort of queue of operations to do (and then return), the messages would get sent out, and the operations marked as finished when the appropriate responses arrived. When the queue is empty, then a notification is sent up to class1 / the UI.

       -Steve