Binary File Transfer 'handshake'

[nigelr]

Hi,

I am trying to make my pda<>pc file transfer 100% robust so have been playing around with sending a confirmation message back to the pc after transfer is complete. The basic procedure for sending data to the pda from 'server' is as follows (all initially based on file transfer example code):

Server received request for data then sends a file as follows:
1. Send one byte to start
2. Send file size
3. Send file name
4. Send file contents
5. Read back confirmation of file size received from pda

The client sends a 'request' byte to start then waits for data from the server as follows:
1. Read byte to start
2. Read file size
3. Read file name
4. Read file contents
5. Write back file size of received file

This is fine in principle but in practice, what happens very often is that the server has finished transmitting the file contents and sits there waiting for the confirmation value. I have implemented a 'timeout' whilst waiting but this is regularly more than 10 seconds before the pda responds. This means that although the file has transferred correctly, the server thinks it hasn't because no confirmation was received in the time given limit. I could just increase this but surely 10 seconds is more than enough? Why is there such a delay?

I started playing around with the buffer size but my head is not calm enough to work out what the best size would be. The largest file size I have at the moment for testing is 78kB and in reality they would not be a lot larger than this.

Does anyone have any ideas on how to make this reliable or if there's a better way of ensuring data that is sent is received and confirmed the other end? I need to make sure that loss of signal / transmission errors are picked up somehow so there's absolutely no loss of data possible.

Thanks in advance

 

[agraham]

I don't know why there is that delay but you don't say how you are actually sending the data? The Network library uses TCP/IP which should be reliable, are you using that?

EDIT:- Sorry, missed the file transfer example reference. Are you using an ActiveSync connection or just your network?

 

[nigelr]

Network

Hi,
I'm using network connection which works perfectly over a lan but when testing on a pda I get the timeout issue. I may not have mentioned - it's not every time I get this problem but once is enough! Any ideas welcome.
Thanks

 

[derez]

If this may be of help - check this application here http://www.b4x.com/forum/share-your...chat-file-transfer-application.html#post19297 - it is also based on the help example, and works on all types of tcp\ip networks that I could test on :
- PPC to PPC using Wifi
- two computers on lan with router
- ppc to pc using Activesync

I had no experience with delays as you describe.

 

[nigelr]

Readbytes Confusion? Would love an Explanation please.

Thanks for the reply.. looked at your chat application and I don't see anything radically different to my file transfer code. One major cause of curiosity to me is the readbytes command. I have included extra stages in my logs to see what's going on and if you look at the log examples below you'll see...

1. When sending a file from the server, the data is sent in blocks as set by my buffer size constant (in this case 32768 bytes).
2. When a desktop client is receiving, it reads in the same size blocks and works fine.
3. When a PPC client is receiving, it's reading in smaller blocks, normally 1360 bytes but this can be larger.
4. The client is still reading the small blocks of data when the server is waiting for a confirmation message which times-out .. even after 20 seconds! This means the server thinks there's been a problem yet the client completes the reading and thinks everything is ok! This is my major problem.

Can anyone explain this.. I'm running the same client code in both cases. Does my buffer size need to be a lot smaller (I've tried various values, 4096, 8192 16384 etc.)?

Thanks.

1. Server sending to desktop client

   Server Log:
   10/09 09:52:01,srv,started
   10/09 09:52:09,srv,Client Connected
   10/09 09:52:10,srv,File Request received
   10/09 09:52:10,srv,Sending... consumer.csv
   10/09 09:52:10,xxx,writing 79181
   10/09 09:52:10,xxx,writing 46413
   10/09 09:52:10,xxx,writing 13645
   10/09 09:52:10,xxx,waiting for confirmation data
   10/09 09:52:10,srv,OK

   Desktop Client Log:
   10/09 09:51,2,started
   10/09 09:51,2,Connected to server
   10/09 09:51,2,Requesting...
   10/09 09:51,2,Receiving... consumer.csv (79181)
   10/09 09:51,2,Deleting existing file consumer.csv
   10/09 09:51,xxx,reading 79181
   10/09 09:51,xxx,reading 46413
   10/09 09:51,xxx,reading 13645
   10/09 09:51,xxx,writing confirmation 
   10/09 09:51,2,OK


2. Server sending to PPC Client

   Server Log:
   10/09 10:04:23,srv,Client Connected
   10/09 10:04:23,srv,File Request received
   10/09 10:04:23,srv,Sending... consumer.csv
   10/09 10:04:23,xxx,writing 79181
   10/09 10:04:24,xxx,writing 46413
   10/09 10:04:25,xxx,writing 13645
   10/09 10:04:25,xxx,waiting for confirmation data
   10/09 10:04:26,xxx,waiting for confirmation data
   10/09 10:04:27,xxx,waiting for confirmation data
   10/09 10:04:28,xxx,waiting for confirmation data
   10/09 10:04:29,xxx,waiting for confirmation data
   10/09 10:04:30,xxx,waiting for confirmation data
   10/09 10:04:31,xxx,waiting for confirmation data
   10/09 10:04:32,xxx,waiting for confirmation data
   10/09 10:04:33,xxx,waiting for confirmation data
   10/09 10:04:34,xxx,waiting for confirmation data
   10/09 10:04:35,xxx,waiting for confirmation data
   10/09 10:04:36,xxx,waiting for confirmation data
   10/09 10:04:37,xxx,waiting for confirmation data
   10/09 10:04:38,xxx,waiting for confirmation data
   10/09 10:04:39,xxx,waiting for confirmation data
   10/09 10:04:40,xxx,waiting for confirmation data
   10/09 10:04:41,xxx,waiting for confirmation data
   10/09 10:04:41,xxx,waiting for confirmation data
   10/09 10:04:43,xxx,waiting for confirmation data
   10/09 10:04:44,xxx,waiting for confirmation data
   10/09 10:04:45,xxx,waiting for confirmation data
   10/09 10:04:45,srv,Error (confirm timeout)
   10/09 10:04:45,srv,Error-Exit sending loop
   10/09 10:04:46,srv,Connection ended

   PPC Client Log:
   10/09 10:03,2,started
   10/09 10:03,2,Connected to server
   10/09 10:03,2,Requesting...
   10/09 10:03,2,Receiving... consumer.csv (79181)
   10/09 10:03,2,Deleting existing file consumer.csv
   10/09 10:03,xxx,reading 79181
   10/09 10:03,xxx,reading 77842
   10/09 10:03,xxx,reading 76482
   10/09 10:03,xxx,reading 75122
   10/09 10:03,xxx,reading 73762
   10/09 10:04,xxx,reading 72402
   10/09 10:04,xxx,reading 71042
   10/09 10:04,xxx,reading 69682
   10/09 10:04,xxx,reading 68322
   10/09 10:04,xxx,reading 66962
   10/09 10:04,xxx,reading 65602
   10/09 10:04,xxx,reading 64242
   10/09 10:04,xxx,reading 41122
   10/09 10:04,xxx,reading 39762
   10/09 10:04,xxx,reading 38402
   10/09 10:04,xxx,reading 37042
   10/09 10:04,xxx,reading 35682
   10/09 10:04,xxx,reading 34322
   10/09 10:04,xxx,reading 31602
   10/09 10:04,xxx,reading 30242
   10/09 10:04,xxx,reading 28882
   10/09 10:04,xxx,reading 27522
   10/09 10:04,xxx,reading 26162
   10/09 10:04,xxx,reading 24802
   10/09 10:04,xxx,reading 23442
   10/09 10:04,xxx,reading 22082
   10/09 10:04,xxx,reading 20722
   10/09 10:04,xxx,reading 19362
   10/09 10:04,xxx,reading 18002
   10/09 10:04,xxx,reading 16642
   10/09 10:04,xxx,reading 15282
   10/09 10:04,xxx,reading 13922
   10/09 10:04,xxx,reading 12562
   10/09 10:04,xxx,reading 9842
   10/09 10:04,xxx,reading 8482
   10/09 10:04,xxx,reading 7122
   10/09 10:04,xxx,reading 5762
   10/09 10:04,xxx,reading 4402
   10/09 10:04,xxx,reading 3042
   10/09 10:04,xxx,reading 1682
   10/09 10:04,xxx,reading 322
   10/09 10:04,xxx,writing confirmation 
   10/09 10:04,2,OK

 

[Erel]

I've used a code similar to the FileTransfer code to transfer a 150mb file to the device and it worked fine.
The TCP protocol which is the protocol used by the Network library is supposed to do similar checks and resend packets if required.
If the connection get broken you should get an exception (eventually). I'm not sure that your procedure will really make the transfer more robust.

 

[moster67]

Erel:

Wow! You managed to transfer a 150mb file to the device! Was this over a stable active sync connection or over a TCP/IP-network (using wireless)?

I have been experimenting a bit with socket-programming and I have problems with transferring large files (2-3MB) which very often gives me out-of-memory exceptions on the device. How did you that?

I mean, any time we're working with larger amounts of data, one wants to make sure one doesn't hold the whole thing in memory at once. So I guess, since the download pipe is a stream, you loaded the data in chunks and at the time of writing the file to disk (storage I presume), you wrote it a chunk at a time.

Erel, can you give some example code?

I've used a code similar to the FileTransfer code to transfer a 150mb file to the device and it worked fine.

nigelr: sorry for hijacking your thread

 

[nigelr]

Confirmation of Receipt?

No problem moster67

I'd like to know Erel.. you say my procedure would proably not make the transfer more robust but without any sort of check after completion, how does the client knpw the data has got there completely?

It seems the client can send all the data but the server not read it all in? This is my experience but if anyone knows better I'd gladly listen. My app must know if the data has been received at the other end,

Can we clarify exactly what's going on in the background? I'd really like to understand the results of my log entries previously shown in this thread.

Thanks

 

[Erel]

@moster67, did you try this code: http://www.b4x.com/files/FileTransfer.zip
I don't remember if it was Activesync or a wireless connection. I guess that a regular wireless connection should be more stable then Activesync.

@nigelr, The TCP protocol is supposed to do these checks for you.
The actual implementation is part of the .Net Framework not Basic4ppc, so I can't really explain the actual numbers.
One thing that you can do is to let the client send a message (can be one byte) to the server when the file transfer was finished. On the server side read this message after sending the file. The server will be blocked on this reading until the message arrive.

 

[nigelr]

Already tried that with probelmatic results..

One thing that you can do is to let the client send a message (can be one byte) to the server when the file transfer was finished. On the server side read this message after sending the file. The server will be blocked on this reading until the message arrive.

Thanks Erel,

My code already attempts what you suggested. It sends the received file size back after the data. This is my main problem however as the server sends all the data then waits for the confirmation message (the size) - the client however is still busy reading the data and it can take over 20 seconds before the it sends the confirmation.. by which time the server thinks there's been a problem so triggers an 'error'. The client however sends the confirmation and thinks everything's fine with the transfer.. I need both ends to know the transfer is successful.

Is there a better way of transferring a file and knowing 100% that it's there or not? Maybe http or ftp?

Any help much appreciated as I'm under pressure to get this going.

 

[agraham]

I'd like to know the explanation for that delay because that doesn't seem to tally with my experience of playing with the Network library. However, because it is TCP/IP if the server doesn't get an error then you can assume that the client got the data OK. That is the reason for using TCP/IP for transfers, it has timeouts and retries built in to it.

 

[nigelr]

Am I missing something?

because it is TCP/IP if the server doesn't get an error then you can assume that the client got the data OK. That is the reason for using TCP/IP for transfers, it has timeouts and retries built in to it.

Am I thinking too much and trying too hard with this or missing the obvious? I want to send a file from a client and know for definite if the server gets it. How does the client know if the server has an error or not?

 

[agraham]

Sorry, I probably didn't read too carefully, I thought the server was sending the file. In that case the client should get an error of some sort if the transfer does not succeed. In general under TCP/IP the sender of the data will be notified if things didn't work out.