Android Question Lost data using AsyncStreams

[Marcob]

Hello,

I'm using an AsyncStreams object to write data packets to a file:

...
Dim asyncFile As AsyncStreams  
Dim outFile as OutputStream
outFile = File.OpenOutput(dir, fileName, True)
asyncFile.Initialize(Null,outFile,"AStreamWriter")  
...
Sub Packet_Ready(data() As Byte)
  packet_counter = packet_counter+1
  asyncFile.Write(data)
End Sub

Sometimes I noted that a few packets get lost (the number of packets inside outFile is lower than packet_counter).
I don't know how this happens, I suspect this could be due to a sort of buffer overrun inside the AsyncStreams thread (may be the internal queue overruns?).
In any case AStreamWriter_Error() subroutine is never called.

What can I do to avoid this loss of data?
Is there a way to know if asyncFile can't accept further data so to avoid to call asyncFile.Write(data) during this condition?
Any idea on how to fix this issue will be greatly appreciated

 

[Erel]

AsyncStreams.Write will return False if the queue is full.

Why do you need to use asyncstreams here? Where does the data come from? What is the size of the file?

 

[Marcob]

AsyncStreams.Write will return False if the queue is full.

Does that mean that when it returns False no bytes at all of data() are written?
So, in order to be sure all data chuncks are correctly written, could I arrange something like:?

Do While asyncFile.Write(data)=False
Loop

Why do you need to use asyncstreams here? Where does the data come from? What is the size of the file?

I'm receiving a data stream from the USB interface in a dedicated thread (I'm using a Thread object). Every time that 4000 bytes are received, they will be saved to a file by calling Packet_Ready(data). The data rate is quite high (up to 5000kbit/s) but I tested that generally it can be sustained correctly. Unfortunately sometimes a few packets are lost, I guess because the operating system or the file access is temporarily busy and perhaps the AsyncStreams queue becomes full. For example in the last test I've made I've written a 240MB file with about 60000 data packets and I've lost 32 packets.

I tried to write directly to an outputstream with outFile.WriteBytes(data,0,data.length) but I found asyncFile.Write(data) works better. I'm not sure why but It seems outFile.WriteBytes(data,0,data.length) would block the program flow and downgrade in some way the receipt of the USB data.

 

[Erel]

It is never a good idea to block the main thread.

I think that a better solution will be to initialize a global output stream with Out.InitializeToBytesArray.

Write the data with Out.WriteBytes.
If the size is larger than 5mb then write it to the file with AsyncStreams (and initialize out again).

Why do you need a thread object? If you are using UsbSerial or felUsbSerial then the data is already managed by a background thread.