Android Question aSyncStream not flushing data.

[synasir]

Hi all,

I am using a fast timer to send sensor telemetry data back to my VB6 server using aSyncStreamtext. However, the data sometimes will all merge (not always).

TimerSvc1.Initialize ("TimerSvc1",50)

Dim mymessage As String
DateTime.DateFormat = "dd/MMM/yyyy HH:mm:ss"
dt = DateTime.Date(DateTime.now)
mymessage = "$$" & "@" & Round2(SensorData,2) & "@" & dt & "!#" & CRLF


If Socket1.Connected = True Then

TCPFlusso.Write (mymessage)

End if

Several mymessage will be sent together.



I tried the https://www.b4x.com/android/forum/threads/asyncstream-and-flush.146655/ to flush the buffer and force send a single mymessage but it does not seem to work (same behaviour as the code above).


Dim mymessage As String
DateTime.DateFormat = "dd/MMM/yyyy HH:mm:ss"
dt = DateTime.Date(DateTime.now)
mymessage = "$$" & "@" & Round2(SensorData,2) & "@" & & dt & "!#" & CRLF


If Socket1.Connected = True Then


r.Target=TCPFlusso
r.Target=r.GetField("aout")
aout=r.GetField("out")
TCPFlusso.Write2 (mymessage.GetBytes("UTF8"),0,mymessage.Length)
aout.RunMethod("flush",Null) ' force send

End if

I am using just regular Winsock in my VB6 server to receive the data.


Any ideas on how to solve this problem?

Thanks.

 

[OliverA]

It might also be that your VB6 app is not processing the received messages fast enough and therefore messages pile up and therefore multiple messages arrive at the next "read". To proof/disproof the issue you may need to wireshark the connection to see what is really happening.

See solution to below link for better explanation on how little control you may have over send/receive buffers:

[Solved] Handling multiple messages on a socket - CodeProject

www.codeproject.com

 

[synasir]

Hi OliverA, thanks for the suggestion to use Wireshark. Anyway, after checking, it is from the Android app. Usually, the length of the message is less than 100 characters but from Wireshark, the message length becomes 4 to 6 times longer when combined.

 

[drgottjr]

there is an interesting discussion re: outputstream flushing here
a quick look at the asyncstreams class makes me think we're using the base implementation of outputstream (as mentioned in the link), not one of the referred-to buffered classes (even though a buffer is used). according to the link, those classes override a noop flush() method.

 

[synasir]

https://www.b4x.com/android/forum/threads/asyncstream-and-flush.146655/

The implementation of the flush function is supposed to send the data in the buffer regardless of if filled or not but it still does not.

 

[OliverA]

It may be more complicated than that. The flush() method is only against the given output stream, but the underlying socket may not be triggered to send the data. Looking at Android's Socket docs (1), the setTcpNoDelay (2) method indicates that Android uses Nagle's algorithm (3) to determine how to fill/flush a buffer. Even though the setTcpNoDelay method can be used to turn off this algorithm, some people report that this has no effect (4) so individual testing may be needed. Another trick may be to decrease the buffer size via setSendBufferSize (5) to a size that would allow space for only one message. Still, Android may not honor the given size (as mentioned in the docs for setSendBufferSize) and one needs to verify the buffer size via getSendBufferSize (6).

To reiterate, this is not a B4A issue but a limitation imposed by the Socket implementation of Android's java.net.Socket API. Therefore your server application may need to be modified in such a way as to accommodate this limitation when receiving data from Android devices via java.net.Socket (no matter what development environment is used for the Android application).

Links:
1) https://developer.android.com/reference/java/net/Socket
2) https://developer.android.com/reference/java/net/Socket#setTcpNoDelay(boolean)
3) https://www.google.com/search?client=firefox-b-1-d&q=Nagle's+algorithm
4) https://www.google.com/search?client=firefox-b-1-d&q=android+not+honoring+settcpnodelay
5) https://developer.android.com/reference/java/net/Socket#setSendBufferSize(int)
6) https://developer.android.com/reference/java/net/Socket#getSendBufferSize()

 

[synasir]

To reiterate, this is not a B4A issue but a limitation imposed by the Socket implementation of Android's java.net.Socket API. Therefore your server application may need to be modified in such a way as to accommodate this limitation when receiving data from Android devices via java.net.Socket (no matter what development environment is used for the Android application).

Yes, unfortunate. I am taking steps to remedy this. Instead of just processing the data when the server receives it, I will create another buffer for the data and use a timer to "smooth" out the data.

Thanks for the advice.

 

[emexes]

the data sometimes will all merge (not always).

The solution might be for the VB end to parse the incoming stream using the "$$@" start-of-packet and "!#" & CRLF end-of-packet markers.

Rather than relying on the network chunking aligning with your packets, instead just treat it as a stream of bytes.

Usually you'd do this by, for each incoming VB data_received event:
- append the received data to a string buffer
- while the string buffer contains an end-of-packet marker
- cut the (potential) packet from the front of the string buffer into maybe_packet
- if maybe_packet contains a start-of-packet marker then
- cut off any guff from the front of maybe_packet up to the start-of-packet marker
- handle maybe_packet

 

[emexes]

I will create another buffer for the data

Yes. ?

and use a timer to "smooth" out the data.

No. No no no no no. Just no. Nein. Nyet. Non. ?

 

[synasir]

Yes, unfortunate. I am taking steps to remedy this. Instead of just processing the data when the server receives it, I will create another buffer for the data and use a timer to "smooth" out the data.

The timer is to process the data buffer.

 

[synasir]

It may be more complicated than that. The flush() method is only against the given output stream, but the underlying socket may not be triggered to send the data. Looking at Android's Socket docs (1), the setTcpNoDelay (2) method indicates that Android uses Nagle's algorithm (3) to determine how to fill/flush a buffer. Even though the setTcpNoDelay method can be used to turn off this algorithm, some people report that this has no effect (4) so individual testing may be needed. Another trick may be to decrease the buffer size via setSendBufferSize (5) to a size that would allow space for only one message. Still, Android may not honor the given size (as mentioned in the docs for setSendBufferSize) and one needs to verify the buffer size via getSendBufferSize (6).

setTcpNoDelay has no effect.

 

[emexes]

The timer is to process the data buffer.

That timer is going to add complexity, fragility and latency.

Instead of just processing the data when the server receives it

What is wrong with continuing to do this?

 

[drgottjr]

https://www.b4x.com/android/forum/threads/asyncstream-and-flush.146655/

The implementation of the flush function is supposed to send the data in the buffer regardless of if filled or not but it still does not.

for reasons mentioned in the link i posted, there may be no flush implementation in this case. that's why it doesn't work.
oracle's dox (https://docs.oracle.com/javase/8/docs/api/java/io/OutputStream.html#flush--) says:
"The flush method of OutputStream does nothing."
if, on the other hand, you look at the dox for, eg, bufferedoutputstream, it does not say that flush does nothing.
that's what the link i posted was about.

 

[synasir]

What is wrong with continuing to do this?

The server will still process single messages.

That timer is going to add complexity, fragility and latency.

This is still ok for my purpose.

Thanks for the help.

 

[emexes]

Well, I quite enjoyed writing this anyway ? plus it might be a useful starting point for others, so:

If you use this packet parsing routine:

Sub Process_Globals
    Dim StartOfPacket As String = "$$@"
    Dim EndOfPacket As String = "!#" & CRLF
    Dim PacketBuffer As String
End Sub

Sub HandleIncomingData(Chunk As String)
    LogColor("In = " & Chunk.Replace(CRLF, "CRLF"), 0xFFc0c0c0)
    PacketBuffer = PacketBuffer & Chunk
   
    Do While True
        Dim P As Int = PacketBuffer.IndexOf(EndOfPacket)    'check for packet in PacketBuffer
        If P < 0 Then Exit    'exit if not
       
        P = P + EndOfPacket.Length    'point to end of packet, not start of EndOfPacket marker
        Dim Packet As String = PacketBuffer.SubString2(0, P)    'cut off front bit
        PacketBuffer = PacketBuffer.SubString(P)    'keep the rest
       
        P = Packet.LastIndexOf(StartOfPacket)
        If P > -1 Then    'if packet present
            If P <> 0 Then    'if guff before StartOfPacket
                Packet = Packet.SubString(P)    'then chop it off
            End If
            HandlePacket(Packet)
        End If
    Loop
End Sub

and test it in B4J with:

Sub AppStart (Args() As String)
    Dim sb As StringBuilder
    sb.Initialize

    DateTime.DateFormat = "dd/MMM/yyyy HH:mm:ss"

    Dim NumSamplePackets As Int = 6   'whatever
    For I = 1 To NumSamplePackets
        sb.Append("noise")
        sb.Append(StartOfPacket)
        sb.Append(Round2(Rnd(0, 5000) / 1000, 2))
        sb.Append("@")
        sb.Append(DateTime.Date(DateTime.now + I * 1000 + Rnd(-100, 100)))
        sb.Append(EndOfPacket)
    Next
   
    DateTime.DateFormat = DateTime.DeviceDefaultDateFormat

    Dim SampleData As String = sb.ToString

    Log(SampleData)
    Log(SampleData.Length)

    Dim ChunkSize As Int = 19    'whatever
    For I = 0 To SampleData.Length - 1 Step ChunkSize
        HandleIncomingData(SampleData.SubString2(I, Min(I + ChunkSize, SampleData.Length)))
    Next
End Sub

Sub HandlePacket(Packet As String)
    LogColor("Packet = " & Packet.Replace(CRLF, "CRLF"), 0xFF008000)
End Sub

then with luck you'll get Log output something like:

 

[emexes]

The server will still process single messages.

And...? Isn't that what you were doing already, until we ran off the rails when:

However, the data sometimes will all merge (not always).

---

We are your friends here ?

That timer is going to add complexity, fragility and latency.

This is still ok for my purpose.

and this hurts because we can see you heading into dark territory I'm not saying it's going to end in tears, but there will certainly be tears along the way ?

 

[synasir]

Thanks for the help and sample code. Will look into it.

 

[emexes]

this hurts because we can see you heading into dark territory

Sorry if I'm sounding like your grandpa ?

Bonus: note that the sample code filters out inter-packet line noise eg when you hot-plug a device in or out

 

[drgottjr]

i realize this thread may have reached its end, but i wanted dwell on flush for a minute. attached please find 2 images of jdk source code for outputstream and bufferedoutputstream. the flush() method for outputstream is a noop. the flush() method for bufferedoutputstream is something quite different. i'm not saying there is no i/o for outputstream, but what there is happens at a different level and is not generated by flush().

 

[OliverA]

dwell on flush for a minute

You make a good point. But even if one were to use a BufferedOutputStream, its flush would write to the underlying buffer of the Socket and there is still no guarantee that this will cause the Socket's buffer to flush (due to Nagle's algorithm, etc -> see post #6 above). Again, testing would be required to see how java.net.Socket would behave when using a BufferedOutputStream and its corresponding flush method.

Link:
Post #6 above: https://www.b4x.com/android/forum/threads/asyncstream-not-flushing-data.158403/post-972369