B4J Question Reading serial ports

[stevewidget]

Hello,
I have a system that sends data from a micro( various, so not B4R compatible) down serial in response to a request from the PC, all in plain ASCII.
Example Request = ? ( No CRLF needed)
Response is in form of a number of channels (Label)(ASCII Value+ or-)( : delimiter) again no CRLF needed
eg send ? response A-123;B65;C123245678; etc
This all seemed to work fine when I did it via keyboard, after I modified to send CRLF by using Chat example and doing a select case looking for specific labels as in

Sub AStream_NewData (Buffer() As Byte)

    Dim s As String = BytesToString(Buffer, 0, Buffer.Length, "UTF8")
    Dim i As Int = 0
    Dim length As Int
    length = s.Length
    Dim temp As String
        Select s.CharAt(i)
            Case "X"
                Do While (s.CharAt(i+1)<>";")
                    i=i+1
                    temp = temp & s.CharAt(i)
                Loop
                lblXval.Text=temp
            
          'add extra cases
        End Select
    Log(length)   
    'LogMessage("You",  temp)
End Sub

So I then set up a timer to issue a '? CRLF' every 200mS, and it crashes after 6-12 readings with an out of range error in i in temp =temp bit. ( Im running at 115200 baud so should be able to cope with 20 characters in this time)
When run the log shows a length of 20 ( correct ) most of time, but sometimes reports 2 numbers that add up to 20 eg, 13 then 7. I don't understand how the buffer is being filled. I sorta assumed that it wait till it got a CFLF so length should always be same.
I cant see a way of controlling the reading of the bytes, or putting in to a smart string to extract each channel data, though Im sure its dead easy when you know what you are doing. Any suggestions please.
Steve

 

[agraham]

I sorta assumed that it wait till it got a CFLF so length should always be same.

Common mistaken assumption for serial comms on any machine - not just Android. What you think of as a line can arrive as several NewData events so your assumption that there will be a semi-colon is wrong. You need to assemble the line somewhere until you see a semi-colon then use it.

 

[stevewidget]

Hello agraham,
Thanks for that. Im really too long in the tooth to get caught by assuming, I should remember assume makes an ASS out of yoU and ME, well at least me. Would be much easier if could specify buffer length, but as you cant are there any clever tricks that can be applied or do I solve using MSbasic/C type string handling?
Thanks
Steve

 

[bdunkleysmith]

I suggest you have a look at the AsyncStreamsText class.

 

[stevewidget]

Hello bdunkleysmith,
Thanks for your response, and I have looked at this, in fact I am using it in non prefix mode, as I am not using B4R compatible micros, and a variant of this sort of protocol exists on commercial products where I have no control of the data.
In prefix mode as I understand it the sender says 'heres X bytes' and the receiver waits until its received X bytes before raising event. For ASCII streams it would be nice to have a delimiter mode where it sends up to the first delimiter found.
In C I would either set up an interrupt per byte, or more likely set up a while loop looking at the port building packets one by one.
It's just messy sorting current stream where it will stop mid packet, save what the packet is, where it is in the packet and wait for next event. I'm calling a packet the data from my label to my semicolon, What I really need is a function that gets this packet without emptying the port buffer. Is there a way for me to do a raw read of the port without using events? Or do I have to look at modifying the library to do this? I don't 'speak' Java and really don't want to learn as I have enough on my plate.
Thanks
Steve

 

[bdunkleysmith]

Hi Steve,

Your comment re: prefix and non-prefix mode suggests you are using AsyncStreams whereas I suggested using the AsyncStreamsText class.

I suggested that because as I understand your requirement, you want to recover each piece of data which terminates with CRLF, which as stated in AsyncStreamsText:

"AsyncStreamsText can help you if:
1. The data sent is text and not binary data.
2. Each message ends with an end of line character (both Unix and Windows formats will work).

AsyncStreamsText works by handling the NewData event and looking for the end of line characters."

As per the example, capturing the data is simply handled by the AsyncStreamsText NewText event:

Sub ast_NewText(Text As String)
   Log("Text: " & Text)
   Log(Text.Length)
End Sub

I'm sorry if I've misunderstood your requirement and this isn't suitable for you, but I've used the AsyncStreamsText class successfully in a variety of B4J apps which consumed text data terminated with an end of line character sent from a variety of sources.

Bryon

 

[agraham]

A raw read of the port will not help. You will still have to assemble the incoming characters till you see your terminator. I don't see it as particularly messy to declare a global string variable, append the data to it and check with Strindex if there is a semi-colon. When there is parse the string and clear the variable retaining any characters after the semi-colon as part of the next packet. As Bryon suggests AsyncStreamsText does in fact do this for you. You will need a slight mod to check for semi-colon instead of new line.

What does get complicated is dealing with character corruption and other comms errors but if you have a short serial connection in a reasonably benign environment you may be able to assume a perfect connection.

 

[bdunkleysmith]

@stevewidget, looking back I did misread your original post because I thought by your comment "after I modified to send CRLF" that you'd changed from using a semi-colon terminator to CRLF and so AsyncStreamsText would be OK as is. But I realised from the post by @agraham that you still want to use the semi-colon terminator and so as he says, you'd need to modify the AsyncStreamsText class to handle your semi-colon terminator in lieu of the "standard" EOL. However hopefully you can use this as the basis to solve your problem.

 

[stevewidget]

Gents,
Thank you both for your replies, greatly appreciated. I had a look at the class for text stream, and I think the solution is pretty much as I had imagined, and is what I am afraid I called messy, ( a poor choice of wording for which I apologise) but it is a lot more complex than the following (psuedo code)
string=""
While (serialinput char<> ";")
string=string+serialinputchar
EndWhile
I suppose this could be considered blocking as if you didn't send a ";" (or whatever terminator) it would sit there forever, but in my case I know I will get a response in a reasonable time, as I should only ever read 16 char. I'm really a hardware guy trying to do software, and not talking to the HW directly is new to me.
The original library on which Jserial is based you could define bytes read ( so could set to 1

jSerial library

The jSerial library allows you to open and communicate with other devices through the computer COM ports. It can also be used to communicate with Bluetooth devices over a virtual com port. (jBluetooth library is now available: https://www.b4x.com/android/forum/threads/jbluetooth-library.60184/)...

www.b4x.com

Regarding complications and ASSUMING (I hope I have re learnt my lesson on that!!!), fortunately the data per packet is averaged and a completely wrong error (or 2) would quickly get corrected.
I shall spend the rest of the day looking at implementing text class, or variant of.
Thanks again
Steve

 

[emexes]

Give this a burl.

Sub Process_Globals

    Dim SerialBuffer As String    'agglomerate incoming serial data

End Sub

Sub AStream_NewData (Buffer() As Byte)

    Dim s As String = BytesToString(Buffer, 0, Buffer.Length, "UTF8")

    SerialBuffer = SerialBuffer & s
    '''Log("SerialBuffer = [" & SerialBuffer & "]")

    Do
        Dim P As Int = SerialBuffer.IndexOf(";")
        If P < 0 Then
            Exit
        End If

        Dim ThisField As String = SerialBuffer.SubString(0, P)    'could be off by one here, aiming to drop the ";"
        SerialBuffer = SerialBuffer.SubString(P + 1)    'keep everything after the ";"

        ThisField = ThisField.Replace(Chr(13), "").Replace(Chr(10), "").Trim.ToUpperCase    'general cleanup
        '''Log("ThisField = [" & ThisField & "]")

        If ThisField.Length >= 2 Then
            Dim FieldLetter As String = ThisField.CharAt(0)
            Dim FieldNumber As String = ThisField.SubString(1)

            If IsNumber(FieldNumber) Then
                HandleFieldData(FieldLetter, FieldNumber)    'or whatever it is you need to do
            End If
        End If
    Loop

End Sub

 

[stevewidget]

Hi Emexes,
Thanks so much for that, I did give it a good burling, and think I now have a working solution with a couple of tweaks. Biggest issue was it kept leaving serial buffer at '010' ( which I guess is end of array terminator), which then became start of next serial buffer, which then wasn't getting sorted as Fieldnumber was non numeric.
My solution is included to aid anyone else, but maybe there is a better way to stop the 10.

BIG thanks to all who helped, hope I can do same for someone else one day

Sub AStream_NewData (Buffer() As Byte)

    Dim s As String = BytesToString(Buffer, 0, Buffer.Length, "UTF8")
    s = s.Replace(Chr(13), "").Replace(Chr(10), "").Replace(" 10","").Trim.ToUpperCase    'general cleanup
    SerialBuffer = SerialBuffer & s
    Log("SerialBuffer = [" & SerialBuffer & "]")

    Do While(True)
        Dim P As Int = SerialBuffer.IndexOf(";")
        If P < 0 Then
            If (s.Length<=2) Then
                SerialBuffer=""
                Log("bNULL"& p)
            End If
            Exit
        End If
        Log("SerialBuffer Point= [" & SerialBuffer & "]  P=" & p)


        Dim ThisField As String = SerialBuffer.SubString2(0, P)    'could be off by one here, aiming to drop the ";"
        SerialBuffer = SerialBuffer.SubString(P+1 )    'keep everything after the ";"

       'ThisField = ThisField.Replace(Chr(13), "").Replace(Chr(10), "").Trim.ToUpperCase    'general cleanup
        Log("ThisField = [" & ThisField & "]")

        If ThisField.Length >= 2 Then
           Dim FieldLetter As String = ThisField.CharAt(0)
           Dim FieldNumber As String = ThisField.SubString(1)
'
           If IsNumber(FieldNumber) Then
                SortPacket(ThisField)
'                HandleFieldData(FieldLetter, FieldNumber)    'or whatever it is you need to do
            Else
                Log("ERRRRRRRORR")
           End If
          
        End If
    Loop

End Sub

 

[emexes]

Biggest issue was it kept leaving serial buffer at '010' ( which I guess is end of array terminator), which then became start of next serial buffer,

This makes me nervous. Is that three bytes of values 48 49 48 ie ASCII/UTF "010", or a single byte of value 10 ie ASCII/UTF linefeed control code?

which then wasn't getting sorted as Fieldnumber was non numeric.

I was hoping your packet handling was a Select statement that ignored/logged packets with id characters that it didn't recognize. ?

maybe there is a better way to stop the 10.

I'd be very happy to look further into this, if you're up for the mission as well. First step is to log precisely what data is being received. Add five lines to the top of the sub, run it for a while, post what gets logged (the more, the merrier).

Sub AStream_NewData (Buffer() As Byte)

    Dim LogLine As String = "srx " & DateTime.Now & " " & Buffer.Length & " :"
    For I = 0 to Buffer.Length - 1
        LogLine = LogLine & " " & Buffer(I)
    Next I
    Log(LogLine)

    Dim s As String = BytesToString(Buffer, 0, Buffer.Length, "UTF8")

 

[emexes]

ps B4X Bytes are signed, so don't panic if you see negative numbers in the srx log lines ✌

 

[stevewidget]

Hello Emexes,
Thanks for the concern. I'll have to 'unfix' it to show you the ' 10', but this is exactly what was showing SOMETIMES in my log output for serial buffer
ie correct SerialBuffer = [X1946;Y0;Z-143;A0;]
bad SerialBuffer = [X1946;Y0;Z-143;A0;]

 

[emexes]

I'll have to 'unfix' it to show you the ' 10', but this is exactly what was showing SOMETIMES in my log output for serial buffer
ie correct .... SerialBuffer = [X1946;Y0;Z-143;A0;]
.. bad ........ SerialBuffer = [X1946;Y0;Z-143;A0;]

So the extraneous 10 isn't visible? Or perhaps it is from a previous field but not removed from SerialBuffer. Although that doesn't gel with the leading space before the "10", given that the device doesn't seem to be sending any spaces.

Anyway, if we could see the raw data and an instance of the problem, all will probably become clear. I have a feeling that your fix has replaced a rare problem with an even rarer problem that is infrequent enough to usually not show up during testing. ?

 

[stevewidget]

Sorry wrong button , didnt mean to post
bad SerialBuffer = [X1946;Y0;Z-143;A0; 10]
My SortPacket is a select case and it does pick up the fact its corrupt, id rather not have them.
When it did go wrong the next buffer would look like
SerialBuffer = [10X1946;Y0;Z-143;A0; ]
hence the first ThisField would become 10X1946
Another issue ( after my 'fix' Ive just noticed)is that if the serial only picks up a few characters the following happens

SerialBuffer = [X1]
bNULL-1
SerialBuffer = [946;Y0;Z-143;A0;]
SerialBuffer Point= [946;Y0;Z-143;A0;] P=3
ThisField = [946]
Error 46
SerialBuffer Point= [Y0;Z-143;A0;] P=2
ThisField = [Y0]
SerialBuffer Point= [Z-143;A0;] P=5
ThisField = [Z-143]
SerialBuffer Point= [A0;] P=2
ThisField = [A0]
*
I shouldn't be nulling the serial buffer unless its this rogue 10.

Steve

 

[emexes]

My SortPacket is a select case and it does pick up the fact its corrupt, id rather not have them.

Amen to that. ?

 

[emexes]

Another issue ( after my 'fix' Ive just noticed)is that if the serial only picks up a few characters the following happens

That's easy to fix, just remove these four lines:

If P < 0 Then
    '''If (s.Length<=2) Then
    '''    SerialBuffer=""
    '''    Log("bNULL"& p)
    '''End If
    Exit
End If

and ideally remove this bit too:

.Replace(" 10","")

and then run it until the problem occurs. Once we can see the raw data that produced the problem, the solution will probably jump off the page at us. ?

It could be that the serial device you're reading is producing the rogue 10 (nicely named!) and the true fix might be better done on the sending side of the serial link, if you have access to that code.

 

[stevewidget]

Just for clarification, I have been running the hardware that creates the string, and reading with a terminal on a PC as well as in to another micro without a hitch for over a year now, so I dont think its that. Was a doddle in C just reading one byte at a time, just built up required packets as I went. ( A LOT of other stuff is not so easy in C!!!!)
OK I'll break it and send you the log
Thanks
Steve

 

[stevewidget]

Here it is, didnt take long to go wrong
Steve