B4J Question Reading serial ports

[stevewidget]

Hello,
I have a system that sends data from a micro( various, so not B4R compatible) down serial in response to a request from the PC, all in plain ASCII.
Example Request = ? ( No CRLF needed)
Response is in form of a number of channels (Label)(ASCII Value+ or-)( : delimiter) again no CRLF needed
eg send ? response A-123;B65;C123245678; etc
This all seemed to work fine when I did it via keyboard, after I modified to send CRLF by using Chat example and doing a select case looking for specific labels as in

Sub AStream_NewData (Buffer() As Byte)

    Dim s As String = BytesToString(Buffer, 0, Buffer.Length, "UTF8")
    Dim i As Int = 0
    Dim length As Int
    length = s.Length
    Dim temp As String
        Select s.CharAt(i)
            Case "X"
                Do While (s.CharAt(i+1)<>";")
                    i=i+1
                    temp = temp & s.CharAt(i)
                Loop
                lblXval.Text=temp
            
          'add extra cases
        End Select
    Log(length)   
    'LogMessage("You",  temp)
End Sub

So I then set up a timer to issue a '? CRLF' every 200mS, and it crashes after 6-12 readings with an out of range error in i in temp =temp bit. ( Im running at 115200 baud so should be able to cope with 20 characters in this time)
When run the log shows a length of 20 ( correct ) most of time, but sometimes reports 2 numbers that add up to 20 eg, 13 then 7. I don't understand how the buffer is being filled. I sorta assumed that it wait till it got a CFLF so length should always be same.
I cant see a way of controlling the reading of the bytes, or putting in to a smart string to extract each channel data, though Im sure its dead easy when you know what you are doing. Any suggestions please.
Steve

 

[agraham]

Looks like the bad ones have " 13 10 49 48" in the raw data. Looks remarkably like a Windows CRLF pair followed by "10". That looks to me like the CRLF is the end of the current packet and "10" is the start of the following packet. Why didn't you try modifying AsyncStreamsText instead of rolling your own. It should take of this.

PS. It looks to me that each packet is actually terminated by CRLF and not by semi-colon.
"10X1946;Y0;Z-38;A0;CRLF"

 

[emexes]

PS. It looks to me that each packet is actually terminated by CRLF and not by semi-colon.
"10X1946;Y0;Z-38;A0;CRLF"

I agree, except that the original post stated "again no CRLF needed".

But if it is guaranteed that the incoming stream has CRLF-terminated lines, and we're not worried about field processing being delayed until the entire line has been received, then AsyncStreamsText would be a simpler way of doing it.

Presumably the over-a-year successful parsing by other programs skips the "10" at the start of (most?) lines when searching for the first/next field's (uppercase?) letter tag.

 

[agraham]

You can see it in the raw data, every packet is separated by three SPACE CR LF characters. If he's been looking at it on a terminal it would need the CRLF to scroll the display.

 

[emexes]

Was a doddle in C just reading one byte at a time, just built up required packets as I went. ( A LOT of other stuff is not so easy in C!!!!)

Even more of a doddle in B4X. ?

Sub AStream_NewText(Text As String)

    Dim m As Matcher = Regex.Matcher("([A-Z])(.*?);", Text)    'fields start with uppercase letter, terminated by semicolon
    Do While m.Find
        HandleField(m.Group(1), m.Group(2))    'letter, value
    Loop

End Sub

Here's the longer and more self-explanatory version, with a bonus numeric safety-net.

Sub AStream_NewText(Text As String)

    Dim FieldPattern As String = "([A-Z])(.*?);"    'fields start with uppercase letter, terminated by semicolon
  
    Dim m As Matcher = Regex.Matcher(FieldPattern, Text)
    Do While m.Find
        Dim FieldLetter As String = m.Group(1)
        Dim FieldValue As String = m.Group(2)

        If IsNumber(FieldValue) Then    'if value is supposed to be numeric
            HandleField(FieldLetter, FieldValue)
        End If
    Loop

End Sub

 

[stevewidget]

Hi Gents,
OK I perhaps need to further expand on my explanation. As I said in my first post ( and picked up by emexes) I did say that not every block of data HAS to be terminated by a CRLF. Truth is I put that in to help with reading in Xojo, but I don't want to use, and don't when using Bluetooth or to other micros. I genuinely didn't believe I would have all this convolution to read the data in B4X. I am reading a number of sensors and each sensor is defined by a label (in this case XYZA, but equally ABCDEFG....). Each sensors value is then defined as an ASCII string terminated by a ';' The value is sent in response to a value sent from PC and may be for an individual sensor without a CRLF or a send all with a CRLF ( Mostly for diagnostic purposes to see easily on terminal) . so AstreamText wouldn't help. Bottom line at this point is emexes solution seems to work with or without the CRLF.
The 10 at the start of the string is as a result of being left over at the end of first read, and was cleared by putting my replace" 10" in.
So to prove (or otherwise) a point I removed the CRLF from my micro, and this 10 is still there ( the * is a new send block request)
*
srx 1592851290578 3 : 88 49 50
SerialBuffer = [X12]
srx 1592851290593 16 : 53 49 59 89 48 59 90 45 57 48 59 65 48 59 49 48
SerialBuffer = [X1251;Y0;Z-90;A0;10]
SerialBuffer Point= [X1251;Y0;Z-90;A0;10] P=5
ThisField = [X1251]
X 1251
SerialBuffer Point= [Y0;Z-90;A0;10] P=2
ThisField = [Y0]
SerialBuffer Point= [Z-90;A0;10] P=4
ThisField = [Z-90]
SerialBuffer Point= [A0;10] P=2
ThisField = [A0]
*
srx 1592851291093 19 : 88 49 50 53 49 59 89 48 59 90 45 57 48 59 65 48 59 49 48
SerialBuffer = [10X1251;Y0;Z-90;A0;10]
SerialBuffer Point= [10X1251;Y0;Z-90;A0;10] P=7
ThisField = [10X1251]
ERRRRRRRORR
SerialBuffer Point= [Y0;Z-90;A0;10] P=2
ThisField = [Y0]
SerialBuffer Point= [Z-90;A0;10] P=4
ThisField = [Z-90]
SerialBuffer Point= [A0;10] P=2
ThisField = [A0]
*
So then I took out all the processing (from do while to end, and just assigned Serial buffer to s, and that darn 10 still appears.
Program started.
*
srx 1592852107800 8 : 88 49 55 57 52 59 89 48
SerialBuffer = [X1794;Y0]
srx 1592852107805 10 : 59 90 45 50 59 65 48 59 49 48
SerialBuffer = [;Z-2;A0;10]
*
And for the sake of completeness, I dumped same data down to teraterm, and not a single 10 in sight.
Where is this coming from?? I really don't like workarounds I don't understand.

Steve

 

[stevewidget]

Hi emexes,
sorry posted my previous reply without looking at yours. Well I knew there would be a clever way of doing it.
Had enough of this for one day. Will try again tomorrow, but as a read is not guaranteed to start or finish with a letter or a ; I'm going to miss data arent I?
Thanks for all your input.

Steve

 

[emexes]

sorry posted my previous reply without looking at yours.

No worries.

Had enough of this for one day.

Surely not.

Will try again tomorrow

 

[emexes]

as a read is not guaranteed to start or finish with a letter or a ; I'm going to miss data arent I?

No. It's all going to work perfectly, we just need to find out where this extraneous "10" is coming from.

When you did the Teraterm capture, was that on the same computer that the B4J receiving program runs on, with the same hardware and everything else?

My understanding is that your sensor device, as it stands right now, is sending a squirt of serial data every half-a-second, and that serial data is always of four fields labelled X, Y, Z and A, and that the CRLF has been removed (super! - test what you fly, fly what you test). Could you please post a ~10 second capture by Teraterm and by the B4J log file, using the same exact same serial device and cabling setup for both?

My mind picture of the serial link is:

microcontroller - UART - 3.3 or 5 volt 9600N81 bps serial data - RS232 driver - regular 9-pin port -
serial cable - serial-to-USB converter - USB port - Windows - Sub AStream_NewData - B4J log window

Is there any Bluetooth involved? Some serial-over-BLE converters don't cope well with burst of data longer than eg 16 bytes. Perhaps what looks like extra bytes, is actually the dregs of missing bytes. Although that doesn't explain Teraterm not seeing them. Hmm. ? Interesting. ?

 

[emexes]

If you get to the point where you want to "fix" this and move on to other stuff, this should do the trick:

Sub Process_Globals

    Dim SerialBuffer As String    'agglomerate incoming serial data

End Sub

Sub AStream_NewData (Buffer() As Byte)

    Dim s As String = BytesToString(Buffer, 0, Buffer.Length, "UTF8")

    SerialBuffer = SerialBuffer & s
    '''Log("SerialBuffer = [" & SerialBuffer & "]")

    Dim P As Int = SerialBuffer.LastIndexOf(";")

    If P >= 0 Then    'found semicolon, -1 if not found
        Dim FieldPattern As String = "([A-Z])(.*?);"    'fields start with uppercase letter, terminated by semicolon
 
        Dim m As Matcher = Regex.Matcher(FieldPattern, Text)
        Do While m.Find
            Dim FieldLetter As String = m.Group(1)
            Dim FieldValue As String = m.Group(2)

            '''HandleField(FieldLetter, FieldValue)
            Log("Letter = [" & FieldLetter & "], value = [" & FieldValue & "]")
        End If

        SerialBuffer = SerialBuffer.SubString(P + 1)
    End If

End Sub

I'm still keen to work out wtf the "10" is coming from, so whenever you're ready to get back on to that: no worries ?

 

[Erel]

Haven't read all posts so hopefully I'm not writing nonsense.
Why aren't you using B4XBytesBuilder? Converting random bytes to string is a big mistake.

 

[emexes]

Why aren't you using B4XBytesBuilder? Converting random bytes to string is a big mistake.

Not random bytes. Is ASCII text, eg: "X1251;Y0;Z-90;A0;". I think they're readings from or commands for CNC machinery.

admittedly, it does look a *little* bit random ✌

 

[stevewidget]

OK thought my brain might be clearer in the morning, apparently not!
Well I cant see a single random 10 anywhere in the data from teraterm. Can post if you want but its a pretty boring read. I also tried with some VCP port monitoring software

Serial Port Monitor. Dedicated tool for serial communication.

Serial Port Monitor. Professional-grade software for establishing a serial connection with COM-based software and devices. COM Port Monitor allows you to debug, test and scan of serial communication

www.virtual-serial-port.org

Good question Erel, why arent I using byte builder. Why is Converting random bytes to string a big mistake? I simply don't know enough about B4X. ( and I did try reading forum on it)
Even if I rewrote all my code to work with B4R in Prefix mode ( which I cant unless you support PIC18 as well) I am not sure that it would then be compatible with other software. Does prefix mode send the first byte as the length of data, or does it make sure that the buffer has the right amount of data. It would be so much simpler to build strings byte by byte, but that's only because I don't know what Im doing. Every send may well be a 'random' length dependant on the value from the sensor, and number of sensors connected.
I will look at modifying code to use Stringbuilder, but that seems to suggest that $String1=$String1 & $String2 is bad idea, as opposed to sb.Append.
The problem looks more like buffer contains the 10 IF you believe TeraTerm and Eltima software

srx 1592851290578 3 : 88 49 50
SerialBuffer = [X12]
srx 1592851290593 16 : 53 49 59 89 48 59 90 45 57 48 59 65 48 59 49 48
SerialBuffer = [X1251;Y0;Z-90;A0;10]

I have downloaded the Eltima log to show that I believe my data is Clean.
But will carry on as you suggest Emexes with an sb.
Thanks
Steve

 

[agraham]

The 0x10 looks like the byte count of your packet. Is that clue?

 

[stevewidget]

The 0x10 looks like the byte count of your packet. Is that clue?

Regrettably not. I tried altering the buffer length by varying sensors, and sending just the one sensor and its always a 10. Thought it might be something random in SrialBuffer, so initialised in appStart to "". Heres what you get with just X

Waiting for debugger to connect...
Program started.
*
srx 1592908976001 5 : 88 45 49 59 49
SerialBuffer = [X-1;1]
srx 1592908976017 1 : 48
SerialBuffer = [10]
*
srx 1592908976514 6 : 88 45 49 59 49 48
SerialBuffer = [10X-1;10]
ERRRRRRRORR10X-1
*
Thanks
Steve

 

[emexes]

Why is Converting random bytes to string a big mistake?

If you are dealing with a binary protocol, then Byte Arrays are better.

If you are dealing with text, and particularly variable lengths of ASCII text, then Strings are better.

Your protocol is ASCII. Stick with Strings.

Does prefix mode send the first byte as the length of data

Close. The length is a 32-bit number, ie 4 bytes, not just one. Given the networking origin of Java, the byte order is probably big-endian, ie most significant byte first.

or does it make sure that the buffer has the right amount of data.

After receiving the length, it then builds up the specified number of bytes in a buffer, and presents it to your program when complete.

It would be so much simpler to build strings byte by byte

Difficult to get much simpler than SerialBuffer = SerialBuffer & s. I used to worry about efficiency, but when I actually tested the speed of Java string operations, I was surprised. They are amazingly fast. Erel was correct, as always mostly.

But will carry on as you suggest Emexes with an sb.

Don't worry about StringBuilder for the time being. The code in this post should get you going, with or without the spurious "10"s or make-it-readable CR's or LF's.

 

[emexes]

*
srx 1592908976001 5 : 88 45 49 59 49
SerialBuffer = [X-1;1]
srx 1592908976017 1 : 48
SerialBuffer = [10]
*
srx 1592908976514 6 : 88 45 49 59 49 48
SerialBuffer = [10X-1;10]
ERRRRRRRORR10X-1
*

This is super-interesting because it shows that the "10"s are coming in on the AStream_NewData events, seemingly after each set of fields (although here, the sets are just one field long, ie X field only), and can be split across two AStream_NewData events. The incoming Byte Array Lengths (here 5, 1 & 6) indicate that the "10"s aren't being added by the Byte-Array-to-String conversions.

Are you using the exact-same setup for receiving serial data over COM16 with Tera Term as with your B4J program? Like, if Tera Term and that Serial Port Monitor program aren't seeing the "10"s then... it points to the Java serial port interface being the culprit, and that doesn't seem plausible. The problem must be something else.

Btw, is there a serial-over-BLE device in the chain?

Now I'm going to fire up a ESP32 to generate some serial data into my serial-portless laptop.

 

[stevewidget]

Hi Emexes,
Thanks for your replies.
Well Im still puzzled First thing we do is convert a byte array to a string!!
OK so prefix mode is upping the data packet size a lot. Worry about these things with micros.
As for simplicity, I guess its what you are used to, but for my mind psuedo code I posted way back was very simple. Anyway just want something that works.
As you say data is interesting, maybe I am doing something wrong, maybe Teraterm ignores the '10' or just maybe there is a problem with B4J. I'll go with a solution but would really like an answer. Will test out this and on an ESP32 BT board later, other duties call
Steve

 

[emexes]

maybe Teraterm ignores the '10'

Lol. It definitely has a quantum theory observation-affects-reality feel about it.

 

[emexes]

Well, I've gotten as far as an ESP32 connected via a USB cable that shows up as COM11 at 115200 bps, sending out some sample data once per second.

The following log is from a B4J program that logs the AStream_NewData event data. Sometimes the lines/records get split up, because AStream doesn't know about the structure of the data, and has to hand over the data eventually, but that's ok as long as we know it's happening. I forgot to log the Byte Array Lengths and the Byte Array data numbers, but it all looks ok nonetheless.

The same data appears in Tera Term ok. No "10"s anywhere, though.

I too have some other duties, so I'll apply the parsing code later.

1592923933799 : X16;Y2;Z0;A0;
1592923934799 : X17;Y2;Z0;A0;
1592923935799 : X18;Y2;Z0;A0;
1592923936799 : X19;Y2;Z0;A0;
1592923937799 : X20;Y2;Z0;A0;
1592923938802 : X21;Y2;Z0;A0;
1592923939794 : X
1592923939795 : 22
1592923939796 : ;Y2;Z0;A0;
1592923940798 : X23;Y2;Z0;A0;
1592923941799 : X24;Y2;Z0;A0;
1592923942801 : X25;Y2;Z0;A0;
1592923943796 : X26;Y2;Z0;A0;
1592923944797 : X27;Y2;Z0;A0;
1592923945796 : X28;Y2;Z0;A0;
1592923946797 : X29;Y2;Z0;A0;
1592923947798 : X30;Y2;Z0;A0;
1592923948801 : X11;Y3;Z0;A0;
1592923949801 : X12;Y3;Z0;A0;
1592923950797 : X13;Y3;Z0;A0;
1592923951797 : X14;Y3;Z0;A0;
1592923952798 : X15;Y3;Z0;A0;
1592923953798 : X16;Y3;Z0;A0;
1592923954798 : X17;Y3;Z0;A0;
1592923955798 : X18;Y3;Z0;A0;
1592923956803 : X19;Y3;Z0;A0;
1592923957793 : X
1592923957803 : 20;Y3;Z0;A0;
1592923958793 : X
1592923958803 : 21;Y3;Z0;A0;
1592923959794 : X22;Y3;Z0;A0;
1592923959804 : 
1592923960797 : X23;Y3;Z0;A0;
1592923961798 : X24;Y3;Z0;A0;
1592923962802 : X25;Y3;Z0;A0;
1592923963802 : X26;Y3;Z0;A0;
1592923964799 : X27;Y3;Z0;A0;
1592923965799 : X28;Y3;Z0;A0;
1592923966802 : X29;Y3;Z0;A0;
1592923967802 : X30;Y3;Z0;A0;
1592923968802 : X11;Y4;Z0;A0;
1592923969802 : X12;Y4;Z0;A0;
1592923970795 : X13;Y4;Z0;A0;
1592923971799 : X14;Y4;Z0;A0;
1592923972799 : X15;Y4;Z0;A0;
1592923973800 : X16;Y4;Z0;A0;
1592923974793 : X17;Y4;Z0
1592923974803 : ;A0;
1592923975798 : X18;Y4;Z0;A0;
1592923976793 : X19;Y4;Z0;
1592923976803 : A0;
1592923977793 : X20;Y4;Z0;A
1592923977803 : 0;
1592923978794 : X21;Y4;Z0;A
1592923978804 : 0;
1592923979796 : X22;Y4;Z0;A0;
1592923980798 : X23;Y4;Z0;A0;
1592923981798 : X24;Y4;Z0;A0;
1592923982799 : X25;Y4;Z0;A0;
1592923983799 : X26;Y4;Z0;A0;
1592923984793 : X27
1592923984793 : ;Y4;Z0;A0;
1592923984803 : 
1592923985793 : X28;Y4;Z0;A0;
1592923986794 : X29;Y4;Z0;A0;
1592923987797 : X30;Y4;Z0;A0;
1592923988797 : X11;Y5;Z0;A0;
1592923989797 : X12;Y5;Z0;A0;
1592923990798 : X13;Y5;Z0;A0;
1592923991799 : X14;Y5;Z0;A0;
1592923992799 : X15;Y5;Z0;A0;
1592923993799 : X16;Y5;Z0;A0;
1592923994791 : X
1592923994801 : 17;Y5;Z0;A0;
1592923995796 : X18;Y5;Z0;A0;
1592923996798 : X19;Y5;Z0;A0;
1592923997799 : X20;Y5;Z0;A0;
1592923998800 : X21;Y5;Z0;A0;

 

[stevewidget]

My Humble apologies to all, though I still don't fully understand the mechanism. I was issuing
Dim req As String ="?" & CRLF
astream.Write(req.GetBytes("UTF8")
To request data packet. My micro was just responding to ? & CR leaving a LF CHR(10 ) lying around in my buffer which I echoed back for when using a Terminal. Anyway taking out the echo fixes issue. Dont understand why the analyser didnt pick it up. Glad I only used trial version. Really sorry, once again bug is between programmers ears!!!
Thanks to all , especially emexes for all your help. I did learn some neat stuff about Regex in the process.
Steve