Android Question felUsbSerial v1.12 Error at random intervals ArrayIndexOutOfBoundsException, and lost characters

W

weighment

Member
Licensed User
Using felUsbSerial - v1.12 on Android 8.1 with FTDI PID 0x6010

The serial communication works for a variable length of time before an exception in the library.

Connecting to a serial device, requesting data from it every half second or so, data out is 10 or so characters, data in is usually 10 or so characters, occasionally 100 characters in a burst.
Randomly, anywhere from within 10 seconds of connection, to as long as 5 minutes later I get a Array index out of bounds error shown below. For the input buffer size, I see the "Default value is 16 * 1024", and this app has no where near enough data to overflow the buffer. (I doubled it to 32*1024 and still had the error) Is the buffer argument sent to the event only 60 bytes? In observation I never see it have more than 20 characters at a time sent to the event in normal operation of the app.

What is the best way to trap this exception and prevent it from taking down the app? Ideally I'd like to detect the error and recover.

Also, regularly losing 2 or three characters from messages. Occurs on a similar random interval. May or may not be related.

Is source code for this library available to help solve it?

B4X: 
    Public usbserial As felUsbSerial

Error message: 
java.lang.ArrayIndexOutOfBoundsException: length=61; index=61
    at com.felhr.usbserial.FTDISerialDevice$FTDIUtilities.copyData(FTDISerialDevice.java:594)
    at com.felhr.usbserial.FTDISerialDevice$FTDIUtilities.adaptArray(FTDISerialDevice.java:502)
    at com.felhr.usbserial.UsbSerialDevice$WorkerThread.run(UsbSerialDevice.java:248)
 
Sort by date Sort by votes

Daestrum

Expert
Licensed User
Longtime User
I have written a new SLC from scratch (still has a few kinks) and this is what I get from felUsbSerial sources. (notice its using java 22)
 
Upvote 0

Daestrum

Expert
Licensed User
Longtime User
I applied your changes and produced this (not tested it though) - was compiled with java 11
Also never changed the version number.
 

Attachments

  • felUsbSerial.zip
    70.6 KB · Views: 51
Upvote 0
W

weighment

Member
Licensed User
Many thanks @Daestrum (and @agraham for pointing me to @Daestrum 's post)

Side observation, the original felUsbSerial xml only has the anywheresoftware.b4a.objects.usb.felUsbSerial class in it. (8k)
The XML from your SLC also includes all of the classes in com.felhr.usbserial and deviceids and utils and their classes (117k)
Not necessarily a problem. It didn't appear to affect the library import or build.

Was this built with your SLC version from here?
https://www.b4x.com/android/forum/t...ters-slc-xml-output-java11.161916/post-993141

Did you have to do your bat file step from here?
https://www.b4x.com/android/forum/t...ters-slc-xml-output-java11.161916/post-995009

Did you have to change anything else in the library to get it to compile with your new SLC?

My project builds with your compiled library. Now I test to see if the FTDI problem is solved with the mod ...

There is a chance it needs to pull in the entire file
https://github.com/felHR85/UsbSeria...ava/com/felhr/usbserial/FTDISerialDevice.java
 
Upvote 0

Daestrum

Expert
Licensed User
Longtime User
weighment said:
Was this built with your SLC version from here?
Click to expand...
No it's a brand new SLC that I wrote. (does B4J and B4A in one app)
weighment said:
Did you have to do your bat file step from here?
Click to expand...
The new SLC doesn't need a bat file for the xml creation.

Did you have to change anything else in the library to get it to compile with your new SLC?

No just pointed it to the src files.


weighment said:
There is a chance it needs to pull in the entire file
Click to expand...
let me know and I will try it.

the xml with -b4aignore com.felhr
 

Attachments

  • felUsbSerial.xml
    7.3 KB · Views: 45
Last edited:
Upvote 0
W

weighment

Member
Licensed User
Thanks @Daestrum .

With the new function it is better but not solved. It appears to have eliminated the dropped characters but not the strange mid-boundary value issue. Same error manifested in the arraycopy function in the new function instead of the array assignment in the old function.

ArrayIndexOutOfBoundsException: 
java.lang.ArrayIndexOutOfBoundsException: src.length=65 srcPos=2 dst.length=61 dstPos=0 length=62
    at java.lang.System.arraycopy(System.java:521)
    at com.felhr.usbserial.FTDISerialDevice$FTDIUtilities.copyData(FTDISerialDevice.java:592)
    at com.felhr.usbserial.FTDISerialDevice$FTDIUtilities.adaptArray(FTDISerialDevice.java:508)
    at com.felhr.usbserial.UsbSerialDevice$WorkerThread.run(UsbSerialDevice.java:253)

Length was 65 (>64) so it lopped off two headers that were not really there.

I'm certain I can fix it, but to iterate I need to be able to build the library like you did. (so as not to trouble you)

It is worth trying replacing the entire file instead of just the function, as that file is the only change in the felhr project since @Erel built it.

https://github.com/felHR85/UsbSeria...ava/com/felhr/usbserial/FTDISerialDevice.java
replacing the file of the same name in <project>\felUsbSerial\src\com\felhr\usbserial\FTDISerialDevice.java"
then build. (I would build it if I had your cool new SLC, but I will wait patiently until you are ready to release it.)

If that does not work, I will make it pass the headers instead of filtering them to repeat it, then go fix it. (i.e. make ftdiUtilities.adaptArray just return the unchanged array temporarily)
 
Upvote 0

Daestrum

Expert
Licensed User
Longtime User
Just compiling now for you.

But the new file is generating some errors.
 
Upvote 0

Daestrum

Expert
Licensed User
Longtime User
Sorry couldn't get it to compile - loads of what look like silly errors - variables not defined etc.

I downloaded the whole of felusbserial and it wont compile - chasing too many missing packages.
 
Last edited:
Upvote 0

Daestrum

Expert
Licensed User
Longtime User
Got it to compile eventually (my fault was trying to be clever and screwed up a path variables)

You need the stream-1.2.2.jar in your extralibs as it uses parts of it.

Hope it works (fingers crossed)

(this is compiled from fthe latest felusbserial sources)
 

Attachments

  • felUsbSerial.zip
    378.3 KB · Views: 45
Last edited:
Upvote 0
W

weighment

Member
Licensed User
Thanks @Daestrum, Getting NoClassDefFoundError with that newest build. I did copy that stream-1.1.2.jar file into additional libraries too but didn't really include it in project given it did not have an XML file

java.lang.NoClassDefFoundError: Failed resolution of: Lokio/Buffer;: 
*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create (first time) **
autoversion: 1.0
compilation time: 12/04/2020 19:45:50
2 devices
/dev/bus/usb/001/004
1004
3823 1
1
/dev/bus/usb/001/003
1003
1027 24592
2
Connecting to /dev/bus/usb/001/003
java.lang.NoClassDefFoundError: Failed resolution of: Lokio/Buffer;
    at com.felhr.usbserial.SerialBuffer$SynchronizedBuffer.<init>(SerialBuffer.java:97)
    at com.felhr.usbserial.SerialBuffer.<init>(SerialBuffer.java:21)
    at com.felhr.usbserial.UsbSerialDevice.<init>(UsbSerialDevice.java:58)
    at com.felhr.usbserial.FTDISerialDevice.<init>(FTDISerialDevice.java:110)
    at com.felhr.usbserial.UsbSerialDevice.createUsbSerialDevice(UsbSerialDevice.java:77)
    at anywheresoftware.b4a.objects.usb.felUsbSerial.Initialize2(felUsbSerial.java:91)
    at anywheresoftware.b4a.objects.usb.felUsbSerial.Initialize(felUsbSerial.java:79)
    at com.tswa.indicator.wiredscalemanager._findandconnect(wiredscalemanager.java:264)
    at com.tswa.indicator.main._reconnect(main.java:615)
    at com.tswa.indicator.main$ResumableSub_Activity_Create.resume(main.java:553)
    at com.tswa.indicator.main._activity_create(main.java:419)
    at java.lang.reflect.Method.invoke(Native Method)
    at anywheresoftware.b4a.shell.Shell.runMethod(Shell.java:732)
    at anywheresoftware.b4a.shell.Shell.raiseEventImpl(Shell.java:351)
    at anywheresoftware.b4a.shell.Shell.raiseEvent(Shell.java:255)
    at java.lang.reflect.Method.invoke(Native Method)
    at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:157)
    at com.tswa.indicator.main.afterFirstLayout(main.java:108)
    at com.tswa.indicator.main.access$000(main.java:20)
    at com.tswa.indicator.main$WaitForLayout.run(main.java:86)
    at android.os.Handler.handleCallback(Handler.java:790)
    at android.os.Handler.dispatchMessage(Handler.java:99)
    at android.os.Looper.loop(Looper.java:164)
    at android.app.ActivityThread.main(ActivityThread.java:6518)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:438)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:807)
Caused by: java.lang.ClassNotFoundException: Didn't find class "okio.Buffer" on path: DexPathList[[zip file "/data/app/com.tswa.indicator-13PjMIZ5bYEwyOyCnsRRQQ==/base.apk"],nativeLibraryDirectories=[/data/app/com.tswa.indicator-13PjMIZ5bYEwyOyCnsRRQQ==/lib/arm64, /system/lib64, /vendor/lib64]]
    at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:125)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:379)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:312)
    ... 27 more
** Activity (main) Resume **

I think if we do the earlier build, but replace the ftdiUtilities.adaptArray function with this code, it will pass me the read buffer headers along with the data to help diagnose it.

adaptArray: 
    static byte[] adaptArray(byte[] ftdiData)
    {
        return Arrays.copyOfRange(ftdiData, 0, ftdiData.length);
    }

I'd like to find the FTDI docs that describe the buffer header, but I don't see them in the chip datasheet
https://ftdichip.com/wp-content/uploads/2020/08/DS_FT2232D.pdf
Or in the FTDI programmers guide
https://ftdichip.com/wp-content/uploads/2023/09/D2XX_Programmers_Guide.pdf
 
Upvote 0

Daestrum

Expert
Licensed User
Longtime User
So do I change all of this
B4X: 
        public byte[] adaptArray(byte[] ftdiData)
        {
            int length = ftdiData.length;
            if(length > 64)
            {
                int n = 1;
                int p = 64;
                // Precalculate length without FTDI headers
                while(p < length)
                {
                    n++;
                    p = n*64;
                }
                int realLength = length - n*2;
                byte[] data = new byte[realLength];
                copyData(ftdiData, data);
                return data;
            }else
            {
                return Arrays.copyOfRange(ftdiData, 2, length);
            }
        }

To what you posted

or just this part of it to the new code
B4X: 
            {
                return Arrays.copyOfRange(ftdiData, 2, length);
            }
 
Upvote 0
W

weighment

Member
Licensed User
Yes please, the first option, replace that whole function with the single line function.

This may not be useful to others in this form, but it makes it leave the headers in the buffer so I can monitor them over a time and strip them in B4A until we discover the bad values. I'm certain it is when the length is 65, but knowing the header contents will reveal the right way to process a 65 character input buffer.
 
Upvote 0

Daestrum

Expert
Licensed User
Longtime User
Ok will do now and post the jar & xml
changed code ( i made it public like original not static)
B4X: 
        // Special treatment needed to FTDI devices
   /*     public byte[] adaptArray(byte[] ftdiData)
        {
            int length = ftdiData.length;
            if(length > 64)
            {
                int n = 1;
                int p = 64;
                // Precalculate length without FTDI headers
                while(p < length)
                {
                    n++;
                    p = n*64;
                }
                int realLength = length - n*2;
                byte[] data = new byte[realLength];
                copyData(ftdiData, data);
                return data;
            }else
            {
                return Arrays.copyOfRange(ftdiData, 2, length);
            }
        }
*/
     public byte[] adaptArray(byte[] ftdiData)
        {
                return Arrays.copyOfRange(ftdiData, 0, ftdiData.length);
       }

Compile cleanly
 

Attachments

  • felUsbSerial.jar
    72.8 KB · Views: 42
  • felUsbSerial.xml
    7.3 KB · Views: 38
Last edited:
Upvote 0
W

weighment

Member
Licensed User
@Daestrum that was exactly what we needed to verify the theory.

The original 1.12 library works fine when the read buffer is 64 or less. It loses exactly two characters when the read buffer is 66 or more, and generates an exception when the read buffer is 65. This is because it is attempting to remove a two byte header on each 64 byte boundary, when in my experience with FTDI FT2232 PID 0x6010, there is only the front two byte header and no more in the buffer, up to the longest read buffer in my test of 176, and many other values above 64. In short there is no later header to remove, at least for this 0x6010 FT2232 chip.

Furthermore, the two byte header always contained 2 (0x2), 96(0x60) in those first two bytes, regardless of the message length. I don't find anything in the FTDI docs describing this header, so it may be added in some USB driver.

For my purposes, we could just strip the first two bytes only. Changing adaptArray to this:

adaptArray: 
public byte[] adaptArray(byte[] ftdiData)
        {
           if (ftdiData.length <2)
               {
                   return EMPTY_BYTE_ARRAY;
               }
               else {
                   return Arrays.copyOfRange(ftdiData, 2, ftdiData.length-2);
               }
       }

(or I can just use the library you sent me and strip the bytes myself.)

Not sure what chips if any had the second header at 65th and 66th bytes. If found, maybe it is worth adding a Mode to the header stripping in case someone thinks the other method works on some chip, or maybe those other chips put a different number in the header, perhaps related to the maximum buffer length possible.

Many thanks for the help on this. I will continue to test other FTDI chips as I get them, but this found the root cause.
 
Upvote 0

agraham

Expert
Licensed User
Longtime User
This is ringing a very faint bell. When I wrote the original UsbSerial library I found an extra two bytes at the start of each packet which apparently the chip adds and looking back at the source I see that I just ignored the first two bytes of any packet. At the time I thought this was a bug but I just found this.
It states
"The first 2 bytes of every packet are used as status bytes for the driver. This status is sent every 16 milliseconds, even when no data is present in the device."
 
Upvote 0

MicroDrie

Well-Known Member
Licensed User
Longtime User
This is the compiled version with Eclipse Version: 2024-06 (4.32.0) with Build id: 20240606-1231 using jdk-1.8.421 and Android API – 33.

M3felUsbSerial version 1.13 is based on version 1.12 of Erel and has the following fixes to made it free of error and warning message. I have fixed the following things:


  1. felUsbSerial class: no change;
  2. Full replacement of the FelHR85 software with the FelHR85/UsbSerial software version 6.0.6 from September 11, 2022 plus;
  3. Suppressed all "deprecated" warnings with @Deprecated in all classes of this library;
  4. Three bugfixes to fix the outdated API level 26 buffer usage errors there in plus which are firstly indicated as a warning;
  5. Vulnerabilities bugfix version updated to okio version 3.9.0.
What happend if the current felUsbSerial is replaced with M3felUsbSerial library (only different names to simple delete it if it doesn’t work).
 

Attachments

  • M3felUsbSerial.jar
    93.6 KB · Views: 33
  • M3felUsbSerial.xml
    7.3 KB · Views: 34
Upvote 0
W

weighment

Member
Licensed User
Many thanks @MicroDrie for building that.

Same error as @Daestrum 's rebuild at felUsbSerial.Initialize

error message on felUsbSerial.Initialize: 
/dev/bus/usb/001/003
1003
1027 24592
2
Connecting to /dev/bus/usb/001/003
java.lang.NoClassDefFoundError: Failed resolution of: Lokio/Buffer;
    at com.felhr.usbserial.SerialBuffer$SynchronizedBuffer.<init>(SerialBuffer.java:97)
    at com.felhr.usbserial.SerialBuffer.<init>(SerialBuffer.java:21)
    at com.felhr.usbserial.UsbSerialDevice.<init>(UsbSerialDevice.java:55)
    at com.felhr.usbserial.FTDISerialDevice.<init>(FTDISerialDevice.java:111)
    at com.felhr.usbserial.UsbSerialDevice.createUsbSerialDevice(UsbSerialDevice.java:73)
    at anywheresoftware.b4a.objects.usb.felUsbSerial.Initialize2(felUsbSerial.java:112)
    at anywheresoftware.b4a.objects.usb.felUsbSerial.Initialize(felUsbSerial.java:98)
    at com.tswa.indicator.wiredscalemanager._findandconnect(wiredscalemanager.java:265)
    at com.tswa.indicator.main._reconnect(main.java:616)
    at com.tswa.indicator.main$ResumableSub_Activity_Create.resume(main.java:554)
    at com.tswa.indicator.main._activity_create(main.java:420)
    at java.lang.reflect.Method.invoke(Native Method)
    at anywheresoftware.b4a.shell.Shell.runMethod(Shell.java:732)
    at anywheresoftware.b4a.shell.Shell.raiseEventImpl(Shell.java:351)
    at anywheresoftware.b4a.shell.Shell.raiseEvent(Shell.java:255)
    at java.lang.reflect.Method.invoke(Native Method)
    at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:157)
    at com.tswa.indicator.main.afterFirstLayout(main.java:108)
    at com.tswa.indicator.main.access$000(main.java:20)
    at com.tswa.indicator.main$WaitForLayout.run(main.java:86)
    at android.os.Handler.handleCallback(Handler.java:790)
    at android.os.Handler.dispatchMessage(Handler.java:99)
    at android.os.Looper.loop(Looper.java:164)
    at android.app.ActivityThread.main(ActivityThread.java:6518)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:438)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:807)
Caused by: java.lang.ClassNotFoundException: Didn't find class "okio.Buffer" on path: DexPathList[[zip file "/data/app/com.tswa.indicator-0NrUkJJ3Zj5ZYOZmCnF52g==/base.apk"],nativeLibraryDirectories=[/data/app/com.tswa.indicator-0NrUkJJ3Zj5ZYOZmCnF52g==/lib/arm64, /system/lib64, /vendor/lib64]]
    at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:125)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:379)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:312)
    ... 27 more

** Activity (main) Resume **
--------- beginning of crash
java.lang.RuntimeException: java.net.SocketException: Connection reset
    at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:188)
    at anywheresoftware.b4a.objects.Timer$TickTack.run(Timer.java:105)
    at android.os.Handler.handleCallback(Handler.java:790)
    at android.os.Handler.dispatchMessage(Handler.java:99)
    at android.os.Looper.loop(Looper.java:164)
    at android.app.ActivityThread.main(ActivityThread.java:6518)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:438)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:807)
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:209)
    at java.net.SocketInputStream.read(SocketInputStream.java:139)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:248)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:267)
    at java.io.DataInputStream.readByte(DataInputStream.java:268)
    at anywheresoftware.b4a.shell.Shell.raiseEventImpl(Shell.java:344)
    at anywheresoftware.b4a.shell.Shell.raiseEvent(Shell.java:255)
    at java.lang.reflect.Method.invoke(Native Method)
    at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:157)
    ... 8 more

I wonder where this "okio.Buffer" class is located
 
Last edited:
Upvote 0
W

weighment

Member
Licensed User
Great find @agraham . FTDI is bad about not providing source code for drivers even in Linux and Android.

This chip FTDI FT2232 PID 0x6010 has a 128 byte tx buffer and 384 byte rx buffer. That doc may have originated on a device with a smaller buffer.

Though that may amount to the driver transfer size
https://ftdichip.com/Support/Knowledgebase/an232b_03transfersize.htm

Nagging question: What is in the status bytes? Is it modem status and line status?


Doesn't seem like it is the Queue status (bytes in the queue) because it always has the same value for my test


Maybe the status header from the part datasheet, which only accounts for one of the bytes, and would logically hold the 0x02


Event status is only one byte and it just tells what events are armed
Notification Events (see FT_SetEventNotification)
FT_EVENT_RXCHAR = 1
FT_EVENT_MODEM_STATUS = 2
FT_EVENT_LINE_STATUS = 4

At the core so far, it appears both of these errors (lost chars and array exceptions) are caused by trying to process status headers in the middle of the buffer, that I have yet to see exist in practice.
 
Upvote 0

agraham

Expert
Licensed User
Longtime User
weighment said:
FTDI is bad about not providing source code for drivers even in Linux and Android
Click to expand...
Used to be worse! When I was writing UsbSerial the chip datasheets were treated as proprietary and you had to reverse engineer somebody's driver to get an inkling about how to interface to the chip.
 
Upvote 0

MicroDrie

Well-Known Member
Licensed User
Longtime User
weighment said:
I wonder where this "okio.Buffer" class is located
Click to expand...
Good question, its a import Kotlin library class loaded in the public class SerialBuffer.
 
Upvote 0
You must log in or register to reply here.

Similar Threads

  • Question
Android Question Trying to map a JSON entry error (SOLVED)
Replies
13
Views
1K
Colin Evans
  • Question
Android Question for next loop, set value of an array
Replies
10
Views
2K
LucaMs
Menu
Log in
Register
Cookies are required to use this site. You must accept them to continue using the site. Learn more…