Android Question Using RandomAccessFiles written by B4J with B4A

[GiovanniPolese]

Hi to All
I have huge files to be processed by B4A on a Tablet. I need to have them in binary format and manage them with RandomAccessFile. Excluding the possibility to write them by Windows program, I thought to do it with B4J, starting from original text files on a Desktop. Of course I can copy the original files on the tablet and convert to binary on it with B4A, but it will take many hours to do this conversion. Therefore my idea was to write RandomAccessFiles using B4J on a desktop and copy the converted binary files on the tablet. This doesn't work. Trying to open the RandomAccessFile (done on the desktop by B4J), on the Tablet, I get "java.lang.NullPointerException: null receiver" at the B4A line which initializes the RandomAccessFile. Therefore there is no compatibility between binary files written by B4J and B4A, I guess. Or I hope to miss something ..
Thanks in advance

 

[GiovanniPolese]

Anyway, it seems to me that running same code in B4J on DeskTop or in B4A on tablet, is not so different in terms of time. Maybe this is obvious. It was not obvious for me. At the end, it took a reasonable time to execute on the tablet.. so my idea to Use B4J for this task was not so good. Thanks for reading.

 

[LucaMs]

I get "java.lang.NullPointerException: null receiver"

The error message seems to be about a receiver so it has nothing to do with text and binary files.

Anyway, what code do you use to do the conversions?

 

[GiovanniPolese]

I said "conversion" but it is not actually a conversion. I just read a text files with coordinates of points and write them using RandomAccessFile. This on B4J side. Then I copy the files to the Tablet and try to read them with B4A. Code in B4J is the same than in B4A. Just using Initialize and WriteInt and WriteFloat to write and analogous functions to read. BTW same code works if I write and read the files directly on tablet. Anyway, as I wrote, the time for doing the operation all in the Tablet is not so longer than on Tablet. The file produced by B4J is attached. It starts with a long variable (the number of points) followed by the sequence of 3 floats.

 

[GiovanniPolese]

"longer than on DeskTop"

 

[LucaMs]

Having only the binary file we can't help you either prevent the error or improve the app performance.

 

[teddybear]

You'd better upload a small project

 

[GiovanniPolese]

Thanks for your patience. Actually there is not much to say. Just open a B4A RandomAccessFile on that file. Nothing else. The error happens on Initializing the RandomAccessFile. For further discussin, I should upload a B4J project for writing, and a B4A project for reading, but using absolutely same code .. You understand that it is a little nonsense. It may be that I am completely crazy (possible), or did an absurd mistake (possible too) or that B4A simply doesn't read B4J RandomAccessFile binaries. Don't waste your good will for this problem. I have resolved anyway. Forget about. Thanks again

 

[emexes]

It starts with a long variable (the number of points) followed by the sequence of 3 floats.

Minor point, but: your example file seems to start with long value 100,000 but then followed by 100,001 points (of 3 floats = 12 bytes).

Or does the 100,000 count not include the first or last point?

 

[GiovanniPolese]

Hi. I did various trials and some things may be not perfect. Forget about this details. In fact the actual number of points is about 100 millions and I posted a reduced version of the file, for obvious reasons, and perhaps I did the classical +/- 1 mistake. Anyway, if you are able to read the file with B4A, you don't confirm what happens to me. So it may be a problem of mine, here.

 

[emexes]

if you are able to read the file with B4A

You might be on to something. In B4A, reading the file as an InputStream from the Assets folder, it reads the first 75 points no problem, but then on the 76th point there are an additional three zero bytes (ie 0x00 0x00 0x00) appearing in the stream, which knocks all the subsequent point values out.

I'm still poking around to try pinpoint wtf is happening.

I do vaguely remember that the Assets folder (derived from the Files tab in the IDE) is a kind-of virtual file system, and that the files in it are possibly extracted on-the-fly from the APK compressed installed resources, and some normal file operations don't work quite like you'd otherwise be expecting.

 

[emexes]

I'm still poking around to try pinpoint wtf is happening.

Ok, it reads the first 917 bytes ok.

After it has read 916 bytes, and goes to read the next 4 bytes ie IEEE single float, the buffer comes back with the first byte correct and the other three bytes remain 0x00 from when the 4-byte buffer was initialized by Dim b(4) As Byte.

So the same code that just worked 227 times to successfully read that many Floats, now chokes on the 228th read.

If I just read the first 1000 bytes in one chunk - which should be no problem, because BytesAvailable indicates the entire 1.2 MB file is available - then the read stops at the same place, ie returns 917 bytes of data followed by 83 unfilled/unread 0x00.

Wtf?

I suppose the easiest fix would be to copy the file from the Assets folder into a real folder, and use RandomAccessFile to read it, but still, I'm keen to find out why the obvious way of doing this, is not working. Lol first step being to update my B4A, I'm sure I'm at least 6 months behind the times. Version 12.80.

 

[emexes]

This is as far as I've gotten.

Started new default B4A project.
Put example file Points1.bin from post #4 above into Files tab.
Ticked ByteConverter library.
Added this code:

Sub Button1_Click

    '''xui.MsgboxAsync("Hello world!", "B4X")
   
    Dim fh As InputStream = File.OpenInput(File.DirAssets, "Points1.bin")

    For I = 1 To 12
        Dim b(100) As Byte
        fh.ReadBytes(b, 0, b.Length)
   
        Dim bc As ByteConverter
        Log(I & TAB & bc.HexFromBytes(b))
    Next
       
End Sub

Got this output, reading chunks of 100 bytes, first 9 reads are ok, 10th read seems to only read 17 bytes (34 hex digits):

Logger connected to:  HMD Global Nokia C01 Plus
--------- beginning of system
--------- beginning of main
1     00000000000186A0C752AC9AC80E8EF341DE147BC752AC66C80E8EF341DE147BC752AC33C80E8EF341DE147BC752AC00C80E8EF341DE147BC752ABCDC80E8EF341DE147BC752AB9AC80E8EF341DE28F6C752AB66C80E8EF341DE8F5CC752AB33C80E8EF3
2     41DEF5C3C752AB00C80E8EF341DF5C29C752AACDC80E8EF341DFAE14C752AA9AC80E8EF341E0147BC752AA66C80E8EF341E07AE1C752AA33C80E8EF341E0E148C752AA00C80E8EF341E13333C752A9CDC80E8EF341E15C29C752A99AC80E8EF341E1851F
3     C752A966C80E8EF341E1999AC752A933C80E8EF341E1AE14C752A900C80E8EF341E1C28FC752A8CDC80E8EF341E1D70AC752A89AC80E8EF341E1EB85C752A866C80E8EF341E2147BC752A833C80E8EF341E23D71C752A800C80E8EF341E26666C752A7CD
4     C80E8EF341E28F5CC752A79AC80E8EF341E2B852C752A766C80E8EF341E2E148C752A733C80E8EF341E30A3DC752A700C80E8EF341E31EB8C752A6CDC80E8EF341E347AEC752A69AC80E8EF341E370A4C752A666C80E8EF341E3999AC752A633C80E8EF3
5     41E3C28FC752A600C80E8EF341E3EB85C752A5CDC80E8EF341E4147BC752A59AC80E8EF341E43D71C752A566C80E8EF341E46666C752A533C80E8EF341E4A3D7C752A500C80E8EF341E4E148C752A4CDC80E8EF341E50A3DC752A49AC80E8EF341E53333
6     C752A466C80E8EF341E570A4C752A433C80E8EF341E5999AC752A400C80E8EF341E5D70AC752A3CDC80E8EF341E60000C752A39AC80E8EF341E628F6C752A366C80E8EF341E63D71C752A333C80E8EF341E651ECC752A300C80E8EF341E66666C752A2CD
7     C80E8EF341E66666C752A29AC80E8EF341E67AE1C752A266C80E8EF341E68F5CC752A233C80E8EF341E68F5CC752A200C80E8EF341E68F5CC752A1CDC80E8EF341E6A3D7C752A19AC80E8EF341E6B852C752A166C80E8EF341E6CCCDC752A133C80E8EF3
8     41E6E148C752B533C80E8EE641D6F5C3C752B500C80E8EE641D70A3DC752B4CDC80E8EE641D73333C752B49AC80E8EE641D770A4C752B466C80E8EE641D7999AC752B433C80E8EE641D7D70AC752B400C80E8EE641D80000C752B3CDC80E8EE641D828F6
9     C752B39AC80E8EE641D851ECC752B366C80E8EE641D87AE1C752B333C80E8EE641D8A3D7C752B300C80E8EE641D8CCCDC752B2CDC80E8EE641D8F5C3C752B29AC80E8EE641D90A3DC752B266C80E8EE641D947AEC752B233C80E8EE641D9851FC752B200
10    C80E8EE641D9AE14C752B1CDC80E8EE6410000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
11    D9EB85C752B19AC80E8EE641DA51ECC752B166C80E8EE641DAA3D7C752B133C80E8EE641DACCCDC752B100C80E8EE641DAF5C3C752B0CDC80E8EE641DB1EB8C752B09AC80E8EE641DB47AEC752B066C80E8EE641DB70A4C752B033C80E8EE641DB999AC7
12    52B000C80E8EE641DBC28FC752AFCDC80E8EE641DBEB85C752AF9AC80E8EE641DC147BC752AF66C80E8EE641DC3D71C752AF33C80E8EE641DC6666C752AF00C80E8EE641DCA3D7C752AECDC80E8EE641DCCCCDC752AE9AC80E8EE641DCF5C3C752AE66C8

 

[emexes]

Read from DirInternal using RandomAccessFile rather than from DirAssets using File, no longer chokes like it did in previous post.

Sub Button1_Click

    '''xui.MsgboxAsync("Hello world!", "B4X")
 
    File.Copy(File.DirAssets, "Points1.bin", File.DirInternal, "Points1.bin")
 
    Dim raf As RandomAccessFile
    raf.Initialize(File.DirInternal, "Points1.bin", True)
 
    For I = 1 To 12
        Dim b(100) As Byte
        Dim Result As Int = raf.ReadBytes(b, 0, b.Length, raf.CurrentPosition)
 
        Dim bc As ByteConverter
        Log(I & TAB & Result & TAB & bc.HexFromBytes(b))
    Next
     
End Sub

Logger connected to:  HMD Global Nokia C01 Plus
--------- beginning of main
--------- beginning of system
1    100    00000000000186A0C752AC9AC80E8EF341DE147BC752AC66C80E8EF341DE147BC752AC33C80E8EF341DE147BC752AC00C80E8EF341DE147BC752ABCDC80E8EF341DE147BC752AB9AC80E8EF341DE28F6C752AB66C80E8EF341DE8F5CC752AB33C80E8EF3
2    100    41DEF5C3C752AB00C80E8EF341DF5C29C752AACDC80E8EF341DFAE14C752AA9AC80E8EF341E0147BC752AA66C80E8EF341E07AE1C752AA33C80E8EF341E0E148C752AA00C80E8EF341E13333C752A9CDC80E8EF341E15C29C752A99AC80E8EF341E1851F
3    100    C752A966C80E8EF341E1999AC752A933C80E8EF341E1AE14C752A900C80E8EF341E1C28FC752A8CDC80E8EF341E1D70AC752A89AC80E8EF341E1EB85C752A866C80E8EF341E2147BC752A833C80E8EF341E23D71C752A800C80E8EF341E26666C752A7CD
4    100    C80E8EF341E28F5CC752A79AC80E8EF341E2B852C752A766C80E8EF341E2E148C752A733C80E8EF341E30A3DC752A700C80E8EF341E31EB8C752A6CDC80E8EF341E347AEC752A69AC80E8EF341E370A4C752A666C80E8EF341E3999AC752A633C80E8EF3
5    100    41E3C28FC752A600C80E8EF341E3EB85C752A5CDC80E8EF341E4147BC752A59AC80E8EF341E43D71C752A566C80E8EF341E46666C752A533C80E8EF341E4A3D7C752A500C80E8EF341E4E148C752A4CDC80E8EF341E50A3DC752A49AC80E8EF341E53333
6    100    C752A466C80E8EF341E570A4C752A433C80E8EF341E5999AC752A400C80E8EF341E5D70AC752A3CDC80E8EF341E60000C752A39AC80E8EF341E628F6C752A366C80E8EF341E63D71C752A333C80E8EF341E651ECC752A300C80E8EF341E66666C752A2CD
7    100    C80E8EF341E66666C752A29AC80E8EF341E67AE1C752A266C80E8EF341E68F5CC752A233C80E8EF341E68F5CC752A200C80E8EF341E68F5CC752A1CDC80E8EF341E6A3D7C752A19AC80E8EF341E6B852C752A166C80E8EF341E6CCCDC752A133C80E8EF3
8    100    41E6E148C752B533C80E8EE641D6F5C3C752B500C80E8EE641D70A3DC752B4CDC80E8EE641D73333C752B49AC80E8EE641D770A4C752B466C80E8EE641D7999AC752B433C80E8EE641D7D70AC752B400C80E8EE641D80000C752B3CDC80E8EE641D828F6
9    100    C752B39AC80E8EE641D851ECC752B366C80E8EE641D87AE1C752B333C80E8EE641D8A3D7C752B300C80E8EE641D8CCCDC752B2CDC80E8EE641D8F5C3C752B29AC80E8EE641D90A3DC752B266C80E8EE641D947AEC752B233C80E8EE641D9851FC752B200
10   100    C80E8EE641D9AE14C752B1CDC80E8EE641D9EB85C752B19AC80E8EE641DA51ECC752B166C80E8EE641DAA3D7C752B133C80E8EE641DACCCDC752B100C80E8EE641DAF5C3C752B0CDC80E8EE641DB1EB8C752B09AC80E8EE641DB47AEC752B066C80E8EE6
11   100    41DB70A4C752B033C80E8EE641DB999AC752B000C80E8EE641DBC28FC752AFCDC80E8EE641DBEB85C752AF9AC80E8EE641DC147BC752AF66C80E8EE641DC3D71C752AF33C80E8EE641DC6666C752AF00C80E8EE641DCA3D7C752AECDC80E8EE641DCCCCD
12   100    C752AE9AC80E8EE641DCF5C3C752AE66C80E8EE641DD1EB8C752AE33C80E8EE641DD5C29C752AE00C80E8EE641DD851FC752ADCDC80E8EE641DDAE14C752AD9AC80E8EE641DDD70AC752AD66C80E8EE641DE147BC752AD33C80E8EE641DE147BC752AD00

 

[emexes]

Looking good:

Sub Button1_Click

    '''xui.MsgboxAsync("Hello world!", "B4X")
  
    Dim bc As ByteConverter
  
    File.Copy(File.DirAssets, "Points1.bin", File.DirInternal, "Points1.bin")
  
    Dim raf As RandomAccessFile
    raf.Initialize(File.DirInternal, "Points1.bin", True)
  
    Dim b(8) As Byte
    raf.ReadBytes(b, 0, b.Length, raf.CurrentPosition)
  
    Dim NumPoints As Int = bc.LongsFromBytes(b)(0)
    Log(NumPoints)
  
    Dim PointNumber As Int = 0
    Do While True
        Dim b(12) As Byte
        Dim Result As Int = raf.ReadBytes(b, 0, b.Length, raf.CurrentPosition)

        If Result < 12 Then Exit
              
        PointNumber = PointNumber + 1
      
        If PointNumber < 6 Or PointNumber > NumPoints - 6 Then
            Dim fff() As Float = bc.FloatsFromBytes(b)

            Dim xf As String = NumberFormat2(fff(0), 1, 5, 5, False)
            Dim yf As String = NumberFormat2(fff(1), 1, 5, 5, False)
            Dim zf As String = NumberFormat2(fff(2), 1, 5, 5, False)
          
            Log(PointNumber & TAB & xf & TAB & yf & TAB & zf)
        End If
    Loop
          
End Sub

Logger connected to:  HMD Global Nokia C01 Plus
--------- beginning of system
--------- beginning of main
100000
1        -53932.60156    -145979.79688    27.76000
2        -53932.39844    -145979.79688    27.76000
3        -53932.19922    -145979.79688    27.76000
4        -53932.00000    -145979.79688    27.76000
5        -53931.80078    -145979.79688    27.76000
99995    -54153.00000    -145968.20312    18.60000
99996    -54152.80078    -145968.20312    18.60000
99997    -54152.60156    -145968.20312    18.60000
99998    -54152.39844    -145968.20312    18.59000
99999    -54152.19922    -145968.20312    18.59000
100000   -54152.00000    -145968.20312    18.58000
100001   -54151.80078    -145968.20312    18.58000

 

[GiovanniPolese]

Hi. You did a great and interesting job, but, anyway I never use File.DirAssets or File.DirInternal for files which are not fixed and independent from users (and small too), so I cannot say anything on any problem connected to the use of such folders. I only use GetSafeDefaultExternal and directories internal to it. This said, my problem was before reading: the App crashed on initializing the RandomAccessFile, with the weird message about "null receiver" that I reported on first post. Anyway it is interesting the fact that you were able to extract the data. The remarkable result is that you read the raw bytes and convert them to long or float (etc.) to get the real data. Therefore we can say that B4J and B4A may exchange binary data, but the exchange needs a ByteConversion, if I understood well. Paradoxically my initial problem remains unsolved. Thanks for your attention and work.

 

[Erel]

Why aren't you using B4XSerializator?

 

[GiovanniPolese]

Why aren't you using B4XSerializator?

Hi. I used RandomAccessFile, simply because I never used B4XSerializator. I saw that RandomAccessFile was available in both systems and thought that it was compatible. It was an experiment to pre-process data on desktop with B4J and use results on Tablet with B4A. I stopped immediately because of the weird error that I reported in my first post. If you say that B4XSerializator must be used, I will give it a try.
Thanks a lot.

 

[Erel]

RandomAccessFile can be used in all platforms, however B4XSerializator is built for this and is easier to use. Example: https://www.b4x.com/android/forum/t...-asyncstreams-b4xserializator.119011/#content