Android Question Best way to upload and download files to internet? Simple example?

[RB Smissaert]

Have a large amount of data that would be too much to go into the .apk file.
What would be the best way to upload this data (ideally from the phone) and then download it and process it?
The data will be in a simple .csv or .txt file.
Once I have the data in File.DirRootExternal I can process the .csv or .txt file and move the data to SQLite.
Did quite a bit of searching the forum, but not clear to me what the best way would be.

RBS

 

[RB Smissaert]

Which reminds me: didn't the APK compression (pretty sure it's a Zip file) work as (un)advertised? You might be doing stuff you don't need to do. Although it'll chew up storage on the phone compared to if you decompress on-the-fly whilst reading, so...

Unless you can read directly from File.DirAssets... tbh I don't think I've ever tried that... always copied to File.DirInternal first... I think was because RandomAccessFile operations didn't work with DirAssets files. Maybe sequential reading via a Stream works ok.

OK, if the the files are already zipped to .apk then I will just have a go at adding all that (unzipped) to the .apk and see how it all goes.

RBS

 

[emexes]

OK, if the the files are already zipped to .apk then I will just have a go at adding all that (unzipped) to the .apk and see how it all goes.

The main problem I noticed with a big file in the Assets folder, was that the automatic regular timed backup would slow the computer down whilst it compressed the big file. It was bearable when the file was pre-shrunk.

 

[RB Smissaert]

Which reminds me: didn't the APK compression (pretty sure it's a Zip file) work as (un)advertised? You might be doing stuff you don't need to do. Although it'll chew up storage on the phone compared to if you decompress on-the-fly whilst reading, so...

Unless you can read directly from File.DirAssets... tbh I don't think I've ever tried that... always copied to File.DirInternal first... I think was because RandomAccessFile operations didn't work with DirAssets files. Maybe sequential reading via a Stream works ok.

I added 14 .csv files, uncompressed 102 Mb to the Assets folder and compiled.
.apk size went from 14.2 Mb to 38.7 Mb.
The size of those 14 files, zipped with 7-zip is 11.5 Mb, so it seems 7-zip compresses about twice as much as the .apk zipper.
Still, not sure if it is worth the extra trouble to add a zipped file to the .apk.
Also 38.7 Mb seems OK for the .apk, so I probably go with adding unzipped files.
Happy to be proven wrong here.

RBS

 

[RB Smissaert]

I added 14 .csv files, uncompressed 102 Mb to the Assets folder and compiled.
.apk size went from 14.2 Mb to 38.7 Mb.
The size of those 14 files, zipped with 7-zip is 11.5 Mb, so it seems 7-zip compresses about twice as much as the .apk zipper.
Still, not sure if it is worth the extra trouble to add a zipped file to the .apk.
Also 38.7 Mb seems OK for the .apk, so I probably go with adding unzipped files.
Happy to be proven wrong here.

RBS

Two problems I encountered with this:
1. Not sure if it is to do with the size of the .apk or the size of the project, but the B4A IDE sometimes goes unresponsive. It recovers OK, but
it is somewhat worrying.
2. Although the .csv are added in the Files tab and do show in the Assets folder I get this error message:

java.lang.RuntimeException: java.io.FileNotFoundException: AssetsDir/postcode_lsoa11cd$1.csv: open failed: ENOENT (No such file or directory)
at anywheresoftware.b4a.keywords.Common$14.run(Common.java:1750)
at android.os.Handler.handleCallback(Handler.java:938)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loopOnce(Looper.java:226)
at android.os.Looper.loop(Looper.java:313)
at android.app.ActivityThread.main(ActivityThread.java:8751)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:571)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1135)
Caused by: java.io.FileNotFoundException: AssetsDir/postcode_lsoa11cd$1.csv: open failed: ENOENT (No such file or directory)
at libcore.io.IoBridge.open(IoBridge.java:575)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:289)
at anywheresoftware.b4a.randomaccessfile.RandomAccessFile.Initialize2(RandomAccessFile.java:103)
at anywheresoftware.b4a.randomaccessfile.RandomAccessFile.Initialize(RandomAccessFile.java:96)
at b4a.exampleljjll.b4xmainpage$ResumableSub_GetCharacterEncodingFromFile.resume(b4xmainpage.java:18949)
at b4a.exampleljjll.b4xmainpage._getcharacterencodingfromfile(b4xmainpage.java:18904)
at b4a.exampleljjll.b4xmainpage$ResumableSub_CSV2SQLiteTable.resume(b4xmainpage.java:13106)
at b4a.exampleljjll.b4xmainpage._csv2sqlitetable(b4xmainpage.java:12818)
at b4a.exampleljjll.b4xmainpage$ResumableSub_PopulateTable.resume(b4xmainpage.java:41600)
at anywheresoftware.b4a.keywords.Common$14.run(Common.java:1748)
... 8 more
Caused by: android.system.ErrnoException: open failed: ENOENT (No such file or directory)
at libcore.io.Linux.open(Native Method)
at libcore.io.ForwardingOs.open(ForwardingOs.java:567)
at libcore.io.BlockGuardOs.open(BlockGuardOs.java:273)
at libcore.io.ForwardingOs.open(ForwardingOs.java:567)
at android.app.ActivityThread$AndroidOs.open(ActivityThread.java:8611)
at libcore.io.IoBridge.open(IoBridge.java:561)
... 17 more

I do use a custom class to move data from .csv to SQLite and this uses RandomAccessFile.
Could this be the problem?

RBS

 

[RB Smissaert]

Two problems I encountered with this:
1. Not sure if it is to do with the size of the .apk or the size of the project, but the B4A IDE sometimes goes unresponsive. It recovers OK, but
it is somewhat worrying.
2. Although the .csv are added in the Files tab and do show in the Assets folder I get this error message:

java.lang.RuntimeException: java.io.FileNotFoundException: AssetsDir/postcode_lsoa11cd$1.csv: open failed: ENOENT (No such file or directory)
at anywheresoftware.b4a.keywords.Common$14.run(Common.java:1750)
at android.os.Handler.handleCallback(Handler.java:938)
at android.os.Handler.dispatchMessage(Handler.java:99)
at android.os.Looper.loopOnce(Looper.java:226)
at android.os.Looper.loop(Looper.java:313)
at android.app.ActivityThread.main(ActivityThread.java:8751)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:571)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1135)
Caused by: java.io.FileNotFoundException: AssetsDir/postcode_lsoa11cd$1.csv: open failed: ENOENT (No such file or directory)
at libcore.io.IoBridge.open(IoBridge.java:575)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:289)
at anywheresoftware.b4a.randomaccessfile.RandomAccessFile.Initialize2(RandomAccessFile.java:103)
at anywheresoftware.b4a.randomaccessfile.RandomAccessFile.Initialize(RandomAccessFile.java:96)
at b4a.exampleljjll.b4xmainpage$ResumableSub_GetCharacterEncodingFromFile.resume(b4xmainpage.java:18949)
at b4a.exampleljjll.b4xmainpage._getcharacterencodingfromfile(b4xmainpage.java:18904)
at b4a.exampleljjll.b4xmainpage$ResumableSub_CSV2SQLiteTable.resume(b4xmainpage.java:13106)
at b4a.exampleljjll.b4xmainpage._csv2sqlitetable(b4xmainpage.java:12818)
at b4a.exampleljjll.b4xmainpage$ResumableSub_PopulateTable.resume(b4xmainpage.java:41600)
at anywheresoftware.b4a.keywords.Common$14.run(Common.java:1748)
... 8 more
Caused by: android.system.ErrnoException: open failed: ENOENT (No such file or directory)
at libcore.io.Linux.open(Native Method)
at libcore.io.ForwardingOs.open(ForwardingOs.java:567)
at libcore.io.BlockGuardOs.open(BlockGuardOs.java:273)
at libcore.io.ForwardingOs.open(ForwardingOs.java:567)
at android.app.ActivityThread$AndroidOs.open(ActivityThread.java:8611)
at libcore.io.IoBridge.open(IoBridge.java:561)
... 17 more

I do use a custom class to move data from .csv to SQLite and this uses RandomAccessFile.
Could this be the problem?

RBS

In debug mode I can see where the error occurs:

Log(tFF.strFolder & " _ " & tFF.strFile)
RAF.Initialize(tFF.strFolder, tFF.strFile, True)

The log shows OK:
AssetsDir _ POSTCODE_LSOA11CD$1.csv

I can see now though that RAF.Initialize can't be used on files in the Asset folder, so maybe I need to copy them or have them
somewhere else eg File.DirInternal?

RBS

 

[RB Smissaert]

In debug mode I can see where the error occurs:

Log(tFF.strFolder & " _ " & tFF.strFile)
RAF.Initialize(tFF.strFolder, tFF.strFile, True)

The log shows OK:
AssetsDir _ POSTCODE_LSOA11CD$1.csv

I can see now though that RAF.Initialize can't be used on files in the Asset folder, so maybe I need to copy them or have them
somewhere else eg File.DirInternal?

RBS

With a copy to File.DirInternal all works fine.
I can delete the files after having moved them to SQLite.

RBS

 

[emexes]

I can see now though that RAF.Initialize can't be used on files in the Asset folder

I could well believe that RandomAccessFile doesn't work with files read directly from the assets folder, if those files require decompressing.

Decompression is usually a sequential process, where later data depends on earlier data.

But it might work if you read it sequentially with an InputStream.

Might even be able to use TextReader with it, to read it line-by-line.

Although now I'm having flashbacks to you and I working on reading large CSV files and using RandomAccessFile because of some anomaly with the EOL bytes, or long lines, or... maybe it was a speed issue.

 

[emexes]

it seems 7-zip compresses about twice as much as the .apk zipper.

Lol good point. Compressors usually have a setting for the speed-vs-size tradeoff, and I'd expect - in debug mode at least - that the APK compressor is set to maximum speed. Also, 7-Zip format does "solid" compression (I think) which helps when compressing multiple similar files.

I'll recompress a random APK here, see what happens. No, better, I'll dig up a GNAF file, stick that in it. Lol my problem will be finding a file small enough, I'm pretty sure they're over 6 GB now.

 

[emexes]

finding a file small enough

GNAF files are along lines of what postcode data would look like.
PSV = Pipe Separated Values

>find "UPALONG" *psv

---------- GNAF_CORE_RETIRED.PSV
GAVIC414546897|29-04-2004|12A UPALONG RD, MOUNT DANDENONG VIC 3767|||||||12A|||UPALONG|ROAD||MOUNT DANDENONG|VIC|3767|2\PS542209|20648430000|PRINCIPAL||||FRONTAGE CENTRE SETBACK|145.35421241|-37.83442417
GAVIC414546895|29-04-2004|1 UPALONG RD, MOUNT DANDENONG VIC 3767|||||||1|||UPALONG|ROAD||MOUNT DANDENONG|VIC|3767||20648430000|PRINCIPAL||||STREET LOCALITY|145.35499422|-37.83491969

>pkzip25 -add -speed testfast *retired*
>pkzip25 -add -maximum testsmall *retired*
>7z a -mx1 testfast *retired*
>7z a -mx9 testsmall *retired*

That last compression took ~2 minutes compared to the other three were ~10 seconds.

>dir /od

17/08/2023  01:52 PM       135,994,541 GNAF_CORE_RETIRED.psv
20/02/2025  12:26 PM        50,067,680 testfast.zip
20/02/2025  12:27 PM        43,284,277 testsmall.zip
20/02/2025  12:31 PM        41,776,507 testfast.7z
20/02/2025  12:33 PM        15,988,445 testsmall.7z

 

[RB Smissaert]

I could well believe that RandomAccessFile doesn't work with files read directly from the assets folder, if those files require decompressing.

Decompression is usually a sequential process, where later data depends on earlier data.

But it might work if you read it sequentially with an InputStream.

Might even be able to use TextReader with it, to read it line-by-line.

Although now I'm having flashbacks to you and I working on reading large CSV files and using RandomAccessFile because of some anomaly with the EOL bytes, or long lines, or... maybe it was a speed issue.

> Although now I'm having flashbacks to you and I working on reading large CSV files and using RandomAccessFile

Indeed, for some reason I needed to have something more advanced than the standard text file parser and you helped with some code to get the character encoding
of the files:

Sub GetCharacterEncodingFromFile(tFF As tFolderAndFile, lLookAtXBytes As Long) As ResumableSub
    
    Dim i As Long
    Dim RAF As RandomAccessFile
    Dim lBytes As Long
    Dim PreviousByte As Int    'actually just the top 2 bits
    Dim ThisByte As Int = 0    'actually just the top 2 bits
    Dim InvalidUTF8Flag As Boolean = False
    
    '------------------------------------------------
    'this can't be used on files in File.DirAssets !!
    '------------------------------------------------
    RAF.Initialize(tFF.strFolder, tFF.strFile, True)
    lBytes = lLookAtXBytes
    Dim TextBytes(lBytes) As Byte
    RAF.ReadBytes(TextBytes, 0, lBytes, 0)
    
    For i = 0 To TextBytes.Length - 1
        PreviousByte = ThisByte
        ThisByte = Bit.And(TextBytes(i), 0xC0)    'top 2 bits

        If PreviousByte = 0xC0 Then      'if multibyte start
            If ThisByte <> 0x80 Then     'not followed by multibyte continuation
                InvalidUTF8Flag = True
                Exit
            End If
        else if ThisByte = 0x80 Then       'if multibyte continuation
            If PreviousByte < 0x80 Then    'not preceded by multibyte start or continuation
                InvalidUTF8Flag = True
                Exit
            End If
        End If
    Next
    
    RAF.Close
 
    If InvalidUTF8Flag Then
        Return "windows-1252"
    Else
        Return "UTF-8"    'or plain ASCII
    End If
    
End Sub

RBS

 

[emexes]

I was on a roll with your original approach ie downloading from web.

I gzip'd that GNAF file, put it on my gdrive, made a download link, tested download link in browser, works great.

But HttpJob isn't downloading the exact same link. Nor coming up with an error. I am wondering if it is timing out - command line WGET download took minutes cf browser took seconds.

View link (human-friendly):
https://drive.google.com/file/d/13WGOgiVH3vkrl6RbMSo_xTgbENO5sa_c/view

Download link:
https://drive.usercontent.google.co...6RbMSo_xTgbENO5sa_c&export=download&confirm=t

 

[emexes]

But HttpJob isn't downloading the exact same link. Nor coming up with an error.

Lol I remembered halfway through mowing front yard that B4J command line programs exit prematurely on Wait Fors.

Sub Button1_Click
    '''xui.MsgboxAsync("Hello World!", "B4X")
   
    Dim fdir As String = File.DirApp
    Dim fname As String = "GNAF_CORE_RETIRED.psv.gz"

#IF DEBUG
    If File.Exists(fdir, fname) Then
        File.Delete(fdir, fname)    'force download during testing
    End If
#END IF

    If Not(File.Exists(fdir, fname)) Then
        'https://drive.google.com/file/d/13WGOgiVH3vkrl6RbMSo_xTgbENO5sa_c/view        Dim url As String = "https://drive.usercontent.google.com/download?id=13WGOgiVH3vkrl6RbMSo_xTgbENO5sa_c&export=download&authuser=0&confirm=t&uuid=dbd77f3f-99a0-4e8e-a7f2-51dadc7e872a&at=AEz70l6Ng30IuPp8vVw7-RfYxQZ6%3A1740019218647"
        Dim url As String = "https://drive.usercontent.google.com/download?id=13WGOgiVH3vkrl6RbMSo_xTgbENO5sa_c&export=download&authuser=0&confirm=t&uuid=dbd77f3f-99a0-4e8e-a7f2-51dadc7e872a&at=AEz70l6Ng30IuPp8vVw7-RfYxQZ6%3A1740019218647"
        Dim url As String = "https://drive.usercontent.google.com/download?id=13WGOgiVH3vkrl6RbMSo_xTgbENO5sa_c&export=download&confirm=t"
        '''Dim url As String = "https://www.redlightcemetery.com/rbs/" & fname
       
        Dim job As HttpJob
        job.Initialize("DownloadJob", Me)
        job.Download(url)
       
        Log("Downloading from: " & url)

        Dim StartTime As Long = DateTime.Now
        Wait For (job) JobDone(job As HttpJob)
        Dim EndTime As Long = DateTime.Now
       
        Log(job.Success)
        If job.Success Then
            ' Save the downloaded file
            Dim out As OutputStream = File.OpenOutput(fdir, fname, False)
            File.Copy2(job.GetInputStream, out)
            out.Close
           
            Log("Downloaded " & File.Size(fdir, fname) & " bytes in " & ((EndTime - StartTime + 999) / 1000).As(Int) & " seconds")
        Else
            Log("Error downloading file: " & job.ErrorMessage)
        End If
       
        job.Release
    End If

    If File.Exists(fdir, fname) Then
        Dim in As InputStream = File.OpenInput(fdir, fname)

        Dim compress As CompressedStreams
        Dim ingzip As InputStream = compress.WrapInputStream(in, "gzip")

        Dim tr As TextReader
        tr.Initialize(ingzip)
       
        Dim StartTime As Long = DateTime.Now
       
        Dim LineNumber As Int = 0
        Do While True
            Dim L As String = tr.ReadLine
            If L = Null Then Exit
           
            LineNumber = LineNumber + 1
           
            If L.Contains("UPALO") Then
                Dim F() As String = Regex.Split("\|", L)
                Dim HeaderFlag As Boolean = False
                For I = 9 To 17
                    If I < F.Length Then
                        If F(I).Length <> 0 Then
                            If Not(HeaderFlag) Then
                                Log("Found on line " & LineNumber)
                                HeaderFlag = True
                            End If
                            Log(I & TAB & F(I))
                        End If
                    End If
                Next
            End If
        Loop
       
        Dim EndTime As Long = DateTime.Now
        Dim ElapsedTime As Int = EndTime - StartTime
        Log("Searched " & LineNumber & " lines in " & ElapsedTime & " ms direct from gzip file")
        If ElapsedTime <> 0 Then
            Log((LineNumber / ElapsedTime * 1000).As(Int) & " lines per second")
        End If

        tr.Close
        ingzip.Close
        in.Close      
    End If
End Sub

Waiting for debugger to connect...
Program started.
Downloading from: https://drive.usercontent.google.com/download?id=13WGOgiVH3vkrl6RbMSo_xTgbENO5sa_c&export=download&confirm=t
true
Downloaded 44409330 bytes in 12 seconds
Found on line 215037
9     12A
12    UPALONG
13    ROAD
15    MOUNT DANDENONG
16    VIC
17    3767
Found on line 216442
9     1
12    UPALONG
13    ROAD
15    MOUNT DANDENONG
16    VIC
17    3767
Searched 720650 lines in 1391 ms direct from gzip file
518080 lines per second

 

[emexes]

So the upshot of all that is that your original idea looks good, for files that gzip to up to 2 GB, provided that your users have a working internet connection when they first run your app, and are ok with the download bandwidth cost (if any, eg international roaming). Then again, they'd still be doing much the same size download even if it's buried inside the APK, so... I guess it's not really a factor.

The only downside I can see is: will the same gdrive download link still work if the gdrive file is updated?

 

[RB Smissaert]

So the upshot of all that is that your original idea looks good, for files that gzip to up to 2 GB, provided that your users have a working internet connection when they first run your app, and are ok with the download bandwidth cost (if any, eg international roaming). Then again, they'd still be doing much the same size download even if it's buried inside the APK, so... I guess it's not really a factor.

The only downside I can see is: will the same gdrive download link still work if the gdrive file is updated?

My file size will not reach 100 Mb, so I should be OK.
Solved one more problem and that was this:

            'note here that the file name needs to be in lower case, otherwise the result will be False!!
            If File.Exists(File.DirAssets, "postcode_lsoa11cd$1.csv") Then

I did see some Java code to check this, but didn't work for me.

RBS

 

[RB Smissaert]

My file size will not reach 100 Mb, so I should be OK.
Solved one more problem and that was this:

            'note here that the file name needs to be in lower case, otherwise the result will be False!!
            If File.Exists(File.DirAssets, "postcode_lsoa11cd$1.csv") Then

I did see some Java code to check this, but didn't work for me.

RBS

Forgot to say that this code line should be this:

If File.Exists(File.DirAssets, "postcode_lsoa11cd$1.csv") Then  'ignore

The ignore is needed to avoid a warning, which I think is due to a little bug, I mentioned before in the bug forum.

RBS

 

[emexes]

the file name needs to be in lower case

I thought I checked that, and what looked like an uppercase I was actually a lowercase l.

Out of interest, what the heck is that file? Can you post a couple of lines from it?

The letters "cd" were a mild worry, had me expecting files in the order of 100's of MB.

 

[RB Smissaert]

I thought I checked that, and what looked like an uppercase I was actually a lowercase l.

Out of interest, what the heck is that file? Can you post a couple of lines from it?

The letters "cd" were a mild worry, had me expecting files in the order of 100's of MB.

POSTCODE,LSOA11CD
AB1 0AA,S01006514
AB1 0AB,S01006514
AB1 0AD,S01006514
AB1 0AE,S01006853

It links larger LSOA geographical areas to the smaller postcode areas.
LSOA11CD will kind of give access to health related parameters and other parameters, such as income.

Just found another problem:
I took those files out of the Files Manager tab, but somehow they were still there, so on startup of the app it again populated the tables
after having run the table delete, eg: delete from LSOA11CD.

RBS

 

[RB Smissaert]

POSTCODE,LSOA11CD
AB1 0AA,S01006514
AB1 0AB,S01006514
AB1 0AD,S01006514
AB1 0AE,S01006853

It links larger LSOA geographical areas to the smaller postcode areas.
LSOA11CD will kind of give access to health related parameters and other parameters, such as income.

Just found another problem:
I took those files out of the Files Manager tab, but somehow they were still there, so on startup of the app it again populated the tables
after having run the table delete, eg: delete from LSOA11CD.

RBS

Not sure what went on there, but those files are gone now from assets folder.

RBS

 

[emexes]

Just found another problem:
I took those files out of the Files Manager tab, but somehow they were still there

Yeah, that's a problem I've seen before too. I think the fix was to both clean the project, and to delete the app off the phone so that it reinstalls afresh.

I'll be back here with the link to that saga in a couple of minutes. Was like two years ago, and totally mystifying. I think it turned out that, in debug mode, the program was half- or fully- running on the Windows PC. Which sounded doubtful, but at that point I was just happy the original problem was fixed.

This seems the best discussion:

File.ListFiles(File.Assets) in Debug

I'm using the DirAssets folder to store text files which are used in my App. If I look in the files explorer there are 21 of these files. If I look in the files tab of the B4A IDE there are also 21 files. I have synchronised the Files tab, and used clean files and clean project. 'Get...

www.b4x.com

Also this post is a two-for-the-price-of-one winner:

I don't know the full details but Debug mode uses a virtual assets folder instead of embedding the files in the apk. Most of the code in Debug mode is executed on the PC and not on the device. The device just executes a cutdown shell version of the app that mainly contains UI code to interact with the user.

You can apparently force normal assets behaviour by the IDE attribute
#DebuggerForceStandardAssets: True
But this is only necessary for libraries that try to load files from DirAssets, otherwise it should not be needed.

 

[emexes]

Not sure what went on there, but those files are gone now from assets folder.

LOL. $^@&ing technology. It could be that you swapped from debug to release mode or vice-versa. Or maybe B4A-Bridge lost and restarted connection. Like, mostly when I F5, I get asked about scanning the app etc, but sometimes - no idea why - it skips that and just runs. Still a mystery. But as long as the app runs, I'm happy.

Back in 2018, sometimes B4A would go into a mode where it wouldn't run the app, ever after closing and restarting everything. Only seemed to happen when a delivery deadline was approaching. Nothing would fix it, and then a day or two later, it would start working again, all by itself. Happened on different phones, too. Also not just to me. Anyway, whatever that $&#* was, seems to have been fixed.