FTP Download greater than 20 files

Smee

Well-Known Member
Licensed User
Longtime User
Yep i am getting the java.util.concurrent.RejectedExecutionException error when i hit the download limit.

I have spent quite a few hours looking at the httputils code but i do not think that it will suit my tasks.

I have to log onto a website, download some csv files, then recurse thru an unknown number of directories downloading the latest files and leaving the old ones alone.

I then have to put those files into specific or matching directories within the Android device so a lot of info has to be saved.

I have coded all this in using the FTP download module which shows the user the file(s) being downloaded and the progress of each download file.

It is quite a bit of code.

Now because of the error i tried yesterday to download using a "sleep" sub. This worked to a certain degree but it is neither efficient or pretty to watch. I had to use a 2000ms delay to be able to download all the files.

At the moment there are around 60 files to download.

Does anyone have an elegant solution I can use to achieve my aims?

I intend to do a demo in a couple of days so i do not think i will have the time to re-code the httputils to my purposes even if it was suitable because of the time to learn exactly what is going on and what i need to achieve.

so again, does anyone have an elegant solution I can use to achieve my aims downloading many files via ftp

Thanks for any answers
 

thedesolatesoul

Expert
Licensed User
Longtime User
I have a solution that uses something similar HttpUtils.
It downloads only 1 file at a time, but allows you to queue as many files (or even directories) into a list.
I wrote this for Dropbox however it is not for FTP. It is unfinished and not even completely tested.
If you want I can send it to you (when I get home).
 
Upvote 0

Erel

B4X founder
Staff member
Licensed User
Longtime User
Instead of downloading the files directly you should add the file paths to a List object.
Once you finished mapping all the files you should start processing the list one item after another.
Download the first file. When download completes (successfully or not), download the next file. It will be efficient and will not break the limit.
 
Upvote 0

Smee

Well-Known Member
Licensed User
Longtime User
Thanks to both for the replies and thanks desolate for the offer. I may take you up on it even to check out the coding. I can always learn from other ppls solutions.

Erel, I tried something similar to that using an array and then downloading after all the directories were processed. It didn't work so i guess i must have done something wrong.

I will retry

Thanks again
 
Upvote 0

Smee

Well-Known Member
Licensed User
Longtime User
Yes i have just completed thje code and about to test it. That is exactly what i did. Although when i have processed the last file how do i terminate the service correctly?
 
Upvote 0

Smee

Well-Known Member
Licensed User
Longtime User
:sign0060:

Thanks for the pointers Erel,

That works beautifully. Now to tidy up the code and take all the excess code routines etc out.

final thing does the service need to be terminated when it is no longer required to remove it from memory and what is the correct way to do so

:sign0060:
 
Upvote 0

Smee

Well-Known Member
Licensed User
Longtime User
All i intend doing with errors is notify the user of the error. They would then re-initiate the download and any missed files should be re-download. I would probably write them to an error file so that they could be easily handled
 
Upvote 0

Rusty

Well-Known Member
Licensed User
Longtime User
FTP Download Files/folders

Any chance you can provide source? I have the FTP download working for files, but would like an example of a simple folder download. (presumably without having to parse the files within each folder.)
Thanks,
 
Upvote 0

Smee

Well-Known Member
Licensed User
Longtime User
Hi Rusty,

I am not 100% happy with my code and on the fourth rewrite. However you do have to parse the files in each directory. At least that is what i found.

I have used an array but i am going to switch to a list object. I started off using the ftp example and just expanded from there. Learning more and more each rewrite


Joe
 
Upvote 0

Rusty

Well-Known Member
Licensed User
Longtime User
FTP Download

After I asked for source, I went ahead and threw together a sample. It seems to work pretty well and can probably be cleaned up a bit. I have used it to download many folders containing hundreds of files with no problem. I expect that I could start multiple downloads simultaneously, but didn't try.
This code needs FTP and core libs.
It also needs a .bal with btnOK, btnDownload, btnCancel, lblDownload, lblDownloaded, pbProgress (btn- button, lbl- label, pb- progressbar)

Thanks for your response
B4X:
SOURCE CODE BELOW
'Activity module
' Libraries:  FTP, Core

Sub Process_Globals
    'These global variables will be declared once when the application starts.
    'These variables can be accessed from all modules.
        
    Type DLD (RemoteFile As List, RemoteFolder As List, LocalFile As List, LocalFolder As List, Size As List)
    Dim Download As DLD
    
    ' FTP
    Dim MyFTP As FTP
    Dim FTPFiles() As FTPEntry
        
    Dim WiFiAvailable As Int            '0 - Maybe, don't know, -1 - yes it is, we've checked   1 - No it is not, we've checked
End Sub

Sub Globals
    'These global variables will be redeclared each time the activity is created.
    'These variables can only be accessed from this module.
    Dim DownloadFilePtr As Int
    Dim FolderLocation As String
    Dim DownloadFolderLocation As String
    
    Dim pbProgress As ProgressBar
    Dim btnOK As Button
    Dim btnCancel As Button
    Dim btnDownload As Button
    Dim lblDownload As Label
    Dim lblDownloaded As Label
    
End Sub

Sub Activity_Create(FirstTime As Boolean)
    If FirstTime Then
        Activity.LoadLayout("Installer")
    
        Download.Initialize
        DownLoad.Remotefolder.Initialize
        Download.Remotefile.Initialize
        Download.Localfolder.Initialize
        download.LocalFile.Initialize
        Download.Size.Initialize
    End If
    
End Sub
' Via FTP, lists all files and folders on the FTP site.  list complete will cause downloads to begin
Sub btnOK_Click
    lbldownload.Text = ""
    pbProgress.Visible = True
    ftpconnect
                                                                            'VERY IMPORTANT:  case is important on folder names (upper/lower)
    folderlocation = "your/FTPFOLDER/location/"                                'your FTP location in the format "fffff/fffff/fffff/"  (note: no beginning slash, but an ending slash (/))
    downloadfolderlocation = "YOUR/localfolder/location/"                    'your local file location in the format "fffff/fffff/fffff/"  same as above 
    Myftp.List(folderlocation)                                                'you can repeat this as you wish adding more FTP folders to download, 
                                                                            'they will asynchronously add folders/files to the list
End Sub
'now we have a complete list of folders/files, let's download them
Sub btnDownload_Click
    If Download.RemoteFile.Size > 0 Then                                    'if there are no files, we can't download
        pbProgress.Visible = True
        DownloadFilePtr = 0
        If myftp.IsInitialized Then
            ftpdisconnect
        End If
        ftpconnect
        If File.Exists(download.LocalFolder.Get(Downloadfileptr), Download.LocalFile.Get(downloadfileptr)) Then    'delete the file, if it already exists
            File.Delete(download.LocalFolder.Get(Downloadfileptr), Download.LocalFile.Get(downloadfileptr))
        End If
        'the below will start the download and then download complete will start the next download of a file, if one exists within the download list
        myftp.DownloadFile(Download.RemoteFolder.Get(DownloadFilePtr) & Download.RemoteFile.get(DownloadFilePtr), False, Download.LocalFolder.get(DownloadFilePtr), Download.LocalFile.get(DownloadFilePtr))    'download complete will disconnect
    Else
        Msgbox ("No new files to install","Notice")
        btnCancel_Click
    End If
    
End Sub

Sub btnCancel_Click
    If myftp.IsInitialized Then
        ftpdisconnect
    End If
    activity.Finish
    ExitApplication
End Sub

Sub FTPConnect
    txtFTP = "your ftp location"
    txtPort = 21
    txtLogin = "your user id here"
    txtPassword = "your password"
    ' connect with the ftp
    myFTP.Initialize("MyFTP", txtFtp, txtPort, txtLogin, txtPassword)
End Sub

Sub FTPDisconnect
    myftp.CloseNow
End Sub

Sub MyFTP_ListCompleted (ServerPath As String, Success As Boolean, Folders() As FTPEntry, Files() As FTPEntry)
    If Success Then
        pbprogress.Visible = True
        Dim i As Int
        Dim SubFolder As String
        SubFolder = serverpath.Replace(downloadfolderlocation.Replace(" ","_"), "") 'won't work if there are embedded spaces

        For i = 0 To folders.Length -1                                                'roll through and find folders
            If Folders(i).Name <> "." AND Folders(i).Name <> ".." Then                'don't add folder control
                myftp.List(serverpath & "/" & folders(i).name)                        'add them to the list (function occurs on list complete
            End If
        Next

        If files.Length >= 1 Then                                                    'we are listing files to go get
            AddFileToDownloadList(serverpath, File.DirRootExternal & "/" & Subfolder, Files)    'add the files to the queue (list) to download
        End If
        btndownload.Enabled = True
        pbprogress.Visible = False
    Else
        Msgbox("Online folder not found, check spelling..." & ServerPath, " Installer")
    End If
End Sub

Sub AddFileToDownloadList(RemoteFolder As String, LocalFolder As String, Files() As FTPEntry)
    Dim i As Int

    For i = 0 To files.Length -1                                                    'roll through the files array and add them to the download list
        DownLoad.RemoteFile.add(Files(i).Name)
        DownLoad.RemoteFolder.add(RemoteFolder & "/")
        Download.LocalFile.Add(Files(i).Name)
        Download.LocalFolder.Add(LocalFolder)
        Download.Size.Add(Files(i).Size)
        DownLoadListLength = download.RemoteFile.Size
    Next
    lblDownload.Text = download.LocalFile.Size & " files identified."
    DoEvents
End Sub

Sub MyFTP_DownloadCompleted (ServerPath As String, Success As Boolean)
    ' Downloadfilelist.size is the number of entries ie 1 entry 1, no entry 0
    If Download.RemoteFile.Size - 1 > DownloadFilePtr   Then                            'we have more files to transmit
        DownloadFilePtr = DownloadFilePtr + 1
        If File.Exists(download.LocalFolder.Get(downloadfileptr),"") = False Then        'if the folder doesn't exist, let's create it
            File.MakeDir(download.LocalFolder.Get(downloadfileptr),"")
        End If
        If File.Exists(download.LocalFolder.Get(Downloadfileptr), Download.LocalFile.Get(downloadfileptr)) Then    'delete the existing file if any
            File.Delete(download.LocalFolder.Get(Downloadfileptr), Download.LocalFile.Get(downloadfileptr))
        End If
        myftp.DownloadFile(Download.RemoteFolder.Get(DownloadFilePtr) & Download.RemoteFile.get(DownloadFilePtr), False, Download.LocalFolder.get(DownloadFilePtr), Download.LocalFile.get(DownloadFilePtr))    'download complete will disconnect
    Else
        Dim FileCt As Int
        Filect = Downloadfileptr + 1    
        Msgbox("Transmission complete " & FileCt & " files received", "Download Status")

        pbProgress.Enabled = False
        pbProgress.Visible = False
        pbProgress.Indeterminate = False
        
        lblDownloaded.Visible = False
        lblDownload.Visible = False
        
        Download.LocalFile.Initialize                                            'don't really need to do all this
        Download.LocalFolder.Initialize
        Download.RemoteFile.Initialize
        Download.RemoteFolder.Initialize
        DownloadFilePtr = 0
        
        FTPDisconnect
    End If
    
End Sub
Sub MyFTP_DownloadProgress (ServerPath As String, TotalDownloaded As Long, Total As Long)
    lblDownload.Text = "Downloading: " & serverpath.SubString(serverpath.LastIndexOf("/") + 1)
    lblDownloaded.text = Totaldownloaded
End Sub


Sub Activity_Resume

End Sub

Sub Activity_Pause (UserClosed As Boolean)

End Sub
 

Attachments

  • installer.bal
    5.5 KB · Views: 350
Last edited:
Upvote 0

Smee

Well-Known Member
Licensed User
Longtime User
Rusty,

That looks like a neater solution than i have. Mine is different in that the following will apply

1) Unknown number of folders
2) Unknown number of files
3) Unknown number of files in each directory
4) Only changed or new files should be downloaded
5) Files on the server are 'mirrored' to the device and kept in their own directories.
6) Root directory contains csv file(s) which if downloaded requires that the corresponding folder to that file should be parsed and changed or new files should be downloaded.

So it is a little complex. (at least for me). Hence the many re-writes to get it correct.

Regards

Joe
 
Upvote 0

Rusty

Well-Known Member
Licensed User
Longtime User
FTP Download

The code I sent will copy unlimited folders/files etc. If you want to only download changed files/folders you would have to compare the FTP file/date with the local File/date and then copy only the changed files.
I have a file structure:
Root - folder 1 - 100 files; folder 2- subfolder 1- subfolder 2 (each subfolder has files and subfolders within)
I can download all folders and all files with one ftp.list.
The list complete then sees folders other than . and ..
each folder is then added to the list by executing another ftp.list for that new folder.
This is a "natural" parsing of the folders so it works for me. I have put it in production.
If the code I sent doesn't do this for you I will send my production code. I have removed my FTP user/password and tried to make it generic before I posted it.
Let me know if you want a copy of "production" code to try.
You can get the file time/date with the Files().timestamp and the local with the File.lastmodified(folder, file) (note: lastmodified is FILE not FILES)
 
Last edited:
Upvote 0

Smee

Well-Known Member
Licensed User
Longtime User
Thanks Rusty,

Please do post code if you are willing. I am sure i will at the very least learn from it. I like the way you used the Type array in your previous code.



Regards

Joe
 
Upvote 0

Rusty

Well-Known Member
Licensed User
Longtime User
FTP Download Files/folders

This will create an exact copy of the folder structure (and files) from your ftp site onto your Android device. I've commented pretty heavily.
I've attached the whole project in a zip file for you.
Let me know if you have problems/questions. Good luck
 
Last edited:
Upvote 0

Smee

Well-Known Member
Licensed User
Longtime User
Thanks Rusty,

I have downloaded the zip. Hopefully i will find time over the next few days to check it out.

Thanks again

Joe
 
Upvote 0

gapi

Active Member
Licensed User
Longtime User
Why FTP_LIST and FTP_Download processes can't live in the same sub ?

thanks
 
Last edited:
Upvote 0
Top