B4J Question An FTP enigma

[Brian Dean]

I like FTP. I have been using a free server for twenty years - only 200MB available storage, but my files are only 20-30k each. I use it to share data between apps and devices, even more now since sharing data via (Android) device storage is so restrictive. But recently the server was shut down for a while because of misuse by some users, and although I found a temporary alternative I realised that FTP is going out of fashion and I could not rely on a free service any more.

I am now setting up on a paid cloud storage provider (Ionos HiDrive) - still using FTP. I have my own small library that standardises my FTP interface and provides basic data synchronisation. All that is required to switch providers is to change the url etc - except that HiDrive provides SFTP and FTPS but not simple old FTP. I don't really understand how these two work, and that might be the source of my problem, which is this ...

I have a testbed for the library - it can exchange short or longer ASCII files (200byte and 2Mbyte) and a random binary data file (2Mbyte). Everything tested well, so I started deploying the updated library in my apps. Everything was still fine until I reached an app that runs on three devices (phone, tablet and PC) and shares a 34kbyte SQLite file. Occasionally the file transfers successfully but usually not; the file is truncated to 4k - 8k.

Here is the code that I am using to set up FTP (@schimanski helped me with this - he also uses HiDrive.)

' Initialise contact with FTP server
Private Sub setupFTP
    FTP.Initialize("ftp", "ftp.hidrive.ionos.com", 21, "account", "password")
    FTP.PassiveMode = True
    FTP.UseSSLExplicit = True
    makeFileList            ' Check connection and list files
End Sub

Here are some other facts that might or might not be helpful . . .

1. I have transmitted 2MByte random binary files both ways and checked that the files returned match the files sent byte-for-byte; they always do.

2. I have transmitted and retrieved a 30MByte *.avi file without any problem - also *.mp3 files.

3. My desktop FTP client (I use CoffeeCup) transfers the SQLite file without problem. It says that it uses this protocol - "SFTP over SSH, standard port is 22".

4. HiDrive offers FTPS (explicit with TLS encryption) on port 21 and SFTP on port 22.

5. B4X gives me three options "UseSSL", "UseSSLExplicit" or neither of these. I do not know what these mean - I am just following the example given by @schimanski.

I have been working on and off on this for two weeks - any suggestions would be welcome. I am sure that I am missing something obvious.

 

[LucaMs]

Occasionally the file transfers successfully but usually not; the file is truncated to 4k - 8k.

Since sometimes everything is fine, I would say that the problem is not SFTP/FTPS, otherwise it would never work.

I would check the project carefully (Wait for?)

 

[teddybear]

FTPS and SFTP are totally different protcols. you only use ftp or ftps to do that if you are using ftp library.
How do you send and receive the file? which transfer data type is set， Binary or Ascii?

 

[Brian Dean]

Here is the upload routine . . .

Private Sub uploadJob(dir As String, filename As String, isASCII As Boolean)
    Log("uploadJob : " & filename)
    Dim localtime As Long = 0
    If File.Exists(dir, filename) Then                ' Check for existing local file
        localtime = normalise(File.LastModified(dir, filename))
        Log("Local file = " & DateTime.Date(localtime) & " " & DateTime.Time(localtime))
        Dim hosttime As Long = hostTimestamp(filename)
        Log("Host file = " & DateTime.Date(hosttime) & " " & DateTime.Time(hosttime))
        If (localtime > hosttime) Then                    ' Local file is newer than host file
            filesize = (File.Size(dir, filename))    ' Needed for progress measure
            status("Uploading", 0)
            Log("FTP.UploadFile")
            FTP.UploadFile(dir, filename, isASCII, hostPath & filename)
            Wait For FTP_UploadCompleted(serverpath As String, success As Boolean)
            Log("FTP_UploadCompleted")
            If success Then
                status("Finished", 100)
            Else
                status("Upload failed", -3)
                Log(LastException.Message)
            End If
            Complete(success)
        Else
            status("Not uploaded", -7)                        ' Server file is newer
            Complete(True)                                                ' The correct outcome
        End If
    Else
        status("Local file not found", -5)                        ' Local file not found
        Complete(False)
        Return
    End If
End Sub

I would say that I have been using this library for years, but of course in recent days I have tried many things and maybe I have changed something and forgotten to change it back, or made an "improvement" that was unwise. However I have just tested it by changing only the FTP initialisation, reverting back to my old, free FTP server. The SQLite file ttansfers reliably and repeatedly, both ways. I changed the FTP initialisation back to HiDrive and the library reports all 36.0kb sent, but only 7.0kb shows up at the server.

To answer your specific question @teddybear - the transfer type is set to binary : ie "isACSII" is false in the code above.

 

[teddybear]

Is transferring the sqlite file failed every time or occasionally？
Try using another FTP client to transfer the sqlite file and see if it fails.

 

[Brian Dean]

Try using another FTP client to transfer the sqlite file and see if it fails.

See item 3 in my first post.

Is transferring the sqlite file failed every time or occasionally？

I think that it usually succeeds on the first transfer and is more likely to fail on successive transfers. In fact I am working on this avenue right now. Maybe I am not opening and closing FTP cleanly, and although my previous provider tolerated this the new one will not. Anyway I am writing a bare bones class to let me check this possibility. I will report back in due course.

 

[teddybear]

3. My desktop FTP client (I use CoffeeCup) transfers the SQLite file without problem. It says that it uses this protocol - "SFTP over SSH, standard port is 22".

The item 3 you said is that client transfers the sqlite file using ssh protocol,did you do ftp or ftps with the client?

 

[Brian Dean]

did you do ftp or ftps with the client?

The client provides a straight FTP option, but using that fails to connect with the host at all, so is unusable. The client has only one SFTP connection option, which I am using with complete success. There are three FTPS options - I have tried them all and none connect to the server. [EDIT : Ignore this last sentence - I need to try them again with a different url]

I have run some B4J trials now with an ultra-simple upload routine (see below), and switched my test file to a 40 kByte jpeg file. This uploads sucessfully/unsuccessfully on a roughly 50/50 ratio. An unsuccessful upload transmits around 8 kByte. Sending my randomly generated 2 Mbyte binary file has never failed yet!

Public Sub setupFTP
    FTP.Initialize("ftp", "ftp.hidrive.ionos.com", 21, "<account>", "<password>")
    FTP.PassiveMode = True
    FTP.UseSSLExplicit = True                                        ' This is mandatory for HiDrive
End Sub

Public Sub uploadFile(dir As String, filename As String, isASCII As Boolean)
    Log("uploadFile : " & filename)
    If Not(FTP.IsInitialized) Then setupFTP
    If File.Exists(dir, filename) Then                ' Check for existing local file
        Log("Uploading file")
        Dim sf As Object = FTP.UploadFile(dir, filename, isASCII, hostPath & filename)
        Wait For (sf) FTP_UploadCompleted(serverpath As String, success As Boolean)
        Log("FTP_UploadCompleted : " & success)
        If success Then
            Log("Upload success")
        Else
            Log("Not uploaded")
            Log(LastException.Message)
        End If
    Else
        Log("Local file not found")
    End If
End Sub

Thanks for your interest. I'm closing down now for today.

 

[aminoacid]

@Brian Dean I share your frustration. I too have been using FTP for a long time but almost every time I use it on a new server or client computer I encounter issues - primarily with the firewall requirements to get it to work properly. I have come to the conclusion that it's time to retire my use of FTP and move on to SFTP (SCP/SSH) instead. Why SFTP? because as someone else said, it's completely different from FTP, FTPS, etc. which were designed before firewalls became a necessity and use multiple ports. I'm not saying that your issue may be firewall related, or maybe it is. I don't know. But I would suggest you switch over to using an SFTP client. It should work seamlessly with Ionos HiDrive. I use it with a couple of Ionos VPSs. The transition seems to be pretty easy. In fact I am doing that right now using the jSCH Library. See link below. I'm writing some test code to test out the different file transfer operations and familiarize myself with the library and so far the code appears to be very similar to the regular FTP client. I will be glad to share this with you if you like. Let me know.

Android SFTP based on JSch tutorial

JSch is an open source Java implementation of SSH2. It supports many features. Currently the Basic4android JSch library supports the SFTP protocol which is SSH File Transfer Protocol or Secured File Transfer Protocol. SFTP is similar to FTP with the difference that the communication is done...

www.b4x.com

PS: It says "Android" but this library can also be used with B4J

 

[Brian Dean]

I have come to the conclusion that it's time to retire my use of FTP and move on to SFTP (SCP/SSH) instead.

Hi @aminoacid - thanks for this information. I didn't know about this and it certainly looks worth investigating - I am going to give it a try. A bit too soon to mark your post as a "solution" but if I get it to work then I will mark it up!

 

[aminoacid]

Hi @aminoacid - thanks for this information. I didn't know about this and it certainly looks worth investigating - I am going to give it a try. A bit too soon to mark your post as a "solution" but if I get it to work then I will mark it up!

It really turned out to be quite simple to convert my FTP code to SFTP. I have attached the example I wrote up which really helped test it out completely before incorporating the code into other projects. Just install the jsch library from the link in my previous post, change the login credentials on Line 24 and then in the Button1_Click event comment out/in the test code you want to try.

For Ionos HiDrive it would be:

Here is the Test Project:

 

[Brian Dean]

Hi @aminoacid - THIS IS THE SOLUTION! Thank you very much - and please check the conversation message that I sent to you.