Android Question [Code snippet] Transfer files from Android to ESP8266-ESP32 micro SD Card in plain TCP (no FTP)

[max123]

Hi all,

this is a code snippet but even an help request to improve it, so I post here before move on code snippets section.

For my 3D printing app host I had necessity to transfer files from Android to ESP8266-ESP32 micro sd card (over network).

Because I don't wanted to use an FTP library on ESP side to just upload a file, I wrote a raw code that read a file in chunks and then
send one by one using a TCP socket.

Because I even want to pass some infos from Android to ESP before send a file I send an info string that contain some variables I want to pass,
it even contain a file name, size etc... and other variables I've put here just as an example, you can put yours.

Here is how it works....
- B4A code send a request to ESP with a file size
- ESP check for a space in the SD Card, if there is space to save a file send a string SPACEOK and B4A will continue the upload, if there is no space it send a string NOSPACE and B4A stop the upload
- If there is a space to save a file on ESP SD Card, then B4A send the info string containing the file name, size and other stuff (variables) not used here. The info string is a splited delimiter string and ESP split it to extract all infos
- after this B4A start read byte chunks (buffer) from file and send to ESP that collect it and append chunk by chunk to a file on micro sdcard.

This worked well for a long time in my app, now I've changed something and sometime the transfer fails,
sometime Android (TCP CLIENT) send info string and ESP (TCP SERVER) do not receive it or receive but then do not receive file chunks,
when this happen sometime Android app completely stall, no errors, completely hang, the only way is to use Android menu to close it.

Because I need to make it always working I've decided to reproduce it in a small project and post it so maybe someone can help to improve this piece of code.
I've attached a working B4A (client) and C++ (server) code, errors only happen on my real app where I have other stuffs in the ESP loop() function.

Note that here there is a sort of data encryption (just for string commands) that I just disabled on this test.
Note that here in the B4A code I open the socket connection before send all data, then send all data, finally I close it. In my app I cannot do this, just I open the socket and mantain it always open, there is a button to connect/disconnect.

Note that I cannot convert ESP side to B4R because it is a big project I started in 2018, so I cannot use socket stream in prefix mode.

I've used a timer to send file chunks, probably this is not good, I noticed that when Android send a chunk of bytes, then wait ESP to download it, if ESP do not read bytes from the stream, Android just wait until ESP read it, so probably using a Timer is not a good option here ?

Maybe it is best send chunks in a Do While loop awaiting ESP that read a chunk before send another one ?

If you see B4A code, I set the first timer interval to 100 ms, send info string, then decrease the timer interval to send file chunks.

Probably it should be wtitten in a different way.....

Actually I still test it on ESP32, but the code already adapted to ESP8266.

Note that C++ code can easlly managed to use other file systems, eg upload a file permanently on the ESP flash using SPIFFS or better LittleFS, FatFS etc.
It already support (but not tested) SDMMC eg on ESP32CAM, it can reach a transfer speed up 3MB sec.

Please help to make it always working (it always should work.... it is TCP, not UDP, it shouldn't miss packets, any file upload should complete successfully) .
Even please help to increase the upload speed, my results are around 250-300-350 Kbps, but sometime I've seen 500-600 and more Kbps on the log, so it can, it depends on many factors like a transmitter and receiver buffer size and a timer interval and a sd card write speed that increase tuning a buffer size. In 3D printing sometime there are relatively big files, like max 100 MB so I need to speed up a bit.

Many thanks

 

[max123]

I've managed it to work on my real application.

The problem seem to be the ESP sketch, because it receive strings as commands and then a file byte stream in chunks, I tried to move the byte stream receive part to another function and now it seem to always work.

Now the goal is to improve the transfer rate.

Sometime I had an insane transer rate like 700-800 Kbs, but it worked without problems, I'm surprised expecially from ESP side where it write to SD Card, I just used the SD library, so this is an incredible result for write mode on a microconttoller using SPI bus, no SD MMC.

It sometime start high then decrease, this depends from a timer interval that send the chunks on android side (one by one until file is fully read), and a trasmitter (B4A) and receiver (ESP) buffer size.

As you can see I placed on B4A app 2 progress bars, the first is the progress for a transfer on B4A side, the second is the progress that show the real transfer on ESP side, it changes when "Progress: xxx" string received on B4A side socket, and where xxx is the percentual, is ESP that inform android on real transfer progress (really received from socket and writed to SD Card).

I think that timer is not good here, on small files the first progress finish when second progress is around 40-50%, so B4A send it too fast and ESP read slow. On big files I have the opposed, B4A send the stream, ESP read it and wait until new data are going on, this decrease the transfer speed.

 

[max123]

Ah, note that both, my app and the code I've posted here, works on local network and over Internet using a Public IP address or DDNS.

On ESP side you can manage with WiFiManager library, so if you move ESP anywhere (not in your home but outside) if it cannot connect to last saved network, ESP start as Access Point exposing a captive portal where you can view all networks and connect to the right one by placing the password and press the connect button. If you never used WiFiManager library read more about it.

 

[max123]

At the end the problem was to remove WaitFor when send data on B4A side, now seem to always work well, in my app I've a visual graph of transfer average.

Note that this code is adaptable to B4J too, so you can upload files from PC to ESP sdcard or flash, in local area network and using public IP or dynamic DDNS over internet.

This even is adaptable to transfer files form an Android device to a PC and viceversa or from one ESP to another ESP, manually or by code.

 

[max123]

I do not see any iteraction here.... Someone tried this code I published ?

 

[amorosik]

The subject is very particular
In addition to this you need to have the hardware described to try it, and I think it's quite difficult to find those who do it with the B4R systems
If it can be of encouragement, I always follow these topics with interest because I also deal with electronics
But at the moment I don't have any esp32 card and even if I want to, I won't be able to try the code

 

[amorosik]

Why you dont like to use ftp?

 

[max123]

@amorosik I've managed with ESP32 (normal, no S3) with USB Host Soft HID Driver. I can connect mice, keyboard, playstation 2 controller and it read all, with PS controller it read all 15 buttons, two analog sticks (so 4 values = X, Y, X, Y from 0 to 255). No USB chip needed, just ESP32 up 4 USB connectors can be attached and read at same time, just need 2 GPIOs every USB port. Read even a wireless keyboard with a small USB key as receiver. Maybe convert it to B4R can be convenient. Tested on breadboard

 

[max123]

Why you dont like to use ftp?

Because I just need to download a file, no need a full FTP library. Actually my app use 78% of ESP32 memory. And I've tried some but not worked so well.

 

[avateur]

Because I just need to download a file, no need a full FTP library. Actually my app use 78% of ESP32 memory. And I've tried some but not worked so well.

Hey Max123. This looks like a cool project, would like to try it. How could I get a copy of the APK.

 

[max123]

Hi, I attached the project in the first post, as I wrote it sometime failed to upload.

Now I found a solution to this, the projects I've attached is working like a charm now.
What do you need is just to remove the Wait For while send data over socket because we write data on the data buffer stream and no need Wait For.

After you change this it should work all times. I use it in my app to transfer files to ESP and it save to uSD card. I tried it with relatively big files, 30-40 MB and works, now my app show the upload graphic line with transfer rate during upload.

 

[avateur]

Hi, I attached the project in thile first post, as I wrote it sometime failed to upload.

Now I found a solution to this, the projects I've attached is working like a charm now.
What do you need is just to remove the Wait For while send data over socket because we write data on the data buffer stream and no need Wait For.

After you change this it should work all times. I use it in my app to transfer files to ESP and it save to uSD card. I tried it with relatively big files, 30-40 MB and works, now my app show the upload graphic line with transfer rate during upload.

Yes I have the files but I don't recognize the files in the client folder. Are they files to compile the client app. I just stumbled upon this site while looking for an ftp server for http. Also would really appreciate it if you could upload the updated file that you say now runs better. Thanks.