B4R Question Suggestion of idea on how to make a backup

[Gerardo Tenreiro]

Hello,
I am finishing the ESP32 application and the client now tells me that he wants to have a backup copy of all the parameters of each of the devices in the cloud.
Each device has hundreds of parameters that can be modified from an application made in B4R and that are stored in the EEPROM memory of the ESP32.

I have a serial number for each ESP32 that is unique, it is made up of the WIFI MAC and an algorithm based on the year, month, day, hour, minute that the original software was installed for the first time. This number is 14 digits long. The application generates CSV files of the parameters and settings so the idea is to upload these files to a WEB and leave them inside a directory that is made up of the serial number.

Here I have the first problem. How do I know if the directory exists on the WEB and if not, create it?

I have a lot of trouble with the WEB so please, some suggestions
Thank you very much

 

[hatzisn]

Why don't you use IFTTT to send the parameters to a Google sheet? Different approach but already done.

 

[Gerardo Tenreiro]

¿Por qué no usas IFTTT para enviar los parámetros a una hoja de Google?

The client wants to have a copy of the parameters on his WEB page and be able to consult them in real time with his management system.
There are many devices to manage and each of them has hundreds of parameters that are divided into several CVS type files.
I think this limits me a lot when looking for alternatives, so I have to be able to know how to detect and create a directory on a WEB.

Thanks for the help.

 

[hatzisn]

The client wants to have a copy of the parameters on his WEB page and be able to consult them in real time with his management system.
There are many devices to manage and each of them has hundreds of parameters that are divided into several CVS type files.
I think this limits me a lot when looking for alternatives, so I have to be able to know how to detect and create a directory on a WEB.

Thanks for the help.

Then if I were you I would create a webapp with B4J that receives a json where one of the values is the serial number. It then checks if the directory exists and if not it creates it and saves the json in a FILE_YYYYMMDD.dat in it. That's all.

 

[Gerardo Tenreiro]

It's a good idea unless I don't have access to the client's WEB server and it doesn't allow me to install any application on it. That was the original idea, but his server uses it for many things and he doesn't want me to install anything on it, just upload the files to a directory that he indicated to me, within that directory is where I have to create a folder for each of the ESP32s.
The task seems simple but the problem I have is in detecting if the directories already exist or not, if they don't exist I have to create them from the ESP32 and then upload the files one by one.

Thanks and I will continue trying to find a solution, in the forum there are many good people with a lot of knowledge that will surely give me the right path.

Thank you very much

 

[hatzisn]

It's a good idea unless I don't have access to the client's WEB server and it doesn't allow me to install any application on it. That was the original idea, but his server uses it for many things and he doesn't want me to install anything on it, just upload the files to a directory that he indicated to me, within that directory is where I have to create a folder for each of the ESP32s.
The task seems simple but the problem I have is in detecting if the directories already exist or not, if they don't exist I have to create them from the ESP32 and then upload the files one by one.

Thanks and I will continue trying to find a solution, in the forum there are many good people with a lot of knowledge that will surely give me the right path.

Thank you very much

You don't have to use his server. You can very well use a cloud server for this and it is much better, since the local server sometimes is off the internet because of the provider. The B4J app once a day will create a zip with all the data and once a day he can use ftp to get this file from the cloud server. Also, you cannot create the folder from esp32. I cannot think of a way this can be done directly. The esp32 will have to give a command to an agent inside the server and the agent will create the folder.

 

[hatzisn]

You don't have to use his server. You can very well use a cloud server for this and it is much better, since the local server sometimes is off the internet because of the provider. The B4J app once a day will create a zip with all the data and once a day he can use ftp to get this file from the cloud server. Also, you cannot create the folder from esp32. I cannot think of a way this can be done directly. The esp32 will have to give a command to an agent inside the server and the agent will create the folder.

Please beside the previous also you might think of more than one server and a load balancer since I suppose there will be a lot of esp32 devices

 

[Gerardo Tenreiro]

Please beside the previous also you might think of more than one server and a load balancer since I suppose there will be a lot of esp32 devices

That is one of the reasons why I have to connect to their website, they have several servers distributed by zones depending on which modules are connected.
This is something new that they are setting up and at the moment it is not yet available for testing, thank goodness, because I have nothing to test.

I am trying to connect to the rHttpUtils2 library with this code:

Sub Conecta(Nada As Byte)
Log("Conectando...")

HttpJob.Initialize("Leer")
HttpJob.Download("https://ota-puerta.000webhostapp.com/ICDOOR06")
End Sub
'
Sub JobDone (Job As JobResult)
Log("*******************************")
Log("JobName: ", Job.JobName)
If Job.Success Then
Log("Response: ", bc.SubString2(Job.Response, 0, Min(200, Job.Response.Length))) 'truncate to 200 characters
If Job.JobName = "Leer" Then
Log("JobStatus=",Job.Status)
'send another request
'This time it is a POST request and we set the Content-Type header
HttpJob.Initialize("Example2")
'add headers before calling Post or Download (this is different than the standard HttpUtils2 library).
HttpJob.AddHeader("Content-Type", "application/x-www-form-urlencoded")
HttpJob.Post("https://www.b4x.com/print.php?key1=value1", "PostKey1=PostValue2&abc=def")
End If
Else
Log("ErrorMessage: ", Job.ErrorMessage)
Log("Status: ", Job.Status)
Log(Job.Response)
CallSubPlus("Conecta",5000,0)

End If
End Sub

but it always gives me an error,
any suggestions?
thanks

 

[hatzisn]

What is the error?

 

[candide]

if parameters are stored in eeprom memory, one way should be to make aperiodic backup of eeprom on google drive , dropbox or other server...

 

[hatzisn]

if parameters are stored in eeprom memory, one way should be to make aperiodic backup of eeprom on google drive , dropbox or other server...

Can you do this from a microprocessor? How is this done? I mean google drive and dropbox.

 

[byz]

You can check whether the file already exists based on the http status code.

 

[emexes]

his server uses it for many things and he doesn't want me to install anything on it, just upload the files to a directory that he indicated to me

some suggestions

FTP? Email?

Or - best sit down for this one - embed data in HTTP requests, and then retrieve from HTTP server log file?

How large is a typical settings file? Is it capped by the EEPROM size? How large is that on the ESP32 devices you're using?

 

[emexes]

There are many devices to manage

Are we talking hundreds, or millions?

As in: if it's hundreds, then receiving a daily backup from each one sounds practical, could be done via email or FTP. But if it's millions, with potentially thousands of devices sending simultaneously at peak periods, then that'd be more of a challenge.

and each of them has hundreds of parameters that are divided into several CVS type files.

CVS? Or CSV (comma-separated values) or KVS (key-value store)?

And if CSV... why are the parameters separated into several files? Is that structure specified by the customer? Does that mean that a backup snapshot of one device can be several files? How would the receiving end know that all files have been received for a given device backup snapshot?

Lol sorry about all the questions. I understand that there might be confidentiality limits about how many clues you can give us. But the better description you give, the better suggestions you'll get back.

 

[candide]

Can you do this from a microprocessor? How is this done? I mean google drive and dropbox.

after a check, i didn't found examples with arduino or B4R.

the best example i found it a transfer of pictures from esp32 to Firebase, it should be possible to make something similar with B4R.

ESP32-CAM Save Picture in Firebase Storage | Random Nerd Tutorials

Take and upload a picture to Firebase Storage using the ESP32-CAM. Create a Firebase project with Storage that allows you to store your files, go to the console to visualize the pictures or create a web app to display them

randomnerdtutorials.com

 

[Gerardo Tenreiro]

I'll go by parts, first I'll answer and then I'll inform you:
A.- The number of ESP32 devices is in the thousands, each order is for 500 units and I have served several orders.
B.- I make a backup copy once a day if any parameter was changed. I have the data organized in such a way that I generate several files. One for parameters, another for values, two for adjustments and one for statistics (Number of maneuvers, errors, and day-to-day data). The cards connect to a NIST server to obtain the date and time, so depending on a parameter at a certain time the cards of a group are updated and at other times the cards of other groups, this is to minimize the server load.
C.- The files that I generate are CSV. This is an express request from the client to be compatible with their control and management system. It really is as if it were a key and value since most of the files only use "Explanation"; "Current Value"; "Previous Value" ; "Limits"

At the moment I am trying to use HPPTs to perform a POST and create the directory and upload the file in question, this implies creating a PHP script to make the server work, it is the only viable option I find. My client does not want me to have anything on the server but I cannot find another way to do it.

Regarding the PHP SCRIPT I have some problems, but if anyone has any suggestions they would be very welcome

Sorry for not answering before but I am very busy with the project and I did not dedicate time to the group

Thank you very much

 

[emexes]

My client does not want me to have anything on the server but I cannot find another way to do it.

What method was your client thinking to use to enable you to add/modify the backup data to their server, without having "anything on the server"?

Like, if you can do it without "anything on the server", then what's to stop any other random person from doing it (filling up client's server) too?

 

[Gerardo Tenreiro]

My client doesn't want me to have an application on his server, he only allows me to use PHP to download the file and create the directories.
I know very little about PHP and that's the problem, I have to make a PHP script so he can upload files from B4R.
I proposed an application in B4J with SOCKET and to be able to manage everything from the application, but they don't want to.
So I continue with the PHP routine

Thanks

 

[Computersmith64]

Not really my thing, but presumably if the directory exists there will be a configuration backup in it? If that's the case, can't you just do a HTTP GET for the backup & if you get an error then you know it doesn't exist. If the issue is that you don't know the name of the backup file (because the name is a timestamp or UUID that's generated when the file is created), then couldn't you just put a "marker" file (eg: "exists.txt") in each directory you create & then do a GET for that? If you get a successful response you know the directory exists, but if you get an error you know it doesn't.

- Colin.

 

[Gerardo Tenreiro]

Indeed, the file name is composed of the date and time of the update, so the file name changes.
I think it's a very good idea to leave a file in the directory indicating whether the copy has been uploaded or not.
I'm going to follow this path a bit to determine which files have been uploaded and which have not.
Thank you very much