B4R Question Suggestion of idea on how to make a backup

Why don't you use IFTTT to send the parameters to a Google sheet? Different approach but already done.

hatzisn said:

¿Por qué no usas IFTTT para enviar los parámetros a una hoja de Google?

Click to expand...

The client wants to have a copy of the parameters on his WEB page and be able to consult them in real time with his management system.
There are many devices to manage and each of them has hundreds of parameters that are divided into several CVS type files.
I think this limits me a lot when looking for alternatives, so I have to be able to know how to detect and create a directory on a WEB.

Thanks for the help.

Gerardo Tenreiro said:

The client wants to have a copy of the parameters on his WEB page and be able to consult them in real time with his management system.
There are many devices to manage and each of them has hundreds of parameters that are divided into several CVS type files.
I think this limits me a lot when looking for alternatives, so I have to be able to know how to detect and create a directory on a WEB.

Thanks for the help.

Click to expand...

Then if I were you I would create a webapp with B4J that receives a json where one of the values is the serial number. It then checks if the directory exists and if not it creates it and saves the json in a FILE_YYYYMMDD.dat in it. That's all.

It's a good idea unless I don't have access to the client's WEB server and it doesn't allow me to install any application on it. That was the original idea, but his server uses it for many things and he doesn't want me to install anything on it, just upload the files to a directory that he indicated to me, within that directory is where I have to create a folder for each of the ESP32s.
The task seems simple but the problem I have is in detecting if the directories already exist or not, if they don't exist I have to create them from the ESP32 and then upload the files one by one.

Thanks and I will continue trying to find a solution, in the forum there are many good people with a lot of knowledge that will surely give me the right path.

Thank you very much

Gerardo Tenreiro said:

It's a good idea unless I don't have access to the client's WEB server and it doesn't allow me to install any application on it. That was the original idea, but his server uses it for many things and he doesn't want me to install anything on it, just upload the files to a directory that he indicated to me, within that directory is where I have to create a folder for each of the ESP32s.
The task seems simple but the problem I have is in detecting if the directories already exist or not, if they don't exist I have to create them from the ESP32 and then upload the files one by one.

Thanks and I will continue trying to find a solution, in the forum there are many good people with a lot of knowledge that will surely give me the right path.

Thank you very much

Click to expand...

You don't have to use his server. You can very well use a cloud server for this and it is much better, since the local server sometimes is off the internet because of the provider. The B4J app once a day will create a zip with all the data and once a day he can use ftp to get this file from the cloud server. Also, you cannot create the folder from esp32. I cannot think of a way this can be done directly. The esp32 will have to give a command to an agent inside the server and the agent will create the folder.

hatzisn said:

You don't have to use his server. You can very well use a cloud server for this and it is much better, since the local server sometimes is off the internet because of the provider. The B4J app once a day will create a zip with all the data and once a day he can use ftp to get this file from the cloud server. Also, you cannot create the folder from esp32. I cannot think of a way this can be done directly. The esp32 will have to give a command to an agent inside the server and the agent will create the folder.

Click to expand...

Please beside the previous also you might think of more than one server and a load balancer since I suppose there will be a lot of esp32 devices

hatzisn said:

Please beside the previous also you might think of more than one server and a load balancer since I suppose there will be a lot of esp32 devices

Click to expand...

That is one of the reasons why I have to connect to their website, they have several servers distributed by zones depending on which modules are connected.
This is something new that they are setting up and at the moment it is not yet available for testing, thank goodness, because I have nothing to test.

I am trying to connect to the rHttpUtils2 library with this code:

Sub Conecta(Nada As Byte)
Log("Conectando...")

HttpJob.Initialize("Leer")
HttpJob.Download("https://ota-puerta.000webhostapp.com/ICDOOR06")
End Sub
'
Sub JobDone (Job As JobResult)
Log("*******************************")
Log("JobName: ", Job.JobName)
If Job.Success Then
Log("Response: ", bc.SubString2(Job.Response, 0, Min(200, Job.Response.Length))) 'truncate to 200 characters
If Job.JobName = "Leer" Then
Log("JobStatus=",Job.Status)
'send another request
'This time it is a POST request and we set the Content-Type header
HttpJob.Initialize("Example2")
'add headers before calling Post or Download (this is different than the standard HttpUtils2 library).
HttpJob.AddHeader("Content-Type", "application/x-www-form-urlencoded")
HttpJob.Post("https://www.b4x.com/print.php?key1=value1", "PostKey1=PostValue2&abc=def")
End If
Else
Log("ErrorMessage: ", Job.ErrorMessage)
Log("Status: ", Job.Status)
Log(Job.Response)
CallSubPlus("Conecta",5000,0)

End If
End Sub

Click to expand...

but it always gives me an error,
any suggestions?
thanks

What is the error?

if parameters are stored in eeprom memory, one way should be to make aperiodic backup of eeprom on google drive , dropbox or other server...

candide said:

if parameters are stored in eeprom memory, one way should be to make aperiodic backup of eeprom on google drive , dropbox or other server...

Click to expand...

Can you do this from a microprocessor? How is this done? I mean google drive and dropbox.

You can check whether the file already exists based on the http status code.

Gerardo Tenreiro said:

his server uses it for many things and he doesn't want me to install anything on it, just upload the files to a directory that he indicated to me

Click to expand...

Gerardo Tenreiro said:

some suggestions

Click to expand...

FTP? Email?

Or - best sit down for this one - embed data in HTTP requests, and then retrieve from HTTP server log file?

How large is a typical settings file? Is it capped by the EEPROM size? How large is that on the ESP32 devices you're using?

Gerardo Tenreiro said:

There are many devices to manage

Click to expand...

Are we talking hundreds, or millions?

As in: if it's hundreds, then receiving a daily backup from each one sounds practical, could be done via email or FTP. But if it's millions, with potentially thousands of devices sending simultaneously at peak periods, then that'd be more of a challenge.

Gerardo Tenreiro said:

and each of them has hundreds of parameters that are divided into several CVS type files.

Click to expand...

CVS? Or CSV (comma-separated values) or KVS (key-value store)?

And if CSV... why are the parameters separated into several files? Is that structure specified by the customer? Does that mean that a backup snapshot of one device can be several files? How would the receiving end know that all files have been received for a given device backup snapshot?

Lol sorry about all the questions. I understand that there might be confidentiality limits about how many clues you can give us. But the better description you give, the better suggestions you'll get back. ?

hatzisn said:

Can you do this from a microprocessor? How is this done? I mean google drive and dropbox.

Click to expand...

after a check, i didn't found examples with arduino or B4R.

the best example i found it a transfer of pictures from esp32 to Firebase, it should be possible to make something similar with B4R.

I'll go by parts, first I'll answer and then I'll inform you:
A.- The number of ESP32 devices is in the thousands, each order is for 500 units and I have served several orders.
B.- I make a backup copy once a day if any parameter was changed. I have the data organized in such a way that I generate several files. One for parameters, another for values, two for adjustments and one for statistics (Number of maneuvers, errors, and day-to-day data). The cards connect to a NIST server to obtain the date and time, so depending on a parameter at a certain time the cards of a group are updated and at other times the cards of other groups, this is to minimize the server load.
C.- The files that I generate are CSV. This is an express request from the client to be compatible with their control and management system. It really is as if it were a key and value since most of the files only use "Explanation"; "Current Value"; "Previous Value" ; "Limits"

At the moment I am trying to use HPPTs to perform a POST and create the directory and upload the file in question, this implies creating a PHP script to make the server work, it is the only viable option I find. My client does not want me to have anything on the server but I cannot find another way to do it.

Regarding the PHP SCRIPT I have some problems, but if anyone has any suggestions they would be very welcome

Sorry for not answering before but I am very busy with the project and I did not dedicate time to the group

Thank you very much

Gerardo Tenreiro said:

My client does not want me to have anything on the server but I cannot find another way to do it.

Click to expand...

What method was your client thinking to use to enable you to add/modify the backup data to their server, without having "anything on the server"?

Like, if you can do it without "anything on the server", then what's to stop any other random person from doing it (filling up client's server) too?

My client doesn't want me to have an application on his server, he only allows me to use PHP to download the file and create the directories.
I know very little about PHP and that's the problem, I have to make a PHP script so he can upload files from B4R.
I proposed an application in B4J with SOCKET and to be able to manage everything from the application, but they don't want to.
So I continue with the PHP routine

Thanks

Not really my thing, but presumably if the directory exists there will be a configuration backup in it? If that's the case, can't you just do a HTTP GET for the backup & if you get an error then you know it doesn't exist. If the issue is that you don't know the name of the backup file (because the name is a timestamp or UUID that's generated when the file is created), then couldn't you just put a "marker" file (eg: "exists.txt") in each directory you create & then do a GET for that? If you get a successful response you know the directory exists, but if you get an error you know it doesn't.

- Colin.

Indeed, the file name is composed of the date and time of the update, so the file name changes.
I think it's a very good idea to leave a file in the directory indicating whether the copy has been uploaded or not.
I'm going to follow this path a bit to determine which files have been uploaded and which have not.
Thank you very much