Android Question Best way to send loads of small files

[JDS]

We've created an app but at this point there is an issue with the amount of calls the phone is making. It's an app for a delivery man with small packages.

To track the mobile phone (and packages) we create every 30 seconds an GPS-file and send it to our server in the background. After succesfull sending the file gets deleted.
During the route the delivery man does a lot of stuff on the phone, resulting in several small and sometimes larger files.

Sometimes the server isn't reachable (for example when server is down). Then all files get buffered on the device. If there is a new file saved then the device tries to send it files.

Is there a way to control the amount of files that are being send? For example if the server is down 2 hours and there could be ~240 files waiting to send. If this is the case and there are 200 devices waiting we talk about 48.000 calls in a matter of seconds. This overloads our server. Is there a way to regulate the data? Some kind of manager?

Below I've included our current upload of the gps-files.

        For i = 0 To lFiles.Size-1
            sNewFile =  lFiles.Get(i)
            Try
                If ((sNewFile.Contains("JSON"))) Then
                    sParameter = Globaal.ReturnJsonFromFile(Globaal.Map_GPS,sNewFile)
               
                    Soap.Initialize("PutGPS",Me)
                    Soap.PostString(Globaal.MainRESTUrl&"PutGPS",sParameter.Replace("[","").Replace("]",""))
                   
                    Soap.GetRequest.SetHeader("lmmobile",Globaal.PDAID.Trim)
                    Soap.GetRequest.SetHeader("wachtwoord",Globaal.wachtwoord.Trim)
                    Soap.GetRequest.SetHeader("Content-Type", "application/json")
                    wait for (Soap) jobdone(Soap As HttpJob)
                    If Soap.Success Then
                        Soap.Release
                        Globaal.DeleteAFile(Globaal.Map_GPS,sNewFile)
                    Else
                        If Soap.ErrorMessage.Trim<>"" Then
                            Soap.Release
                            Exit
                        End If
                        Soap.Release
                    End If
                End If
               
            Catch
                If Soap.IsInitialized Then Soap.Release
                Globaal.DeleteAFile(Globaal.Map_GPS,sNewFile)
               
            End Try
        Next

 

[OliverA]

Zip the buffered files? Then make a different call to the server (to let it know to process the zip file instead of individual file). The server can then batch process one zip file at a time. Just an idea...

 

[OliverA]

When the server comes up, have a special "handle one client at a time" mode where the server only takes a client's request at a time. Server was down anyways, so the other clients would not be affected (they just keep buffering). Once a client is processed, it is whitelisted for communication and the next client is added to process it's stored messages.

 

[DonManfred]

For example if the server is down 2 hours

~240 files waiting to send. If this is the case and there are 200 devices waiting we talk about 48.000 calls in a matter of seconds. This overloads our server.

HOW if the service IS DOWN????

 

[JDS]

HOW if the service IS DOWN????

For maintanance or possible the sim-card of the device is defect and the phone can only send when at the WiFi at the office.

 

[cimperia]

A quick question. Is it imperative to upload all the files if the server has been down or can you just send the last ones? Second question, is it imperative that they arrive in order, if the server was down, or would processing the most recent/current ones be a priority?

Your uploader routine is on a 30s timer right? Therefore the uploader routine should always check how many files there are and never send more than 10 for example, at a time. This would be configurable. An easy way to implement this is to always use a for loop, and therefore most of the time one file will be sent every 30 seconds, and other times 10 at once, or whatever threshold you decide on.

There are other implementations that would require to package the JSON files in batches of 10 say but it's more difficult to code and you'd have to modify the way the server manipulates the files (ie unpack them). But in the long run, it would be the smartest solution.

 

[JDS]

A quick question. Is it imperative to upload all the files if the server has been down or can you just send the last ones? Second question, is it imperative that they arrive in order, if the server was down, or would processing the most recent/current ones be a priority?

Your uploader routine is on a 30s timer right? Therefore the uploader routine should always check how many files there are and never send more than 10 for example, at a time. This would be configurable. An easy way to implement this is to always use a for loop, and therefore most of the time one file will be sent every 30 seconds, and other times 10 at once, or whatever threshold you decide on.

There are other implementations that would require to package the JSON files in batches of 10 say but it's more difficult to code and you'd have to modify the way the server manipulates the files (ie unpack them). But in the long run, it would be the smartest solution.

All the files are needed but it is not allways needed send them in order. For some files the can be send with the latest first (and then the others). Other files should be send in the create order. Each file has as prefix followed with datetime.now as a filename.

I'll have a look at sending a batch of files (zipped) or just put them together when sending (and there are multiple files). This both reduces the amount of files (and therefor the amount of calls).

 

[KMatle]

How big are the files? Maybe you should use a db.

 

[udg]

I agree with @KMatle : why not use a local sqlite DB to store the same info you now place in those small files?
Once a connection with the remote server is established, you sync local data to remote and mark it locally as "done". You could delete "done" data at any time.