Android Question HttpUtils2 large files

[tomoshea]

Hi,
I use HttpUtils2 (based on Erel's LargeFileDownload example) to access a csv file from a WIFI SD card in another device.
Is there a way to read the file without downloading it as the time taken increases as the file size increases.
Records are added to the csv file every second and the app needs to read each new record as soon as it is inserted and process it. So I only need to access one record at a time (last one written).
I cannot change the file type etc as the records are written by a Third Party application.
The file size can reach a size of 10mb approx.
Thank You,
Kind Regards,
Tom

 

[Reviewnow]

Do you have control of the server that you are downloading the file from?
if so I would create an application the exports the last line of the csv file to a new csv file then just download the new one

 

[tomoshea]

Thank you for your quick reply.
Maybe I could store a php script on the card and have the app run this script which would extract the last line and return it to the app.
Would that work?

 

[DonManfred]

it would work, yes. but it would hard to be a real life update.... but a php-script should be really fast to get the last line from the log...

 

[Reviewnow]

This Service Module may work for you without saving the file then opening it again
the Code below reads the downloaded input stream and gives you the last line,
it is also setup to run as a service every five minutes if needed

Sub Process_Globals
    'These global variables will be declared once when the application starts.
    'These variables can be accessed from all modules.
Dim shouldScheduleNextTask As Boolean
    shouldScheduleNextTask = True 'by default we are scheduling the next update
    Dim csvUrl As String : csvUrl = "Your Url Here"
    Dim lastline As String

End Sub
Sub Service_Create
   
   
End Sub

Sub Service_Start (StartingIntent As Intent)
If shouldScheduleNextTask = True Then
      'Check for csv every 5 minutes
        StartServiceAt("", DateTime.Now + 5 * DateTime.TicksPerMinute,True)
    End If
  ToastMessageShow("Loading CSV " & csvUrl, True)
  Dim getCSV As HttpJob
  getCSV.Initialize("getCSV", Me)
  getCSV.Download(csvUrl)
End Sub

Sub Service_Destroy

End Sub
Sub JobDone (Job As HttpJob)
Dim In As InputStream

Dim strLine As String


  Log("JobName = " & Job.JobName & ", Success = " & Job.Success)
 
  If Job.Success = True Then
 
    ' Log(Job.GetString)
    In = Job.GetInputStream
    Dim tr As TextReader
    tr.Initialize(In)
   
      Do While tr.ReadLine() <> Null
      strLine = tr.ReadLine
     
      If strLine <> Null Then
        'Log(strLine)
          lastline = strLine
      End If
       
      Loop
         
    In.Close
    'Display the last line
    'The last line variable is here
    Log(lastline)

  Else
      Log("Error: " & Job.ErrorMessage)

  End If
  Job.Release
 
End Sub

 

[DonManfred]

@Reviewnow This may work, sure... But you have to download the hole csv (even if 100mb in size) just to get the last line.
A php-script called with an httputils-job just need a second to answer (cpu-power of webserver is used) but it just returns one line.
In time where networktraffic on mobile-phones are limited i would suggest to use a php-script as bridge

<?php
$lines = file ('mydata.csv');
echo $lines[count($lines)-1];
?>

But this could give problems when files are to big... >16mb or so...


Alternatively something like this could be of use:

$r = readBackward('mydata.csv', 1); print_r($r);
function readBackward($file, $length = 0 ){ $buffer = 8 * 1024; $nl = "\n"; $lines = array(); $last = ''; $fh = fopen($file, 'r');

if(!$fh) { die('Fehler beim öffnen von:' . $file . ' => ' . $php_errormsg);}
  fseek($fh, 0, SEEK_END); $pos = ftell($fh);

while($pos > 1){ $pos -= $buffer;
if($pos < 0) { $buffer += $pos; $pos = 0;
} fseek($fh, $pos, SEEK_SET); $tmp = explode($nl, fread($fh, $buffer) . $last);
if($pos > 0) {$last = array_shift($tmp);} $lines = array_merge($tmp, $lines);
if($length && count($lines) >= $length) { break; }
}
if($length && count($lines) >= $length) {  $lines = array_slice($lines, -$length);
}
return $lines;
}

 

[Reviewnow]

I agree with that scenario , this was just an alternate way incase he could not the php script created

 

[tomoshea]

Many Thanks to all.
I will let you know how I get on with the php scenario.
Kind Regards,
Tom

 

[tomoshea]

Hi DonManfred,
I have written a simple PHP script to test this scenario - but how do I use the php-script as bridge?
Can you give me a B4A example please?
Thank you,
Tom

 

[DonManfred]

You need httputils2-Library

Sub Activity_Create(FirstTime As Boolean)
   'Do not forget to load the layout file created with the visual designer. For example:
   'Activity.LoadLayout("Layout1")
   Dim php As HttpJob
   php.Initialize("phptest",Me)
   php.Download("http://www.domain.com/phpscript.php")
End Sub

Sub JobDone(Job As HttpJob)
   ProgressDialogHide
   If Job.Success Then
     Dim res As String
     res = Job.GetString
     Log("JobName: "&Job.JobName)
     If Job.JobName = "phptest" Then
       Log("Returned: "&res) ' Here is the output of phpscript. It should be only one line if the php works correctly
     End If
   Else
     ToastMessageShow("Error: " & Job.ErrorMessage, True)
   End If
   Job.Release
End Sub

 

[tomoshea]

Thanks Manfred

 

[DonManfred]

You´re welcome

If you find my answer useful please click on like ;-)

 

[tomoshea]

Hi Manfred,
Just a short update on how things went.
The running of a PHP script does not work with the code above.
I am using a WIFI SD card and linking to this card over WIFI.
I have copied the PHP script to the card and I use
php.Download("http://ip.address/sd/readlastline.php").
But it actually downloads the PHP script file instead of running it..
Also if I use http://ip.address/sd/readlastline.php in a browser it displays the script.
Thanks again,
Tom

 

[DonManfred]

The called php must be run on an system where php is running. A webserver in usual cases. for example; any webhosting with php capacities. A apache server with php compiled. On private networks and for testcases an Windows XAMPP for example.
If the "server" does not know how to handle (run) php-script then the file is treated as any other file. it is downloaded... This will happen on your PC´s browser too if you call a url with a php-file-.extension and the server dont know php.


The system where this sdcard is built in have any kind of webserver running? I mean; any running with can run php or any other things...

 

[tomoshea]

From the results of my tests it seems it does not have one?

 

[DonManfred]

Then it seems the only thing you can do is to download the hole file

 

[tomoshea]

Hi Manfred,
I think that I have run out of options.
Many thanks for all the help.
Tom

 

[Reviewnow]

Tom
not sure if this will help you http://www.phpforandroid.net/doku.php (I know nothing about php) if not, The solution I posted does get you the last line, but the whole file has to be downloaded, the only difference is that you don't have to save the file and open it again to get the final result.

Good Luck
Bill

 

[tomoshea]

Thanks Bill
Tom

 

[tomoshea]

Hi,
I have come up with a solution to my problem.
I have written some html with some javascript embedded in it to find the record i require.
this works fine - I have tested it with a Webview as follows

WebView1.LoadUrl("http://flashair/SD_WLAN/rwlast1.htm")

This writes the record to localstorage which I need to access/process.
How do I read this data from localstorage in my B4A application or should I use something other than a Webview to do this?
Thanks again for all your help.
Regards,
Tom

<!DOCTYPE html>
<html>
  <head>
    <title>reading file</title>
    <meta charset="UTF-8">
    <script type="text/javascript" src="/SD_WLAN/js/jquery.js"></script>
    <script type="text/javascript" src="/SD_WLAN/js/modernizr.custom.14927.js"></script>
    <script type="text/javascript">
    function getFileFromServer(url, doneCallback) {
            var xhr;
            xhr = new XMLHttpRequest();
            xhr.onreadystatechange = handleStateChange;
            xhr.open("GET", url, true);
            xhr.send();
            function handleStateChange() {
                    if (xhr.readyState === 4) {
                            doneCallback(xhr.status == 200 ? xhr.responseText : null);
                    }
            }
    }
    function WriteToFile(that) {
        if (Modernizr.localstorage) {
            // window.localStorage is available!
            localStorage["last"] = JSON.stringify(that);
            //document.getElementById ('main1').innerHTML= localStorage["last"];
        } else {
            // no native support for HTML5 storage :(
            // maybe try dojox.storage or a third-party solution
        }   
    }   
    </script>
</head>
<body>
    <div id="main"></div>
    <div id="main1"></div>
    <div id="file"></div>
    <script type="text/javascript">
          getFileFromServer("http://flashair/sd_wlan/log1.csv", function(text) {
                if (text === null) {
                        // An error occurred
                        document.getElementById ('file').innerHTML="http://flashair/sd_wlan/log1.csv";
                }
                else {
                        // `text` is the file text
                        var lines = text.split('\n');
                        var size = lines.length;
                        document.getElementById ('main').innerHTML= lines[size-2];
                        WriteToFile(lines[size-2]);
                        document.getElementById ('file').innerHTML="http://flashair/sd_wlan/log1.csv";
                }
        });
    </script>

</body>
</html>