B4J Question jOkHttpUtils2 getstring stops download after 3 dots (spread operator)

gezueb · Sep 28, 2024

Hi All, I try to read a local html website said utils and parse some data. However, the job.Getstring truncates everything after (including) the first occurrence of three dots (...).
Size of website is some 250'000 characters, the string is only 1100 chars.

B4X:

Sub getLocalVartaData As ResumableSub

    Dim job As HttpJob
    job.Initialize("",Me)
    job.Download("http://192.168.1.18/home.html")
    Wait For (job) JobDone(job As HttpJob)
    If job.Success Then
        Dim result As String
        result = job.GetString
        Log("String-Length: "&result.Length)
        Log(result)
    End If
    
    job.Release
End Sub

gezueb · Oct 1, 2024

OK, here comes the code. However, dont spend too much time on it. I run several html validators and they report many errors. I have come to the conclusion that this page is crap.

Daestrum · Oct 1, 2024

If you get %C3 and %9C as single characters - you probably need to read the string as windows-1252 as opposed to utf-8

windows-1252 %C3
utf-8 %C3%83

windows-1252 %9C
utf-8 %C5%93

gezueb · Oct 1, 2024

Tried

B4X:

result = job.GetString2("windows-1252")

no change however?

DonManfred · Oct 1, 2024

gezueb said:
I have come to the conclusion that this page is crap.

The page is reloading data with javascript. As the javascript is minified it is hard to read.

Not all data shown in a browser are inside the html. DATA are probably missing.

aeric · Oct 1, 2024

No content is truncated.

B4X:

Sub DownloadPage
    Dim job As HttpJob
    job.Initialize("", Me)
    job.Download("http://127.0.0.1/varta/home.html")
    Wait For (job) JobDone(job As HttpJob)
    If job.Success Then
        Log("Response Content Length=" & job.GetString.Length)
        Dim result As String = job.GetString
    End If
    job.Release
    File.WriteString(File.DirApp, "output.txt", result)
    ' Verify content length
    Dim FileContent As String = File.ReadString(File.DirApp, "output.txt")
    Log("File Content Length=" & FileContent.Length)
End Sub

Logs:

B4X:

Waiting for debugger to connect...
Program started.
Response Content Length=269601
File Content Length=269601

aeric · Oct 1, 2024

This also prove that the content after ... is not truncated and can be read.

B4X:

Sub DownloadPage
    Dim job As HttpJob
    job.Initialize("", Me)
    job.Download("http://127.0.0.1/varta/home.html")
    Wait For (job) JobDone(job As HttpJob)
    If job.Success Then
        Dim result As String = job.GetString
    End If
    job.Release
    
    Dim occurence As Int
    Dim start As Int = -3
    Dim found As Int
    Do Until found = -1
        found = result.IndexOf2("...", start+3)
        If found > -1 Then
            Log(found & "->" & result.SubString2(found, found+30)) ' Log 30 characters after ...
            occurence = occurence + 1
            start = found           
        End If
    Loop
    Log("occurence=" & occurence)
End Sub

Logs:

B4X:

Waiting for debugger to connect...
Program started.
5494->...c){for(const b of c)if(a=a[
6203->...e){}const b={};new Map;b.is
6286->...a){return this.logModule(nu
6319->...a)};b.logModule=function(a,
6349->...c){return d("log",a,...c)};
6372->...c)};b.logInfo=function(...a
6398->...a){return this.logModuleInf
6435->...a)};b.logModuleInfo=functio
6469->...c){return d("info",a,...c)}
6493->...c)};b.logWarning=function(.
6522->...a){return this.logModuleInf
6559->...a)};b.logModuleWarning=func
6596->...c){return d("warning",a,...
6623->...c)};b.logError=function(...
6650->...a){this.logModuleError(null
6683->...a)};b.logModuleError=functi
6718->...c){return d("error",a,...c)
6743->...c)};b.proxyObject=function(
occurence=18

aeric · Oct 1, 2024

By the way, I have replaced the string

http://192.168.1.18/

to

http://127.0.0.1/varta/

in home.html
to load into my local server.

gezueb · Oct 1, 2024

Ok, I see that the page works. Thats good news. But how do I get http://192.168.1.18/ programmatically into http://127.0.0.1/varta/ ?

aeric · Oct 1, 2024

gezueb said:
Ok, I see that the page works. Thats good news. But how do I get http://192.168.1.18/ programmatically into http://127.0.0.1/varta/ ?

I am not sure what you are trying to do is correct and whether my mod gives what you are looking for.
I use Laragon development server. Just drop the files into a folder name varta inside www folder and refresh the server.

gezueb · Oct 1, 2024

Ok, sorry I have to be more precise. I can access the page on my local network on 192.168.1.18. (or any other local url defined by my router) But I cannot change the contents, its read only. I guess the problem might be file naming and file structure. I wonder how I can access the files from the my embedded server without copying and zipping manually.

aeric · Oct 1, 2024

I am not sure you are familiar with server, HTML and JavaScript. The content of the HTML or DOM can be modified by JavaScripts.
I am also not sure what device it is hosted the web pages. I think you have to access the File Manager inside the server using FTP tool.

gezueb · Oct 1, 2024

I have no other access to the server than the unmodified local url. No FTP. If I must change the html code, I will have to copy the contents by opening the webpage with a browser and store it from there and then possibly edit. That's how I made the zip file. But I cannot use this procedure periodically (say every 15 min). Thats why I am looking for a simpler way with a B4J (or eventually a B4A) app.

teddybear · Oct 1, 2024

gezueb said:
Hi All, I try to read a local html website said utils and parse some data. However, the job.Getstring truncates everything after (including) the first occurrence of three dots (...).
Size of website is some 250'000 characters, the string is only 1100 chars.

As DonManfred said at #24

page is reloading data with javascript.

The page is dynamic and you can‘t entirely download it using jOkHttpUtils2.
You may use webview to load the url and then parse html code.

aminoacid · Oct 1, 2024

aeric said:
I am not sure you are familiar with server, HTML and JavaScript. The content of the HTML or DOM can be modified by JavaScripts.
I am also not sure what device it is hosted the web pages. I think you have to access the File Manager inside the server using FTP tool.

I see what you are saying, @aeric. If the file "home.html" that GetString is reading from his device server contains javascript that produces the HTML that he is seeing in his browser, then the size of the string read from the server is definitely going to be different (and perhaps shorter) that the size of the HTML code displayed in the browser.

@gezueb, I think what you should do is use @aeric 's code in post#25 and save the string to a file. Then look at the javascript in the file to see what it's doing.

aminoacid · Oct 1, 2024

teddybear said:
The page is dynamic and you can‘t entirely download it using jOkHttpUtils2.
You may use webview to load the url and then parse html code.

Exactly!

gezueb · Oct 3, 2024

Thanks, I will try to learn webview and its functions

B4J Question jOkHttpUtils2 getstring stops download after 3 dots (spread operator)

gezueb

Active Member

gezueb

Active Member

Attachments

Daestrum

Expert

gezueb

Active Member

DonManfred

Expert

aeric

Expert

aeric

Expert

aeric

Expert

gezueb

Active Member

aeric

Expert

Attachments

gezueb

Active Member

aeric

Expert

gezueb

Active Member

teddybear

Well-Known Member

aminoacid

Active Member

aminoacid

Active Member

gezueb

Active Member

Similar Threads

B4J Question jOkHttpUtils2 getstring stops download after 3 dots (spread operator)

Active Member

Active Member

Attachments

Expert

Active Member

Expert

Expert

Expert

Expert

Active Member

Expert

Attachments

Active Member

Expert

Active Member

Well-Known Member

​

Active Member

Active Member

​

Active Member

Similar Threads

Privacy & Transparency

Privacy & Transparency