B4J Question What is the best way to download image (httpjob) in a loop?

[Anno Jo]

Hi
I have IDs in excel file. then with those IDs I make download request with same url. yes it is image.

    For aloop=2 To reader.BottomRight.Row0Based+1
        Dim Ids As String = reader.Get(XL.AddressOne("B",aloop))

        downloadImageEvi(Ids)
        Sleep(IIf(chkSlow.Checked,100,0))
        prog2=(aloop-2)/xCount * 100
        ProgressBar2.Progress=(aloop-2)/xCount
        lblProgEvidence.Text="Requesting photo.... " & prog2 & "%"
    Next

Private Sub downloadImageEvi (IDs As String)
    Try
        Dim jobFoto As HttpJob
        jobFoto.Initialize(IDs, Me)
        jobFoto.Download(UrlD & IDs)
        jobFoto.GetRequest.Timeout=60000
        jobFoto.GetRequest.SetHeader("cookie",cookie)
        jobFoto.GetRequest.SetHeader("User-Agent","Mozilla/5.0 (Windows NT 6.3; WOW64)")
'        jobFoto.getrequest.setheader("Cache-Control","no-cache")
'        jobFoto.GetRequest.SetHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0")
        Sleep(50)

        Wait For (jobFoto) JobDone (jobFoto As HttpJob)
        If jobFoto.Success Then
            
            Dim bmp As B4XBitmap = jobFoto.GetBitmap
            If bmp.Height=0 Then
                Log("0?")
                ClearCookies
                Sleep(1000)
            Else
                Dim Out As OutputStream
                Out = File.OpenOutput(folderTemp, IDs, False)
                bmp.WriteToStream(Out, 60, "JPEG")
                Out.Close
            End If
            bmp=Null
        Else
            Log(jobFoto.ErrorMessage)
        End If
            
    Catch
        Log(LastException.Message)
    End Try
'    Log("release")
    jobFoto.Release
End Sub

I put in loop n it worked well in few request.
The problem is that the more requests in a loop (sometimes it can be 1000 req in a time), the more images fail to download successfully. despite jobdone is success.
I tried using sleep but it didn't work.
I tried to put all in a sub, but also didn't work. some images saved, some empty (0) then saved again n empty again. not stable

what's the best trick? I mean how to make it stable and all requests successful (image not empty)
does this have anything to do with memory usage? or cookie?

Thank You

 

[Anno Jo]

update:
I have tried calling process_gc method and it really makes a difference. maybe because of the excel loading process and image handling which requires memory. but still within a certain period of time the image is still 0 and will continue to be 0.
currently I am trying SMM n save the image. I need 'stable' non empy response

so (still) what is the best way to do this?

 

[Erel]

The problem might be on the server side, assuming that you are downloading from the same server.
Your code sends all requests at once. The progress implementation is wrong.

 

[Anno Jo]

The problem might be on the server side, assuming that you are downloading from the same server.
Your code sends all requests at once. The progress implementation is wrong.

true, that progress will not be able to show the exact value as the time the image was successfully saved. I will fix it.

the main purpose is how to make sure all images are successfully downloaded in a short time. maybe not in very short time.
okay it might be on the server side, I'll try to make it slower request.
I am very sure calling process_gc is very helpful
is it safe and okay to call it often before executing download?

Jun 15, 2025 2:33:57 PM com.sun.javafx.tk.quantum.PrismImageLoader2$PrismLoadListener imageLoadWarning
WARNING: Corrupt JPEG data: 1 extraneous bytes before marker 0xd9

also I have this message sometime, is it okay?

Thank You

 

[Erel]

I am very sure calling process_gc is very helpful

It has no effect.

I never seen a case where Job.Success is true and the response wasn't downloaded properly. I've seen many cases where the server response itself was corrupt. The "ClearCookies" call looks suspicious. Maybe it breaks something.

 

[Anno Jo]

I need cookies because the client asks for evidence reports from the application. while there is no photo export menu.
so I must login from webview, take the cookie and add it in the download photo request.

You are so right Erel, the problem has nothing to do with the code and memory.
it comes from the server and cookies. somehow it's not stable and break sometimes.
I have tested 1000 records using static photo urls and 100% successfully downloaded properly. I learned a lot again this time.
Thank you so much for the help