B4J Question DoEvents alternative

[rtek1000]

I saw a recommendation not to use DoEvents:

DoEvents should be avoided in all platforms. Use CallSubDelayed if you want to run something after the current sub finishes.

Source: https://www.b4x.com/android/forum/threads/doevents-alternative.68918/#post-437288

But the window freezes if you don't use something to wait a while. And Windows O.S. notifies you that the window is not responding.

I managed to read thousands of lines in a text file without freezes the window like this:
- I use a CallSubDelayed
- Inside the long loop I use a Sleep(0) command

I don't know if there's a better way to do this, but it worked here. If you don't use the Sleep(0) command, the window continues to freeze.

I'm converting a text file of tens of thousands of lines into an Excel spreadsheet.

Is there a more efficient way than using Sleep(0)?

Thank you.

 

[peacemaker]

more efficient way

Threading. But just find where in your loop there is most long code and use Sleep(0). Now it looks that you use Sleep(0) in wrong point.

 

[Erel]

First step is to check the actual task and see whether it can be optimized or be done asynchronously.

I'm converting a text file of tens of thousands of lines into an Excel spreadsheet.

Please post the relevant code.

 

[agraham]

- I use a CallSubDelayed

Not needed

- Inside the long loop I use a Sleep(0) command

This is OK but you need to be aware that the user may raise other events while you are processing because you are keeping the UI awake by returning to the message loop so you might need to disable parts of the UI until the processing is complete. Note that this is also a consideration if you are doing the processing asynchronously in a thread. It is simpler to stick with Sleep(0) unless it spoils the user experience in some way.

 

[rtek1000]

Code:

Sub File_Export
    Dim myTextReader As TextReader
    Dim components() As String
    Dim XLAddress1 As XLAddress
    Dim component_index As Int = 0
   
    ProgressBar1.Progress = 0

    XL.Initialize
    XLAddress1.Initialize

    Dim Workbook As XLWorkbookWriter = XL.CreateWriterBlank
    Dim sheet1 As XLSheetWriter = Workbook.CreateSheetWriterByName("Sheet1")
    Dim TitleStyle As XLStyle = Workbook.CreateStyle
   
    TitleStyle.ForegroundColor(XL.COLOR_GREY_80_PERCENT).FontBoldColor(12, XL.COLOR_WHITE).HorizontalAlignment("CENTER")
   
    Try
        myTextReader.Initialize(File.OpenInput(SourceFileFolder, SourceFileName) )
        Dim LineCounter As Int = 0
        Dim line As String = "0"
        Do While line <> Null
            line = myTextReader.ReadLine
           
            Try
                If(line <> Null) Then
                    components = Regex.Split("\t",line)
                End If
            Catch
                Log(LastException)
            End Try

            If(line <> Null) Then
                If(components.Length > 0) Then
                    component_index = 0
                    For Each data_string In components
                        XLAddress1.Col0Based = component_index
                        XLAddress1.Row0Based = LineCounter
                        sheet1.PutString(XLAddress1, data_string)

                        component_index = component_index + 1
                    Next
                End If
            End If

            ProgressBar1.Progress = LineCounter / TotalLineCounter
           
            LineCounter = LineCounter + 1
            Sleep(0) ' <----------------------------- Sleep(0) is here
        Loop
        myTextReader.Close
    Catch
        Log("Error --> " & LastException.Message)
        If myTextReader.IsInitialized Then
            myTextReader.Close
        End If
    End Try
   
    'save the workbook
    Dim f As String = Workbook.SaveAs(File.DirApp, "Products.xlsx", True)
    Wait For (XL.OpenExcel(f)) Complete (Success As Boolean)
'    StopMessageLoop 'non-ui
End Sub

Reference to the progress bar:

    Dim myTextReader As TextReader

    Try
        myTextReader.Initialize(File.OpenInput(SourceFileFolder, SourceFileName) )

        Dim line As String = "0"
        Do While line <> Null
            TotalLineCounter = TotalLineCounter + 1
            line = myTextReader.ReadLine
        Loop
        myTextReader.Close
    Catch
        Log("Error --> " & LastException.Message)
        If myTextReader.IsInitialized Then
            myTextReader.Close
        End If
    End Try

 

[Erel]

What is the output of this code, in release mode:

Dim n As Long = DateTime.Now
 Dim lines As List = File.ReadList(SourceFileFolder, SourceFileName) 
      For Each line As String In lines
                If(components.Length > 0) Then
                    component_index = 0
                    For Each data_string In components
                        XLAddress1.Col0Based = component_index
                        XLAddress1.Row0Based = LineCounter
                        sheet1.PutString(XLAddress1, data_string)
                        component_index = component_index + 1
                    Next
                End If
            End If
        Next
Log("Time: " & (DateTime.Now - n)")

 

[rtek1000]

Yes I was curious to know the times too!

I got these times:

Thread1.Start(Null, "File_Export", args1): 37 seconds

12:41:37 - Start
12:42:14 - Stop

CallSubDelayed(Me, "File_Export"): 1 minute, 16 seconds
12:44:02 - Start
12:45:18 - Stop

I used Log(DateTime.Time(DateTime.Now)) instead of subtracting, but I believe it's a similar time:

Sub File_Export
    Dim myTextReader As TextReader
    Dim components() As String
    Dim XLAddress1 As XLAddress
    Dim component_index As Int = 0
  
    ProgressBar1.Progress = 0
  
    Log(DateTime.Time(DateTime.Now))

    XL.Initialize
    XLAddress1.Initialize

    Dim Workbook As XLWorkbookWriter = XL.CreateWriterBlank
    Dim sheet1 As XLSheetWriter = Workbook.CreateSheetWriterByName("Sheet1")
    Dim TitleStyle As XLStyle = Workbook.CreateStyle
  
    TitleStyle.ForegroundColor(XL.COLOR_GREY_80_PERCENT).FontBoldColor(12, XL.COLOR_WHITE).HorizontalAlignment("CENTER")
  
    Try
        myTextReader.Initialize(File.OpenInput(SourceFileFolder, SourceFileName) )
        Dim LineCounter As Int = 0
        Dim line As String = "0"
        Do While line <> Null
            line = myTextReader.ReadLine
          
            Try
                If(line <> Null) Then
                    components = Regex.Split("\t",line)
                End If
            Catch
                Log(LastException)
            End Try

            If(line <> Null) Then
                If(components.Length > 0) Then
                    component_index = 0
                    For Each data_string In components
                        XLAddress1.Col0Based = component_index
                        XLAddress1.Row0Based = LineCounter
                        sheet1.PutString(XLAddress1, data_string)

                        component_index = component_index + 1
                    Next
                End If
            End If

            ProgressBar1.Progress = LineCounter / TotalLineCounter
          
            LineCounter = LineCounter + 1
            Sleep(0) ' <----------------------------- Sleep(0) is here (Comment this line for use in Thread)
        Loop
        myTextReader.Close
    Catch
        Log("Error --> " & LastException.Message)
        If myTextReader.IsInitialized Then
            myTextReader.Close
        End If
    End Try
  
    Log(DateTime.Time(DateTime.Now))
  
    'save the workbook
    Dim f As String = Workbook.SaveAs(File.DirApp, "Products.xlsx", True)
    Wait For (XL.OpenExcel(f)) Complete (Success As Boolean)
'    StopMessageLoop 'non-ui
End Sub

Ref.: #29

 

[rtek1000]

Note: For Thread: "Sleep(0)" was commented out and the window did not freeze.

 

[Erel]

How many lines are in the text file?

Please test it with my code and run it on the main thread. You cannot run arbitrary code on a background thread. It will not work as you expect.

 

[rtek1000]

Please test it with my code and run it on the main thread. You cannot run arbitrary code on a background thread. It will not work as you expect.

Time: 214

 

[rtek1000]

How many lines are in the text file?

22.932

 

[Erel]

So it takes 200 milliseconds. Nothing more needs to be done (unless I've missed something).

 

[rtek1000]

What is the output of this code, in release mode:

Dim n As Long = DateTime.Now
 Dim lines As List = File.ReadList(SourceFileFolder, SourceFileName)
      For Each line As String In lines
                If(components.Length > 0) Then
                    component_index = 0
                    For Each data_string In components
                        XLAddress1.Col0Based = component_index
                        XLAddress1.Row0Based = LineCounter
                        sheet1.PutString(XLAddress1, data_string)
                        component_index = component_index + 1
                    Next
                End If
            End If
        Next
Log("Time: " & (DateTime.Now - n)")

It didn't work yet, this step is not met, this whole block is skipped so it's so fast:

If(components.Length > 0) Then

 

[Erel]

True. I've missed this line: components = Regex.Split("\t",line)

Please add it and test.

 

[rtek1000]

True. I've missed this line: components = Regex.Split("\t",line)

Please add it and test.

Yes I managed to find this need:

Sub time_test
   
    Dim components() As String
    Dim XLAddress1 As XLAddress
    Dim component_index As Int = 0
    Dim LineCounter As Int = 0
   
    Dim Workbook As XLWorkbookWriter = XL.CreateWriterBlank
    Dim sheet1 As XLSheetWriter = Workbook.CreateSheetWriterByName("Sheet1")
    Dim TitleStyle As XLStyle = Workbook.CreateStyle
   
    TitleStyle.ForegroundColor(XL.COLOR_GREY_80_PERCENT).FontBoldColor(12, XL.COLOR_WHITE).HorizontalAlignment("CENTER")

    Dim n As Long = DateTime.Now
   
    ProgressBar1.Progress = 0
   
    XL.Initialize
    XLAddress1.Initialize
   
    Dim lines As List = File.ReadList(SourceFileFolder, SourceFileName)
   
    'Log("lines.Size: " & lines.Size)
   
    For Each line As String In lines
'        If(LineCounter = 0) Then
'            Log("line: " & line)
'        End If
       
        If(line <> Null) Then
            components = Regex.Split("\t",line)
        End If
       
        'Log("components.Length: " & components.Length)
   
        If(components.Length > 0) Then
            component_index = 0
           
'            If(LineCounter = 0) Then
'                Log("components.Length > 0")
'            End If
           
            For Each data_string In components
'                If(component_index = 0) Then
'                    Log("data_string: " & data_string)
'                End If
               
                XLAddress1.Col0Based = component_index
                XLAddress1.Row0Based = LineCounter
                sheet1.PutString(XLAddress1, data_string)
                component_index = component_index + 1
            Next
        End If
       
        LineCounter = LineCounter + 1
       
        ProgressBar1.Progress = LineCounter / TotalLineCounter
    Next
    Log("Time: " & (DateTime.Now - n))
   
    'save the workbook
    Dim f As String = Workbook.SaveAs(File.DirApp, "Products.xlsx", True)
    Wait For (XL.OpenExcel(f)) Complete (Success As Boolean)
'    StopMessageLoop 'non-ui
End Sub

Time: 2745

Fast code:

Dim lines As List = File.ReadList(SourceFileFolder, SourceFileName)

Slow code:

line = myTextReader.ReadLine

What about the memory resources that File.ReadList uses, how can I see the difference?

 

[rtek1000]

If it is a very large file, I believe the system should allocate memory to perform this task, and in case it is a computer with HDD and there is little RAM memory available, the HDD will be used as swap memory, this can cause more slowdowns than reading line by line as a heavily used HDD is very slow.

 

[rtek1000]

I found this post below,

This article will be a collection of Java performance measurement pointer. It describes how memory works in general and how Java use the heap and the stack. The article describes how to set the available memory for Java. It discusses then how to get the runtime and the memory consumption of a Java application.

Java Performance - Memory and Runtime Analysis - Tutorial

But I think I'll leave that subject for another opportunity, if I need to work with very large files, at the moment I expect it to be only around 50 to 100MiB.

Thank you for your help!

 

[agraham]

You are over analyzing and prematurely optimizing for a problem that may never exist. Go with the far faster File.ReadList until you get a problem - then optimize it.

 

[rtek1000]

Yes, the difference between stepping on the brake pedal at the time of an accident or before an accident is the use of thought.

 

[Erel]

BTW, the progress bar won't show anything without a call to Sleep. Better to remove it.