Android Question Number of lines in a text file

Sergey_New · Nov 19, 2023

I need to read a text file.
For the progressbar to work, I need to know the number of lines scount.

B4X:

    Try
        Dim rd As TextReader:
        rd.Initialize(File.OpenInput(Starter.myFolder, FileName))
        scount=Regex.Split(CRLF,rd.ReadAll).Length
        rd.Close
    Catch
        Log(LastException)
    End Try

If the number of lines is less than 90,000, then everything works, but if, for example, there are 1,500,000 lines, the application crashes on line #4.

What restrictions might there be?

DonManfred · Nov 21, 2023

klaus said:
(ErrnoException) android.system.ErrnoException: open failed: ENOENT (No such file or directory)

did you rename the file to be lowercased?
It is named Royal.ged in original... must be royal.ged when reading from assets

Edit: I did not tested in debug at all. I am always using Release

klaus · Nov 21, 2023

DonManfred said:
It is named Royal.ged in original... must be royal.ged when reading from assets

In Release mode it works with Royal.ged and with royal.ged.
In Debug mode, even with royal.ged the error message is the same.

Daestrum · Nov 21, 2023

What if you change

B4X:

 Dim strLines As String= rd.ReadLine

to

B4X:

 Dim strLines As Object= rd.ReadLine

and then add that to the list

Sergey_New · Nov 30, 2023

In reality, populating a special database is much more difficult than

B4X:

     Dim lst As List
     lst.Initialize
     lst=File.ReadList(Starter.myFolder, FileName)
     Log(lst.Size)

Reading of a large file stops depending on its size at 30% read or more.
All attempts to catch a memory overflow exception have failed.
Is it possible to get an exception when memory is full?

LucaMs · Nov 30, 2023

You could try using the Application_Error event in the Starter service, but then... What?

DonManfred · Nov 30, 2023

Sergey_New said:
Is it possible to get an exception when memory is full?

your app crash when the memory is full. You can not intercept that.
I already told you that you are better of with writing a parser for the textfile which will parse the file line by line and create the db entries. Especially when the files can be big.
You are just ignoring such hints.

It is´nt fun to read this thread too as it has 3 pages already

Sergey_New · Nov 30, 2023

No, I don't ignore it.
Exactly as you advise, the application analyzes each line and fills the database. Below is the entire code for reading the file:

B4X:

Sub ReadFile(FileName As String, cpb As CircularProgressBar)
    TagsMap.Initialize
    ecount=0
    Starter.IsChanged=False
    Dim c1, progress=1, scount As Int, c2, c3 As String
    Dim isconc As Boolean
    myStack.Initialize
    db.Initialize
    UserTags.Initialize
    UncnowMap.Initialize
    Dim h As Int=0
    fileError=FileName
    Try
        Dim rd As TextReader
        scount=File.ReadList(Starter.myFolder, FileName).Size
        rd.Initialize(File.OpenInput(Starter.myFolder, FileName))
        Dim strLines As String= rd.ReadLine
        Dim r As record
        r.Initialize
        Do While strLines <> Null
            ecount=ecount+1
            Dim sb As StringBuilder: sb.Initialize
            Dim m As Matcher = Regex.Matcher(pat,strLines)
            Do While m.Find
                c1=m.Group(1): c2=m.Group(2): c3=m.Group(3)
            Loop
            Select Case c2
                Case "CONC"
                    r.c3=sb.Append(r.c3).Append(c3)
                    strLines = rd.ReadLine
                Case "CONT"
                    r.c3=sb.Append(r.c3).Append(CRLF).Append(c3)
                    strLines = rd.ReadLine
                Case Else
                    If isconc Then
                        FillDB(r)
                    End If
                    r.c1=c1: r.c2=c2: r.c3=c3
                    isconc=True
                    strLines = rd.ReadLine
            End Select
            If cpb.IsInitialized Then
                Dim x As Int=100*progress/(scount)
                If x>h Then
                    cpb.Value=x
                    Sleep(0)
                    h=x
                End If
                progress=progress+1
            End If
        Loop
    Catch
        Log(LastException)
    End Try
    rd.Close
End Sub

DonManfred · Nov 30, 2023

Sergey_New said:
Below is the entire code for reading the file

do you still get OOM with it?

Sergey_New · Nov 30, 2023

DonManfred said:
do you still get OOM with it?

I don't understand what it means to get OOM (Out-Of-Memory). I don't find anything similar in the debug log. The application just stops.

MicroDrie · Nov 30, 2023

In addition to the Out-Of-Memory (OOM) problem, there is another problem, namely time. The OS considers running your program to be a problem after a long time and quietly kills it after showing a pop-up message several times. With text reader I could read about 22,698 lines, but without text reader it was after 142,730 lines that the OS thought it had enough and quietly killed my debug mode program.

agraham · Nov 30, 2023

Sergey_New said:
All attempts to catch a memory overflow exception have failed.

You can't catch an OOM exception as the only thing that can be done is to kill the process as it is now in a probably unstable state.

Sergey_New · Nov 30, 2023

MicroDrie said:
and quietly kills it after showing a pop-up message several times

There were no messages.

Sergey_New · Nov 30, 2023

agraham said:
kill the process

How do you tell an application or user that a process needs to be terminated?

RB Smissaert · Nov 30, 2023

Sergey_New said:
I don't understand what it means to get OOM (Out-Of-Memory). I don't find anything similar in the debug log. The application just stops.

Likely if there are no error messages then there are OOM problems or as suggested by MicroDrie

Sergey_New said:
There were no messages.

If there are no messages then likely there are OOM problems or as MicroDrie suggested the OS kills the app as it considers it to be stuck/frozen or whatever it thinks it is doing wrong.
If you are interested I can post code that produces in a loop, small enough lists of arrays that you can then process with no problems.
Maybe best to start a new thread with a more accurate title.

RBS

73Challenger · Nov 30, 2023

This isn't exactly what you asked but... could you estimate the number of lines by getting the SIZE of the file instead of trying to read al the lines?
FileSize = 1,000 bytes
Line size = 100 bytes
Lines = 10
If the lines are all similar in length, this might work.

Second thought, could you keep a count of the lines before you write them out to the file? If you need to store this across sessions you could put it in a sqllite table or a simple text file to restore when the app opens?

A progress bar, especially if you are talking about a massive amount of lines doesn't have to be exactly perfectly accurate. If it's off by 10 your users won't notice.

Android Question Number of lines in a text file

Sergey_New

Well-Known Member

DonManfred

Expert

klaus

Expert

Daestrum

Expert

Sergey_New

Well-Known Member

LucaMs

Expert

DonManfred

Expert

Sergey_New

Well-Known Member

DonManfred

Expert

Sergey_New

Well-Known Member

MicroDrie

Well-Known Member

agraham

Expert

Sergey_New

Well-Known Member

Sergey_New

Well-Known Member

RB Smissaert

Well-Known Member

73Challenger

Active Member

Similar Threads

Android Question Number of lines in a text file

Well-Known Member

Expert

Expert

Expert

Well-Known Member

Expert

Expert

Well-Known Member

Expert

Well-Known Member

Well-Known Member

Expert

Well-Known Member

Well-Known Member

Well-Known Member

Active Member

Similar Threads

Privacy & Transparency

Privacy & Transparency