B4A Library Introducing liteCsvParse: A Lightweight Library for Efficient CSV Processing

Hello everyone,

I’d like to share with you a tool I’ve developed called liteCsvParse. It’s a lightweight library I created to make working with CSV files easier, especially on older devices or those with limited resources.

The library consists of two main functions:

• CsvToList: This function takes a CSV file and converts it into a list of arrays, allowing for easy manipulation of the contained data.
• ListToCsv: Conversely, this function takes an array of arrays and exports it to a CSV file.

I was motivated to develop this library because when processing large CSV files on devices with low memory, existing tools like CvsParser couldn’t complete the task. Even on more modern devices, if the files are very large, similar issues arise.

To my surprise, liteCsvParse worked much better than expected. Not only does it handle files efficiently on devices with limited resources, but it’s also remarkably fast. In my tests, I found that the processing time is comparable to LoadCsv and up to ten times faster than csvParser.

While it doesn’t have all the advanced features of Csvparser, liteCsvParse requires significantly less memory and offers greater speed. So far, I’ve tested it with dozens of files and files up to 1 million rows with satisfactory results.

Additionally, I am attaching the project files with which I conducted the tests of CsvParser vs LoadCsv vs liteCsvParse, as well as the library itself. This is so you can conduct your own tests if you wish and also see how the library is used.

I hope this library will be as useful to you as it has been for me.

Notes: The library version is 1.07, a couple of bugs were fixed

If you liked the library and wish to support my work, you can do so through my PayPal account: paypal.me/Carlos7000

 

[carlos7000]

Recently, I encountered an issue in my B4A application while running multiple performance tests to evaluate three CSV file processing algorithms: csvParser, LoadCsv, and liteCsvParse. After executing several tests consecutively, the application would either freeze or close due to excessive memory consumption.

The root cause was that I had defined the Lista variable (of list type) outside the test function. In each test, the application allocated memory to store the processed data but failed to release it afterward. This accumulation of allocated memory eventually led to the application freezing.

The solution was straightforward: I moved the Lista definition inside the test function. This way, when each test finished, the garbage collector properly released the memory allocated for that variable. Now I can run consecutive tests without encountering any blocking issues.

The updated version of RunCsvToFile is a direct result of the issue I mentioned earlier. In this new version, the data for the Lista variable is obtained within the RunCsvToFile function. This approach ensures that when the function completes its execution, memory is properly released. As a result, I can now run multiple tests consecutively without the operating system complaining about excessive memory consumption.

Public Sub RunCsvToFile
    Dim Lista As List
    Funciones.PDS("ListToCsv...")
  
    Try
        Lista        = liteCsvParse.CsvToList(File.DirAssets, ArchivoStr, ",", False)
      
        IniLong        = DateTime.Now
        liteCsvParse.ListToCsv(Directorio.Get, "file.csv", Lista, ",")
        FinLong        = DateTime.Now
  
        LabelRunTimeListToCsv.Text = Funciones.CalculaTiempo(IniLong, FinLong)
    Catch
        Log(LastException)
    End Try
End Sub

In summary, the project can now be executed multiple times without the application freezing. However, if the amount of data is very large, there is a possibility that the application may eventually freeze due to memory consumption.

I am attaching the project with the necessary corrections.