Android Question [solved] missing rows in saved CSV file?

[Dave O]

After 2 days of bafflement, I'm asking for fresh eyes.

I'm exporting data to a CSV file, but it seems to be losing the last few lines of the data.

There are 190 rows of data, but I'm only seeing 170 in the resulting file. The last line in the file is incomplete (missing part of a value in a column, and the rest of the columns in that row).

I've checked the data, but it looks fine - no weird characters, no delimiters, no line feeds.

And I've tried this on 2 different devices with 2 different Android versions - same result. I even switched from using StringUtils.saveCSV2 to using CSVParser - same result.

My logging shows that the list I'm creating and exporting does contain all 190 items.

The only other clue I have is that the CSV file is apparently encoded as "Windows 1252: Western European" instead of the UTF-8 I expected, but that may be a consequence of reading the file off the connected device using Windows?

Here's the export code:

Sub exportListToFile As String
    Dim headerList As List
    headerList.Initialize2(Array As String("Name", "Quantity","Needed" ,"Order in list", "Note", _
        "Group", "Group position", "Order in group", "Crossed out"))

    Dim itemList As List
    itemList.Initialize
    For Each tempItem As shopItem In currentShopList.items
        itemList.Add(buildExportItem(tempItem))
    Next

    Dim strippedName As String = _
        stripToAllowedCharacters(currentShopList.name, ALLOWED_FILENAME_CHARACTERS) & ".csv"
    Dim su As StringUtils
    su.SaveCSV2(importExportDir, strippedName, SEPARATOR_CHAR, itemList, headerList)
        
    Return strippedName
End Sub

Sub buildExportItem(tempItem As shopItem) As String()
    Dim stringArray(9) As String
    stringArray(0) = tempItem.name
    stringArray(1) = tempItem.quantity
    stringArray(2) = tempItem.needed
    stringArray(3) = tempItem.manualIndex + 1
    stringArray(4) = tempItem.note
    stringArray(5) = getGroupNameFromID(tempItem.groupID)
    stringArray(6) = getIndexOfGroupID(tempItem.groupID) + 1
    stringArray(7) = tempItem.manualGroupIndex + 1
    stringArray(8) = tempItem.crossedOut
    Return stringArray
End Sub

...and the last few lines of the CSV file come out as:

tortillas: corn small,2,false,152,,cans & sauces,6,29,false
vacuum bags S-bag,5,false,153,Electrolux/Philips,Cleaning,9,15,false
vanilla extract,1,false,154,,baking,5,17,false
venison,2,false,155,,Meat,2,5,fa

So, the final column value is "false", but truncated to "fa". And the last 18 rows are missing.

Any idea what's going on here? Thanks!

 

[LucaMs]

Any idea what's going on here?

I have no idea about that (as almost aways ?) but... can you use a different method, B4XSerializator?

I think you could create two types in the Main (I change the name of your shopItem because... I like to do it ?, using my "style"):

Type tShopItem(--- all your stuff ---)
Type tShopList(lstShopItems As List)

Then, create a tShopList variable:

Dim ShopList As tShopList ' <--- here you can see ONE reason why I use "t" as prefix for my custom types
' Here fill its lstShopItems (as you did in your "for each loop")

and serialize ShopList:

'serializator is a B4XSerializator variable
Dim arrShopList() As Byte = serializator.ConvertObjectToBytes(ShopList)

After done this, you may convert the serialized object (arrShopList()) to a string (EncodeBase64) , if you want:

Dim b64ShopList As String ' b64 stands for base 64 encoded

' su is a StringUtils variabe
b64ShopList = su.EncodeBase64(arrShopList)

Another way could be use JSON (most used and "standard").

 

[emexes]

The way that it gets cut short near the end, makes me think perhaps it is a number-of-characters vs number-of-bytes issue. Hard to believe such a thing could have lay dormant for so long, but... somebody has to be first to find it, perhaps you are the winner, ie today is your "lucky" day

Can you temporarily change the BuildExportItem routine to just return fixed ASCII-only sample data? See if still truncated, and if so, where.

 

[LucaMs]

Anyway... I love B4XSerializator ?

(I love B4X !!!)

 

[Erel]

StringUtils.SaveCSV2 writes all the data. I'm pretty sure that the problem is somewhere else. If you can try to reproduce it in a small project.

The only other clue I have is that the CSV file is apparently encoded as "Windows 1252: Western European" instead of the UTF-8 I expected, but that may be a consequence of reading the file off the connected device using Windows?

This sounds like a hint that there is a different problem. My guess is that you are reading an old file.

Download the file with B4A-Bridge FTP feature (Tools - B4A-Bridge - File Explorer).

 

[Dave O]

Thanks @Erel , once again your instincts were right. False alarm - the file is not really truncated at all.

I loaded up a CSV file viewer app on both phones so I could view the file directly. This "real" file has the full 190 lines of data.

Connecting to Windows to view the file via NotePad and Excel only shows 170 lines. So yes, using Windows is somehow messing it up (by caching or encoding or something else). Something to watch for if you're using Windows to view/manipulate files on Android devices.

Thanks all! I now return to my regularly scheduled programming.

 

[Jeffrey Cameron]

the file via NotePad and Excel

I haven't used Notepad in years. I strongly recommend Notepad++

 

[Dave O]

Notepad++

I agree that Notepad is only useful for a quick look-see like this. I bought EditPad Pro years ago and rely on it for anything substantial.

 

[LucaMs]

I haven't used Notepad in years. I strongly recommend Notepad++

As I said a long time ago (the day before yesterday and kidding ?)... even Erel can make a mistake: he only forgot to write "+ +", because that's what he always recommends and uses. Indeed, by now he implies it.

 

[Erel]

The very first thing that I install on a new computer is Notepad++. There is no valid reason for programmers to use Notepad

 

[LucaMs]

The very first thing that I install on a new computer is Notepad++.

What plugins did you install?
(new thread? ? Chit chat???)

 

[LucaMs]

What plugins did you install?
(new thread? ? Chit chat???)

Notepad ++ best plugins

(Also, tried to create B4X language: failed on start ? )

 

[Mahares]

The very first thing that I install on a new computer is Notepad++

The Notepad ++ team would love to get hold of your above comment. It can be the best endorsement for their product they would ever hope for. A strong statement coming from a powerhouse like Erel; that is all they need.