Android Question Max items of a List - list alternatives?

kostefar

Active Member
Licensed User
Longtime User
Dear All,

I´m using b4x to do some calculations involving setting up a list with around 6 mio. items, which seems to be more than a normal list can hold.
The data is loaded line by line from a txt file, where 5 elements are put into a map, which is then added to the list.
This is prolly something I´d be better off at doing in vb6, as it´s not app related at all, and vb6 won´t suffer from the kind of memory limitations that we have in android, plus it´d prolly be faster too.
However, it´s been years since I touched vb6, so I first thought I´d see if there´s a way for me to do this in b4a.
After 1 mio lines, I get java.lang.OutOfMemoryError: OutOfMemoryError thrown while trying to throw OutOfMemoryError; no stack trace available.
I already have
B4X:
SetApplicationAttribute(android:largeHeap,"true")
set in the manifest.

Alternatively I could perform the calculations while loading the lines from file, not adding them to a list, but I assume it´ll be superslow then as they´re not loaded into memory first.
 

kostefar

Active Member
Licensed User
Longtime User

Didn´t quite understand the part about "goal posts"? I guess the limitations I´m running into are related to genymotions memory handling .. 8GB should be enough to handle a 160 mb file with 6 mio-ish lines. So yeah, I could split the file up into multiple ones and work with it that way. Maybe having an array of lists with the 1 mio entries pr list would actually work.
 
Upvote 0

OliverA

Expert
Licensed User
Longtime User
Upvote 0

stevel05

Expert
Licensed User
Longtime User
Another approach could be to create an SQLite DB and load that to the device or emulator. Search Google for text file convert to SQLite, there are a few available converters, you wouldn't have to write the conversion (or read the text file on the device). Or you could write your own in B4j.
 
Upvote 0

kostefar

Active Member
Licensed User
Longtime User
How large is the file? Are the lines of consistent length lines or random length lines?

160 mb, lines look like this:


11132017;15:56:29;28.9734;28.97;28.98
11132017;15:56:39;28.98;28.98;28.99
11132017;15:56:41;28.98;28.98;29
 
Upvote 0

OliverA

Expert
Licensed User
Longtime User
Upvote 0

kostefar

Active Member
Licensed User
Longtime User

Thanks steve, I think for now it´s the easiest to just split the file into 7 or 8 parts, and load them into an array of lists.
 
Upvote 0

OliverA

Expert
Licensed User
Longtime User
Maybe look at the example lines above.
But the file is static? Are you basing you "close the deal" on this static file or is the file updated in the background?
 
Upvote 0

OliverA

Expert
Licensed User
Longtime User
Have you tried File.ReadList to see if you can load the file in one swoop into memory?
 
Upvote 0

kostefar

Active Member
Licensed User
Longtime User
But the file is static? Are you basing you "close the deal" on this static file or is the file updated in the background?

The file remains the same, but the result of the sim, where parameters with various values are used against the data - hence the multiple runs - are saved to another file.
 
Upvote 0

kostefar

Active Member
Licensed User
Longtime User
The file remains the same, but the result of the sim, where parameters with various values are used against the data - hence the multiple runs - are saved to another file.

Good idea. I got same error as before, but tried in debug mode instead that gives me:

java.lang.OutOfMemoryError: Failed to allocate a 104 byte allocation with 4901600 free bytes and 4MB until OOM; failed due to fragmentation (required continguous free 4096 bytes for a new buffer where largest contiguous free 0 bytes)
 
Upvote 0

OliverA

Expert
Licensed User
Longtime User
How about TextReader's ReadAll? Can always use regular expression(s) to split it all up and load it into lists/maps.
 
Upvote 0

KMatle

Expert
Licensed User
Longtime User
Upvote 0

OliverA

Expert
Licensed User
Longtime User
Guys, please. An Adroid phone isn't made to handle 6 mio. rows/lines. If you really need that data huge amount of data on the phone, put it in a SQlite DB (assuming it will be 500 MB then with some indices).
Understood. Still nice to see what the limits are. Plus, considering that an Android phone is so much more powerful than computers were back in the day, it's interesting to see that you can still hit a brick wall. You sort of get spoiled that you can carry all this computing power with a touch screen attached in your pocket. Also, even though this may be more appropriate for "desktop" or "server", sometimes a phone is the only device available/accessible to an individual and therefore it's still interesting to figure out how to circumvent any remaining limitations of the platform.
Using a Database is most probably the best way. You also can probably do all the calculation-stuff inside a SQL-Query and just get the result from the Database...
Several of us already hinted at that. Now were just seeing if it can be done the old fashion way first (brute force).
Same memory error
Going by this (https://www.b4x.com/android/forum/threads/outofmemoryerror-width-object-serializer.76702/)(and other links on the net that I'm too lazy to list ) splitting up may be the last alternative before moving on to something else.
 
Upvote 0

KMatle

Expert
Licensed User
Longtime User
Understood. Still nice to see what the limits are. Plus, considering that an Android phone is so much more powerful than computers were back in the day

I can understand what you mean but the same device can do magic things if you use a SQLite DB.

What I would do:

1. Use B4J (I've heard this is a good tool) to read the file and store it into a SQlite DB)
2. Use the Batch method to insert the rows
3. After that it is just a file copy to your device (e.g. via Google Drive or so). Do NOT include such a big file in your assets.

For test reasons I've put 10 mio. rows in a Sqlite DB which took just 30-60 secs or so (B4J, WIN 10 64 Bit, 12 GB RAM, SSD). The size (depending on the data) was about 400 MB which isn't that big.
 
Upvote 0

kostefar

Active Member
Licensed User
Longtime User
Ok peepz, I thank you for all the suggestions and knowledge! Never played with Sqlite db in b4x before, so I´m gonna read up on that. At least I know that it works without memory issues, as it was already verified by KMatie.
 
Upvote 0

kostefar

Active Member
Licensed User
Longtime User
Ok, one last question: I thought that setting MaxRamForDex to 1024 or higher could help here, but I can´t start anything with above 512, although I have 5 GB of 16 physical GB left. Could there be differences in java versions causing this?
The one I´m using is jdk1.8.0_121.

Same issue with b4j, which I just installed.
 
Upvote 0
Cookies are required to use this site. You must accept them to continue using the site. Learn more…