Android Question Max items of a List - list alternatives?

[kostefar]

Dear All,

I´m using b4x to do some calculations involving setting up a list with around 6 mio. items, which seems to be more than a normal list can hold.
The data is loaded line by line from a txt file, where 5 elements are put into a map, which is then added to the list.
This is prolly something I´d be better off at doing in vb6, as it´s not app related at all, and vb6 won´t suffer from the kind of memory limitations that we have in android, plus it´d prolly be faster too.
However, it´s been years since I touched vb6, so I first thought I´d see if there´s a way for me to do this in b4a.
After 1 mio lines, I get java.lang.OutOfMemoryError: OutOfMemoryError thrown while trying to throw OutOfMemoryError; no stack trace available.
I already have

SetApplicationAttribute(android:largeHeap,"true")

set in the manifest.

Alternatively I could perform the calculations while loading the lines from file, not adding them to a list, but I assume it´ll be superslow then as they´re not loaded into memory first.

 

[kostefar]

Your changing the goal posts on me! Yes, then it would be more efficient. But that means squat if the memory capacity is exceeded. You then either find a method of dealing with the data in chunks (maybe write a prepped version of the data back to a file) or you find a library that can deal with stuff like that.

Didn´t quite understand the part about "goal posts"? I guess the limitations I´m running into are related to genymotions memory handling .. 8GB should be enough to handle a 160 mb file with 6 mio-ish lines. So yeah, I could split the file up into multiple ones and work with it that way. Maybe having an array of lists with the 1 mio entries pr list would actually work.

 

[OliverA]

Close deal whenever price goes x % above lowest price.

Does that mean you constantly get new data? If so how?

 

[stevel05]

Another approach could be to create an SQLite DB and load that to the device or emulator. Search Google for text file convert to SQLite, there are a few available converters, you wouldn't have to write the conversion (or read the text file on the device). Or you could write your own in B4j.

 

[kostefar]

How large is the file? Are the lines of consistent length lines or random length lines?

160 mb, lines look like this:


11132017;15:56:29;28.9734;28.97;28.98
11132017;15:56:39;28.98;28.98;28.99
11132017;15:56:41;28.98;28.98;29

 

[OliverA]

8GB should be enough

Which phone has 8GB?

 

[kostefar]

Does that mean you constantly get new data? If so how?

Maybe look at the example lines above.

 

[kostefar]

Which phone has 8GB?

None that I´m aware of, this is in genymotion where you can set the base mem to what you want.

 

[kostefar]

Another approach could be to create an SQLite DB and load that to the device or emulator. Search Google for text file convert to SQLite, there are a few available converters, you wouldn't have to write the conversion (or read the text file on the device). Or you could write your own in B4j.

Thanks steve, I think for now it´s the easiest to just split the file into 7 or 8 parts, and load them into an array of lists.

 

[OliverA]

Maybe look at the example lines above.

But the file is static? Are you basing you "close the deal" on this static file or is the file updated in the background?

 

[OliverA]

Have you tried File.ReadList to see if you can load the file in one swoop into memory?

 

[kostefar]

But the file is static? Are you basing you "close the deal" on this static file or is the file updated in the background?

The file remains the same, but the result of the sim, where parameters with various values are used against the data - hence the multiple runs - are saved to another file.

 

[kostefar]

The file remains the same, but the result of the sim, where parameters with various values are used against the data - hence the multiple runs - are saved to another file.

Good idea. I got same error as before, but tried in debug mode instead that gives me:

java.lang.OutOfMemoryError: Failed to allocate a 104 byte allocation with 4901600 free bytes and 4MB until OOM; failed due to fragmentation (required continguous free 4096 bytes for a new buffer where largest contiguous free 0 bytes)

 

[OliverA]

How about TextReader's ReadAll? Can always use regular expression(s) to split it all up and load it into lists/maps.

 

[kostefar]

How about TextReader's ReadAll? Can always use regular expression(s) to split it all up and load it into lists/maps.

Same memory error

 

[KMatle]

Guys, please. An Adroid phone isn't made to handle 6 mio. rows/lines. If you really need that data huge amount of data on the phone, put it in a SQlite DB (assuming it will be 500 MB then with some indices).

 

[DonManfred]

Using a Database is most probably the best way. You also can probably do all the calculation-stuff inside a SQL-Query and just get the result from the Database...

 

[OliverA]

Guys, please. An Adroid phone isn't made to handle 6 mio. rows/lines. If you really need that data huge amount of data on the phone, put it in a SQlite DB (assuming it will be 500 MB then with some indices).

Understood. Still nice to see what the limits are. Plus, considering that an Android phone is so much more powerful than computers were back in the day, it's interesting to see that you can still hit a brick wall. You sort of get spoiled that you can carry all this computing power with a touch screen attached in your pocket. Also, even though this may be more appropriate for "desktop" or "server", sometimes a phone is the only device available/accessible to an individual and therefore it's still interesting to figure out how to circumvent any remaining limitations of the platform.

Using a Database is most probably the best way. You also can probably do all the calculation-stuff inside a SQL-Query and just get the result from the Database...

Several of us already hinted at that. Now were just seeing if it can be done the old fashion way first (brute force).

Same memory error

Going by this (https://www.b4x.com/android/forum/threads/outofmemoryerror-width-object-serializer.76702/)(and other links on the net that I'm too lazy to list ) splitting up may be the last alternative before moving on to something else.

 

[KMatle]

Understood. Still nice to see what the limits are. Plus, considering that an Android phone is so much more powerful than computers were back in the day

I can understand what you mean but the same device can do magic things if you use a SQLite DB.

What I would do:

1. Use B4J (I've heard this is a good tool) to read the file and store it into a SQlite DB)
2. Use the Batch method to insert the rows
3. After that it is just a file copy to your device (e.g. via Google Drive or so). Do NOT include such a big file in your assets.

For test reasons I've put 10 mio. rows in a Sqlite DB which took just 30-60 secs or so (B4J, WIN 10 64 Bit, 12 GB RAM, SSD). The size (depending on the data) was about 400 MB which isn't that big.

 

[kostefar]

Ok peepz, I thank you for all the suggestions and knowledge! Never played with Sqlite db in b4x before, so I´m gonna read up on that. At least I know that it works without memory issues, as it was already verified by KMatie.

 

[kostefar]

Ok, one last question: I thought that setting MaxRamForDex to 1024 or higher could help here, but I can´t start anything with above 512, although I have 5 GB of 16 physical GB left. Could there be differences in java versions causing this?
The one I´m using is jdk1.8.0_121.

Same issue with b4j, which I just installed.