Android Question Unexpected random App vanishing

[GiovanniPolese]

Hi. I have an App that much probably is rather heavy to manage for Android. This App uses OpenGL2, reads and displays huge bitmaps and drawings over them. A very serious tools, which is rather successfully even with high level users. Clearly, when we have big data, we cannot manage them, with the miserable 500 mb allowed to B4A (while the tablet may have 8+8 Gb memory ). As a matter of fact the App displays a memory problem and this is out of question. But it happens, randomly, that the App "vanishes", even with data that, in other moments but same conditiona, were normally managed. Now a first question: (1) The App "vanishing" without any message in debug or release mode (btw I work with USB cable connected) is a symptom of Android unquestionable decision to kill the App or what else? It seems not a programming error, otherwise it should appear anyway. Then some other questions. I am struggling to try to optimize the code, still hoping to minimize this "vanishing" problem, but I don't know some details, related to B4A memory management. For example: (2) I always clear lists and similar stuff, not waiting GarbageCollector. Is this a good practice or it may generate the abovementioned problem, or it is irrelevant? (3) Using classes to encapsulate some parts of code may give any benefit to App behaviour (4) Making a lib of some parts of code may give benefits?
Note: Probably I am fighting against WindMills. Android has its good reasons for what it does, but equipping devices stimulating people to work on them, to arbitrarily kill their Apps, when they try to be serious tools.. is not that fair. Maybe I am simply paranoic and surely frustrated, forgive me and my ignorance. Thanks for any advice.

 

[emexes]

reads and displays huge bitmaps

Can you break the bitmaps down so that you only have in memory the section that is shown on the display? Like with Google Map / OpenStreetMap tiling, which (from memory) work with 256 x 256 pixel chunks,

The App "vanishing" without any message in debug or release mode

No message when it happens in debug mode? That's going to be fun to track down.

My first thought here is to find out whereabouts in your program it happens. Like, even if it is Android killing your program, presumably Android has a reason for doing that, and it would be after your program does something that Android doesn't like. In order to stop upsetting Android, first we have to find out what your program is doing that Android doesn't like or otherwise causes the problem.

If there is no message, then maybe it is running out of some resource, to the point that it doesn't even have enough of that resource available to let you know what the problem is. If the program can't send any diagnostic information after the problem happens, then we'll have to start sending diagnostic information before the problem happens, and then at least we'll know where things were ok, and that that the problem must be between the point where things were ok and the next diagnostic point that it didn't get to because it died (or was killed) before it got there.

I'd write a little Sub that leaves breadcrumbs of when the program passes through various points of the program, eg the start of an event handler, and just before the Returns from and the end of the event handler. That way, when the problem occurs, we'll know that it got into that Sub but never made it out again, and we can add more breadcrumbs inside that Sub to keep narrowing down which line the program vanished at. Like a binary search.

I'd either 1/ write the breadcrumbs to a file, and open and close the file for each breadcrumb, to make sure the breadcrumb is on disk and safe from being lost when the program vanishes. Or 2/ send the breadcrumbs via UDP to simple logging program running on your PC.

Now that I think about it, your breadcrumbs could include other information too, eg, the size of a bitmap that you're about to create, or the selection number of an item chosen from a list.

Once we've located - and fixed - the problem, then you can do a global search and replace to comment out all the calls to Breadcrumb() and the residual overhead of the diagnostic technique will be zero. Or you could leave the breadcrumb calls still in the program, but just put a Return at the top of the Breadcrumb() Sub to turn it into a dummy routine, and the only residual overhead is the calls and returns, without the file or network operations.

How does that sound? Would you go with breadcrumbs-to-file method, or the breadcrumbs-over-network method? Each way has advantages and disadvantages over the other. Eg:

File method:
- how to view the breadcrumb file after program vanishes
- how to not fill up the disk
Network method:
- only works for you, not for your users
- gotta write a receiving app (but then again: for the file method, you gotta write code to view the file, so... the disadvantages balance out)
- won't wear down your flash drive with excessive writes

 

[emexes]

On a related note, you've reminded me of a DOS program I worked on last century, which had a secret option in the date screen that enabled a user to toggle trace mode on and off. If trace mode was on, then the program would display a number on the screen that would tell us whereabouts the program was executing. If the program froze or crashed, then that magic number would tell us where to start looking in the program, like a breadcrumb. But under DOS, when things stop, then what's on the screen at that point remains visible, whereas under Android, then anything that was on the screen when your app vanishes, is lost. Hence the need to write the breadcrumbs out to a file or other computer.

That trace mode was brilliant. It was on the date screen because that was the first screen that came up when the program started, and also could be called up at any point in the program. If user entered the word "magic" instead of a date, then it toggled the trace mode off and on. Helped with customer relations too: they could see that something was being done about whatever the problem was, and it turned problem solving into a team effort of us (the customers and I) versus the problem, rather than the customer feeling alone versus the problem.

 

[emexes]

how to view the breadcrumb file after program vanishes

I think the solution here is that, when your program starts up, it renames the breadcrumb.txt file to breadcrumb.previous, and starts a new breadcrumb.txt file for the current session.

Then, either have
1/ a secret option in your program that shows you breadcrumb.previous in a text box or similar, or brings it up in an Android notepad app using an intent
2/ check breadcrumb.previous and, if it looks like the program only got halfway through doing something, then automatically do option 1

The trace numbers I spoke of: at the start of doing something, like entering a Sub, the trace number was xxxx[0-5]. And when the program finished doing something, like as it was exiting a Sub, the trace number was xxxx[6-9]. So if the last trace number ended in digit 0-6, then we knew that something had gone wrong before the operation was complete.

With benefit of hindsight, maybe nowadays I'd do it more like a stack dump. On the other hand, simple is good. Like, it's easy for a user to tell me that the magic number was 27331 when the problem happened. The more information that the user had to write down and tell me, the more opportunity there was for mistakes.

 

[emexes]

how to not fill up the disk

The solution to this is the make the breadcrumb log file a ring buffer of fixed-length lines (or fixed-length binary records, if you're willing to write corresponding log viewer code).

Then include a sequence number that gets incremented each time you call Breadcrumb() - this number can do double duty:
1/ tells you which slot in the ring buffer that the breadcrumb should be recorded to
2/ lets you identify which was the last breadcrumb recorded

You know, you've got me fired up now: I'll bang out some code so that it's magically ready for you when the sun rises.

edit: today didn't quite pan out as expected; I'm about to start now

 

[GiovanniPolese]

Hi. Thanks for your efforts to help, but the situation is more complex. There is no place where the problem happens. No way : I wrote that vanishing is random and means that there is not a fixed place where it happens. Of course I use tiled images, because I manage OrthoPhotos of several GigBytes. Tiles are several thousands, but this mechanism works well. No problems in it. Problems arise when reading dxf file and here begins the nightmare. I must read hundreds of thousands of lines, polylines, and sometimes thousands of hatches , texts and splines. Complex geometrical issues, expecially for splines and hatches. To render so many primitives, I must use OpenGL: otherwise rendering is very slow (normally in Windows it is so, and I guess here the same.. The reason for exispence of OpenGL and similar libraries is mainly speed, no uquestion about. The reason for speed is the use of GPU and not CPU). Therefore, to get a result reading Dxfs of more than one Gb, I must use tricks: no direct conversion from dxf to OpenGL. This latter only manages lines, for example, so forget about primitives like circles, arcs, spines and also texts, as we discussed in another thread. So, for example a spline with 10 control points and 14 knots (apparently innocent) may generate thousands of points. Then I calculate all the necessary primitives and write them on disk. At this point I feed the openGL buffers. The reason is that OpenGL works on its own buffers (normally in GPU and not in CPU). I cannot fill these buffers directly while reading the DXF, because the data written on DXF are structured in a way that it is impossible to send them directly: OpenGL uses contiguous arrays of lines, while on dxf I can meet first a line, then a text, than a spline (which must be calculated and only after added to the line buffer), etc.. BUT I did all this. Of course I have thousands of lines of code to do this, divided in modules and classes. Things work well ... sometimes ... Of course in such a mass of code one or many bugs maybe. In my rather long experience of doing mistakes, I am used to rely on debugging. And here the nightmare: reading a text file like the dxf, in debug mode is so slow that it may take hours for files of meaningful size some megabytes. Moreover, in debug mode, OpenGL doesn't show the image. Finally: App vanishes, not crashes... sometimes , not always. Conclusion: much probably B4A is not the right tool, but I suspect that these problems are not strictly related to B4A. Sorry f I will not continue the discussion with you. I thinks useless for both of us. I posted here the question but I guess it has no answer, besides your kind observations. What I observe that any trial is useless. For example using global variables instead of local ones, when they are huge lists, to avoid engaging the heap seems useless. But How Android uses Stack and Heap? Is it like Windows or not? Etc. The mitical Garbage collector works well? Probably my questions have no answer. I appreciated B4A exactly to avoid to study Android. Now I am in troubles. thanks for .reading and your kind observations. As I said, now I have to find out an exit way and no time for long discussions. Thanks again.

 

[emexes]

Longest paragraph ever.

But is no problem I copy it to notepad and break it down sentence-by-sentence, which is about the largest chunk my old brain can handle without crashing.

Also, I just finished that breadcrumb project and it is pretty good, even if I do say so myself I will post it soon.

 

[emexes]

Righto. That's a big job. Like, effectively you are recreating a quarter of AutoCad. That's a lot for one programmer.

I am imagining you are displaying construction or engineer DXF files over top of tiled map or aerial imagery.

Is the DXF-to-OpenGL-commands conversion done on the laptop?

Is it done whilst the user is waiting, whilst the app starts up? Or is it done beforehand, possibly by a different app?

Dumb questions:

Can the OpenGL pre-converted data be tiled, like the aerial imagery is tiled, so that your app and OpenGL only need to deal with the stuff that is visible, and not with everything else too that is currently not visible?

Can the DXF overlay be pre-rendered to an image format (with lots of transparent area) rather than to OpenGL data?

Presumably it wouldn't be any larger than the aerial imagery that you're already handling.

 

[GiovanniPolese]

Hi. The problem seems related to OpenGL. As a matter of fact, the OpenGL renderer is attached to a Panel and receives redrawing comands that are not expected. This is maybe ininfluent when the "scene" displays fixed data. But if in the meanwhile of a rendering , the OpenGL buffers are updated, because, for example, you are adding some data, the impact of a rendering call is probably devastating. Of course it seems obvious, but it is not that simple because I didn't call any rendering while updatring the data. For example, being the OpenGL renderer attached to a Panel, simply touching the panel, while reading new data, closes the App, irrespective to having managed the click and touch events of the involved panel. Of course for now this is the maximum that I may guess, from my trials.

 

[emexes]

reading Dxfs of more than one Gb, I must use tricks

reading a text file like the dxf, in debug mode is so slow that it may take hours for files of meaningful size some megabytes.

edit: the movie "The Blind Side" is on tv, and I'm doing this DXF thinking on the side and for fun, so don't panic

Hey I've been mulling over why it would take so long to parse the DXF, and practicing on the largest sample file I have (floorplan.dxf = 1.1 MB). What I've noticed is that many of the fields are of fixed length, according to tag value (or whatever that number is that identifies the following line). Which has gotta be quicker to read.

Eg the tag values seem to all be three characters long, from " 0" to "999", and so instead of reading it byte-by-byte into a string, you could read 5 bytes (the 3 digits and cr and lf), and then I suspect it might be quicker to convert that 3 digit number to an Int with three Bit.Ands and two multiplications and two additions rather than creating a string (which I assume involves a memory allocation) and using the more-general string-to-int conversion. Yeah there are some tag values that are 4 characters long rather than 3, but that doesn't take much extra to work around.

Another idea is to pre-parse the DXF files, probably including pre-converting the tag values and numeric fields into their binary form. I'd expect those DX-binary files to be about half the size of the original DXF eg a coordinate value like 347.36713176115635 CR LF would be 8 bytes when stored as a Double, rather than 20 bytes.

      Num   = Field Length =
Tag#  Tags   Min   Max Fixed  Field Contains These Characters
====  ====  ====  ====  ====  ===============================
   0  4447     3    19        "ABCDEFGHIJKLMNOPQRSTUVWXY_"
   1   642     0   181        " "%&'()*,-./0123456789:;@ABCDEFGHIJKLMNOPQRSTUVWXY\_abcdefghiklmnopqrstuvwxy{|}░"
   2   977     0    39        " #$()*,-.0123456789ABCDEFGHIJKLMNOPRSTUVWXY_abcdefghijklmnoprstuvwxyz{}"
   3   520     0   128        " "$'()*,-./0123456789:ABCDEFGHIKLMNOPRSTUVWXY\_abcdefghijklmnoprstuvwxy|"
   4    23     0    34        ""()*.012568AGINPSV_cehilnprsux"
   5  4318     1    16        "*,0123456789ABCDEFINPRST"
   6    93     0    33        "$-028ABCDEHILNORSTXYacdefhiklnoprstuvxy"
   7     7     5    14        ".NSabcdehmnorst"
   8  3979     1    39        " $-018ABCDEFGHIJLMNOPRSTUVWXadefhilnoprstvwx"
   9   213     5    20        "$12345ABCDEFGHIJKLMNOPQRSTUVWXYZ"
  10  4996     3    19        "-.0123456789"
  11  2351     3    19        "-.0123456789"
  12    69     3    18        "-.0123456789"
  13   132     3    18        "-.0123456789"
  14   132     3    18        "-.0123456789"
  15     6     3    17        ".023456789"
  16     6     3     3     3  ".01"
  17     6     3    18        "-.013456789"
  20  4996     3    19        "-.0123456789"
  21  2351     3    19        "-.0123456789"
  22    69     3    18        "-.0123456789"
  23   132     3    18        "-.0123456789"
  24   132     3    18        "-.0123456789"
  25     6     3    17        ".01456789"
  26     6     3     3     3  ".0"
  27     6     3    18        "-.012456789"
  30  3449     3     3     3  ".0"
  31  2348     3     3     3  ".0"
  32    65     3     3     3  ".0"
  33   128     3     3     3  ".0"
  34   128     3     3     3  ".0"
  35     2     3     3     3  ".0"
  36     6     3     3     3  ".01"
  37     6     3     3     3  ".0"
  40   573     3    18        ".0123456789"
  41   638     3    18        "-.0123456789"
  42   421     3    17        "-.0123456789"
  43   730     3    18        "-.0123456789"
  44   196     3    18        "-.0123456789"
  45    35     3    19        "-.0123456789"
  46    33     3    19        "-.012345689"
  47    38     3    18        ".0123456789"
  48     9     3    18        "-.0123456789"
  49    97     3    18        "-.0123456789"
  50   445     3    17        ".0123456789"
  51    85     3    17        ".0123456789"
  52    27     3     4        ".09"
  53    26     4     5        ".01345"
  62  1321     6     6     6  " 0123456789"
  63     3     6     6     6  " 0"
  64     3     6     6     6  " 0"
  65     4     6     6     6  " 0"
  66     7     6     6     6  " 01"
  67     8     6     6     6  " 01"
  68     6     6     6     6  " 0123"
  69     6     6     6     6  " 0123"
  70   951     6     6     6  " -0123456789"
  71   314     6     6     6  " 01256"
  72   388     6     6     6  " 01256"
  73   407     6     6     6  " 01246"
  74   102     6     6     6  " 012345"
  75    32     6     6     6  " 0136"
  76    45     6     6     6  " 013"
  77    36     6     6     6  " 012"
  78    46     6     6     6  " 01238"
  79    38     6     6     6  " 02"
  90   566     9    11        " -0123456789"
  91   157     9    11        " -0123456789"
  92    74     9     9     9  " -01235678"
  93    74     9    11        " -0145689"
  94    10     9    11        " -01245689"
  95    21     9     9     9  " 012345689"
  97    46     9     9     9  " 0"
  98    26     9     9     9  " 124"
 100  9046     7    25        "ABCDEFHILMOPRSTVWXabcdefghijklmnoprstuwxy"
 102   223     1    33        "027ABCDEFGHIJKLMNOPRSTUVXY_{}"
 105    16     2     4        "012345789BCEF"
 110     4     3     3     3  ".0"
 111     4     3     3     3  ".01"
 112     4     3     3     3  ".0"
 120     4     3     3     3  ".0"
 121     4     3     3     3  ".0"
 122     4     3     3     3  ".01"
 130     4     3     3     3  ".0"
 131     4     3     3     3  ".0"
 132     4     3     3     3  ".0"
 140    44     3    17        ".0123456789"
 141    40     3    17        ".0123456789"
 142    23     3    18        ".0124568"
 143    12     3    17        ".012345679"
 144    21     3    18        ".01234568"
 146    16     3    18        ".012567"
 147    18     3    17        ".012356789"
 148     2     3    17        ".0123456789"
 149     2     3    17        ".01345678"
 170    12     6     6     6  " 0125"
 171     6     6     6     6  " 1"
 172    11     6     6     6  " 012"
 173     4     6     6     6  " 1"
 174     9     6     6     6  " 1"
 175     3     6     6     6  " 1"
 176    16     6     6     6  " 0256"
 177    13     6     6     6  " 0256"
 178    16     6     6     6  " 124"
 179     3     6     6     6  " 235"
 210    39     3     3     3  ".0"
 213     4     3    17        ".023456789"
 220    39     3     3     3  ".0"
 223     4     4    18        "-.012345678"
 230    39     3     4        "-.01"
 233     4     3     3     3  ".0"
 270     3     6     6     6  " 2"
 271    14     6     6     6  " 1235"
 272    14     6     6     6  " 1235"
 273     6     6     6     6  " 34"
 274    13     6     6     6  " -24"
 275     3     6     6     6  " -2"
 276     4     6     6     6  " -12"
 277     8     6     6     6  " -24"
 278     3     6     6     6  " -2"
 279     9     6     6     6  " -2"
 280   139     6     6     6  " 0127"
 281   132     6     6     6  " 01"
 283    10     6     6     6  " 0"
 284    16     6     6     6  " 138"
 285     9     6     6     6  " 13"
 286     9     6     6     6  " 13"
 287     3     6     6     6  " 1"
 288     5     6     6     6  " 1"
 289     5     6     6     6  " 01"
 290    36     6     6     6  " 01"
 291     3     6     6     6  " 1"
 292     3     6     6     6  " 0"
 293     3     6     6     6  " 1"
 294     3     6     6     6  " 1"
 295     3     6     6     6  " 01"
 296     3     6     6     6  " 0"
 297     3     6     6     6  " 01"
 300    32     0    13        "01234568:ACDEFHILMNORSTY_"
 301     3     6     6     6  "AGIMNR"
 302    18    10    10    10  "ADFGIMORT"
 309    30    13    17        "ABCDEFGILMNORSTY_"
 310   408   126   254        "0123456789ABCDEF"
 330  5798     1     4        "0123456789ABCDEF"
 331   287     3     4        "0123456789ABCDEF"
 340   194     1     4        "012345789BCDEF"
 341     9     1     3        "035BEF"
 342    10     2     3        "12378"
 343     4     1     4        "016EF"
 350    77     1     4        "0123456789ABCDEF"
 360    41     3     4        "0123456789ABCDEF"
 370   790     6     6     6  " -0123"
 371    10     6     6     6  " -1"
 372    10     6     6     6  " -1"
 380     1     6     6     6  " 0"
 390    24     1     1     1  "F"
1000    72     0    46        " (),.3ABCDEFGHIJLMNOPRSTVWXY_abcdehilmnoprstuvwx"
1001    49     4    22        "-135ABCDEFGIJKLMNOPRSTUVXY_acdenrty"
1002    74     1     1     1  "{}"
1011     6    15    19        "-.0123456789"
1021     6    17    19        "-.0123456789"
1031     6     3     3     3  ".0"
1040    26     3    17        ".0123456789"
1070    86     6     6     6  " -012346789"
1071     5     9     9     9  " 034"

 

[GiovanniPolese]

Hi. Actually I don't understand what you are doing. Looking for fun with dxf? There are other funny things in the world....

 

[emexes]

I don't understand what you are doing. Looking for fun with dxf?

Yes.

Puzzles that I am forced to solve, I don't enjoy.

Puzzles that intrigue me, and that have a practical purpose, I do enjoy, in my spare time. I have no hobbies or sports and I am unable to watch television without also doing a background activity to fill in the boring bits.

Lol at the moment I'm at my parents' home and they watch television every evening, so I'm sucked in by that. But at home, I almost never watch it. Last time was New Year's Eve ie over a month ago. Before that... I can't remember.

Anyway, the puzzle here is why does it take hours to parse a ≅ 1 GB file? I can see that character-by-character processing will drag things down. Also the constant allocation and garbage collection of memory, and conversion from string to binary numerics etc.

 

[emexes]

I just did some timing on my Android phone.

Whether reading by InputStream or by RandomAccessFile is much-of-a-muchness, so I'm leaning towards RandomAccessFile because it allows you to do stuff like:

1/ check at the end of the DXF file that it is indeed 0 CRLF EOF CRLF (so that we know we don't have to worry about checking for physical EOF)
1b/ and also know for sure whether the line terminator is CR or LF or CR LF - bonus!
1c/ and you could even use it as an indicator of the exact format of the tag (group?) numbers, like: are they padded with leading spaces?

2/ back up a bit if we read too far ahead, eg real numbers are up to 19 characters long, so what we could do is prospectively read 20 bytes, and then parse it directly in the Byte Array, and only advance the file pointer by the correct number of bytes, not by the full 20 bytes.

Anyway, the timing was that each read operation takes (approx) 2500 ns + 11 ns per byte.

If the file is read byte-by-byte then it does 0.4 MB/second = 40 minutes just to read the bytes ie no parsing, no string or numeric conversion.

If the file is read in 100 byte blocks then it does 30 MB/second = 33 seconds to read the bytes

If the file is read in 1000 byte blocks then it does 87 MB/second = 12 seconds to read the bytes

I know there is still parsing to be done, but character-type operations are a heck of a lot quicker on Byte Arrays than on Strings.

Btw those were in Release Mode (not Debug Mode). Let me check Debug Mode.

W T F ?

There is something odd going on. It is like there is a watchdog timer that's killing the button-press event handler if it runs for too long without showing sign of life.

Something to mull about over dinner.