B4R Question Problem with RAM allocated for strings even before Sub is called

[BertI]

I had been having problems due to stack overflow (see https://www.b4x.com/android/forum/t...-when-using-ssl-connection.114046/post-712328)

On further investigation it appeared that my code was using quite a lot of RAM before anything was actually running. In other words, reading AvailableRAM even immediately upon AppStart showed it was already down by a massive 16K! Ok so I could account for some of that being due to my allocating a larger buffer stack (4K) but that still left 12K. The Global variables only accounted for a few hundred bytes, so where was it all going before the code even started doing anything? Somewhat onerously after some trial and error, adding bits of the original code back, compiling, then checking RAM at Appstart I eventually discovered that the main offender was an area where I was using lots of Astream.writes followed by explicit strings, e.g Astream.write("This eats up RAM before you even use this Sub"). I had wrongly assumed that such constructs would be stored in the code flash until called upon by the running code, but evidently it is not so?

So question is - how might I better deal with this type of thing? Perhaps answering my own question, I'm guessing I need to understand and use the rESP8266FileSystem which is probably more sensible all round, but how does one get all the strings into the file system in the first place? Does it mean that I can't do this from within the code/compiler itself because I'd end up with the same RAM allocation issue anyway?

 

[Erel]

The const chars (strings) are handled by ESP8266 compiler and linker.

If you are using many strings then you should create another program that writes them to the flash file system.

 

[tigrot]

https://www.b4x.com/android/forum/threads/using-progmem-to-save-memory.71114/
I have a sample,
https://www.b4x.com/android/forum/threads/various-table-operation-and-lookup-in-progmem.88640/

 

[BertI]

Thank you for your responses. The problem I have is that the strings are mainly HTML constructs sometimes with variables within sections of the string e.g to report some value or response via a web page. So it's not simply a matter of just pulling a few contiguous blocks of characters from the flash (at least as far as I'm seeing it at the moment). I did notice the posts about progmem but I thought these were just suited to arduino rather than ESP8266, which is the platform I'm working on. Anyhow this also looked quite difficult to work with when you have a whole load of strings to convert first into comma delimited bytes representation. As you might imagine, if you are writing HTML code it is easier to see it all there in front of you whilst you are experimenting (and believe me I have just started on the HTML journey so a lot of experimentation is necessary ). So I may end up trying a mix of flash file for larger blocks and perhaps leave other stuff as explicit strings in the code. I did also see the file transfer example so may have to delve into B4J to set something up to help me transfer the strings but as I say just seems like it's going to make development work quite arduous. Still, I guess it might instill more discipline..

I can't really understand why the compiler puts these strings straight into RAM. For a moment I wondered whether it might even do this with the string content of Log statements but thankfully it seems not. Also I wonder if there are any other SSL libraries that might be more RAM efficient?

 

[Erel]

Also I wonder if there are any other SSL libraries that might be more RAM efficient?

Not that I know of.

 

[tigrot]

You can build the HTML string on the fly, dividing it in substrings, send it using basic HTTP protocol one piece at a time. It's like climbing a mirror, but if you need...

 

[BertI]

Thanks to Erel's example I have successfully repurposed the B4R/B4J file transfer code (https://www.b4x.com/android/forum/threads/resp8266filesystem-spiffs.76487/post-485079) to install some of the string content into Flash, but the problem I have now is that initializing the file system itself eats up about 2K of RAM. Is there any way to 'uninitilize' this after use? I did try doing the initilization within a sub that did the file processing, but RAM was still ~2K down after return from the sub. I seem to be digging myself deeper into the hole - hope it'll end up being a U shaped hole, or rather J shaped

 

[BertI]

Sorry tigrot, I only saw your response after my post above. Not sure what you mean? At present I am just using a series of Astream.write("HTML syntax stuff") type of statements to build up a web page. Trouble is the textual content in each such statement appears to get loaded into RAM right at the start of program execution. In other words not just temporarily loaded into RAM when the Astream.write containing subs are called. So there seems to be nowhere to 'hide' the string content from RAM. Like I say above I got caught out by this since I had previously assumed that such strings would simply be stored as part of the code and then copied from the flash when needed by the Astream.write process.

I'd certainly prefer it if there was a way to avoid having to put the strings into the file system because I've now also realised that I will then also have to face the issue of how to OTA update the file system content which will have to be done separate to the code itself since they are treated differently (a firmware update even via the IDE does not change the content of the file system nor the EEprom designated segment)

 

[tigrot]

You are acting exactly in the way I mean. Sorry it doesn't help.

 

[BertI]

Not at all, it's good that people respond. I myself am just finding out by trial and error that the RAM allocation must be done in a non straightforward manner. It looks in some cases like the compiler must be trying to optimise because in some of my experimentation I found that sometimes adding one or two further strings didn't make any difference to RAM usage and conversely when I changed some repeated content statements with a call to a single sub containing those statements it didn't reduce RAM usage. I think this is heading towards a different thread though.

 

[thetahsk]

Not at all, it's good that people respond. I myself am just finding out by trial and error that the RAM allocation must be done in a non straightforward manner. It looks in some cases like the compiler must be trying to optimise because in some of my experimentation I found that sometimes adding one or two further strings didn't make any difference to RAM usage and conversely when I changed some repeated content statements with a call to a single sub containing those statements it didn't reduce RAM usage. I think this is heading towards a different thread though.

Post a "real world" example, so we can check and examine your problems, also we need Arduino Version, ESP vers., B4R Version ....

 

[BertI]

The easy part is the versions: Arduino 1.8.9, ESP is an ESP-12F or 12S, B4R 2.8. I know I should probably update B4R and Arduino, but wanted to avoid too many changes in one go and didn't think these had an impact on the memory allocation issues. I also found this whilst looking around: https://esp8266life.wordpress.com/2019/01/13/memory-memory-always-memory/ which perhaps goes some way towards explaining the variation in memory usage and it's causes. There is a link within this: https://learn.adafruit.com/memories-of-an-arduino/optimizing-sram which may also be of interest. In particular the mention of the F() function which sounds like a way to force strings to only be held in program memory, but it's beyond my present knowledge as to whether or how this might be used in B4R for ESP8266.

As for example code a bit more tricky as I'd have to extricate something meaningful out of my somewhat convoluted program, but I'll try and construct something

 

[BertI]

Ok so here is some example code which illustrates the amount of RAM taken up when taking up an SSL connection. You will obviously have to modify the router credentials to suit. You can test the effect of the Astream.write strings by block commenting out the content of the Astream_NewData and the Header subs and then recompiling and running. It is not a great deal in this case (320 bytes) but soon adds up even if you implement some relatively simple web page based user interfaces (though I'm definitely not the most efficient at this right now). Also in this example I've included the effect of initializing the filesystem which, as I mentioned before, seems not to be reversible. However as you can see the worst offender is the SSL related stuff. Even just introducing the WiFiSSLSocket object into Globals sucks up around 7.6K (you can test by commenting out ssl related stuff), and then the actual connection uses around 21K, though this is temporary I believe. (Note that this code is not actually usable for serving up a web page)

#Region Project Attributes
    #AutoFlushLogs: True
    #CheckArrayBounds: True
    #StackBufferSize: 300
#End Region

Sub Process_Globals
    'These global variables will be declared once when the application starts.
    'Public variables can be accessed from all modules.
    Public Serial1 As Serial
    Private astream As AsyncStreams
    Private wifi As ESP8266WiFi
    Private server As WiFiServerSocket
    Private fs As ESP8266FileSystem
    Private sslsocket As WiFiSSLSocket

End Sub

Private Sub AppStart
    Serial1.Initialize(115200)
    Log("AppStart")
    Log("Available RAM at App start: ",AvailableRAM)
   
    Log("FS Initialize result: ", fs.Initialize())
    Log("Available RAM after fs: ",AvailableRAM)
   
    If wifi.Connect2("xyz","123456789") = False Then
        Log("Error connecting to network")
        Return
    Else
        Log("Connected to network")
    End If
    server.Initialize(80, "server_NewConnection")
    server.Listen
    Log("Available RAM after server init: ",AvailableRAM)
   
    Dim st As Stream = Null
    sslsocket.Close
    If sslsocket.ConnectHost("www.google.com",443) Then
        Log("Connected to google")
        st = sslsocket.Stream
    Else
        Log("Failed to connect to Google")
    End If
    Log("Available RAM after ssl connect: ",AvailableRAM)
End Sub

Sub Server_NewConnection (NewSocket As WiFiSocket)
    astream.Initialize(NewSocket.Stream, "astream_NewData", "astream_Error")
End Sub

Sub Astream_NewData (Buffer() As Byte)
    Header
    astream.Write("Hello World, the quick brown fox jumped over the lazy dog").Write(CRLF)
    astream.Write("</body>").Write(CRLF)
    astream.Write("</html>").Write(CRLF)
    CallSubPlus("CloseConnection",200,0)
End Sub

Private Sub Header()
    astream.Write("HTTP/1.1 200").Write(CRLF)
    astream.Write("Content-Type: text/html").Write(CRLF).Write(CRLF)
    astream.Write("<!DOCTYPE html>").Write(CRLF)
    astream.Write("<html>").Write(CRLF)
    astream.Write("<head>").Write(CRLF)
    astream.Write("<style>").Write(CRLF)
    astream.Write("body {margin-top: 12%; background-color:lightgray; font-family:Helvetica; Font-size:350%; }").Write(CRLF)
    astream.Write("input[type=text] {font-family:Helvetica; Font-size:120%}").Write(CRLF)
    astream.Write("</style>").Write(CRLF)
    astream.Write("</head>").Write(CRLF)
    astream.Write("<body>").Write(CRLF)
End Sub

Private Sub CloseConnection(u As Byte)
    server.Socket.Close
End Sub

Sub Astream_Error
    server.Listen
End Sub

Typical log output for above:
Available RAM at App start: 44128
FS Initialize result: 1
Available RAM after fs: 41952
Connected to network
Available RAM after server init: 41168
Connected to google
Available RAM after ssl connect: 20080

Log output when Astream_NewData and Header sub content is commented out:
Available RAM at App start: 44448
FS Initialize result: 1
Available RAM after fs: 42272
Connected to network
Available RAM after server init: 41488
Connected to google
Available RAM after ssl connect: 20400

 

[BertI]

Well a mixture of using SPIFFS and some bits of pruning eventually got me there. For anyone else struggling with such memory problems, some particular areas of savings included the obvious - such as reducing the stack buffer allocation, reducing the response cache array size in the HttpJob module (the default was 5000 bytes for example), but even when I appeared to have ~23K before heading for the SSL Job I was still having problems. The final thing I tried was to change to the Basic SSL setting in the board manager options. This appeared to release enough further RAM (possibly 2-3K) to give enough headroom. I was able to successfully connect to an https aws server but don't know if using this setting will encounter problems with other SSL connections. I've also yet to find out if I encounter situations where the snipping of buffers and cache was a bit too severe, but anyhow at least it's going for now. My overall feeling is still that some way of getting the compiler to keep strings in flash rather than in static RAM would be a very useful function in the context of these devices. I have read further about the '"F()" macro which appears to be a way to achieve it in C and on the arduino but I don't have enough understanding at present to even determine if this is nonsensical. So I shall transfer to a wish thread for further comments.

 

[miker2069]

Well a mixture of using SPIFFS and some bits of pruning eventually got me there. For anyone else struggling with such memory problems, some particular areas of savings included the obvious - such as reducing the stack buffer allocation, reducing the response cache array size in the HttpJob module (the default was 5000 bytes for example), but even when I appeared to have ~23K before heading for the SSL Job I was still having problems. The final thing I tried was to change to the Basic SSL setting in the board manager options. This appeared to release enough further RAM (possibly 2-3K) to give enough headroom. I was able to successfully connect to an https aws server but don't know if using this setting will encounter problems with other SSL connections. I've also yet to find out if I encounter situations where the snipping of buffers and cache was a bit too severe, but anyhow at least it's going for now. My overall feeling is still that some way of getting the compiler to keep strings in flash rather than in static RAM would be a very useful function in the context of these devices. I have read further about the '"F()" macro which appears to be a way to achieve it in C and on the arduino but I don't have enough understanding at present to even determine if this is nonsensical. So I shall transfer to a wish thread for further comments.

Possibly this write up will help. Pretty much explains PROGMEM and F("") usage on esp8266.

Esp8266 PROGMEM docs

 

[BertI]

I had found something similar in my later post: https://www.b4x.com/android/forum/t...-string-constants-into-ram.114394/post-714629

But I'm not clear on how to use it in B4R context (if it is usable) as my C skills are next to nil (yeah, yeah I know...)

 

[tigrot]

Give a look https://www.b4x.com/android/forum/threads/various-table-operation-and-lookup-in-progmem.88640/

 

[BertI]

Give a look https://www.b4x.com/android/forum/threads/various-table-operation-and-lookup-in-progmem.88640/

Thanks, I hadn't encountered this one in my searches. It does provide further insights.

 

[tigrot]

Thanks, I hadn't encountered this one in my searches. It does provide further insights.

happy to help you. Ask if you need!

 

[tigrot]

One thing: the module is called statica.bas
The code for C isn't optimized, I'm am a basic C programmer, since I rarely use my notions.