B4R Question What max file size to SD card your used?

[peacemaker]

Hi, All

Who practically used SD-cards - what max file size you could write ?

Strange, but my ESP32 is starting to give error like "Failed to open file for appending" when the file is just ... 30966 bytes.
At

#Region Project Attributes
#AutoFlushLogs: True
#CheckArrayBounds: True
#StackBufferSize: 100000 'YES, YES !!!
#End Region

If to try to save 5000 lines - the latest line in the file is 1336.
If to try to save 3 files 500 lines each - all is OK.
The file name structure is 8.3.
And card is formatted to ... 32 KB cluster.... But no, can't be...

What about your experience ?

 

[peacemaker]

Arduino IDE code for SD-card works OK:

  Serial.println("Wait...");
  int i, j;
  String fn, line;
  for (int j = 1; j <= 5; j++) {
    fn = "/log_" + String(j) + ".txt";
    for (int i = 1; i <= 5000; i++) {
      line = "Hello " + String(i) + "\r\n";
      appendFile(SD, fn.c_str(), line.c_str());
    }
    Serial.println(fn + " saved.");
  }
  Serial.println("Finished.");

It makes 5 files of 5000 lines and 58893 bytes each. Looong (about 2 secs for each 100 lines), but 3 times faster than B4R code.
But B4R with StackBufferSize = 100000 cannot finish this work.
How the FOR loops can eat the memory ?

Private Sub SaveToLog As Boolean
    Log("Wait..")
    Dim millis1 As ULong = Millis
    Dim j, i As UInt
    For j = 1 To 3
        Dim fn As String = GenerateFileName(j)
        For i = 1 To 5000
            Dim a() As Byte = GenerateFileLine(i)
            WriteString(fn, a)
        Next
        Log(fn, " saved OK")
    Next
    Dim millis2 As ULong = Millis
    Dim delta As Float = (millis2 - millis1)/1000
    Log("Saved log file OK: ", NumberFormat(delta, 1, 3))
    Return True
End Sub

Sub GenerateFileName(i As UInt) As String
    Dim fn As String = JoinStrings(Array As String("/log_", NumberFormat(i,4,0), ".dat"))
    Return fn
End Sub

Sub GenerateFileLine(i As UInt) As String
    Dim a As String = JoinStrings(Array As String(NumberFormat(i,10,0), ";", "Column_", NumberFormat(i, 2, 0), CRLF))
    Return a
End Sub

Sub WriteString(key_string() As Byte, value() As Byte)
    Dim source As KeyValueType
    source.key = key_string
    source.value = value
    RunNative("save_csv_line", source)
    'Log("Saved to file")
End Sub

#if C
#include "FS.h"
#include "SD.h"
#include "SPI.h"

B4R::Object returnvalue_esp_sp;

void writeFile(fs::FS &fs, const char *path, const char *message) {
  Serial.printf("Writing file: %s\n", path);

  File file = fs.open(path, FILE_WRITE);
  if (!file) {
    Serial.println("Failed to open file for writing");
    return;
  }
  if (file.print(message)) {
    Serial.println("File written");
  } else {
    Serial.println("Write failed");
  }
  file.close();
}

void appendFile(fs::FS &fs, const char *path, const char *message) {
  //Serial.printf("Appending to file: %s\n", path);

  File file = fs.open(path, FILE_APPEND);
  if (!file) {
    Serial.println("Failed to open file for appending");
    return;
  }
  if (file.print(message)) {
    //Serial.println("Message appended");
  } else {
    Serial.println("Append failed");
  }
  file.close();
}

B4R::Object* setup_sd(B4R::Object* o){
    bool begun = SD.begin();
    if (!begun) {
        //Serial.println("Card Mount Failed");
        return returnvalue_esp_sp.wrapNumber((byte)0);
    }else{
        return returnvalue_esp_sp.wrapNumber((byte)1);
    }
}

void save_csv_line(B4R::Object* passedvar){
    _keyvaluetype* tmp = (_keyvaluetype*)B4R::Object::toPointer(passedvar); //structure from B4R
 
    B4R::Array* b = tmp->key;
    char* c = (char*)b->data; //file name, do not forget slash "/file.txt"
    //String key = (String)c;
 
    B4R::Array* b2 = tmp->value;
    char* c2 = (char*)b2->data; //test line for saving to file
    //String value = (String)c2;

    appendFile(SD, c, c2);
}

B4R::Object* freespace_sd(B4R::Object* o){
    float result;
    uint64_t total = SD.totalBytes() / (1024 * 1024);
    Serial.printf("totalBytes: %llu MB\n", total);
    uint64_t used = SD.usedBytes() / (1024 * 1024);
    Serial.printf("usedBytes: %llu MB\n", used);
    result = ((float)total - (float)used) / (float)total * 100;
   
       return returnvalue_esp_sp.wrapNumber((float)result);
}
#End If

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
modeIO, clock div:1
load:0x3fff0030,len:4832
load:0x40078000,len:16460
load:0x40080400,len:4
load:0x40080404,len:3504
entry 0x400805cc
AppStart
Start SD...
totalBytes: 1881 MB
usedBytes: 14 MB
FreeSpace, % = 99.2557
Wait..
Failed to open file for appending
Failed to open file for appending
Failed to open file for appending
Failed to open file for appending

 

[candide]

with several Log("stack: ",StackBufferUsage) in your code, you will see where stack is lost

i think stack usage is due to string usage with loop : stack is not released until sub closed

 

[peacemaker]

Log("stack: ",StackBufferUsage)

Indeed !

stack: 25948
stack: 28348
stack: 30752
/log_0002.dat saved OK
stack: 33176
stack: 35576
stack: 37976
stack: 40376
stack: 42776
stack: 45176
stack: 47576
stack: 49976
stack: 52376
stack: 54776
stack: 57176
stack: 59576
stack: 61976
stack: 64376
stack: 1428
stack: 3828
stack: 6228
stack: 8628
stack: 11028
stack: 13432
/log_0003.dat saved OK
Saved log file OK: 55.549

But strange that that stack is released just sometimes, and JoinStrings is used in separated subs.
How to avoid it ?

 

[candide]

Dim a() As Byte = GenerateFileLine(i)
WriteString(fn, a)

try to move this part in a sub...

 

[peacemaker]

move this part in a sub

Yes, helped, thanks !

stack: 24
stack: 24
stack: 24
stack: 24
/log_0001.dat saved OK
stack: 48
stack: 48
stack: 48
stack: 48
stack: 48
stack: 48
stack: 48
stack: 48
stack: 48
stack: 48
/log_0002.dat saved OK
stack: 72
stack: 72
stack: 72
stack: 72
stack: 72
stack: 72
stack: 72
stack: 72
stack: 72
stack: 72
/log_0003.dat saved OK
Saved log file OK: 28.099