Android Question MQTT Memory consumption

Blueforcer

Well-Known Member
Licensed User
Longtime User
In my current project, I have to send files via mqtt, for this I split them into several 5MB chunks and send them one after the other in a Do While loop as base64 in a json.
Everything works wonderfully. The only problem is that with each mqtt.publish the memory decreases by the size of the current chunk. Only when the file has been transferred completely the memory slowly become free again. But the problem is that if the file is too big I get an OutOfMemory exception at some point. QOS makes no difference here. I tested it by comment the mqtt.publish line, and the free memory stays stable.

Is there a problem with the garbage collector in the MQTT libary or is there a trick i can use to avoid the problem??
 
Sort by date Sort by votes
E

emexes

Expert
Licensed User
Longtime User
10MB chunks in 10s interval:

B4X: 
Waiting 10s
Mem: 374.222 MB
publish chunk #21
Waiting 10s
Mem: 360.644 MB
publish chunk #22
Waiting 10s
Mem: 347.076 MB
publish chunk #23
Waiting 10s
Mem: 347.08 MB
publish chunk #24
Waiting 10s
Mem: 317.041 MB
publish chunk #25
Waiting 10s
Mem: 302.8 MB
publish chunk #26
Waiting 10s
Mem: 302.467 MB
publish chunk #27
Waiting 10s
Mem: 289.032 MB
publish chunk #28
Waiting 10s
Mem: 275.456 MB
publish chunk #29
Waiting 10s
Mem: 275.447 MB
publish chunk #30
Waiting 10s
...

Brilliant!

But why aren't they evenly spaced? Why does the memory sometimes NOT decrease by the usual chunk size 13-14 MB? That makes me think that, during that 10 seconds, a chunk was added to the queue and also a chunk was removed from the queue.

Also mysterious - why did memory decrease by two chunks instead of the usual one chunk, from 347 MB to 317 MB?
 
Last edited:
Upvote 0
E

emexes

Expert
Licensed User
Longtime User
Blueforcer said:
I never said that im waiting for a ACK from the broker.
Click to expand...
Nor did I. I said that I couldn't see any wait-until-confirmed loops.

You said:

Blueforcer said:
The chunk is already received by the requesting client before the next one is sent
Click to expand...

How can that be happening, without somebody or something waiting in-between for confirmation that a chunk has been received, before sending the next chunk? If it's not you, then it must be the MQTT library. Or it's not happening.

Your log above looks to me like it's not happening. 9 chunks were published, but memory usage only went down by 7 chunks, and the simplest explanation for that is that during that same time, MQTT managed to send 2 chunks over the network and then freed up their memory.

Please, please, please try the same test again, but with a 50 second sleep.
 
Upvote 0

Blueforcer

Well-Known Member
Licensed User
Longtime User
Because i can see it in the other client that the chunk was received, this takes about 2s.
It doesnt matter how long i should wait, in reality its simply too slow until the mqtt buffer is released.
I cant wait that much for each chunk.
I expect that the mem will be free as soon the message was sent, at the latest when the other client received it (with QOS 1-2), and not 20-30s later..
 
Upvote 0
E

emexes

Expert
Licensed User
Longtime User
Blueforcer said:
Because i can see it in the other client that the chunk was received, this takes about 2s.
Click to expand...

Super! That sounds plausible over a 100 Mbps network that is operating efficiently.

What about if you transfer a 30-chunk file, as shown in the log? Can you see the chunks arrive in the other client about every 2 seconds?

B4X: 
Mem: 275.447 MB
publish chunk #30
Waiting 10s

Maybe the problem is the write speed to flash media by the other client.

Blueforcer said:
I cant wait that much for each chunk.
Click to expand...

We're in the diagnosis phase. Try with a 50 second Sleep. Do the chunks arrive in the other client abut 2 seconds after they're sent? At one chunk per 50 seconds, you've got time to check the chunk numbers too, ie make sure that it's the same chunk number that is received in the other client about 2 seconds after it's sent.

Blueforcer said:
I expect that the mem will be free as soon the message was sent, at the latest when the other client received it (with QOS 1-2), and not 20-30s later..
Click to expand...

I agree. That's what I'd expect too. But apparently that is not what's happening.

Before we can fix a problem, first we have to understand it.

I'm not proposing 50 second Sleeps as a fix; I am proposing them as a test to confirm that the memory is released after a chunk has been sent.

It could be that the MQTT system has a safety/quality feature than keeps messages for a minute or two aven after they've been sent, just in case there is a network fault and the hub/server/broker comes back with "hey, there was a problem with that last message, can you please send it again?" request. For most MQTT use cases, eg under 100 kB per minute, that wouldn't be a problem.

I'm not saying that *is* what MQTT does, I'm just brainstorming possibilities that could explain what you're seeing.
 
Last edited:
Upvote 0
E

emexes

Expert
Licensed User
Longtime User
emexes said:
brainstorming possibilities that could explain what you're seeing
Click to expand...

Maybe MQTT has metering/rationing/flow control that restricts average usage by a client/publisher to eg its total bandwidth divided by the number of clients it serves.

The first chunk/message goes through at full speed, because at that point your average usage of 0 is under whatever the allowance is, but when you send the second chunk and the broker sees your new average usage of 5 MB per second and says hey, slow down, 200 kB per second should be enough for any typical MQTT purpose... that could explain with what you're seeing.
 
Upvote 0
E

emexes

Expert
Licensed User
Longtime User
Any luck in discovering reason for the unexpected high MQTT memory usage?

Another thought that occurred to me is:

if you are using a third-party broker/server/hub then perhaps there are service limits or caps or throttling being applied

or if your device is using a mobile broadband internet connection, perhaps it is happening there

(eg my parents moved to a wireless broadband internet connection for their home phone service - they have a monthly 20 GB at full speed, and then it is throttled to 2 Mbps ie 200 kB/second for the remainder of the month) (coincidentally similar to your logged 14 MB chunk per 45 seconds speed)
 
Last edited:
Upvote 0
You must log in or register to reply here.

Similar Threads

  • Solved
Android Question Memory leak during screen rotation
Replies
12
Views
2K
mevial
  • Question
Android Question Available memory
Replies
7
Views
2K
Erel
  • Solved
Android Question Error OUT OF MEMORY
Replies
5
Views
2K
amorosik
  • Question
Android Question Crash after Days - do i have a Memory-Problem?
Replies
10
Views
4K
Peter Simpson
  • Question
Out of Memory
2
Replies
30
Views
12K
Erel
Menu
Log in
Register
Cookies are required to use this site. You must accept them to continue using the site. Learn more…