B4J Question Out Of Memory Issue With VPS

[aaronk]

Hi,

Sorry if this post is long, but wanted to make sure I have as much information in the post to help understand my issue..

I have created a B4J Cloud app that I have put in a data centre running on a VPS.

The Server is Linux Ubuntu 16.04 LTS. (64bit)
This server has 768MB of Memory with 15GB SSD Hard Drive.

The Java version running on it is:

First of all, my app is running fine on my development Windows 10 machine (Running in a VM on a Mac)

When my app is running on the Linux VPS it is running out of memory and causing the app to crash.

My app is doing many things, and thought I would create a small test app and help narrow it down to what it could be.

My main app is doing:
- Handling 2 HTTP servers (1 for my front end and one for my backend. I wanted the backend to have a different port to connect with and wanted them both separate from each other)
- WebSocket to allow my B4A app (and eventually for my B4i app) to connect and communicate with it.
- UDP Server
- TCP Socket

I thought it was too much for my VPS so I created a small test app. When I only have the 2 HTTP servers & WebSocket running then it seems to be fine, soon as I add the TCP and UDP sockets and the data starts to come though then it starts to use up the memory.

There are products out in the field that will send UDP packets to my VPS which I need. The same device will also send TCP messages to my server which I also need. Unfortunately I can't change the protocol these products communicate with. (So I am stuck with TCP and UDP)

The UDP socket in my B4J app will listen on a port for the incoming connection and receive the packet. This data is saved to a SQLite database on my VPS. (It only contains ASCII messages - fairly small bits of data)

The TCP socket listens on a different port for incoming data and will log that data in a SQLite database on my VPS. (It only contains ASCII messages - fairly small bits of data)

The TCP socket only ever stays open for 10 seconds after it sends the message. This is closed by the remote device sending the message to my server.

To make sure it's not the data being saved that is causing my issues, in my test app I am not storing anything in the database.

My test app also sends the UDP & TCP messages out to my development PC by a UDP message to I can confirm the data is being received remotely.

I have created a service to run my B4J app on my VPS so that in case the server reboots my app will run again when it boots up. (more on my script I am using here: https://www.b4x.com/android/forum/threads/executable-path-is-not-absolute.91303/#post-577149)

When I run my Service, it starts and I can view the status of the service using the SSH command: systemctl status CloudServer.service

You will notice in the screenshot below that I checked 1 minute after the service had started and it was using 13.9M of memory.

I then checked again approx. 4 minutes later and it was using 24.1M of memory.

I then checked again 13 minutes after it had started and it shows it's using 45.6M of memory.

Then checked again 38 minutes after it had started and it was using 77.0M of memory.

Then checked again 1 hour after it had started and it was using 98.8M of memory.

This is running my test app, which is only listening and receiving TCP and UDP messages. (as per the attached B4J project.)

Although the above might not sound like a lot of memory being used, my main app is also handling the HTTP servers and WebSocket as well. However a lot more memory is being used in that app then my test app as it's filtering the message etc before it saves it to the database.

I guess if I left my test app running overnight, then it will crash by morning as it will run out of memory.

I have left my main app running overnight and it crashed. I attached the error report. (It's in 2 files as I had to split the file in half as it was too big to upload to the forum.)

In my test app (attached), in the Main module I have a sub called SendToDevelopmentPC when a TCP message or UDP message is received I am sending this message to that sub which then sends the message as a UDP message to my development PC over the internet.

I have also attached my test app that I used as per my testing above. Maybe I am doing something wrong?

 

[OliverA]

Are you using some sort of threading library? Unless you are not explaining the scope correctly, over 2000 threads seem to be excessive for the type of work your application does. What is causing that amount of threads?

 

[OliverA]

In Main, your are creating a new UDP socket for every message that you sent to your PC. You only need one.

Sub Process_Globals
   Private UDPSocket1 As UDPSocket   ' jNetwork v1.20
End Sub

Sub AppStart (Args() As String)
   UDPSocket1.Initialize("UDPOutMessage", 0, 0) ' Select random port (0) and we're not receiving (0) on this socket (sent only)

   TCP.Initialize   ' this will make it so it listens for incoming TCP messages
   UDP.Initialize   ' this will make it so it listens for incoming UDP messages
   
   StartMessageLoop
End Sub

' This will send all messages to development PC
Public Sub SendToDevelopmentPC(value As String)
   Dim Packet As UDPPacket
   Dim data() As Byte
       data = value.GetBytes("UTF8")
       Packet.Initialize(data, "My_Development_PC_IP_Was_Here", 5000)   'Send to IP on Port 5000
       UDPSocket1.Send(Packet)
End Sub

 

[aaronk]

Are you using some sort of threading library? Unless you are not explaining the scope correctly, over 2000 threads seem to be excessive for the type of work your application does. What is causing that amount of threads?

Check out the B4J app from post 1. This is the test app that I was using.

If I stop it from sending the messages to my PC then it still increases in memory.

 

[Erel]

As OliverA wrote it is a mistake to create a new UDPSocket for each message. This is probably the source of the memory leak.

If I stop it from sending the messages to my PC then it still increases in memory.

Does it eventually crash?

Can you post the error message?

 

[aaronk]

As OliverA wrote it is a mistake to create a new UDPSocket for each message. This is probably the source of the memory leak.

I have removed that completely now. (still get the same result)

Does it eventually crash?

Yes.

Can you post the error message?

I ended my Linux VPS. I then setup a Windows VPS to see if that made any difference.

The Windows VPS has the same result. It crashed a few hours after it had started.

The error message that the Windows VPS shows is:
(but this shows a different error than what my Linux VPS had)

When my Original Linux VPS was setup and running my B4J app, it used to show an error when it crashed and even created a file. (the file it created is in post 1 above)

I just setup my Linux VPS again (since I ended the original Linux VPS).

Now when I run my new Linux VPS with my B4J app, it runs but then crashes but the service looks like starts it again automatically.

If I stop the service, and run the file manually using:
nohup java -jar /opt/CloudServer/CloudServer.jar > nohup.out &

Then when it crashes, it doesn't log the crash in the nohup.out file. I do see the B4J app log things though in that file.

If I run it like:
Java -jar /opt/CloudServer/CloudServer.jar
Then it runs and I see events display in the SSH window but when the app crashes it just logs Killed.

 

[Erel]

The error message should appear in nohup.out.

Can you upload the updated code?

 

[aaronk]

The error message should appear in nohup.out.

Only my B4J app data is displayed in that file. The error message didn't write to that file for some reason.

Can you upload the updated code?

See attached. This is the tester app.

 

[Erel]

Looks fine.
You need to post the error message and the heap dump.

 

[aaronk]

I will run the server overnight and see if it locks up.

 

[OliverA]

See attached. This is the tester app.

Are you sure Tester.zip contains everything that produced the error messages in post#6? Compiling this in either Release or Debug mode does not produce anything near the 140+lines for the main._appstart (java line: 140) error message to make any sense.

 

[aaronk]

It seem that my test app works fine now on this new Linux VPS. I had it running overnight and it didn't crash, and the memory stayed low.

When I ran my main app, after a few hours it crashed.

You need to post the error message and the heap dump.

It didn't log any errors, other than the word 'Killed'.

Are you sure Tester.zip contains everything that produced the error messages in post#6? Compiling this in either Release or Debug mode does not produce anything near the 140+lines for the main._appstart (java line: 140) error message to make any sense.

Sorry, I should of said, this was from my main app, not my test app.

After running my test app again on this new Linux VPS it seems to be OK. The have setup a new Linux VPS from a different data center.

I think I may of found the memory leak, but want to confirm..
(the following is with my main app, and not my test app)

When a UDP message is received, it checks the SQLite database to see if the device is in the database. If it's not in the database it adds it.

Would every time I check the database cause the memory to increase ?

The reason I ask is I ran a few tests, by making it do nothing when the UDP message arrives. I stopped the code from running just before it wrote and checked the SQLite database and it didn't increase the memory.

I then made it do a SELECT from my SQLite database and it began to increase in memory.

 

[OliverA]

Would every time I check the database cause the memory to increase ?

Seeing some code (especially working app that crashes) would help.

After running my test app again on this new Linux VPS it seems to be OK

I've been running the "test" app all day long on my system (win10) without issues. I created a client that constantly sends UDP messages and ran 3 instances of it, bombarding the test app. Memory went up to about 48MB for server (each client was around that size too) and dropped to 28MB after a while. No crash. So I need something else to reproduce your crashes.

 

[aaronk]

Found my memory leak.. (I hope)

For some reason I was doing:

Sub GetValue(Account As String) as String
    Dim RS As ResultSet = db_sql.ExecQuery("SELECT * FROM `status` WHERE `Account` = '" & Account & "'")
       
    Do While RS.NextRow
        If RS.GetString("value1") = "no" Then Return "no"
        If RS.GetString("value1") = "yes" Then Return "yes"
        If RS.GetString("value1") = "" Then Return "yes"
    Loop 'ignore
    RS.Close

Notice that the return value is being returned before RS.Close.

This meant it could never close and most likely being stored in the memory, and with messages being sent to my VPS it kept opening more and more ResultSets and eventually filling the memory and causing the app to crash since it wasn't being released.

I ended up using:

Sub GetValue(Account As String) as String
Return db_sql.ExecQuerySingleResult("SELECT 'value1' FROM `status` WHERE `Account` = '" & Account & "'")
End Sub

Now it seems to work and fixed the issue from what I can see.

I created a little test app that sent a UDP message every 10ms (I wanted to stress test it even knowing there wouldn't be that many messages being sent) to simulate devices connecting to my VPS, and got it to send the data they will send the field and the memory stayed around approx. 77MB.

Now I will need to go through the rest of the app and make sure I am not doing anything like that else where.

 

[aaronk]

I ran my B4J app for one hour and it was using 84.1M of memory.

I let it run overnight and it didn't crash but it's using 113.6M of memory.

Does this sound normal or is there still a memory leak somewhere ?

 

[Erel]

It sounds normal.