B4J Question JBDC pool connections: how to use correctly ?

peacemaker

Expert
Licensed User
Longtime User
HI, All

If a B4J server app (works on a Host1 of a LAN) uses a big MySQL database on a dedicated Host0 (of this LAN), and also planned to make other apps on different hostsX (and connecting to the same Host0 db) - how correctly to use the connections pool to avoid errors ?

There are tons of subs inside the app where the DB is used - is a new connection to be opened at each sub and closed at sub finish ?
 

peacemaker

Expert
Licensed User
Longtime User
Difference is
1) No pool at all, SQL object is inited instead of the pool
2) The parameters are

$"jdbc:mysql://${DBLocation}/${DBname}?characterEncoding=utf8&allowPublicKeyRetrieval=false&autoReconnect=false&requireSSL=false&useSSL=false&pooling=true"$
 
Upvote 0

Magma

Expert
Licensed User
Longtime User
you mean changed that:
DBM.InitializeMySQL($"jdbc:mysql://${DBLocation}/${DBname}?characterEncoding=utf8&allowPublicKeyRetrieval=false&autoReconnect=true&requireSSL=false&useSSL=false&pooling=true"$, DBUsername, DBPassword, 100)
with ?
 
Upvote 0

OliverA

Expert
Licensed User
Longtime User
Where to start?
1) The JDBC Pooling landscape has changed. At one time, the only way to do pooling was via 3rd party (such as 3CP0, which is provided by the jServer library and is used by ABMaterial), and as of lately, JDBC provides a basic pooling framework that can be used/extended using standard JDBC drivers. In your case, you are mixing both versions:

B4X:
DBM.InitializeMySQL($"jdbc:mysql://${DBLocation}/${DBname}?characterEncoding=utf8&allowPublicKeyRetrieval=false&autoReconnect=true&requireSSL=false&useSSL=false&pooling=true"$, DBUsername, DBPassword, 100)

a) Remove pooling=true. You're doubling up on the pooling, which may cause the issue of too many connections
b) Remove autoReconnect=true. Why remove autoReconnect? Because the 3CP0 library will take care of that

2) When using pooling, you want to
a) Aquire and connection
b) Do your query
c) Close your connection
as close together as possible. Under plain JDBC, you want to get a connection, do as many SQL calls as possible for as long as possible, and then close your connection. But pooling changes all of that. With pooling, fetching a connection is not an expensive operation. Also, every time you fetch a connection and hold on to it for extended periods of time, you are a) robbing other processes of that connection, and b) making it likely that you run out of connections in the pool, leading to "no connection" available issues. So, don't fetch your SQL globally in your Handler. Do it as close to the SQL calls as possible.

So first, rem out your global SQL object
B4X:
DBM.InitializeMySQL($"jdbc:mysql://${DBLocation}/${DBname}?characterEncoding=utf8&allowPublicKeyRetrieval=false&autoReconnect=true&requireSSL=false&useSSL=false&pooling=true"$, DBUsername, DBPassword, 100)
'SQL = DBM.GetSQL ' Nope: Not fetching a SQL connection yet

Then follow up by using local SQL objects (fetched from the pool)
B4X:
'The change is from using a global SQL object to a local one. For EVERY query, we fetch a new pool connection (DBM.GetSQL), execute the SQL
'    and close the pool connection.
'Fetching and closing pool connections are inexpensive and allow us to share the connections as needed.
'
Sub timSystem_Tick
    timSystem.Enabled = False

    
    If machine_id = 0 Then
        Dim filter As Map: filter.Initialize
        filter.Put("scanner_name", others.Get_DeviceID)
        webapi1.API_LOAD("scanners", filter)
        Wait For API_LOADED (L As List)
        If L.Size > 0 Then
            Dim machine_id As Int = L.Get(0).As(Map).Get("machine_id")
            Log("machine_id = " & machine_id)
            timFTPupload_Tick
        Else
            Log("machine_id trouble !")
        End If   

    Else
         'Create a local SQL object here
        dim localSQL As SQL = DBM.GetSQL
        Dim names As ResultCursor
        'names.Initialize(db.SQL, "SELECT * FROM " & db.DBT_NAMES & " WHERE sent = '0'", Null, 1)
        names.Initialize(localSQL, "SELECT * FROM " & db.DBT_NAMES & " WHERE sent = '0'", Null, 1)
        localSQL.close ' Yup, done for now
        For i = 0 To names.RowCount - 1
            names.Position = i
            Dim name As String = names.GetString("name")
            Log("Sending ... " & name)
            
           ' db.SQL.ExecNonQuery("UPDATE " & db.DBT_NAMES & " SET sent = '2' WHERE name = '" & name & "'")    'sending status
            localSQL = DBM.GetSQL
            localSQL.ExecNonQuery("UPDATE " & db.DBT_NAMES & " SET sent = '2' WHERE name = '" & name & "'")    'sending status
            localSQL.Close ' Yup, done once more
            Dim Sending As ResumableSub = api.Send_Vag(name, "")
            Do Until Sending.Completed
                Sleep(50)
            Loop
            '*******
            '******* The loop above can block many SQL connection requests to the pool !!!!!!!!
            '******* That's why we are freeing the connection before we do the Send_Vag method
            '*******
            Dim filter As Map: filter.Initialize
            filter.Put("machine_id", machine_id)
            filter.Put("name", name)
            webapi1.API_LOAD("objs", filter)
            Wait For API_LOADED (L As List)
            If L.Size > 0 Then
                Dim obj_id As Int = L.Get(0).As(Map).Get("id")
                Log("obj_id = " & obj_id)
                Dim obj_img As String = L.Get(0).As(Map).Get("img")
                Log("obj_img = " & obj_img)
                
                
                'img name for server: sample_4_85.jpg = sample_ machine_id _ obj_id .jpg
                Dim img_name As String = "sample_" & machine_id & "_" & obj_id & ".jpg"
                Log("img_name = " & img_name)
                
                Log("Sending2 ... " & name)
                'db.SQL.ExecNonQuery("UPDATE " & db.DBT_NAMES & " SET sent = '2' WHERE name = '" & name & "'")    'sending status
                localSQL = DBM.GetSQL
                localSQL.ExecNonQuery("UPDATE " & db.DBT_NAMES & " SET sent = '2' WHERE name = '" & name & "'")    'sending status
                localSQL.Close ' Yup, done once more
                Dim Sending As ResumableSub = api.Send_Vag(name, img_name)
                Do Until Sending.Completed
                    Sleep(50)
                Loop
                '***** Looks like another killer loop above

                'Dim local_file As String = db.SQL.ExecQuerySingleResult("SELECT x.img FROM (SELECT img, id FROM " & db.DBT_OBJS & " WHERE name = '" & name & "' ORDER BY id DESC, rate DESC LIMIT 10) AS x LIMIT 1")
                 localSQL = DBM.GetSQL
                Dim local_file As String = localSQL.ExecQuerySingleResult("SELECT x.img FROM (SELECT img, id FROM " & db.DBT_OBJS & " WHERE name = '" & name & "' ORDER BY id DESC, rate DESC LIMIT 10) AS x LIMIT 1")
                 localSQL.Close ' Yup, done once more

                Log("local_file = " & local_file)
                local_file = fs.getPath(local_file)
                Log("local_file2 = " & local_file)
                If local_file <> "" Then
                    If File.Exists(File.DirApp, "by_ftp") = False Then
                        File.MakeDir(File.DirApp, "by_ftp")
                    End If
                    Dim local_folder As String = File.Combine(File.Combine(File.DirApp, "by_ftp"), machine_id)
                    
                    If File.Exists(local_folder, "") = False Then
                        File.MakeDir(local_folder, "")
                    End If
                    File.Copy("", local_file, local_folder, img_name)    'save file for next uploading by FTP
                End If
            Else
                Log("no obj_id yet")
            End If
            
            
            Log("Sent: " & name)
            'db.SQL.ExecNonQuery("UPDATE " & db.DBT_NAMES & " SET sent = '1' WHERE name = '" & name & "'")    'sent status finally
            localSQL = DBM.GetSQL
            localSQL.ExecNonQuery("UPDATE " & db.DBT_NAMES & " SET sent = '1' WHERE name = '" & name & "'")    'sent status finally
            localSQL.Close ' Yup, done once more
            
        Next
    End If

    timSystem.Enabled = True
End Sub

'Note: Untested code, but you should get the gist of it.
 
Upvote 0

peacemaker

Expert
Licensed User
Longtime User
Thanks, @OliverA, for the info.
So, my first post question about a SQL connection at any sub - was correct. And it's not a problem. But the code is incompatible with SQLite alternative usage anymore.
 
Last edited:
Upvote 0

Magma

Expert
Licensed User
Longtime User
...If the SQL server runs at the same Server with your app... I think no need to close every time the Database-Connection... for sure need to close the resultsets...

You can do it sometimes into day, or in week for maintenance with some IFs.... or if you wanna be sure that you will not lose anything you can use try ..catch... end try and if you get the error that has to do with no reading or writing to db... close (checking initialization i think/reopen it)

But all these if at the same machine... (my opinion)

Check the loops - or limit the loops with a timer-counter... also in those loops check for errors - if having semantic errors, with your procedure you must inform your log, end-user / what is going wrong...

* You can have the loops outside/before of the For..Next.... and have the result at an array... so will be not killing the resultset...
 
Upvote 0

peacemaker

Expert
Licensed User
Longtime User
The MySQL server is on the same host with the app now. But it's planned to make several app hosts that use one server DB, so migrating to MySQL has to be...
Now it works with MySQL without the pool. Why\where the pool is required ?
 
Upvote 0

Magma

Expert
Licensed User
Longtime User
Connection pools are used to enhance the performance of executing commands on a database. Opening and maintaining a database connection for each user, especially requests made to a dynamic database-driven website application, is costly and wastes resources.
 
Upvote 0

aeric

Expert
Licensed User
Longtime User
But the code is incompatible with SQLite alternative usage anymore.
Again, which library you are referring to?

As I mentioned, SQLite is different which doesn't use pooling. You can't treat it same as MySQL. Both have different engine and features.
In my JRDC2 Server template, I handle the connections differently.
 
Upvote 0

peacemaker

Expert
Licensed User
Longtime User
In my situation - MySQL is needed only due to several app hosts are planned to work together on a shared VPN to a shared DB.
Each host recognizes objects on camera's frames and result of them all should be collected in a single DB. Network accessible DB.
 
Upvote 0

Magma

Expert
Licensed User
Longtime User
Also... using pool... is costly too.. you must count the users if any... or the use (resultsets, updates, queries)... because setting pool is also using a lot of memory (before even asked) and some CPU... not using pool... using much more the CPU... ang gives some lag... (especially at web apps / sites)
 
Upvote 0

Magma

Expert
Licensed User
Longtime User
VPN or VPS ?
 
Upvote 0

peacemaker

Expert
Licensed User
Longtime User
VPN. One plant's workshop has machines that share objects that are controlled by several cameras, each must be processed by a recognition host, all in a LAN (VPN), all results must be collected...
 
Upvote 0

peacemaker

Expert
Licensed User
Longtime User
I was started as usual with local SQLite DB, and when recognition started to work - the DB size is growing like, say, 1 megabyte per minute or so, very fast. Without images.
So, how to explore such huge DB during debugging...
 
Upvote 0

Magma

Expert
Licensed User
Longtime User
I was started as usual with local SQLite DB, and when recognition started to work - the DB size is growing like, say, 1 megabyte per minute or so, very fast. Without images.
So, how to explore such huge DB during debugging...
Well... sometimes need the records to be small as possible.. fields must set to real-life sized... Int, Double, String where need to be - may be with limits, boolean, byte, char... every byte counts on those big data....
 
Upvote 0

amykonio

Active Member
Licensed User
Longtime User
I was started as usual with local SQLite DB, and when recognition started to work - the DB size is growing like, say, 1 megabyte per minute or so, very fast. Without images.
So, how to explore such huge DB during debugging...
I don't think that we can give any suggestion about database size growing so fast, as we have no idea of the schema and what is stored in.
Andreas.
 
Upvote 0

peacemaker

Expert
Licensed User
Longtime User
on those big data
NN recognition things use "big data", say, one area of the image (object on the photo) saved as a digital id (vector) needs 1000 pcs of float values. This one fact does not allow to have compact DB...
Have you tried to make autoreconnect=false also to pool?
Yes, did not help. All latest tests with the MySQL connections pool did not help yet in such intensive working with DB app.
But i did not change yet ALL SQL calls to the pool.
 
Last edited:
Upvote 0

amykonio

Active Member
Licensed User
Longtime User
As other stated we don't have enough information. I did some tests with a ui application using pooling. I created a table in a mysql schema:
TABLE `testdata`:
CREATE TABLE `testdata` (
  `idtestdata` int NOT NULL AUTO_INCREMENT,
  `testdatadescr` varchar(2000) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `testdatadescr2` varchar(5000) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `testdatavalue` float DEFAULT NULL,
  `testdatavalue2` decimal(12,2) DEFAULT NULL,
  PRIMARY KEY (`idtestdata`),
  UNIQUE KEY `idtestdata_UNIQUE` (`idtestdata`)
) ENGINE=InnoDB AUTO_INCREMENT=157605 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
Filled that table with randomly generated data.
Selected in a loop all contents in my table...
All inserts, select statements where issued using connection pooling.
Had no problem!

I added a "delete from testdata" statement. This one took more time to execute. So I've got a socket time out exception. That's logical as this setting was set as: socketTimeout=5000 (5000 -> msec). I increased that setting to 500000 and that issue disappeared.

PS: I attach my simple testing application. It's not perfect. But it was made just to do some testing. Not for production.

Andreas.
 

Attachments

  • MySQLPoolingTests.zip
    19.7 KB · Views: 66
Upvote 0
Cookies are required to use this site. You must accept them to continue using the site. Learn more…