Android Tutorial [B4X] CloudKVS - synchronized key / value store

Widget

Well-Known Member
Licensed User
Longtime User
Erel,

The CloudKVS looks like a really capable solution for simple cloud storage.
Although it can handle a lot of requests compared to most SQL solutions, have you given any thought to making CloudKVS scalable? This will add redundancy should the computer with the database fail, so the client request would then shift over to the next available CloudKVS server. It will also speed things up if one computer isn't fast enough. (The database speed may slow down depending on the size of the values being updated like binary data such as images.)

I only mention this because if an application does catch on and becomes popular, the developer doesn't want to see his work come crashing down because his backend server can't handle the volume or it crashed on him. He'd be pretty embarrassed and his company's reputation would be in tatters.

Would it be possible (or practical) to have several CloudKVS DB servers (running Sqlite for example) with a load balancer computer (CloudKVS LB) up front that is receiving the requests from the client, and redirecting the request to one of several CloudKVS DB servers on the same network?

Here is how I think it could work.

The load balancer computer would itself be a CloudKVS server (called CloudKVS LB where "LB"=Load Balancer) with a SQLite memory database that keeps the connection info of the user and a list of available CloudKVS DB servers that are operational (haven't crashed and are still responding), and redirects the request to the next available CloudKVS Db computer. Every 60 seconds (or even 5 minutes depending how current the data has to be between servers), each CloudKVS DB server would sync with the other CloudKVS DB servers on the network so they have the same data. Every 30 minutes it could even sync with another CloudKVS server that is at a remote location which could also be part of another bank of CloudKVS servers. When the client app running on the phone connects to the server, he would be given an updated list of available remote servers (IP Addresses) should the ISP hosting the CloudKVS go down the next time he tries to connect.

The neat part of all this, I think it can be done with just computers running CloudKVS. It would be like building a CloudKVS server farm from Lego blocks. Yes there will be some overhead every minute or so syncing between servers, but I think it will be small (due to larger transaction size) compared to the improved reliability and throughput for the client.

Will it work? Or do I have my head stuck in the clouds?

TIA
 

Widget

Well-Known Member
Licensed User
Longtime User

I've actually used MySQL years ago on another project. It doesn't scale well either. Its MYISAM engine (fastest) uses table locking. It is great at bulk loading data and building indexes, but it is not good for concurrency with more than around 25 users with heavy updates. I've had MYISAM tables up to 100 million rows and it worked reasonably well for a low number of connections. Its InnoDb engine uses record locking with transactions with better concurrency, but is much slower for < 25 connections because of its greater overhead per update. I don't believe Innodb is free anymore for the community version. Since Oracle bought MySQL, they tightened up its licensing policies. I will look for another solution that has better replication methods.
 

JakeBullet70

Well-Known Member
Licensed User
Longtime User
https://mariadb.org/

It is from the original developers of MySQL.


s InnoDb engine uses record locking with transactions with better concurrency, but is much slower for < 25 connections because of its greater overhead per update.

I think that 25 users regarding the above statement (application does catch on and becomes popular) is not an issue. All in all just toss more RAM - move to SSD drives in your DB server.
 
Last edited:

Widget

Well-Known Member
Licensed User
Longtime User

InnoDb is a CPU hog and requires at least 6 cores. If you want very good performance from InnoDb then 36 cores is ideal. More RAM will of course help, and 32GB to 62GB is recommended for a db under heavy load. Unfortunately even with all this hardware thrown at InnoDb, you're not going to get ACID transactions. You still need to turn off flushing transactions at commit using "innodb_flush_log_at_trx_commit = 0" if you want any type of speed. Turning it on will reduce the speed of InnoDb by nearly 90%. Ouch!

MariaDb from Michael Widenius (MySQL author) is a good alternative to MYISAM and inexpensive. I haven't researched its TPS though. Might be worth a look. Thanks.
 

JakeBullet70

Well-Known Member
Licensed User
Longtime User
flushing transactions at commit using "innodb_flush_log_at_trx_commit = 0" if you want any type of speed. Turning it on will reduce the speed of InnoDb by nearly 90%.

That is a huge OUCH! I myself was MsSQL for many years at work (and still do) and only got into MySQL when I went to b4a.
Oh and @DonManfred has a MariaDB connector lib.
 

Widget

Well-Known Member
Licensed User
Longtime User
That is a huge OUCH! I myself was MsSQL for many years at work (and still do) and only got into MySQL when I went to b4a.
Oh and @DonManfred has a MariaDB connector lib.

I don't want to get off of topic too much, but what finally turned me off of MySQL after using it for a few years, was the licensing change when Oracle took over. After talking with MySQL AB (aka Oracle) sales people about whether I could use the free community license on my own web server, he couldn't tell me for sure (or didn't want to tell me) but strongly suggested I pay for a MySQL license. At $700 (USD) per server, I decided to use only non-royalty based databases because I don't want any licensing hassles to come back and bite me. IMHO, there are better (transactional) performance (and certainly cheaper) alternatives out there like PostgreSQL, MongoDB (NOSQL) and Firebird to name a few. I believe commercial applications that use MariaDB for a database *may* still require a license depending on which libraries are included with your application. This may dissuade me from considering MariaDB (at least for commercial Windows apps that I write) and I will have to look into it further. https://mariadb.com/kb/en/mariadb/licensing-faq/

I am going to look into MongoDB for constructing a web service of my own or perhaps get it to work with CloudKVS. It has the ability to scale well with its built-in replication and is royalty free (Apache License v2.0). It should be a better solution than MySQL as a cloud database.
 

jcohedman

Member
Licensed User
Longtime User
How do I get all the user's name of the table on the server side, from a client?
Thank You.
 

OliverA

Expert
Licensed User
Longtime User
How is it possible to refresh only the given key of a user?
As is, it is not. But it should not be necessary. You will only get keys that have been updated since the last sync. If that key is part of the update, you'll get it synced back down to the client.
 

schimanski

Well-Known Member
Licensed User
Longtime User
Ok, thanks for answer. Than i have to oranize my keys and users in an other way.
 

incendio

Well-Known Member
Licensed User
Longtime User
What's the difference between this and jRDC? And how its performance to send/retrieve huge data compare to jRDC?
 

ocalle

Active Member
Licensed User
Longtime User
Excellent Solution, a little question, can be installed on Linux Servers? as Debian for example
 
Cookies are required to use this site. You must accept them to continue using the site. Learn more…