Android Tutorial [B4X] CloudKVS - synchronized key / value store

CloudKVS solves a difficult and common problem. The user needs to work with online data, however as this is a mobile app we cannot assume that the device will always be connected to the remote server.

SS-2016-02-15_13.23.32.png


With CloudKVS the app always works with a local database. If the device can connect to the remote server then the local store will be synchronized with the online store.

The store is implemented as a key/value store. It is similar to a persistent Map collection.
The values are serialized with B4XSerializator.
The following types are supported as values:
Lists, Maps, Strings, primitives (numbers), user defined types and arrays (only arrays of bytes and arrays of objects are supported).
Custom types should be declared in the main module.
Including combinations of these types (a list that holds maps for example).

This is a cross platform solution. The clients can be implemented with B4A, B4i or B4J and the data can be shared between the different platforms.
Note that ClientKVS class source code is exactly the same on all three platforms.

Working with ClientKVS is almost as simple as working with a regular Map.

User field

To allow more flexibility items are grouped by a "user" field.
For example:
B4X:
ckvs.Put("User1", "Key1", 100)
Log(ckvs.Get("User1", "Key1")) '100
Log(ckvs.GetDefault("User2", "Key1", 0)) '0 because User2/Key1 is different than User1/Key1

The synchronization (from the remote store to the local store) is based on the user field.
The SetAutoRefresh method sets the user(s) that will be fetched from the remote store.

For example if we want to auto-synchronize the data of "User1":
B4X:
ckvs.SetAutoRefresh(Array("User1"), 5)
You can pass multiple users in the array. The second parameter is the interval measured in minutes.
This means that the client will check for new data every 5 minutes.
New data in this case means data that was uploaded from other clients.
The NewData event is raised when new data was fetched from the remote server.
Note that auto-refresh is not relevant for local updates. Local updates are uploaded immediately (if possible).

Multiple clients can use the same 'user name'.

Defaults

GetDefaultAndPut:
B4X:
Dim Score As Int = ckvs.GetDefaultAndPut ("User1", "Score", 0)
If there is a User1/Score value in the local store then it will be returned. Otherwise it will return 0 and also put 0 in the store.
This is useful as it allows us later in the program to get the score with: ckvs.Get("User1", "Score").
Defaults put in the store are treated specially. If there is already a non-default value in the remote store then the default value will not overwrite the non-default value. The non-default value will be synchronized to the local store once there is a connection.

Notes & Tips

- CloudKVS is fault tolerant. The local store includes a 'queue' store which holds the changes that were not yet synchronized.
- For performance reasons it is better to use larger values (made of maps or lists) than to use many small items.
- In B4A it is recommended to initialize ClientKVS in the Starter service.
- The B4J server project can run as is. It accepts a single command line argument which is the port number. If you want to run it on a VPS: https://www.b4x.com/android/forum/threads/60378/#content
- In the examples the auto refresh interval is set to 0.1 (6 seconds). In most cases it is better to use larger intervals (1 minute+).
- The keys and user names are case sensitive.
- On older versions of Anrdoid there is a limit of 2mb per field. You will see the following error if you try to put a larger value: java.lang.IllegalStateException: Couldn't read row 0, col 0 from CursorWindow


Projects

The three client projects and the server project are attached.
If you want to add this feature to an existing project then you need to add:
1. CloudKVS and CallSubUtils modules.
2. The two custom types Item and Task to the main module.
3. The following libraries are required: SQL, RandomAccessFile and HttpUtils2.

The server project depends on jBuilderUtils library. The library is attached.

Development Test Server

You can use this link for the ServerUrl during development:
https://www.b4x.com:51041/cloudkvs
Note that the messages are limited to 100k and more importantly the database is deleted every few days. The clients will stop updating after the database is deleted. You can delete the local database (or uninstall and install the app again) for the clients to work again.
Remember to use unique ids for the user value.
 

Attachments

  • B4J_ServerKVS.zip
    2.8 KB · Views: 2,208
  • B4i_ClientKVS.zip
    8.7 KB · Views: 1,318
  • jBuilderUtils.zip
    2.3 KB · Views: 2,746
  • B4J_ClientKVS.zip
    6.4 KB · Views: 1,700
  • B4A_ClientKVS.zip
    11.6 KB · Views: 2,203
Last edited:

christiantric

Member
Licensed User
Hi all,
I'm a novice in B4X and looking for a client-server/cloud data store solution.
Is this component still valid and supported?
I tried to start the B4JClient (attached in the first post) pointing to the demo server but is seems that doesn't respond.

Thanks in advance.
Christian
 

christiantric

Member
Licensed User
I'm testing CloudKVS to store values with image data (Byte array).
With inserts and updates everything's ok.
If I insert a new object with image I see the db size increasing accordinglly but if I delete the object the db size remains unchanged.

I delete with
B4X:
KVSUtils.Client.Put(KVSUtils.User, itemKey, Null)

Does CloudKVS implement a kind of "soft deletion"?
Isn't storing images a good idea? What's the best approach?
How could I delete data from database (local and remote)?
 

christiantric

Member
Licensed User
Integration to my previous post:
I looked into the db and noticed something weird: after deletion the values are null but the db size is high (see attached image).
How could we explain that?
upload_2019-11-22_12-9-2.png
 

christiantric

Member
Licensed User
Always better to start a new thread for your questions.

This is how SQLite behaves by default. You need to run this command on the server:
B4X:
sql1.ExecNonQuery("VACUUM")

I will update the server code to enable auto vacuum.

Oh sorry, I will start a new thread the next time for my questions about CloudKVS, if any.
Thanks, I didn't know about this SQLite behaviour.
A "Vacuum" function to call from time to time would be usefull even for the client in my opinion.
 

OliverA

Expert
Licensed User
Longtime User
I looked into the db and noticed something weird: after deletion the values are null but the db size is high (see attached image).
How could we explain that?
This is how SQLite behaves by default.
I'm going to go one step further and say that almost all on disk databases behave that way. It is normal operating procedure. Disk access is slow and if a DB would compact (remove empty spaces) every time a record is deleted, it would kill its performance. Please note that freed up space is usually marked in such a way that a DB will reuse it. Of course there may be times when you would want to compact/de-frag/vacuum a DB and SQLite's site gives a good breakdown of these times:
  • Unless SQLite is running in "auto_vacuum=FULL" mode, when a large amount of data is deleted from the database file it leaves behind empty space, or "free" database pages. This means the database file might be larger than strictly necessary. Running VACUUM to rebuild the database reclaims this space and reduces the size of the database file.

  • Frequent inserts, updates, and deletes can cause the database file to become fragmented - where data for a single table or index is scattered around the database file. Running VACUUM ensures that each table and index is largely stored contiguously within the database file. In some cases, VACUUM may also reduce the number of partially filled pages in the database, reducing the size of the database file further.

  • When content is deleted from an SQLite database, the content is not usually erased but rather the space used to hold the content is marked as being available for reuse. This can allow deleted content to be recovered by a hacker or by forensic analysis. Running VACUUM will clean the database of all traces of deleted content, thus preventing an adversary from recovering deleted content. Using VACUUM in this way is an alternative to setting PRAGMA secure_delete=ON.

  • Normally, the database page_size and whether or not the database supports auto_vacuum must be configured before the database file is actually created. However, when not in write-ahead log mode, the page_size and/or auto_vacuum properties of an existing database may be changed by using the page_size and/or pragma auto_vacuum pragmas and then immediately VACUUMing the database. When in write-ahead log mode, only the auto_vacuum support property can be changed using VACUUM.
I will update the server code to enable auto vacuum.
Please note SQLite's doc on this feature
An alternative to using the VACUUM command to reclaim space after data has been deleted is auto-vacuum mode, enabled using the auto_vacuum pragma. When auto_vacuum is enabled for a database free pages may be reclaimed after deleting data, causing the file to shrink, without rebuilding the entire database using VACUUM. However, using auto_vacuum can lead to extra database file fragmentation. And auto_vacuum does not compact partially filled pages of the database as VACUUM does.
So I think that auto_vacuum should be an OPTION for CloudKVS, in such a way that a user can specify its usage and let the user decide if the need the benefits of auto_vacuum.
 

jcohedman

Member
Licensed User
Longtime User
Hi Erel, first of all the cloud_kvs Server its amazing!!, im using it since 2017 in a lot of applications
I have a problem tough, the "log" folder its going bigger and bigger in some raspy servers that I made, and I cant find in the code the way to remove the option wich creates the .txt in the log folder.

Thanks again,

Juan Carlos
 
Top