Android Tutorial [B4X] CloudKVS - synchronized key / value store

CloudKVS solves a difficult and common problem. The user needs to work with online data, however as this is a mobile app we cannot assume that the device will always be connected to the remote server.

With CloudKVS the app always works with a local database. If the device can connect to the remote server then the local store will be synchronized with the online store.

The store is implemented as a key/value store. It is similar to a persistent Map collection.
The values are serialized with B4XSerializator.
The following types are supported as values:
Lists, Maps, Strings, primitives (numbers), user defined types and arrays (only arrays of bytes and arrays of objects are supported).
Custom types should be declared in the main module.
Including combinations of these types (a list that holds maps for example).

This is a cross platform solution. The clients can be implemented with B4A, B4i or B4J and the data can be shared between the different platforms.
Note that ClientKVS class source code is exactly the same on all three platforms.

Working with ClientKVS is almost as simple as working with a regular Map.

User field

To allow more flexibility items are grouped by a "user" field.
For example:

ckvs.Put("User1", "Key1", 100)
Log(ckvs.Get("User1", "Key1")) '100
Log(ckvs.GetDefault("User2", "Key1", 0)) '0 because User2/Key1 is different than User1/Key1

The synchronization (from the remote store to the local store) is based on the user field.
The SetAutoRefresh method sets the user(s) that will be fetched from the remote store.

For example if we want to auto-synchronize the data of "User1":

ckvs.SetAutoRefresh(Array("User1"), 5)

You can pass multiple users in the array. The second parameter is the interval measured in minutes.
This means that the client will check for new data every 5 minutes.
New data in this case means data that was uploaded from other clients.
The NewData event is raised when new data was fetched from the remote server.
Note that auto-refresh is not relevant for local updates. Local updates are uploaded immediately (if possible).

Multiple clients can use the same 'user name'.

Defaults

GetDefaultAndPut:

Dim Score As Int = ckvs.GetDefaultAndPut ("User1", "Score", 0)

If there is a User1/Score value in the local store then it will be returned. Otherwise it will return 0 and also put 0 in the store.
This is useful as it allows us later in the program to get the score with: ckvs.Get("User1", "Score").
Defaults put in the store are treated specially. If there is already a non-default value in the remote store then the default value will not overwrite the non-default value. The non-default value will be synchronized to the local store once there is a connection.

Notes & Tips

- CloudKVS is fault tolerant. The local store includes a 'queue' store which holds the changes that were not yet synchronized.
- For performance reasons it is better to use larger values (made of maps or lists) than to use many small items.
- In B4A it is recommended to initialize ClientKVS in the Starter service.
- The B4J server project can run as is. It accepts a single command line argument which is the port number. If you want to run it on a VPS: https://www.b4x.com/android/forum/threads/60378/#content
- In the examples the auto refresh interval is set to 0.1 (6 seconds). In most cases it is better to use larger intervals (1 minute+).
- The keys and user names are case sensitive.
- On older versions of Anrdoid there is a limit of 2mb per field. You will see the following error if you try to put a larger value: java.lang.IllegalStateException: Couldn't read row 0, col 0 from CursorWindow


Projects

The three client projects and the server project are attached.
If you want to add this feature to an existing project then you need to add:
1. CloudKVS and CallSubUtils modules.
2. The two custom types Item and Task to the main module.
3. The following libraries are required: SQL, RandomAccessFile and HttpUtils2.

The server project depends on jBuilderUtils library. The library is attached.

Development Test Server

You can use this link for the ServerUrl during development:
https://www.b4x.com:51041/cloudkvs
Note that the messages are limited to 100k and more importantly the database is deleted every few days. The clients will stop updating after the database is deleted. You can delete the local database (or uninstall and install the app again) for the clients to work again.
Remember to use unique ids for the user value.

 

[Erel]

1. GMT (ticks are always measured based on GMT).

2. I think that it will be better to use KVS (v2) for the local only store. Later you can switch to Cloud KVS by copying the data from the local store to Cloud KVS.
You need to create your own Get / Put methods that will either access KVS or CKVS.

 

[luke2012]

@Erel I'm thinking to implement CloudKVS within an existing app that store data in solite db tables.

Is it correct if I assume that a single sqlite db tablet could be mapped (ported) to a KVS store?

KVS Key ---> SQLite col. Name
KVS Value ---> SQLite col. Valute

 

[Erel]

You need to read the data from the current database and add it with CKVS.Put. How you map the items depends on your requirements.

The primary key for example can be used as the key and all the records can be stored as the value (with a custom type).

 

[luke2012]

You need to read the data from the current database and add it with CKVS.Put. How you map the items depends on your requirements.

The primary key for example can be used as the key and all the records can be stored as the value (with a custom type).

Ok.
So assume that I have to do a SQLite INSERT statement porting where I have a record primary key set as an auto increment (numeric id) within the SQLite table.
In this case, we assume that the KVS Key must handle (emulate) the SQLite auto increment ID.

Within an INSERT (Put) statement I have to compute the new auto increment ID like : KVSObj.size + 1 ?
In this way (for example) I'm sure that the new numeric id (key) is unique because the KVSObj.size + 1 element doesn't exist within the KVS store. I'm right?

 

[Erel]

You shouldn't write directly to the SQLite database. You need to use ckvs.Put to insert data. Otherwise it will not work properly.

 

[luke2012]

You shouldn't write directly to the SQLite database. You need to use ckvs.Put to insert data. Otherwise it will not work properly.

@Erel I agree with you and I understand it, so assuming that my custom type represent my data record (for example) in order to insert a new element (with a unique key) in the ckvs I have to write:

private myUniqueKey as int = ckvs.size + 1
private MyVal as object = MyCustType
ckvs.put (myUniqueKey, MyVal)

Is it correct ?

 

[Erel]

You are missing the user field (see the example).

A key / value store works better with a real, meaningful key. Your solution can only work if the data (related to the same user) will never be updated by multiple clients at the same time.

 

[luke2012]

You are missing the user field (see the example).

A key / value store works better with a real, meaningful key. Your solution can only work if the data (related to the same user) will never be updated by multiple clients at the same time.

In other words within the kvs cloud I cannot write (.put) a kvs store using multiple clients at the same time using the same user field. It is correct?

 

[Erel]

It is correct?

No.
The whole point of CloudKVS is to synchronize between multiple clients.

 

[luke2012]

You are missing the user field (see the example).

A key / value store works better with a real, meaningful key. Your solution can only work if the data (related to the same user) will never be updated by multiple clients at the same time.

@Erel, thank you for your helpful answer.
In order to let my solution works fine within a scenario With 2 clients that runs a Put at the same time using the same user field (for ex. "User1) I Need to generate a unique key (unique key across All clients that run the Put using the same user field).

Client1
ckvs.put ("User1", Key1, aValue)

Client2
ckvs.put ("User1", Key2, aValue)

Can I generate a unique key across client1 and client2 also if client1 and client2 runs the Put at the same time ?
Is it possibile?

 

[Erel]

You can use a large random number as the key or to generate a random GUID: https://www.b4x.com/android/forum/threads/how-do-you-generate-a-guid.16195/#post-92012

 

[luke2012]

Using CloudKVS can I Put an image (blob) as value within a custom type ?

 

[Erel]

Yes. You can include arrays of bytes in the value.

 

[luke2012]

A little true / false to understand better the KVSCloud solution

1) Each KVSClient handle a "private" store (SQLite db). So ckvs1 handle "store1" and ckvs2 handle "store2" that are two indipendent SQLite db on the device storage.
2) When I put an item (key / value) for a specific user (ex. "User1") I cannot change / switch the user field (ex. from "User1" to "User2") after it has been stored into KVSClient store (SQLite db).
3) When the kvs (server) store is lost (ex. storage failure) all the data that the user putted in the store is lost.
4) The KVSCloud Server can be accessible on internet by KVSClients using a Dynamic DNS service like DUCK DNS

 

[Erel]

I think that you didn't understand it correctly.

1) You should use a single client on every app.
2) Each field is uniquely determined based on the user and key fields. You cannot modify existing items. You can remove and add (or replace) items.
3) The connection can be lost and nothing bad will happen (don't be confused from the online demo). The server database should never be lost. Make a backup if you need.
4) This is not really related to CloudKVS. If the computer doesn't have a static ip address then you need to use a service such as duck dns to manage the ip address.

 

[luke2012]

You need to read the data from the current database and add it with CKVS.Put. How you map the items depends on your requirements.

The primary key for example can be used as the key and all the records can be stored as the value (with a custom type).

I understood your example. This example is valid for one specific table.
I think that if I have more than one table I need to add the table name info within the key (as primary key prefix for example) in order to bind all the values (records) to the corresponding table. It is correct?

 

[Erel]

There are no tables. It is a key/value store. It is not a SQL database (technically it is based on a SQL database however the API exposed is different).

https://en.wikipedia.org/wiki/Key-value_database

 

[luke2012]

There are no tables. It is a key/value store. It is not a SQL database (technically it is based on a SQL database however the API exposed is different).

https://en.wikipedia.org/wiki/Key-value_database

Ok @Erel, thanks for your info. Now It's quite clear.

But my great doubt is:

If I have to do a porting of an old B4A Android App that works with a local Database (many SQL query and some SQL tables) and within this database I have more than one data sets (orders table, catalog table...) to a distributed data platform (where all app instances within all devices share the same App data).
Which is the most advantageous solution considering a balance between development cost (time) and development complexity (porting and project maintenance):

1) KVSCloud
2) Server Data Collection Solution (https://www.b4x.com/android/forum/t...olution-device-desktop-and-web-reports.37254/)

P.S.
Considering that my target is to obtain a cheap development total cost (for my customer)

 

[Erel]

As I see it you have two possible options: KVSCloud or RDC. The advantage of KVSCloud is that it works both offline and online (with built-in synchronization). The advantage of RDC is that you have the full SQL power.

There is no better solution. It depends on your requirements.

 

[Fernando Melo]

As I see it you have two possible options: KVSCloud or RDC. The advantage of KVSCloud is that it works both offline and online (with built-in synchronization). The advantage of RDC is that you have the full SQL power.

There is no better solution. It depends on your requirements.

Suppose i need to sync local off-line data with RDC. Using a flag for deleted rows and date-time columns for sync issues is a way? Or KVSCloud should be better for this scenario? I know that kvs is different. But i see that KVSCloud can act as a transaction log.