Android Tutorial [B4X] CloudKVS - synchronized key / value store

CloudKVS solves a difficult and common problem. The user needs to work with online data, however as this is a mobile app we cannot assume that the device will always be connected to the remote server.

With CloudKVS the app always works with a local database. If the device can connect to the remote server then the local store will be synchronized with the online store.

The store is implemented as a key/value store. It is similar to a persistent Map collection.
The values are serialized with B4XSerializator.
The following types are supported as values:
Lists, Maps, Strings, primitives (numbers), user defined types and arrays (only arrays of bytes and arrays of objects are supported).
Custom types should be declared in the main module.
Including combinations of these types (a list that holds maps for example).

This is a cross platform solution. The clients can be implemented with B4A, B4i or B4J and the data can be shared between the different platforms.
Note that ClientKVS class source code is exactly the same on all three platforms.

Working with ClientKVS is almost as simple as working with a regular Map.

User field

To allow more flexibility items are grouped by a "user" field.
For example:

ckvs.Put("User1", "Key1", 100)
Log(ckvs.Get("User1", "Key1")) '100
Log(ckvs.GetDefault("User2", "Key1", 0)) '0 because User2/Key1 is different than User1/Key1

The synchronization (from the remote store to the local store) is based on the user field.
The SetAutoRefresh method sets the user(s) that will be fetched from the remote store.

For example if we want to auto-synchronize the data of "User1":

ckvs.SetAutoRefresh(Array("User1"), 5)

You can pass multiple users in the array. The second parameter is the interval measured in minutes.
This means that the client will check for new data every 5 minutes.
New data in this case means data that was uploaded from other clients.
The NewData event is raised when new data was fetched from the remote server.
Note that auto-refresh is not relevant for local updates. Local updates are uploaded immediately (if possible).

Multiple clients can use the same 'user name'.

Defaults

GetDefaultAndPut:

Dim Score As Int = ckvs.GetDefaultAndPut ("User1", "Score", 0)

If there is a User1/Score value in the local store then it will be returned. Otherwise it will return 0 and also put 0 in the store.
This is useful as it allows us later in the program to get the score with: ckvs.Get("User1", "Score").
Defaults put in the store are treated specially. If there is already a non-default value in the remote store then the default value will not overwrite the non-default value. The non-default value will be synchronized to the local store once there is a connection.

Notes & Tips

- CloudKVS is fault tolerant. The local store includes a 'queue' store which holds the changes that were not yet synchronized.
- For performance reasons it is better to use larger values (made of maps or lists) than to use many small items.
- In B4A it is recommended to initialize ClientKVS in the Starter service.
- The B4J server project can run as is. It accepts a single command line argument which is the port number. If you want to run it on a VPS: https://www.b4x.com/android/forum/threads/60378/#content
- In the examples the auto refresh interval is set to 0.1 (6 seconds). In most cases it is better to use larger intervals (1 minute+).
- The keys and user names are case sensitive.
- On older versions of Anrdoid there is a limit of 2mb per field. You will see the following error if you try to put a larger value: java.lang.IllegalStateException: Couldn't read row 0, col 0 from CursorWindow


Projects

The three client projects and the server project are attached.
If you want to add this feature to an existing project then you need to add:
1. CloudKVS and CallSubUtils modules.
2. The two custom types Item and Task to the main module.
3. The following libraries are required: SQL, RandomAccessFile and HttpUtils2.

The server project depends on jBuilderUtils library. The library is attached.

Development Test Server

You can use this link for the ServerUrl during development:
https://www.b4x.com:51041/cloudkvs
Note that the messages are limited to 100k and more importantly the database is deleted every few days. The clients will stop updating after the database is deleted. You can delete the local database (or uninstall and install the app again) for the clients to work again.
Remember to use unique ids for the user value.

 

[luke2012]

Hi @Erel.
First of all thanks for your great support about B4X CluodKVS

Is there a way to known all the users (es. User1, User2, User3...) within a CloudKVS store ?
I mean all the users that stored at least an item within the CloudKVS store.

In this way, different apps that deal within the CloudKVS can known all the users within a CloudKVS store.

For example, A "Supervisor" app that have to collect (display) all the items putted by all the users within a CloudKVS.

Thanks in advance for your reply.

 

[Erel]

This line in the server code creates the main table:

sql.ExecNonQuery("CREATE TABLE data (user TEXT, key TEXT, value BLOB, id INTEGER, time INTEGER, PRIMARY KEY (user, key))")

You can access all the users and keys from this table.

 

[luke2012]

This line in the server code creates the main table:

sql.ExecNonQuery("CREATE TABLE data (user TEXT, key TEXT, value BLOB, id INTEGER, time INTEGER, PRIMARY KEY (user, key))")

You can access all the users and keys from this table.

Thanks @Erel! This info helped me very much.

If I want to purge all data (ex. within my dev environment DB) I can run:

sql.ExecNonQuery("delete from data")

Is this correct for the CloudKVS env?

 

[Erel]

Is this correct for the CloudKVS env?

No, it will cause a mess. You shouldn't modify the data from the server. You should instead create a client and set the values to Null. You can use a B4J client that runs on the same computer as the server.

 

[luke2012]

No, it will cause a mess. You shouldn't modify the data from the server. You should instead create a client and set the values to Null. You can use a B4J client that runs on the same computer as the server.

Ok. The client could be also an Android client (B4A client)?

 

[Erel]

Yes.

 

[wimpie3]

1. Updating records is not possible? You can only delete/add them?
2. SSL is not supported on the server side?

 

[Erel]

1. Updating is possible. When you add an item it replaces the previous item with the same <user, key>.

2. Why do you think so? It is supported.

 

[wimpie3]

2. But how? How can you add a certificate?

Will this lib still work when the user has a wrong date or time on their mobile?

 

[Erel]

2) https://www.b4x.com/android/forum/pages/results/?query=b4j+ssl
For further discussion about SSL please start a new thread in B4J questions forum (as it is not specific to cloudkvs).

Will this lib still work when the user has a wrong date or time on their mobile?

It will work but it will have an effect. See the code in DB.AddItem. The time field is used to in cases of conflict to determine which entry is more recent.

 

[luke2012]

You can use a large random number as the key or to generate a random GUID: https://www.b4x.com/android/forum/threads/how-do-you-generate-a-guid.16195/#post-92012

The Good
@Erel I confirm you that I'm following your suggestion and I'm using a UUID for each new item putted in to the Cloud (works very well).
This UUID solve the problem of data updated by multiple clients at the same time (as you suggested) and I can have an solid unique item id.

The problem:
My problem now is that I can't display that UUID (very very long and impossible to remember) to the end user in order to let him identify the item within an item list and probably I need a progressive number to display to user within the App (ex. item n.1, item n.2 etc...).

Some custom item have a item name different from each item within the item list that I show within the app, but there are some custom items that doesn't have a name, but also some data (numeric data for example) but the end user cannot identify items because there haven't name or number (he can't see item n. 1 etc).

Any Idea?

Thanks in advance for your precious help.

 

[Erel]

@luke2012 please start a new thread for this question.

 

[luke2012]

You need to first convert the file to an array of bytes.
This can be done with:

Dim b() As Byte = Bit.InputStreamToBytes(File.OpenInput(<your db file>))

This statement convert a image file (.jpg, .png ...) into a byte array?

 

[Erel]

@luke2012 please start a new thread for this question.

 

[JakeBullet70]

Hi all.

I have everything compiled and running on my VPS but am getting a connection refused error in b4a. I checked the nohup.out on the server and am getting this error. Any ideas?

Thanks.

 

[Erel]

version 52.0 = Java 8. You need to use Java 8 on your server. Note that you don't need to install it. Just copy it: [server] Run a Server on a VPS

 

[luke2012]

Hi to all,
I have an issue putting .jpg images within the CloudKVS.
I explain my issue with the follwing examples:

1) The following code works fine with the CloudKVS

public Sub File2Byte (Dir As String, FileName As String) As Byte()
 Return Bit.InputStreamToBytes(File.OpenInput(Dir, FileName))
End Sub

myCustType.Pic = File2Byte (File.DirAssets, "13kb.png") 'Pic is a byte array declared within the custom type

starter.ckvs.Put("user1", "key1", myCustType)

2) The following code crash the CloudKVS Client (see the attached log)
Note that is the same code but with a 24bit jpg image (4160 x 3120 px - 3,42 MB)

public Sub File2Byte (Dir As String, FileName As String) As Byte()
 Return Bit.InputStreamToBytes(File.OpenInput(Dir, FileName))
End Sub

myCustType.Pic = File2Byte (File.DirAssets, "3mb.jpg") 'Pic is a byte array declared within the custom type

starter.ckvs.Put("user1", "key1", myCustType)

My suspect is the file size.
How to workaround this issue and allow to load high res jpg?

 

[Erel]

Note that it is better to post the error message as text instead of a screenshot (right click to copy).

There is a 2mb limit to SQLite records on older versions of Android: http://stackoverflow.com/questions/...ception-couldnt-read-row-0-col-0-from-cursorw

 

[luke2012]

Note that it is better to post the error message as text instead of a screenshot (right click to copy).

There is a 2mb limit to SQLite records on older versions of Android: http://stackoverflow.com/questions/...ception-couldnt-read-row-0-col-0-from-cursorw

So the only solution in this case is to check the image size (< 2mb) before execute the put ?

 

[Erel]

You can split the array into several chunks.