Android Tutorial [B4X] CloudKVS - synchronized key / value store

CloudKVS solves a difficult and common problem. The user needs to work with online data, however as this is a mobile app we cannot assume that the device will always be connected to the remote server.

With CloudKVS the app always works with a local database. If the device can connect to the remote server then the local store will be synchronized with the online store.

The store is implemented as a key/value store. It is similar to a persistent Map collection.
The values are serialized with B4XSerializator.
The following types are supported as values:
Lists, Maps, Strings, primitives (numbers), user defined types and arrays (only arrays of bytes and arrays of objects are supported).
Custom types should be declared in the main module.
Including combinations of these types (a list that holds maps for example).

This is a cross platform solution. The clients can be implemented with B4A, B4i or B4J and the data can be shared between the different platforms.
Note that ClientKVS class source code is exactly the same on all three platforms.

Working with ClientKVS is almost as simple as working with a regular Map.

User field

To allow more flexibility items are grouped by a "user" field.
For example:

ckvs.Put("User1", "Key1", 100)
Log(ckvs.Get("User1", "Key1")) '100
Log(ckvs.GetDefault("User2", "Key1", 0)) '0 because User2/Key1 is different than User1/Key1

The synchronization (from the remote store to the local store) is based on the user field.
The SetAutoRefresh method sets the user(s) that will be fetched from the remote store.

For example if we want to auto-synchronize the data of "User1":

ckvs.SetAutoRefresh(Array("User1"), 5)

You can pass multiple users in the array. The second parameter is the interval measured in minutes.
This means that the client will check for new data every 5 minutes.
New data in this case means data that was uploaded from other clients.
The NewData event is raised when new data was fetched from the remote server.
Note that auto-refresh is not relevant for local updates. Local updates are uploaded immediately (if possible).

Multiple clients can use the same 'user name'.

Defaults

GetDefaultAndPut:

Dim Score As Int = ckvs.GetDefaultAndPut ("User1", "Score", 0)

If there is a User1/Score value in the local store then it will be returned. Otherwise it will return 0 and also put 0 in the store.
This is useful as it allows us later in the program to get the score with: ckvs.Get("User1", "Score").
Defaults put in the store are treated specially. If there is already a non-default value in the remote store then the default value will not overwrite the non-default value. The non-default value will be synchronized to the local store once there is a connection.

Notes & Tips

- CloudKVS is fault tolerant. The local store includes a 'queue' store which holds the changes that were not yet synchronized.
- For performance reasons it is better to use larger values (made of maps or lists) than to use many small items.
- In B4A it is recommended to initialize ClientKVS in the Starter service.
- The B4J server project can run as is. It accepts a single command line argument which is the port number. If you want to run it on a VPS: https://www.b4x.com/android/forum/threads/60378/#content
- In the examples the auto refresh interval is set to 0.1 (6 seconds). In most cases it is better to use larger intervals (1 minute+).
- The keys and user names are case sensitive.
- On older versions of Anrdoid there is a limit of 2mb per field. You will see the following error if you try to put a larger value: java.lang.IllegalStateException: Couldn't read row 0, col 0 from CursorWindow


Projects

The three client projects and the server project are attached.
If you want to add this feature to an existing project then you need to add:
1. CloudKVS and CallSubUtils modules.
2. The two custom types Item and Task to the main module.
3. The following libraries are required: SQL, RandomAccessFile and HttpUtils2.

The server project depends on jBuilderUtils library. The library is attached.

Development Test Server

You can use this link for the ServerUrl during development:
https://www.b4x.com:51041/cloudkvs
Note that the messages are limited to 100k and more importantly the database is deleted every few days. The clients will stop updating after the database is deleted. You can delete the local database (or uninstall and install the app again) for the clients to work again.
Remember to use unique ids for the user value.

 

[Erel]

You should either use RDC with your own synchronization solution or CloudKVS. Combining both of them will probably be more complicated.

 

[pesquera]

You can see in the code that all the messages sent are first serialized with B4XSerializator.

Sorry if this post is not related..
Why this serialization is needed?
How can I do to see the "Value" data into the SQLite DB with any db tool editor? (in some kind of legible way)
Where can I read about the B4XSerializator logic? I've never used it and trying to find info without luck, may be my brain is not compiling at this time
Thanks!

 

[Erel]

Why this serialization is needed?

You can only send bytes over the network. Serialization is the process of converting data or objects to bytes. B4XSerializator is very useful as it makes it easy to convert complex objects to bytes (and later convert the bytes back to objects).
Using B4XSerializator also allows us to store (almost) any type of value in the data store. For example the value can be a list of custom types.

For more information about B4XSerializator please start a new thread.

You can open the database with any SQL editor you like. The value field is a serialized blob.

 

[pesquera]

Ok. I understand now.. thanks
I just expected to update some simple text value with my sql editor.. because I need to edit all records from serverdb.db, filtering in the user field
As I can see, I should do some B4J UI for that.. correct?
Is there any way to do the server side with B4A?

 

[Erel]

As I can see, I should do some B4J UI for that.. correct?

Yes.

Is there any way to do the server side with B4A?

No. The server is implemented with jServer library that is not available in B4A.

 

[pesquera]

Do you plan to add encryption methods on the future? (like KeyValueStore 2)

 

[Erel]

No. You can run CloudKVS over SSL and then the traffic will be encrypted.

You can also encrypt the values yourself before you add them to the store.

 

[pesquera]

Is possible to put/get a jpg file as a value? (converting to array of bytes)

 

[Erel]

Yes. You can use PutBitmap and GetBitmap.

 

[pesquera]

Yes. You can use PutBitmap and GetBitmap.

Thank you Erel, copied them from KeyValueStore 2 and it's working perfect

Now, I must put/get a SQLite data file.. Could you please recommend me how to do that?
Should I handle the whole .db file? (if possible)
or, Should I handle each record into a List?
db has less than 1000 records

Thanks

 

[Erel]

You need to first convert the file to an array of bytes.
This can be done with:

Dim b() As Byte = Bit.InputStreamToBytes(File.OpenInput(<your db file>))

 

[luke2012]

I have a couple of questions about KVS Cloud.

1) Can I use array of bytes to store (put) audio file within the KVS store?
2) Is it possibile to set an auto-refresh interval like 0.5 or 1 sec ?

 

[pesquera]

yes. you can

 

[luke2012]

yes. you can

thanks for your answer.
Do you mean Yes for both questions?

 

[pesquera]

thanks for your answer.
Do you mean Yes for both questions?

Yes, just take in mind that the the interval is in minutes
1/60=1 second
1/120=0.50 seconds

 

[luke2012]

Hi @Erel.
Regarding your example:

ckvs.Put("User1", "Key1", 100)
Log(ckvs.Get("User1", "Key1")) '100
Log(ckvs.GetDefault("User2", "Key1", 0)) '0 because User2/Key1 is different than User1/Key1

How I can get a list of all values "putted" (for example) by "User1" ?
I mean with "Get" method I can get always one value (putted by the User1), because the key is a unique value within the KVS.

 

[LucaMs]

Hi @Erel.
Regarding your example:

ckvs.Put("User1", "Key1", 100)
Log(ckvs.Get("User1", "Key1")) '100
Log(ckvs.GetDefault("User2", "Key1", 0)) '0 because User2/Key1 is different than User1/Key1

How I can get a list of all values "putted" (for example) by "User1" ?
I mean with "Get" method I can get always one value (putted by the User1), because the key is a unique value within the KVS.

There is a GetAll method which returns a map:

'Returns a map with the keys and values of the given user.
Public Sub GetAll(user As String) As Map
    Dim res As Map
    res.Initialize
    Dim ser As B4XSerializator
    Dim rs As ResultSet = sql.ExecQuery2("SELECT key, value FROM data WHERE user = ? AND value IS NOT NULL", Array As String(user))
    Do While rs.NextRow
        res.Put(rs.GetString("key"), ser.ConvertBytesToObject(rs.GetBlob("value")))
    Loop
    rs.Close
    Return res
End Sub

 

[luke2012]

There is a GetAll method which returns a map:

'Returns a map with the keys and values of the given user.
Public Sub GetAll(user As String) As Map
    Dim res As Map
    res.Initialize
    Dim ser As B4XSerializator
    Dim rs As ResultSet = sql.ExecQuery2("SELECT key, value FROM data WHERE user = ? AND value IS NOT NULL", Array As String(user))
    Do While rs.NextRow
        res.Put(rs.GetString("key"), ser.ConvertBytesToObject(rs.GetBlob("value")))
    Loop
    rs.Close
    Return res
End Sub

Thank you very much @LucaMs

 

[luke2012]

The method GetAll is simple to use and works very well
Now I can put a value, get a value, replace a value and get all values for a gived user but how to delete a value or delete all values for a given user?

 

[Erel]

See the documentation of the Put command. You need to pass Null as the value to delete an existing key.