Android Tutorial [B4X] CloudKVS - synchronized key / value store

CloudKVS solves a difficult and common problem. The user needs to work with online data, however as this is a mobile app we cannot assume that the device will always be connected to the remote server.

SS-2016-02-15_13.23.32.png


With CloudKVS the app always works with a local database. If the device can connect to the remote server then the local store will be synchronized with the online store.

The store is implemented as a key/value store. It is similar to a persistent Map collection.
The values are serialized with B4XSerializator.
The following types are supported as values:
Lists, Maps, Strings, primitives (numbers), user defined types and arrays (only arrays of bytes and arrays of objects are supported).
Custom types should be declared in the main module.
Including combinations of these types (a list that holds maps for example).

This is a cross platform solution. The clients can be implemented with B4A, B4i or B4J and the data can be shared between the different platforms.
Note that ClientKVS class source code is exactly the same on all three platforms.

Working with ClientKVS is almost as simple as working with a regular Map.

User field

To allow more flexibility items are grouped by a "user" field.
For example:
B4X:
ckvs.Put("User1", "Key1", 100)
Log(ckvs.Get("User1", "Key1")) '100
Log(ckvs.GetDefault("User2", "Key1", 0)) '0 because User2/Key1 is different than User1/Key1

The synchronization (from the remote store to the local store) is based on the user field.
The SetAutoRefresh method sets the user(s) that will be fetched from the remote store.

For example if we want to auto-synchronize the data of "User1":
B4X:
ckvs.SetAutoRefresh(Array("User1"), 5)
You can pass multiple users in the array. The second parameter is the interval measured in minutes.
This means that the client will check for new data every 5 minutes.
New data in this case means data that was uploaded from other clients.
The NewData event is raised when new data was fetched from the remote server.
Note that auto-refresh is not relevant for local updates. Local updates are uploaded immediately (if possible).

Multiple clients can use the same 'user name'.

Defaults

GetDefaultAndPut:
B4X:
Dim Score As Int = ckvs.GetDefaultAndPut ("User1", "Score", 0)
If there is a User1/Score value in the local store then it will be returned. Otherwise it will return 0 and also put 0 in the store.
This is useful as it allows us later in the program to get the score with: ckvs.Get("User1", "Score").
Defaults put in the store are treated specially. If there is already a non-default value in the remote store then the default value will not overwrite the non-default value. The non-default value will be synchronized to the local store once there is a connection.

Notes & Tips

- CloudKVS is fault tolerant. The local store includes a 'queue' store which holds the changes that were not yet synchronized.
- For performance reasons it is better to use larger values (made of maps or lists) than to use many small items.
- In B4A it is recommended to initialize ClientKVS in the Starter service.
- The B4J server project can run as is. It accepts a single command line argument which is the port number. If you want to run it on a VPS: https://www.b4x.com/android/forum/threads/60378/#content
- In the examples the auto refresh interval is set to 0.1 (6 seconds). In most cases it is better to use larger intervals (1 minute+).
- The keys and user names are case sensitive.
- On older versions of Anrdoid there is a limit of 2mb per field. You will see the following error if you try to put a larger value: java.lang.IllegalStateException: Couldn't read row 0, col 0 from CursorWindow


Projects

The three client projects and the server project are attached.
If you want to add this feature to an existing project then you need to add:
1. CloudKVS and CallSubUtils modules.
2. The two custom types Item and Task to the main module.
3. The following libraries are required: SQL, RandomAccessFile and HttpUtils2.

The server project depends on jBuilderUtils library. The library is attached.

Development Test Server

You can use this link for the ServerUrl during development:
https://www.b4x.com:51041/cloudkvs
Note that the messages are limited to 100k and more importantly the database is deleted every few days. The clients will stop updating after the database is deleted. You can delete the local database (or uninstall and install the app again) for the clients to work again.
Remember to use unique ids for the user value.
 

Attachments

  • B4J_ServerKVS.zip
    2.8 KB · Views: 2,236
  • B4i_ClientKVS.zip
    8.7 KB · Views: 1,341
  • jBuilderUtils.zip
    2.3 KB · Views: 2,771
  • B4J_ClientKVS.zip
    6.4 KB · Views: 1,722
  • B4A_ClientKVS.zip
    11.6 KB · Views: 2,227
Last edited:

luke2012

Well-Known Member
Licensed User
Longtime User
Hi @Erel,
when the server is up all work fine.

But sometimes the server is not available (down or not in wifi range) and I have an app crash with the following error:

"Error sending task: java.io.InterruptedIOException: timeout" (please see the attached log)

Is it possible to allow user to use App also without a server connection using the local App data?

Thanks in advance for your reply.
 

Attachments

  • log_Error_sending_task.txt
    5.4 KB · Views: 460

Erel

B4X founder
Staff member
Licensed User
Longtime User
Is it possible to allow user to use App also without a server connection using the local App data?
Yes. This is the most important feature of CloudKVS.

Which version of B4A are you using? Have you modified the code in ClientKVS example? Are you running it in Release (obfuscation) mode?
 

luke2012

Well-Known Member
Licensed User
Longtime User
1) B4A: I'm using B4A version: 6.00
2) ClientKVS: yes, but only for add my app code without modify the CloudKVS core code (main & class).
3) Compile mode: yes I'm running in Release (obfuscation) mode
4) I have duplicated (copy, paste and renamed) the project folder starting from the original ClientKVS sample project and than I customized it
 
Last edited:

JakeBullet70

Well-Known Member
Licensed User
Longtime User
My little addition to the ClientKVS.bas file.
Took me like 5 minutes to figure out that to delete a key you use the 'put' function with Null... LOL

B4X:
Public Sub Delete(User As String, key As String)
    Put(User,key,Null)
End Sub
 

luke2012

Well-Known Member
Licensed User
Longtime User
@Erel I have a couple of questions about CloudKVS Server:

1) How to known (programmatically within the client app) if a CloudKVS Client is disconnected from the CloudKVS Server?
2) In this specific case (CloudKVS Server) is it better to distribute the Server using the B4J Packager or only the server .jar executable file?

Thanks in advance for your reply.
 

Erel

B4X founder
Staff member
Licensed User
Longtime User
1. The clients do not maintain an active connection to the server. The clients connect to the server when needed and then disconnect. Job.Success will be false (in JobDone) if it is unable to connect.

2. It is better to distribute the executable jar. The executable is cross platform compatible and doesn't require installation.
 

luke2012

Well-Known Member
Licensed User
Longtime User
1. The clients do not maintain an active connection to the server. The clients connect to the server when needed and then disconnect. Job.Success will be false (in JobDone) if it is unable to connect.

2. It is better to distribute the executable jar. The executable is cross platform compatible and doesn't require installation.

2. The minimun requirement is JAVA jre 8 on the server machine right?
 

luke2012

Well-Known Member
Licensed User
Longtime User
Hi @Erel
Is it possible to access the keys about new items (runtime object that contains the new data) within the "ckvs_NewData" event?
 

Erel

B4X founder
Staff member
Licensed User
Longtime User
This is the code that inserts the new data to the local database (ser_BytesToObject):
B4X:
If Success Then
     Dim items As List = NewObject
     If items.Size > 0 Then
       For Each item1 As Item In items
         InsertItemIntoData(item1, True)
       Next
       sql.ExecNonQueryBatch("getuser")
     End If
You can modify this code and raise another event with the the list of keys (or list of items). Do note that the items were not yet added to the database at this point.

A simpler option is to just get the keys that you are interested in again.
 

luke2012

Well-Known Member
Licensed User
Longtime User
Hi @Erel,
it is possible to know (progammatically) if a new putted KVSCloud item has been synchronized or not?

Thanks in advance for your reply.
 

luke2012

Well-Known Member
Licensed User
Longtime User
DeleteFromQueue is called for each item that was uploaded successfully to the server.

You can get the user and key from the queue table at that point and raise an "ItemWasUploaded (User As String, Key As String)" event.

OK @Erel. I'll check this.

Last thing about Cloud Sync.
Sometime I read this error within the log:

"Error sending task: java.net.SocketTimeoutException: failed to connect to /192.168.0.10 (port 51042) after 30000ms" *

*Where 192.168.0.10 is my server (Cloud) machine

Is there any way to intercept this error and in order to notify the user about the Server connection problem ?
 

miquelr

Member
Licensed User
Longtime User
Hi Erel,
I'm developing an aplicación in B4A with a number of 'activitys'. After add ClientKVS, CallSubUtils and Starter (as is in your example ) -- with kvsCfgMap = Starter.ckvs.GetDefaultAndPut (user, "kvscfgmap" kvsCfgMap) in Main ---, when I switch to another 'activity' (with callSubDelayed2 ...) appears the new activity but in some seconds Main reappears.

I've tried to change in Starter module without success:

Sub ckvs_NewData
' CallSubDelayed(Main, "ckvs_NewData")
' CallSubDelayed(SWUtils, "ckvs_NewData")
SWUtils.ckvs_NewData '
End Sub
SWUtils is a code module with a sub ckvs_NewData
I need your assistance
Miquel
 
Top