Android Tutorial [B4X] CloudKVS - synchronized key / value store

CloudKVS solves a difficult and common problem. The user needs to work with online data, however as this is a mobile app we cannot assume that the device will always be connected to the remote server.

With CloudKVS the app always works with a local database. If the device can connect to the remote server then the local store will be synchronized with the online store.

The store is implemented as a key/value store. It is similar to a persistent Map collection.
The values are serialized with B4XSerializator.
The following types are supported as values:
Lists, Maps, Strings, primitives (numbers), user defined types and arrays (only arrays of bytes and arrays of objects are supported).
Custom types should be declared in the main module.
Including combinations of these types (a list that holds maps for example).

This is a cross platform solution. The clients can be implemented with B4A, B4i or B4J and the data can be shared between the different platforms.
Note that ClientKVS class source code is exactly the same on all three platforms.

Working with ClientKVS is almost as simple as working with a regular Map.

User field

To allow more flexibility items are grouped by a "user" field.
For example:

ckvs.Put("User1", "Key1", 100)
Log(ckvs.Get("User1", "Key1")) '100
Log(ckvs.GetDefault("User2", "Key1", 0)) '0 because User2/Key1 is different than User1/Key1

The synchronization (from the remote store to the local store) is based on the user field.
The SetAutoRefresh method sets the user(s) that will be fetched from the remote store.

For example if we want to auto-synchronize the data of "User1":

ckvs.SetAutoRefresh(Array("User1"), 5)

You can pass multiple users in the array. The second parameter is the interval measured in minutes.
This means that the client will check for new data every 5 minutes.
New data in this case means data that was uploaded from other clients.
The NewData event is raised when new data was fetched from the remote server.
Note that auto-refresh is not relevant for local updates. Local updates are uploaded immediately (if possible).

Multiple clients can use the same 'user name'.

Defaults

GetDefaultAndPut:

Dim Score As Int = ckvs.GetDefaultAndPut ("User1", "Score", 0)

If there is a User1/Score value in the local store then it will be returned. Otherwise it will return 0 and also put 0 in the store.
This is useful as it allows us later in the program to get the score with: ckvs.Get("User1", "Score").
Defaults put in the store are treated specially. If there is already a non-default value in the remote store then the default value will not overwrite the non-default value. The non-default value will be synchronized to the local store once there is a connection.

Notes & Tips

- CloudKVS is fault tolerant. The local store includes a 'queue' store which holds the changes that were not yet synchronized.
- For performance reasons it is better to use larger values (made of maps or lists) than to use many small items.
- In B4A it is recommended to initialize ClientKVS in the Starter service.
- The B4J server project can run as is. It accepts a single command line argument which is the port number. If you want to run it on a VPS: https://www.b4x.com/android/forum/threads/60378/#content
- In the examples the auto refresh interval is set to 0.1 (6 seconds). In most cases it is better to use larger intervals (1 minute+).
- The keys and user names are case sensitive.
- On older versions of Anrdoid there is a limit of 2mb per field. You will see the following error if you try to put a larger value: java.lang.IllegalStateException: Couldn't read row 0, col 0 from CursorWindow


Projects

The three client projects and the server project are attached.
If you want to add this feature to an existing project then you need to add:
1. CloudKVS and CallSubUtils modules.
2. The two custom types Item and Task to the main module.
3. The following libraries are required: SQL, RandomAccessFile and HttpUtils2.

The server project depends on jBuilderUtils library. The library is attached.

Development Test Server

You can use this link for the ServerUrl during development:
https://www.b4x.com:51041/cloudkvs
Note that the messages are limited to 100k and more importantly the database is deleted every few days. The clients will stop updating after the database is deleted. You can delete the local database (or uninstall and install the app again) for the clients to work again.
Remember to use unique ids for the user value.

 

[miquelr]

Please use [code]code here...[/code] tags when posting code.

ClientKVS will not start any activity. It will only call the NewData event in the Starter service. You should check your code and look for any CallSubDelayed call that is causing the Main activity to start.

Hi Erel

In the Activity_Pause of the Main module, it had code to update a remote SQL database when the program was closed:

Activity_Pause sub (UserClosed As Boolean)
   Dim ct As String ="UPDATE usuaris set idLastDisp ='" & deviceId & "', idCurrentDisp = 'ninguno1' WHERE GuidId = '" & idUser &"'"
   job5.Initialize("Job5", Me)
  job5.Download2(sqlSvr, Array As String("query",".....

It is obvious that the code is not correct in Activity_Pause.
Removing this code solves the problem.

Thanks for you fast support and congratulations for the outstanding work of your team, and the contribution of the most of members of the forum.

( I beg your pardon for my googlenglish )

 

[luke2012]

Hi to all. What is the path of local store ?

 

[Erel]

You can see the path in ClientKVS.Initialize sub.

 

[luke2012]

Can I use a PI 3 as CloudKVS Server ?
I can run the compiled Server jar as it is on the PI 3 ?

Thanks in advance for the reply.

 

[Erel]

Yes.

 

[jareal]

Hi,

I have App1 that uses CloudKVS and has an average of 4 PUTs per hour.
Then developed App2, totally independent of App1, that also uses CloudKVS and has about 10 PUTs per hour.
Both apps communicate with the same Server.

Both apps use the original unchanged ClientKVS module. So, both use 'db.db' as local store.

When I installed the two apps on the same device, both continued to work without a problem.

But I am wondering if my approach is correct... or I will have problems when the update rate of the apps increase...

 

[Erel]

The local store name is not important as it is stored in the app's internal folder.

You can use the same CloudKVS server to support multiple applications. However you need to make sure that the <user, key> tuple is unique or data will be synchronized between the apps.

CloudKVS can support much higher rates (many puts per second).

 

[Dey]

Hello to all,
They can asynchronous CloudKVS with a lot of data
Thank you

 

[miquelr]

Hi,
On a computer in the office I tried the example of CloudKVS smoothly. Now I'm trying on another computer in my house and I can not do it.
During compilation I get this message:
***
B4A version: 6.30
Parsing code. (0.01s)
Compiling code. (1.32s)
Compiling code layouts. (0.00s)
Organizing libraries. (0.00s)
Generating R file. (0.11s)
Compiling code debugger engine. (1.21s)
Compiling Java code generated. Error
javac 1.8.0_111
src \ b4a \ example \ starter.java: 84: error: can not find symbol
super.onTaskRemoved (rootIntent);
^
symbol: method onTaskRemoved (Intent)
Note: src \ b4a \ example \ clientkvs.java use unchecked or unsafe operations.
Note: Recompile With -Xlint: unchecked for details.
1 error
*****
First I tried with jdk1.7.0_45 and then width jdk1.8.0_111 getting the same error.
What I can do?

 

[Erel]

You need to use a newer version of android.jar (Tools - Configure Paths).

They can asynchronous CloudKVS with a lot of data

The data is sent in the background. It is already asynchronous.

 

[luke2012]

Hi to all,
My problem is that each time the app is started, I check for a specific item and I create it if not found (in local store).

But when a user remove the app all the related local data is removed (allitems = 0).
So the item is present on the server store but not in the local store and my program can't find it and create a duplicated item (thinking at my app logic).

See the details...

Sub Service_Create

       'This code run just after the app is removed

       ckvs.Initialize(Me, "ckvs", Main.ServerUrl)
       Private allitems As Map = ckvs.GetAll ("user1")

       'here I check for the item but the local store contains nothing (it is on the server store in this moment)...
end sub

So in this case the local store isn't in sync with the server store and my program cannot find this item (that already exist within the cloud) and it create a new item (but it doesn't) within the local store and than it is replicated to server (in my app logic it is an item duplication).
So the final result is that I have two items with the same data within the cloud.

So the point is that I don't known how to force a local store sync (just after ckvs.Initialize) within the Service_Create (Starter) to avoid this problem.
Note that this problem exist only when a user remove and reinstall the app (cKVS client).

Thanks in advance for your reply.

 

[Erel]

This is exactly the purpose of GetDefaultAndPut.

It gets the current value. If the value doesn't exist then it will return the default value. The default value will be synced with the server. The server will first check whether there is an existing item. If there is one then it will not update the value.

 

[luke2012]

This is exactly the purpose of GetDefaultAndPut.

It gets the current value. If the value doesn't exist then it will return the default value. The default value will be synced with the server. The server will first check whether there is an existing item. If there is one then it will not update the value.

So you are suggesting me to use the GetDefaultAndPut instead of Put to avoid this situation.

 

[Erel]

Not exactly. GetDefaultAndPut should only be used for default values.

 

[luke2012]

Not exactly. GetDefaultAndPut should only be used for default values.

Your example of GetDefaultAndPut:
"If there is a User1/Score value in the local store then it will be returned. Otherwise it will return 0 and also put 0 in the store."

In my case the item (value) is a type and I check (within the Starter/Service_Create) it using a field within the type (user gmail).
So if there is an item with the current user GMAIL it doesn't create the item. If this item isn't found (item with a specific gmail within the type) the item will be created.

Each user item (with current user GMAIL) is generated using large number key like bbb80701-bebe-4caa-a0d2-5678220939b4 (see your post #31).
So the problem is that (in the scenario described above) I don't have this data in the local store (only on the server) when the local store is purged (app removed).

So the point is that within the local store (cloudKVS client) I don't have the big number key and all related data (I known only the user gmail from the AccountManager).
So the final result is that my program within the sarter don't find the user item with the current user gmail (because in that moment is only on the server store) and create a new (duplicated item with the same data) with a differrent large number ID (key) like bbb80701-bebe-4caa-a0d2-5678220939b5.

So I have data redundancy and two different unique key related to the same user.

 

[Erel]

You can use GetDefaultAndPut with the new GUID (large number key).
If it returns the new GUID then you need to call RefreshUser and wait for NewData event. Get the GUID again in NewData event. It will be the correct one.

 

[luke2012]

You can use GetDefaultAndPut with the new GUID (large number key).
If it returns the new GUID then you need to call RefreshUser and wait for NewData event. Get the GUID again in NewData event. It will be the correct one.

Please Erel can you show me a example how to check it within the NewData event ?

 

[Erel]

Sub ckvs_NewData
 Dim NewGuid As String = Starter.ckvs.Get(user, "guid")
 If NewGuid <> Starter.TempGuid Then
   'You need to switch to the new guid you received from the server.
 End If
End Sub

 

[luke2012]

Sub ckvs_NewData
Dim NewGuid As String = Starter.ckvs.Get(user, "guid")
If NewGuid <> Starter.TempGuid Then
   'You need to switch to the new guid you received from the server.
End If
End Sub

thanks Erel. Do you confirm that each time I call the RefreshUser the ckvs_NewData event will be raised ?

 

[Erel]

It is expected to be raised in this case as you have updated the local store.