Android Question Remote/local db syncing, best practice

[james_sgp]

Hi, im working on a large scale app (20,000+ users) for a client which is going to have a fair amount of data in a remote DB.

I'm getting a mind block on how to efficiently get the new entries from the remote DB, to insert into the local db.

I was initally just accessing data remotely, but locations with poor internet are causing the app to crash/be unresponsive. So want to move towards moving some of the data locally.

Clients server doeant have VPS so im running thro PHP, which is all working well so far.

Tips and suggestions, would be greatly appreciated.

Thanks, James

 

[walterf25]

Would be worth taking a look at this post
Thread 'Syncing local databases using PHP to access a database on a hosted server' https://www.b4x.com/android/forum/t...-access-a-database-on-a-hosted-server.142126/

 

[james_sgp]

Would be worth taking a look at this post
Thread 'Syncing local databases using PHP to access a database on a hosted server' https://www.b4x.com/android/forum/t...-access-a-database-on-a-hosted-server.142126/

Yeah, this is my current stratergy:

Routine in 'Starter'

> get all records with timestamp greater than 'last sync' (KVS field)
> if record exists, update it
> if record doesnt exist, insert it
> if all sucessful, put above timestamp into 'last sync'

Call the routine at different intervals, depending on table data importance

 

[emexes]

How large is the remote database? How frequently does it change? Is it time-critical, or is an hourly or daily update good enough?

 

[emexes]

> get all records with timestamp greater than 'last sync' (KVS field)
> if record exists, update it
> if record doesnt exist, insert it
> if all sucessful, put above timestamp into 'last sync'

No deletions?

 

[james_sgp]

No deletions?

Im only downloading the datavthat doeant change much, like ads data (can change hourly), products for sale, etc. Other data like user profile i'm putting in a B4XOrderedmap, so its easy/fast to access it.

Mmmm, deletions...there shouldn't be any but will build in the ability jiat on case.

The whole database could be many 10's of Mb, as its also recording tracking information of 10,000+ users on 30 second intervals when they are inside a building.

 

[emexes]

recording tracking information of 10,000+ users on 30 second intervals

That sounds entirely achievable with data held in memory and using UDP packets. ?

You might need to keep an early and close eye on throughput capacity, rather than risk ending up with a beautiful system that works great with 100 simulated users but dies when faced with 10,000.

 

[Erel]

> get all records with timestamp greater than 'last sync' (KVS field)
> if record exists, update it
> if record doesnt exist, insert it
> if all sucessful, put above timestamp into 'last sync'

All of this and more is already implemented in CloudKVS: https://www.b4x.com/android/forum/threads/63536/#content