Android Question Remote/local db syncing, best practice

james_sgp · Jan 21, 2023

Hi, im working on a large scale app (20,000+ users) for a client which is going to have a fair amount of data in a remote DB.

I'm getting a mind block on how to efficiently get the new entries from the remote DB, to insert into the local db.

I was initally just accessing data remotely, but locations with poor internet are causing the app to crash/be unresponsive. So want to move towards moving some of the data locally.

Clients server doeant have VPS so im running thro PHP, which is all working well so far.

Tips and suggestions, would be greatly appreciated.

Thanks, James

walterf25 · Jan 21, 2023

Would be worth taking a look at this post
Thread 'Syncing local databases using PHP to access a database on a hosted server' https://www.b4x.com/android/forum/t...-access-a-database-on-a-hosted-server.142126/

james_sgp · Jan 21, 2023

walterf25 said:
Would be worth taking a look at this post
Thread 'Syncing local databases using PHP to access a database on a hosted server' https://www.b4x.com/android/forum/t...-access-a-database-on-a-hosted-server.142126/

Yeah, this is my current stratergy:

Routine in 'Starter'

> get all records with timestamp greater than 'last sync' (KVS field)
> if record exists, update it
> if record doesnt exist, insert it
> if all sucessful, put above timestamp into 'last sync'

Call the routine at different intervals, depending on table data importance

emexes · Jan 21, 2023

How large is the remote database? How frequently does it change? Is it time-critical, or is an hourly or daily update good enough?

emexes · Jan 21, 2023

james_sgp said:
> get all records with timestamp greater than 'last sync' (KVS field)
> if record exists, update it
> if record doesnt exist, insert it
> if all sucessful, put above timestamp into 'last sync'

No deletions?

james_sgp · Jan 21, 2023

emexes said:
No deletions?

Im only downloading the datavthat doeant change much, like ads data (can change hourly), products for sale, etc. Other data like user profile i'm putting in a B4XOrderedmap, so its easy/fast to access it.

Mmmm, deletions...there shouldn't be any but will build in the ability jiat on case.

The whole database could be many 10's of Mb, as its also recording tracking information of 10,000+ users on 30 second intervals when they are inside a building.

emexes · Jan 21, 2023

james_sgp said:
recording tracking information of 10,000+ users on 30 second intervals

That sounds entirely achievable with data held in memory and using UDP packets. ?

You might need to keep an early and close eye on throughput capacity, rather than risk ending up with a beautiful system that works great with 100 simulated users but dies when faced with 10,000.

Erel · Jan 22, 2023

james_sgp said:
> get all records with timestamp greater than 'last sync' (KVS field)
> if record exists, update it
> if record doesnt exist, insert it
> if all sucessful, put above timestamp into 'last sync'

All of this and more is already implemented in CloudKVS: https://www.b4x.com/android/forum/threads/63536/#content

Android Question Remote/local db syncing, best practice

james_sgp

Active Member

walterf25

Expert

james_sgp

Active Member

emexes

Expert

emexes

Expert

james_sgp

Active Member

emexes

Expert

Erel

B4X founder

Similar Threads

Android Question Remote/local db syncing, best practice

Active Member

Expert

Active Member

Expert

Expert

Active Member

Expert

B4X founder

Similar Threads

Privacy & Transparency

Privacy & Transparency