How to process realtime data flow ?

HI, All

I'm making an app that show the charts of realtime data from sensors. It's educational project for schools.
Some sensors are slow, 1-5 point per second. But most sensors are fast, 100 points per second, as simple ADC measurement is fast.
It should be up to 12 sensors lines on one chart.
I store points into SQLite db.
Minimal time step on the chart should be 0.1 second.

Problem is that fast sensors in several seconds flood the db with thousands of records and charting is getting slower each second. As all info should be got by SQL requests group from the same db.
If to set the time step of X-axes = 1 sec - no problem to update the chart, but 0.1 s step is impossible - fast chart moving is getting slower and PC is freezing. RAM consumption from 300MB per process is going 1GB+...

How to process such realtime data of such volumes correctly ? Without visual interface freeze.

MQTT and Update on Message receive? You can also reduce the sampling sent on the very fast sensors.

It's local app, even without Inet-connection, working with COM-ports.

peacemaker said:

How to process such realtime data of such volumes correctly ? Without visual interface freeze.

Click to expand...

Sounds like a job for... Super Array!



Either limited to say 60 seconds capturing, which at 100 x 16-bit samples per second = 12 kB per channel = 80 channels per MB of RAM.

Continuous recording is usually done by treating the buffer as a ring (ie circular),
and incrementing the start/end pointer as each new sample arrives and is put into the buffer,
rather than moving all 5999 samples along by one to make space for each new sample.

I've used RRDtool for something like that a couple of times. It's a nice tool, for its use. Perhaps it could help you somehow.