Android Question how correct to do long operation in the main thread ?

KMatle

Expert

Licensed User

Longtime User

• Dec 10, 2016

• #2

Do it in the starter service. It's made for that.

 

Upvote 0

peacemaker

Expert

Licensed User

Longtime User

• Dec 10, 2016

• #3

No difference where, the interface must be locked by this process. But i mean - the only way is to use a loop with some condition for interrupting, calcualting persent for the progressbar and using DOEVENTS in some extra condition, to avoid the system "app is not responding" alert ?

 

Upvote 0

M

Mahares

Expert

Licensed User

Longtime User

• Dec 10, 2016

• #4

Did you take a look at sql1.AddNonQueryToBatch and sql1.ExecNonQueryBatch for executing large inserts for instance without tying up the main thread. It is explained in @Erel's tutorial with the following link: https://www.b4x.com/android/forum/threads/sql-tutorial.6736/#content.
It is designed for that purpose in mind.

 

Upvote 0

peacemaker

Expert

Licensed User

Longtime User

• Dec 10, 2016

• #5

Thanks for the prompt.
But the source file is very huge, see the first my post. It's not sure that enought memory to open whole the source file, or collect lots of SQL.AddNonQueryToBatch before INSERT.
So, i'm working line by line.

 

Upvote 0

M

Mahares

Expert

Licensed User

Longtime User

• Dec 10, 2016

• #6

peacemaker said:

But the source file is very huge, see the first my post. It's not sure that enought memory to open whole the source file

Click to expand...

I used it on importing a text file 14 MB in size with 175000 lines into a table on a Galaxy Tab A tablet. It worked.
Can youput a copy of your huge text file ( may be in Dropbox) even with fake data and table structure and give the forum a chance to test it.

 

Upvote 0

KMatle

Expert

Licensed User

Longtime User

• Dec 10, 2016

• #7

You can insert f.e. 1000 rows and then the next 1000 pushed by a timer with 0.5s or so.

 

Upvote 0

peacemaker

Expert

Licensed User

Longtime User

• Dec 10, 2016

• #8

Wow ! How long is your processing 175 K lines ?

 

Upvote 0

M

Mahares

Expert

Licensed User

Longtime User

• Dec 10, 2016

• #9

16 sec on galaxy tab A tablet.

 

Upvote 0

KMatle

Expert

Licensed User

Longtime User

• Dec 10, 2016

• #10

May I ask what you are doing with so many data? Can't be for the user...

 

Upvote 0

peacemaker

Expert

Licensed User

Longtime User

• Dec 11, 2016

• #11

It's commercial request development, i make as requested: first of all a some exported DB in the text file (100 - 200 symbols in 6 fields per line, with "TAB" as separator-symbol) must be imported, and then working with this big database.

Super! Only 16 seconds. At me 3 mins with progressbar for 4 MB file and 28000 lines, if to read line-by-line, parse by RegEx and INSERT into SQLite one by one by DBUtils.
Yes, i can optimise, if without DButils subs, just direct SQLite requests, but, seems, minimal difference will be.

So, now you should understand why my question. Yes, i can try to read whole the file first, maybe 4MB is not a problem, but files will be several times bigger. But the device is a Android based barcode-scanner device with 1GB memory only.

 

Upvote 0

Erel

B4X founder

Staff member

Licensed User

Longtime User

• Dec 11, 2016

• #12

Which part is slow? Reading the file or inserting the data to the database?

 

Upvote 0

peacemaker

Expert

Licensed User

Longtime User

• Dec 11, 2016

• #13

Hmmmm, correct question - it needs to check

 

Upvote 0

RandomCoder

Well-Known Member

Licensed User

Longtime User

• Dec 11, 2016

• #14

How often are you trying to update the progressbar using DoEvents? This will have a significant effect.

 

Upvote 0

peacemaker

Expert

Licensed User

Longtime User

• Dec 11, 2016

• #15

HA ! Reading file line by line without INSERTing into SQLite - is just 6 seconds for 28000 lines !
Progressbar is updated each 200 records, if 500 - it's not confortable, delaying\hangs for the user interface.

So - it needs to improve SQLite inserting

 

Upvote 0

peacemaker

Expert

Licensed User

Longtime User

• Dec 11, 2016

• #16

Re-checked, direct request to SQLite - the same situation - Inserting into SQLite DB is the much more slow process, if line by line.

UPD: and loop with adding SQL.AddNonQueryToBatch 28000 times is the same long

KMatle said:

You can insert f.e. 1000 rows and then the next 1000 pushed by a timer with 0.5s or so.

Click to expand...

How do _you_ make so fast ?

 

Last edited: Dec 11, 2016

Upvote 0

Erel

B4X founder

Staff member

Licensed User

Longtime User

• Dec 11, 2016

• #17

Can you post the code that inserts the data?

 

Upvote 0

peacemaker

Expert

Licensed User

Longtime User

• Dec 11, 2016

• #18

B4X:

Starter.SQL.ExecNonQuery2("INSERT INTO db VALUES (?,?,?,?,?,?,?)", Array As Object(count, p(0), p(1), p(2), p(3), a, b))

each line

 

Upvote 0

Erel

B4X founder

Staff member

Licensed User

Longtime User

• Dec 11, 2016

• #19

Ok. But we cannot help you with only this line.

 

Upvote 0

peacemaker

Expert

Licensed User

Longtime User

• Dec 11, 2016

• #20

It's a loop, reading line by line, parsing and inserting by this SQL request, that's all. But this loop only lines reading without SQL inserting is just 6 secs for 28000 lines.

 

Upvote 0