Scribd
Upload
47 views48 pages

Z4Nu868oQ7mjnCJKWOvg - Using - SQLite - With - QT - Squeeze - Every - CPU - Clock - To - Get - Faster - Gianbattista - Gualeni

- SQLite is a database management system that is implemented as a library rather than a separate server process, allowing for faster access. - The document discusses how to integrate SQLite into a Qt application, including adding the SQLite module to the project file and importing headers. - Various optimizations are tested to improve the speed of inserting over a million images into a SQLite database using Qt, including transaction handling, journal modes, and prepared statements. With all optimizations, over 90,000 images can be inserted per second.

Uploaded by

yoni zerbib
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
47 views48 pages

Z4Nu868oQ7mjnCJKWOvg - Using - SQLite - With - QT - Squeeze - Every - CPU - Clock - To - Get - Faster - Gianbattista - Gualeni

- SQLite is a database management system that is implemented as a library rather than a separate server process, allowing for faster access. - The document discusses how to integrate SQLite into a Qt application, including adding the SQLite module to the project file and importing headers. - Various optimizations are tested to improve the speed of inserting over a million images into a SQLite database using Qt, including transaction handling, journal modes, and prepared statements. With all optimizations, over 90,000 images can be inserted per second.

Uploaded by

yoni zerbib
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 48

Qt And SQLite

… squeeze every CPU clock to get faster

Florence, April 2nd 2019


DBMS
SQLite is a database management system

Qt and SQLite - Gianbattista Gualeni

2
Database management system
A database is an organised collection of data, generally stored and
accessed electronically from a computer system. Where databases
are more complex they are often developed using formal design and
modeling techniques.

The database management system (DBMS) is the software that


interacts with end users, applications, and the database itself to
capture and analyse the data. The DBMS software additionally
encompasses the core facilities provided to administer the database.
The sum total of the database, the DBMS and the associated
applications can be referred to as a "database system". Often the
term "database" is also used to loosely refer to any of the DBMS, the
database system or an application associated with the database.

Wikipedia

Qt and SQLite - Gianbattista Gualeni 2019 3


Not really new!

This was actually a DBMS used 2500 years b.C.


I miss the programmer name, sorry.
I don’t find the project on GitHub!
Qt and SQLite - Gianbattista Gualeni 2019 4
Why SQLite

There are 2 Big Family


•SQL
•NOSQL
Among SQL we have:
•Database server (MySQL, Oracle …)
•Database Library (in-process database)
SQLite is a library, hence lower latency
Speed? It depends.
Implementation? Simpler, no excuse…

Qt and SQLite - Gianbattista Gualeni 2019 5


SQLite and Qt
How we can use a database from Qt application

Qt and SQLite - Gianbattista Gualeni 2019 6


Fast lane to SQLite

The fastest way to play with SQLite database?


DB Browser for SQLite

Qt and SQLite - Gianbattista Gualeni 2019 7


Integration

Add sql module to .pro

Import include

Qt and SQLite - Gianbattista Gualeni 2019 8


addDatabase / removeDatabase

A lot of Static!
(Danger, High Voltage)

Qt and SQLite - Gianbattista Gualeni 2019 9


EXAMPLE
A simple image database application

Qt and SQLite - Gianbattista Gualeni 2019 10


My Sample Project

1Milion Images… Again


I create images using a Python script (6 hrs!)
Images are organised in sub-folder
1000 folders x 1000 images = 1M
4kB for image = 4GB

Qt and SQLite - Gianbattista Gualeni 2019 11


Who I am /1

I am the Author of the book:


Qt 5 Quanto Basta

And I am a Qt enthusiastic user since early 2000.

Currently I work for Datalogic as Machine Vision Product


Specialist and I use Qt to play with images, logs and
… private projects.

I’m working on another book? Yes! Fast and Furious

Qt and SQLite - Gianbattista Gualeni 2019 12


Who I am /2

Qt and SQLite - Gianbattista Gualeni 2019 13


Update the database
Writing images info into the database

Qt and SQLite - Gianbattista Gualeni 2019 14


Create the database

DB Browser
images table
4 cols
Just text

Qt and SQLite - Gianbattista Gualeni 2019 15


INSERT INTO
Static again

For each folder, for each file in the folder…

Qt and SQLite - Gianbattista Gualeni 2019 16


SPEED?

Really, not the right word!


5.4 ms for each image
1h 30 min estimated for 1M images
or 18.5 images for second

Using Core i7, 8th gen, 4 GHz turbo boost

Qt and SQLite - Gianbattista Gualeni 2019 17


Optimisation
Can we speed-up this shame?

Qt and SQLite - Gianbattista Gualeni 2019 18


Speed Limit ?

If I leave just the file scan, query string creation but I disable the
query?

So this means that:


1. We can run up to 50k images/sec
2. The limiting factor is the access to the database

Qt and SQLite - Gianbattista Gualeni 2019 19


Optimisation

During optimisation I use a sub-set of just 100 images

I test every possible options, but I’m going to save you from my
boring experience.

Qt and SQLite - Gianbattista Gualeni 2019 20


Journal_mode = WAL
Write Ahead Log

1.955 ms means 511 Images / sec


faster than 18.5 images /sec
but far from the speed limit (50.000)

Qt and SQLite - Gianbattista Gualeni 2019 21


Journal_mode = MEMORY

We can save the backup in memory and not on a separate file

1709 ms means 585 Images/sec


faster than 18.5 images/sec
but far from the limit (50.000)

Qt and SQLite - Gianbattista Gualeni 2019 22


PRAGMA SYNCHRONOUS OFF
Don’t save to disk in synchro

0.887 ms means 1127 Images / sec


faster than 18.5 images /sec
but far from the limit (50.000)

Qt and SQLite - Gianbattista Gualeni 2019 23


BEGIN / END TRANSACTION
We lock the database for our transaction till the end!

0.114 ms means 8772 Images / sec


faster than 18.5 images /sec
but far from the limit (50.000)

Qt and SQLite - Gianbattista Gualeni 2019 24


Playing with the
MILION
What happens when we play with the real beast?

Qt and SQLite - Gianbattista Gualeni 2019 25


Can we use multiple optimisations?

This is the table with optimisation results


BEGIN / END TRANSACTION is way more better!

Qt and SQLite - Gianbattista Gualeni 2019 26


Sure!
Just apply BEGIN TRANSACTION as last.
This is the optimal configuration I test:

What happens with 1M Images?

Qt and SQLite - Gianbattista Gualeni 2019 27


1AMAZING MILION

Did you see the number? 0.016 ms


This means 62k images per second
Faster than the speed limit.
Why so fast? Maybe the turboboost is
more efficient on a long run.

Qt and SQLite - Gianbattista Gualeni 2019 28


Last but not least: Bind

We need 2 steps: prepare and bind

Qt and SQLite - Gianbattista Gualeni 2019 29


1AMAZING MILION /2

This is INSANE! 0.011 ms per image

This means 90k images per second.


Considering we start from 18.5…

Qt and SQLite - Gianbattista Gualeni 2019 30


1AMAZING MILION /2

Maybe I was not clear

90.000 Images
per second!

Qt and SQLite - Gianbattista Gualeni 2019 31


Let the icon play
What happen if we insert images?

Qt and SQLite - Gianbattista Gualeni 2019 32


Insert small and big icons
We add 2 Binary Large Object to store one small icon and one
thumbnail. In my test I use

Qt and SQLite - Gianbattista Gualeni 2019 33


Creating Icons /1

Read the file, resize and write as JPG into a Byte Array

Just for first image, because we focus on database not on image


conversion

Qt and SQLite - Gianbattista Gualeni 2019 34


Creating Icons /2

This process is painfully slow, compared to the incredible level


we reach

But this optimisation is out of scope

Can I use it in the next QtDay?

Qt and SQLite - Gianbattista Gualeni 2019 35


INSERT INTO + BLOB

How can I insert a blob into a query? Using bind

Qt and SQLite - Gianbattista Gualeni 2019 36


Performances
Considering that now the resulting database is 4GB, instead of
160 MB, I expect the a slower image rate …

I think it is still incredible


We have 0.024 ms for each image (4 seconds for 1M Images)
Or 42k Images per second.

Considering the disk speed is 400 MB/s


It takes 10 secs to write the file (and 14 to do the rest…)

Qt and SQLite - Gianbattista Gualeni 2019 37


And the answer is…

42
(thousand images per second)

Qt and SQLite - Gianbattista Gualeni 2019 38


VISUALIZATION
Can we see the database content?

Qt and SQLite - Gianbattista Gualeni 2019 39


Can we see the database content?

Qt and SQLite - Gianbattista Gualeni 2019 40


Table View and Model
We need a tableView
We need a QSqlQueryModel
We need a delegate to paint icons

Then we need the update event

Qt and SQLite - Gianbattista Gualeni 2019 41


The Delegate

Qt and SQLite - Gianbattista Gualeni 2019 42


Search
How fast is searching an image?

Qt and SQLite - Gianbattista Gualeni 2019 43


SELECT * FROM WHERE
"SELECT * FROM images WHERE FileName LIKE '%1234.png%'
ORDER BY FileName DESC;"

Qt and SQLite - Gianbattista Gualeni 2019 44


Search
12 second to search all images that contains 1234 in the name!

Qt and SQLite - Gianbattista Gualeni 2019 45


Index to speed-up search

Qt and SQLite - Gianbattista Gualeni 2019 46


Index effect

Amazing!

Few milli seconds instead of 12 seconds.

Qt and SQLite - Gianbattista Gualeni 2019 47


Questions?

Qt and SQLite - Gianbattista Gualeni 2019 48

You might also like