arbtt: use of database like sqlite3?

[Joachim Breitner]

Hi,


Am Montag, den 15.12.2014, 11:19 -0500 schrieb Gwern Branwen:
> On Mon, Dec 15, 2014 at 3:35 AM, Joachim Breitner
> <mail at joachim-breitner.de> wrote:
> > But now the insertion is even more expensive: Upon every sample, for
> > every open window, sqlite will have to traverse an index of over a
> > million¹ entries to see if this particular window title has occurred
> > before.
> 
> No; come on, these are *databases*, their raison d'etre is looking up
> stuff. The ideas behind them are now at least 44 years old - give them
> a little credit, they can do better than a linear scan.

that’s why I said “an index”!

> Suppose inserts were as dumb as a binary tree - then the traverse only
> involves ~14 lookups (log_2(1million) = 13.8 levels of a tree). I
> would be surprised if sqlite3 couldn't do thousands of linked inserts
> per second, and I'd expect it to require microbenchmarking to show the
> difference between a regular insert and an index-linked insert like
> proposed.
> It's really not an issue.

You might be right...

> The real question is whether anyone wants the faster queries enough to
> rewrite the backend for and make everyone convert their old logs over.

I don’t think this will be too hard. Maybe I should simply give it a try
over the holidays.

Also, this would maybe make it easier to cache the actual tags. This
might require some refactoring and redesign, though. E.g. currently,
tagging may easily depend on the command line.

Anyways, I also added a ticket for it
https://bitbucket.org/nomeata/arbtt/issue/19/use-sqlite3-as-a-backend

> (The existing append-only logs may be terrible for queries, but all
> the code exists and presumably is debugged.)

presumably.... unfortunately, they are not bullet proof (otherwise there
would be no need for arbtt-recover).


-- 
Joachim “nomeata” Breitner
  mail at joachim-breitner.de • http://www.joachim-breitner.de/
  Jabber: nomeata at joachim-breitner.de  • GPG-Key: 0xF0FBF51F
  Debian Developer: nomeata at debian.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <https://lists.nomeata.de/pipermail/arbtt/attachments/20141215/29e1c0c8/attachment.asc>