arbtt: use of database like sqlite3?

[Gwern Branwen]

So I was waiting on arbtt to --dump-samples for the past 100 hours to
write a rule classifying a web serial I read as recreational, and I
began wondering: what is arbtt doing that it takes so long?

Is it because of the log structure that it has to read through, parse,
and classify my full 85M arbtt log just to get the last 100 hours of
data? I know from working with an 18GB sqlite3 db for Mnemosyne that
date range queries in databases can be *extremely* fast, and
arbtt-capture dumping into a db would probably be more reliable and
durable (ACID rather than arbtt-recover), and sqlite3 has had multiple
Haskell bindings for half a decade now.

Would switching to sqlite3 be an improvement?

-- 
gwern
http://www.gwern.net