Large (multi-gb) sqlite db support

[hubert3]

This looks neat but it takes several minutes to open a 10 GB sqlite file containing 40 million rows.

It does eventually render the interface, but it's taking 38 GB memory according to macOS Activity Monitor.

DB Browser for SQLite is able to work with this DB without loading the entire DB into memory (it's consuming 800 MB RAM).

Loading the entire DB into memory would be OK if it's done in a more memory-efficient way (my system could handle taking 1.5x the size of the sqlite db file but not 3.8x)

Once the interface loaded, I got this crash when I pressed a button:

panic: runtime error: slice bounds out of range [1450:61]-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------┤ 1450, 0

goroutine 561 [running]:
github.com/mathaou/termdbms/viewer.DisplaySelection(0xc203347600)
	/Users/hubert/go/pkg/mod/github.com/mathaou/[email protected]/viewer/ui.go:250 +0x5f3
github.com/mathaou/termdbms/viewer.AssembleTable(0xc203347600?)
	/Users/hubert/go/pkg/mod/github.com/mathaou/[email protected]/viewer/tableutil.go:18 +0x7b
github.com/mathaou/termdbms/viewer.TuiModel.View.func1(0xc203e34880)
	/Users/hubert/go/pkg/mod/github.com/mathaou/[email protected]/viewer/viewer.go:152 +0x2e
created by github.com/mathaou/termdbms/viewer.TuiModel.View
	/Users/hubert/go/pkg/mod/github.com/mathaou/[email protected]/viewer/viewer.go:151 +0x219

[mathaou]

I am a much better developer now than when I first wrote this project, so ideally I would rewrite this from the ground up to account for things like this. I am also 100% not a great writer of SQL (which may be paradoxical given the nature of this project). How in the world would I query for a set range of data in SQLITE so I could cycle things in and out? I see info on other DBMS (MySQL, etc.), but nothing really for SQLITE.

Unlike other issues that I haven't gotten around to implementing (mainly UI related bugs), this is an atomic enough problem that I actually might go ahead and fix this if I can just figure out that bit.

Depending on if I can get a range query top work and how easy the rest of the rework is, I might not have to do a from-scratch rewrite.

[hubert3]

sqlite> select * from files limit 10 offset 0;
sqlite> select * from files limit 10 offset 10;
sqlite> select * from files limit 10 offset 20;