Create SQLite extensions pack installer for Windows

[karim]

I created this as a PR instead of pushing directly to master cause it needs to be discussed first.

There was a discussion some time ago about adding SQLite extensions to DB4S. I wanted to add them all to the same installer as DB4S but Justin said that he liked how the current installer size is small and we should move the extra bits outside of the main installer.

So I made this installer for collecting all SQLite extensions that are known to work with DB4S.

• It is made to check if DB4S is already installed and abort if it is not.

• Also it will detect if you tried to install 32-bit over a 64-bit DB4S and vice versa.

• At the moment, it has SpatiaLite only. More will be added (or moved from DB4S to it).

Installed extensions are not loaded automatically, and a note is provided at the end of setup.

Finally, how it look in Control Panel.

Things to discuss:

• Is this the right approach? A main DB4S installer and an extensions pack (only) installer.
• Should we make Light/Full versions better? A main DB4S (light) installer with the application only, and another (full) installer with the application + extensions?

[justinclift]

Awesome @karim, this is Very Cool! 😄

What's the expected way to have this generated? Be called from the existing win32/64 batch file, or be done separately in their own tasks, or ? 😄

Is this the right approach? A main DB4S installer and an extensions pack (only) installer.

This PR is likely a very good starting point. 😄

We could also do some kind of "show list of extension on a server (ext.sqlitebrowser.org?), then install user selected ones".

But not every installation has internet access anyway, so some kind of offline-suitable thing (like this pack) would be needed regardless.

Should we make Light/Full versions better? A main DB4S (light) installer with the application only, and another (full) installer with the application + extensions?

Hmmm, not sure. That'd probably confuse things more than keep them straight forward.

Just my opinion anyway. 😄

[karim]

What's the expected way to have this generated? Be called from the existing win32/64 batch file, or be done separately in their own tasks, or ? smile

Either way is fine. But I think we shouldn't rebuild them daily with the nightly, since they are mostly (at least this SpatiaLite) binary-only files that we need to generate once. I would say build it once now and be done with it.

Or maybe create a repo for them (github.com/sqlitebrowser/extensions)? This way we can do a rebuild/new version when the repo commits change.

We could also do some kind of "show list of extension on a server (ext.sqlitebrowser.org?), then install user selected ones".

Yes, I like this idea. Having a list of extensions on their own subdomain would give them more exposure and increases DB4S functionality and usage.

You are right, moving the extensions out of DB4S will give us more freedom to add more and more extensions, as a different package, without affecting most of our users who just want to use DB4S. Experienced and advanced users can install the extensions if they wanted more from DB4S.

We can now add all the extensions from there (https://www.sqlite.org/src/file/ext/misc) and search the Internet for more. We can even add experimental ones too or create our own. 😃

Final notes about SpatiaLite:

• The latest stable version (4.3.0a) is about 4 years old. We probably need an updated version.
• You should get the 32/64-bit versions from here and extract them and change the path in variables.wxi to point to them.
• Don't forget to update libstdc++-6.dll with a newer version according to the wiki.
• We probably should build it ourselves, with newer compiler/libraries, since all that seems very hacky. 😃

[mgrojo]

Is this the right approach? A main DB4S installer and an extensions pack (only) installer.

Yes, I think this is a good approach.

We can now add all the extensions from there (https://www.sqlite.org/src/file/ext/misc) and search the Internet for more. We can even add experimental ones too or create our own. smiley

Looking at the list of extensions included in Ubuntu 18.04, I see some interesting options:

libsqlite3-mod-blobtoxy/bionic,now 0.9995-1 amd64 [instalado]
  SQLite3 extension module for read-only BLOB to X/Y mapping

libsqlite3-mod-csvtable/bionic,now 0.9995-1 amd64 [instalado]
  SQLite3 extension module for read-only access to CSV files

libsqlite3-mod-impexp/bionic,now 0.9995-1 amd64 [instalado]
  SQLite3 extension module for SQL script, XML, JSON and CSV import/export

libsqlite3-mod-rasterlite2/bionic,now 1.0.0~rc0+devel1-6 amd64 [instalado]
  SQLite 3 module for huge raster coverages

libsqlite3-mod-spatialite/bionic,now 4.3.0a-5build1 amd64 [instalado]
  Geospatial extension for SQLite - loadable module

libsqlite3-mod-xpath/bionic,now 0.9995-1 amd64 [instalado]
  SQLite3 extension module for querying XML data with XPath

libsqlite3-mod-zipfile/bionic,now 0.9995-1 amd64 [instalado]
  SQLite3 extension module for read-only access to ZIP files

This packages come from the Debian project, which is also including libsqlite3-mod-virtualpg. In the Debian page for each package we can get the link to the upstream source codes.

[karim]

Looking at the list of extensions included in Ubuntu 18.04, I see some interesting options

Thanks for this list, Manuel. I didn't know about it before, and I see some interesting ones too.

That also got me thinking about a different approach for simple extensions. How about we statically link some of them to DB4S? SQLite allows us to link extensions statically (https://www.sqlite.org/c3ref/auto_extension.html) and load them with every database.

How about we link some of them to DB4S and add some checkmarks to the Extensions tab, like..

• Math (sin, cos, tan, ...)
• File IO (readfile, writefile)
• zLib (compress, uncompress)

On the app startup, we check the config file for the enabled extensions and then use sqlite3_auto_extension function to enable them (all will be off by default), so they will be available to the user with every database without any extra hassle.

Most of the extensions on SQLite website are in the public domain, so we don't a have a problem with licenses, and some of them are very useful and used often.

It will also increase the "wow factor" with new users. 😄

Example...

User: I want to fill a BLOB column with a file content, how can I do this?
Us: You can use readfile function. Go to Edit > Preferences > Extensions and enable File IO.
User: Wooooooow ... MIND BLOWN

[mgrojo]

@karim, I think there are two separable ideas in your comment:

• Provide a list of available known extensions and an easy way, through a check button, to load them.
• Link statically some of the extensions we are compiling.

I don't think you need the second to implement the first. Then I don't see any advantage in statically linking the extensions if we already have the code to load extensions in shared libraries.

To implement the first point, we could define a standard location to look for extensions (maybe this isn't easy to make it portable, but it seems feasible). In order to recognize the available extensions, so a description can be provided, we would only need a resource that matches the expected filenames with the description (maybe there's a better ways to recognize the extension based on the symbols it exports, but might be difficult to implement portably).

[karim]

I don't think you need the second to implement the first. Then I don't see any advantage in statically linking the extensions if we already have the code to load extensions in shared libraries.

What happens when the user check the button when the extension isn't there? Not installed during installation (skipped), not available for that platform. I believe we only build extensions for Windows and macOS.

If this PR is accepted and we moved all extensions to their own installer, then users will be confused when they check the button and nothing happens.

Linking some extension statically will add them to our code base, and make them available to every platform DB4S can be built on. If DB4S is available for that platform then users can be sure that their non-standard SQL function they use are also available to them.

We are kinda doing the same with regexp function and adding it statically to DB4S so the REGEXP operator will work.

When we add an extension statically and know the functions it creates we can also provide better error messages, like "readfile function is disabled, please enable it from ..." instead of the current "Result: no such function: readfile".

We can't do the same for every extension. I'm talking here about the official SQLite (https://sqlite.org/src/file/ext/misc) and very popular (e.g. math) extensions. They are in the public domain so we don't have a problem with licenses.

[mgrojo]

What happens when the user check the button when the extension isn't there? Not installed during installation (skipped), not available for that platform. I believe we only build extensions for Windows and macOS.

If this PR is accepted and we moved all extensions to their own installer, then users will be confused when they check the button and nothing happens.

I was thinking in scanning the directory where the extensions are installed and showing (or enabling) only the check boxes for the extensions that are there and we can identify.

Linking some extension statically will add them to our code base, and make them available to every platform DB4S can be built on. If DB4S is available for that platform then users can be sure that their non-standard SQL function they use are also available to them.

We can provide extension builds for the platform we'd like. If we're not doing it already, it's because Windows is our most used platform.

We are kinda doing the same with regexp function and adding it statically to DB4S so the REGEXP operator will work.

You're right, but it's minimal code that wouldn't make sense in a separate extension, because it would add a dependency to Qt. I don't say statically linking the libraries is unfeasible or doesn't make sense, but it'd need additional code for something that would provide less flexibility.

Users could prefer to load their own version of the extension, because it's provided by their Linux distribution or because they have compiled their own version.

When we add an extension statically and know the functions it creates we can also provide better error messages, like "readfile function is disabled, please enable it from ..." instead of the current "Result: no such function: readfile".

That would require parsing the statements annd having a list of functions provided for each extension. Both are costly but do not depend on having the extension loaded statically.

We can't do the same for every extension. I'm talking here about the official SQLite (https://sqlite.org/src/file/ext/misc) and very popular (e.g. math) extensions. They are in the public domain so we don't have a problem with licenses.

Why make a difference between them? The only difference might be whether we know the extension and can then provide a friendlier way to load it, compared to the current approach, where we only show the filename.

Look at how Pidgin (a chat application) provides plugin loading. They're still dynamic libraries:

[karim]

I was thinking in scanning the directory where the extensions are installed and showing (or enabling) only the check boxes for the extensions that are there and we can identify.

That should work, yes.

You're right, but it's minimal code that wouldn't make sense in a separate extension, because it would add a dependency to Qt.

Most extensions that we could link statically should fit the same criteria.

Off-topic question: C++ standard library does not have regexp support? I thought everything should be in C++ and Qt is used for the multi-platform bits (e.g. GUI).

You can ignore the above question. After a quick search I found this commit 37aaa4c. 😃

I don't say statically linking the libraries is unfeasible or doesn't make sense, but it'd need additional code for something that would provide less flexibility.

Linking some extensions statically actually simplify things for us, since we only have to call sqlite3_auto_extension function when the box is checked, and call sqlite3_cancel_auto_extension when it is unchecked.

SQLite library will handle everything from now on, including loading/unloading all of them whenever a database is opened/closed, so functions will appear as if they were part of the library itself.

Moreover, statically linked extension should work (I'm not sure) while dynamically loaded extensions (using C API sqlite3_load_extension or SQL function load_extension) are disabled. Not sure how this might help us/users though. 😄

Users could prefer to load their own version of the extension, because it's provided by their Linux distribution or because they have compiled their own version.

Users can uncheck the box and use another version if they don't like the one we provide.

But I see your point. We must also keep them maintained/updated/bug-free or users will disable them and use their own version and ours will be just dead code.

That would require parsing the statements annd having a list of functions provided for each extension. Both are costly but do not depend on having the extension loaded statically.

You are correct. It is probably over-thinking/over-engineering too.

Why make a difference between them? The only difference might be whether we know the extension and can then provide a friendlier way to load it, compared to the current approach, where we only show the filename.

I think my main concern was, if we are going to provide check boxes to enable/disable some SQL function (extensions) then we should link (bundle) them with DB4S. This way they will be always there, and on every platform. Loading other extensions dynamically should work the same alongside this method.

[mgrojo]

It seems we're reaching some agreements in this discussion. We might want to think in a future extension format in the lines of what we already discussed in #1198 (comment)

Using that file format, which would include, among other metadata, a description and a list of provided functions, we could implement the list of selectable extensions that you have in mind. Seems an effort for a 3.13 version, though 😄

In the short term, @karim, do you think this installer could enable loading of the extension after installation in the way described in this comment?
#1716 (comment)

As commented by @justinclift, the same approach could not be used in a future macOS installer. Nevertheless, is that enough for avoiding this solution at hand for Windows?

[karim]

We might want to think in a future extension format in the lines of what we already discussed in #1198 (comment)

I don't know how I missed this. 😃

Yes, I get the idea. You are proposing a new XML file that acts as a DB4S extension? It will add syntax highlighting to functions, helper message, and a new display format? The choice for XML file format is to make all this platform-independent?

That sounds great. I'm just a little bit confused by this..

What I'm thinking could be solved simply by using an XML file. The only compiled part (in a platform specific shared library) would be an optional SQLite extension

What do you mean by optional here? Can this XML work/provide functionality without depending on a SQLite extension? Can we have a full XML extension that does not need any complied code at all?

In the short term, @karim, do you think this installer could enable loading of the extension after installation in the way described in this comment?

Yes, I think it is possible. Auto loaded extensions are saved in Windows Registry and it is possible to read/write the Registry during installation. I will get it done.

[mgrojo]

Yes, I get the idea. You are proposing a new XML file that acts as a DB4S extension? It will add syntax highlighting to functions, helper message, and a new display format? The choice for XML file format is to make all this platform-independent?

Yes, but it could be any other format. Even an SQLite DB, but I think XML is the best option because we are already using it for our project files.

What I'm thinking could be solved simply by using an XML file. The only compiled part (in a platform specific shared library) would be an optional SQLite extension

What do you mean by optional here? Can this XML work/provide functionality without depending on a SQLite extension? Can we have a full XML extension that does not need any complied code at all?

No, unless we developed some kind of interface between the SQLite extension API and an interpreted language that could be embedded in the XML. But I wasn't thinking in something so sophisticated, but in XML extensions that would only add display formats, for example. The other possible extendable objects might not make sense, though. Why would someone add functions for highlighting if we already provide that for the base SQLite functions? Only if we have missed something or they load their own SQLite3 DLL, newer or with statically loaded extensions, for example. Being optional is probably not very useful, but I don't see why we should make the SQLite-extension-library part mandatory.

In the short term, @karim, do you think this installer could enable loading of the extension after installation in the way described in this comment?

Yes, I think it is possible. Auto loaded extensions are saved in Windows Registry and it is possible to read/write the Registry during installation. I will get it done.

Yes, you can either run the application with the settings arguments or directly write the Windows Registry, whatever you consider better or easier, because it is already specific for Windows in any way.

[karim]

but in XML extensions that would only add display formats, for example.

Yes, that's a good usage for it that doesn't depend on anything.

What I wanted with statically linking some extensions (or functions) is to save ourselves from a lot of overhead with extension building/loading/installing. Some functions are just one liners, and others are just a couple of lines long.

pi function for example is just sqlite3_result_double(context, 3.14159265358979323846);. We just add syntax highlighting for it, helper message, auto-complete, put it behind a toggle (to enable/disable), and be done with it.

I know this will add more code to DB4S and will force us to maintain it, and we have a working extension-loading system already. It is just very tempting. 😄

For the time being, we just migrate the extensions to their own installer/package (this PR). Maybe put them on a server also (Justin idea) and pull them from there, using a dialog/GUI like Pidgin screenshot, too for better discoverability.

Yes, you can either run the application with the settings arguments or directly write the Windows Registry, whatever you consider better or easier, because it is already specific for Windows in any way.

When I add an extension to Extensions tab, DB4S writes its path in list value under HKCU\software\sqlitebrowser\sqlitebrowser\extensions key in Windows Registry, so I think this is a good start.

[justinclift]

Oh wow, totally forgot about this. Should get to it some time this week. 😄

[karim]

No worries. 😃 Sorry for being inactive lately.

[mgrojo]

What about this pull request? Is it not going to enter 3.12? I don't know which is the current state of affairs with extensions in Windows, but I understood that this pull request was already an improvement over the current state.

[justinclift]

Good point. Yeah, we should get this in for 3.12. I just need to sit down and think the relevant bits through. 😄

[justinclift]

Leaving this here, as a reminder it should probably be looked at for the extensions pack (whenever we get to that):

    https://www.sqlite.org/floatingpoint.html#decext

[huastecodecorazon]

• [ ]

[justinclift]

@huastecodecorazon ???

[mgrojo]

@karim, @justinclift, if this is not ready for merging, maybe we should convert it to a draft and add the help-requested label, so maybe someone interested can move it forward?

[lucydodo]

As a side note, we recently added the sqlean library for macOS builds, which includes many commonly known SQLite extensions. I'll be adding it for Windows builds as well after I work out some pressing issues. :) (Related: #3357)