How should extensions be installed for packages?

[shah]

I love using pkgx - great job on the developer experience! One of my projects uses sqlite3 (which of course pkgx supports nicely) but I need to also install and use numerous SQLite loadable extensions. Currently I use SqlPkg SQLite 3 extension manager but since sqlpkg is basically copying files into a well-known location I figured it might be easier to do it through new pkgx and the Pantry.

For example, when I write SQL for sqlite3 I might need to do the following:

.load /home/me/.sqlpkg/asg017/ulid/ulid0
select ulid();

Without the extension ulid() will fail but with the .load ... line it will work.

What are your suggestions about how to proceed with dynamic extensions acquisition, download, and referencing for packages that pkgx supports? Hopefully this is a very generalizable problem so a general-purpose solution to support plugins, extensions, etc. makes sense for pkgx especially for dev environments.

[mxcl]

Firstly, super glad you like pkgx, we're glad you’re using it!

In answer to your Q. if there's a good PM then generally we would say: package that and use that.

However we totally are up for expanding what pkgx can do if required.

An example of what we already do is eg with git where git foo will search pkgx for git-foo and automatically install and use it, if we have it packaged. I don‘t think that easily applies here, I'm just using it as an example of what we think is fine for pkgx to do.

We would go as far as to patch sqlite with new “commands” or whatever sqlite calls them—because we believe packaging should truly be abstracted away because it is tedious—but hopefully we can do something simpler and less intrusive.

I think we need some idea of how sqlite extensions work before we can proceed. Probs they are just dynamic libraries right?

[shah]

Yes @mxcl you're right -- SQLite loadable extensions are very similar to the git-foo example you gave. Each extension is a platform-specific *.so or *.dll etc. and is used as follows:

1. You download the extension binaries (.so, .dll, etc.) and place them anywhere on the local file system
2. You use the dowloaded binaries by giving the path to the extension as follows (in this example ulid0.so is the extension on Linux and /home/me/.sqlpkg/asg017/ulid/ is where it was downloaded):

.load /home/me/.sqlpkg/asg017/ulid/ulid0
select ulid();

The job of pkgx in this situation would be to grab the platform-specific *.so, *.dll, etc. binaries and put them in a well known location (e.g. ~/.pkgx/a/b/c and then it would be up to the SQL author to add the proper .load into the SQL.

Does the spec above help?

[mxcl]

well sounds like a standard package then.

googling suggests LD_LIBRARY_PATH is searched by sqlite, if that's true then packaging them and including them in the env will automatically load them without having to specify full paths!