More granular inclusions

[LukeLongworth]

Kia ora,

Following my seminar on the linear algebra contrib libraries @sangwinc and I got to chatting about the include mechanism. The existing system works fine for me but there are some slightly awkward hurdles:

• Dependencies between inclusions require me to run several inclusion lines (we avoided this by putting linearalgebra_core.mac into the main installation). This would be worse if there were defined variables (such as return_vect in vectorcalculus.mac) that then required inclusions to happen in a certain order.
• stack_include is a hammer and sometimes we want a scalpel. Just today I wrote a question in which I used 4 different inclusions to get access to 4 different functions and the load times when compiling spiked noticeably. Being able to include individual functions would avoid this, but keeping every function in a separate file is not feasible.
• There are some code snippets I copy around all the time that aren't suitable for broader STACK use. For instance, I have a validator I call validate_set_of_vecs which checks that the student has precisely followed the input syntax I need, but that other institutions won't necessarily use. Local inclusion libraries would be wonderful for this and other applications.

Also noted in our brief discussion was that version control will eventually become a hurdle.

Chris's thought before suggesting we move the discussion here was to "develop a stack_depends function [or similar] which would auto-load (once) other libraries from the local contrib".

I recognise that this could be a substantial bit of reworking! I'm mostly curious to hear what other devs think about this sort of project first.

[aharjula]

Let's just say that I have some thoughts... The first being that stack_include in its current form needs to stay in its current form, we need a light "include this here" logic, and we must not mess up existing stuff. But we sure do need some sort of "require"/"depends" feature that can pick relevant bits of old single-file stack_include libs and can extend to picking from more complex libraries spanning multiple files.

If we accept that not all that the new thing can do can be done with stack_include, we will start from the correct place. If we further accept that running the new logic in a development setup directly referencing raw code files will be slow during the compile, and that it is meant for stable libraries with pre-compiled manifests, that the compilation process can use, then we will probably start this process with a sensible heading.

After your seminar, I too thought about things and sketched something. This is what I came up with:

stack_require("rand_matrix",["rand_diagonalizable","rand_orth"]);

Which would be a require statement that refers to a library called "rand_matrix" somewhere (locally, or maybe in published branch of git), notably it would not refer to a .mac file but a named library that has a pre-compiled "manifest" and based on that it would then compose an inclusion of the requested identifiers, in this case those two and the required rand_integer_invertible and rand_invertible and nothing else.

Now things get complicated if one had this:

stack_require("rand_matrix",["rand_diagonalizable"]);
stack_require("rand_matrix",["rand_integer_invertible"]);

Would the first require statement include rand_integer_invertible and then the second include it again? Either we would need to have visibility over the inclusions and, in the worst case, over the parts of the questions to optimise that away, or we would need to force the inclusion of everything in a singular statement, which is not an option, as people will want to leave some inclusions to the PRTs. In this case, we will probably have to accept unnecessary duplication, but at the very least, we could make things simpler for ourselves (STACK devs) by making it simple to join inclusions, even if we can not force it:

stack_require(
  "rand_matrix", ["rand_diagonalizable"],
  "rand_matrix", ["rand_integer_invertible"],
  "vectorspaces", ["orthogonalize"]
);

Thus, the composition logic could see all the requirements at the same time and drop duplicates. Well, at least for this single place where those inclusions happen.

Now obviously "rand_matrix" is a good library of suitably self-contained nature, but once we start having cross-file requirements inside the library and direct inclusions of singular files are no longer sensible, we might have "packages" with catchy names and abstracting away the actual file that contains the implementation, e.g., "L3" for Luke's Linear Library:

stack_require("L3", ["rand_orth", "orthogonalize", "vector_in_spacep"]);

Now, as "L3" sure does not map to a .mac file, it must map to a manifest instead. So what is a manifest?

1. A manifest is a pre-compiled listing of all the things the library contains.
2. One of those things might be a common preamble like stack_linear_algebra_declare(true) in "L3".
3. Each thing provides a list of other things it might require, primarily from a default scope of within this manifest, but also from other libraries that have manifests.
4. Unlike a real manifest in the real sense of the word, ours will also contain the code for each thing. We do not read/download a manifest just to make N requests to fetch the parts we need from elsewhere.
5. Based on the requirements, we can simply concatenate all unique bits related to the requested bits, the bits they require and the preamble. We can somewhat order these, but cycles are a possibility. Luckily, we don't need to care about those, as most of these are function definitions.

Once that concatenated bit is included like a normal stack_include thing, things should run as normal. But we might go further and seek new troubles...

In my view, the manifest is always a product of compilation, starting from a set of .mac files and ending with a manifest with a sensible name and place in some directory of manifests. Anything that affects the generation should exist in those .mac-files as annotations, i.e., requirements over the manifest-scope, and the compilation should be able to identify local dependencies on its own. But what else should that compilation do? Is it like the STACK 4.12 tool that turns library code to the modified executable bits, the tests and even documentation? Would it do simillar code rewriting, if it would rename variables to avoid collisions how badly would that mess up peoples ability to trace errors should they appear?

If such a tool existed, would this mean that anyone wanting to generate libraries at the manifest level must have suitable technical rights to run that tool? Or do we provide some public website where one dumps a set of .mac files and it spits out a manifest? Or how complex can that tool be? If we go back to the development case, how significant should the checking be? Obviously, loading in something that does not yet have a manifest would require the generation of a manifest, but not tests or documentation, but it would still be a significant step, not to mention the complications of defining multi-file sourced things, or delivering compilation errors related to manifest building through the question editor:

stack_require(
 "genmanifest: https://.../rand_matrix.mac https://.../vectorspaces.mac", ["orthogonalize","rand_diagonalizable"]
);

So in short I do have some ideas; however, I would really like to see the new core-library thing grow and see how the tooling for that develops before building a parallel tool for separate libraries.

[aharjula]

Further complications that come to mind and needs a decision before any implementation. Should the inclusion logic have any visibility related to the context in which the inclusion is being done? Suppose we have a library "L" having a function "A" depending on a function "B" and a configuration variable "C". If we bring in "A", we would get "B" and "C" at the same time, but what if someone does this:

B(x):=something;
C: something;
stack_require("L", ["A"]);

Should the inclusion logic be aware that "B" and "C" are already defined in this context? Should it:

1. Silently overwrite them with the dependencies it has, i.e., no knowledge necessary.
2. Silently accept them as the dependencies and not load versions from the manifest.
3. Make some warnings about overwriting and overwrite them.
4. Error out.
5. Require explicit overwriting, i.e., stack_require("L", ["A","B","C"]);
   Basically, the question is: If an included thing has a requirement for a function named X will an existing function X fulfil that, or does it have to be the very implementation the manifest refers to?

Forward visibility would not be a thing; this would simply load the things from the library and then overwrite them. We simply do not have the static analysis capability to see if there is a redefinition before use:

stack_require("L", ["A"]);
B(x):=something;
C: something;