Unnecessary GAPDoc dependencies cause a lot of texlive installs

[stertooy]

Out of the 172 packages distributed with gap, 67 have GAPDoc listed as a needed package (and a further 7 have it as a suggested package).

In many cases this dependency is not necessary. But it causes the installation of some texlive-related packages during the CI tests that aren't needed at all. And due to dependencies cascading, this ends up happening for many more packages than just these 72 that explicitly depend on GAPDoc.

This is a bit of a waste (both in time and electricity), but it also increases the chance that some CI tests will fail due to apt-get hanging. Perhaps we should try to identify which packages don't really need GAPDoc, and open PR's in the respective repositories to remove the dependency? (I'm happy to help with this)

---

For reference, the list of packages with GAPDoc as Needed/Suggested package. I didn't bother with extensions.

images needs GAPDoc version >= 1.5
agt needs GAPDoc version >= 1.6
atlasrep needs GAPDoc version >= 1.6.2
autodoc needs GAPDoc version >= 1.6.3
automata suggests GAPDoc version >= 1.2
browse needs GAPDoc version >= 1.6
cddinterface needs GAPDoc version >= 1.5
circle needs GAPDoc version >= 1.5.1
congruence needs GAPDoc version >= 1.5.1
crypting needs GAPDoc version >= 1.5
ctbllib needs GAPDoc version >= 1.6.2
curlinterface needs GAPDoc version >= 1.5
cvec needs GAPDoc version >= 1.2
datastructures needs GAPDoc version 1.5
deepthought needs GAPDoc version >= 1.5
design needs GAPDoc version >=1.6
difsets needs GAPDoc version 1.5
digraphs suggests GAPDoc version >=1.6.3
edim needs GAPDoc version >= 1.6
example needs GAPDoc version 1.5
examplesforhomalg needs GAPDoc version >= 1.1
factint needs GAPDoc version >=1.6
ferret needs GAPDoc version >= 1.5
fining needs GAPDoc version >= 1.6.3
float needs GAPDoc version >=1.0
fr needs GAPDoc version >=1.0
francy needs GAPDoc version >= 1.6.1
gauss suggests GAPDoc version >= 1.0
gaussforhomalg needs GAPDoc version >= 1.0
gbnp needs GAPDoc version >= 1.5
generalizedmorphismsforcap needs GAPDoc version >= 1.5
genss needs GAPDoc version >= 1.5
gradedmodules needs GAPDoc version >= 1.0
gradedringforhomalg needs GAPDoc version >= 1.0
guarana needs GAPDoc version >=1.3
hapcryst suggests GAPDoc version >= 0.99
homalg needs GAPDoc version >= 1.0
homalgtocas needs GAPDoc version >= 1.0
ibnp needs GAPDoc version >= 1.6.1
inducereduce needs GAPDoc version >= 1.5
intpic needs GAPDoc version 1.5
io_forhomalg suggests GAPDoc version >= 1.0
json needs GAPDoc version >= 1.5
jupyterkernel needs GAPDoc version >= 1.6.1
jupyterviz needs GAPDoc version 1.3
laguna needs GAPDoc version >= 1.6.1
localizeringforhomalg needs GAPDoc version >= 1.0
majoranaalgebras needs GAPDoc version >= 1.5
matricesforhomalg needs GAPDoc version >= 1.0
modules needs GAPDoc version >= 1.0
nock suggests GAPDoc version 1.5
openmath needs GAPDoc version >= 1.6.0
packagemanager suggests GAPDoc version >= 1.6.1
patternclass needs GAPDoc version >= 1.5
primgrp needs GAPDoc version 1.5
profiling needs GAPDoc version >= 1.5
qdistrnd needs GAPDoc version 1.5
rcwa needs GAPDoc version >=1.5.1
repndecomp needs GAPDoc version >= 1.6.1
resclasses needs GAPDoc version >=1.5.1
ringsforhomalg needs GAPDoc version >= 1.0
sco needs GAPDoc version >= 1.0
scscp needs GAPDoc version >= 1.5
semigroups suggests GAPDoc version >=1.6.3
simpcomp needs GAPDoc version >=0.9999
smallantimagmas needs GAPDoc version 1.5
smallgrp needs GAPDoc version >= 1.5
spinsym needs GAPDoc version >=1.5
standardff needs GAPDoc version >= 1.6.5
thelma needs GAPDoc version 1.5
unitlib needs GAPDoc version >= 1.6
uuid needs GAPDoc version >= 1.5
walrus needs GAPDoc version >= 1.5
wedderga needs GAPDoc version >= 1.5.1
zeromqinterface needs GAPDoc version >= 1.6.1

[fingolfin]

Almost all of these will be unnecessary. A very few packages define functions that invoke GAPDoc to build their manuals (let's not discuss here whether that's a good idea or not). One should be able to identify those by grepping them for "MakeDocDoc" or whatever that name was.

We could then do two things:

1. Short term: add a workaround here that removes the gapdoc dependencies "on the fly" if they afe unnecessary (by having a table of the few necessary cases)
2. long term: get all packages that don't need it to change

[fingolfin]

Thinking some more, perhaps we should just special case GAPDoc and only install texlive when testing GAPDoc itself; never install it "transitively".

[fingolfin]

Some searching:

rg --type=gap -l MakeGAPDocDoc pkg/*/{lib,gap}
pkg/gapdoc/lib/Make.g
pkg/autodoc/gap/Magic.gd
pkg/autodoc/gap/Magic.gi
pkg/qpa/lib/documentation.gi
pkg/intpic/gap/ip_utils.gi
pkg/hap/lib/TitlePage/makeHapMan.gi
pkg/hap/lib/HapPrime/happrime.gi

So it seems the only "real" dependencies on GAPDoc are in

• AutoDoc
• intpic
• hap
• qpa

And I'd argue that for all but AutoDoc, this dependency could and should be removed, too.

Of course some packages might also have dependencies on AutoDoc -- we should remove those as well.

A long time ago, @olexandr-konovalov made a visualization of the package dependency graph. This is in fact the "avatar" graphic used on https://github.com/gap-packages. Anyway, I guess it'd be useful to make a new version of that, and put the code for updating it in a place we can easily find. The output might be helpful for understanding various things about the package distro.

[olexandr-konovalov]

Here it is for GAP 4.15.1 - enjoy! You can use the .png file directly, or put both .png and .html file in the same directory and open the .html file in your browser, to be able to click on packages and navigate to their websites.

Fun facts: 176 nodes, 374 edges, 19 connected components.

The code I have generates input file for the yEd graph editor. It does not apply any layout - you have to open it in yEd and try to apply various automated layouts. One of them is shown here. The "avatar" was a result of some other one.

Happy to create a repository and add some README, please suggest where.

gap2yed4_15_1.html

[olexandr-konovalov]

The log also reports packages which are specified in the dependencies but are missing from the distribution. Those are indicated using grey boxes:

Warning: package ctblocks is missing
Warning: package mfer is missing
Warning: package CompilerForCAP is missing
Warning: package chevie is missing
Warning: package NautyTracesInterface is missing
Warning: package homology is missing
Warning: package Carat is missing
Warning: package ParGAP is missing
Warning: package Char0Gauss is missing
Warning: package TopcomInterface is missing

[stertooy]

Thinking some more, perhaps we should just special case GAPDoc and only install texlive when testing GAPDoc itself; never install it "transitively".

That shouldn't be too bad to implement, and it would indeed take care of the problem.

Here it is for GAP 4.15.1 - enjoy! You can use the .png file directly, or put both .png and .html file in the same directory and open the .html file in your browser, to be able to click on packages and navigate to their websites.

Oh, that's amazing. And that big blob of black illustrates quite well how many packages depend on GAPDoc right now.

The log also reports packages which are specified in the dependencies but are missing from the distribution. Those are indicated using grey boxes:

Well, that's already proved to be helpful just now. I submitted gap-packages/hapcryst#24 to rename the carat dependency in HAPCryst to CaratInterface.

[stertooy]

So it seems the only "real" dependencies on GAPDoc are in

• AutoDoc
• intpic
• hap
• qpa

PackageManager also suggests GAPDoc, but actually requires it (and a texlive install) to pass tests (hence #1405). Reason is that, when installing a package via git, it always tries to compile documentation and gives an error if that fails.

A quick grep indicates that Cap and Ctbllib use TextAttr, which I believe comes from GAPDoc?

[fingolfin]

Yes TextAttr is from GAPDoc. So we should suggest to the respective package authors to add GAPDoc as an explicit dependency -- or else stop using TextAttr, or only use it conditionally (the latter is what the GAP library does: it only uses TextAttr if it is defined)

[olexandr-konovalov]

Also OpenMath package has a "real" dependency using ParseTreeXMLString from GAPDoc.

[ThomasBreuer]

A quick grep indicates that Cap and Ctbllib use TextAttr, which I believe comes from GAPDoc?

CTblLib uses the variable TextAttr only if it is bound, this is not really a problem.
However, CTblLib also uses the GAPDoc functions InitialSubstringUTF8String and WidthUTF8String, that's why GAPDoc is a needed package of CTblLib.

I think we have to distinguish between using GAPDoc's utility functions and calling GAPDoc for creating a package manual.
And in the latter case, wouldn't it be enough to leave out the creation of the PDF version of the manual from CI tests?

Currently one can achieve this by an optional "nopdf" argument for MakeGAPDocDoc.
Would it perhaps help if one could alternatively set a global option "nopdf", which could then be set in CI tests for packages? If yes then no texlive-related packages need be installed for these CI tests.

[fingolfin]

Just talked a bit more about this with @ThomasBreuer.

I think that the "dependency" on LaTeX in GAPDoc is a bit different from other packages: it is "only" needed if one really wants to re-generate the manuals; but for most of our packages, that is is irrelevant. Perhaps GAPDoc itself needs it for its test suite, fine; but TeX probably simply shouldn't be treated as a transitive dependency at all?

Do we know of any other package that really needs TeX in its tests? Even if, though: might be best to only install it for these few packages, explicitly?

[stertooy]

I think we're talking about two situations here:

• Unnecessary texlive installs in the packages' repositories.
• Unnecessary texlive installs in this repository.

I think we have to distinguish between using GAPDoc's utility functions and calling GAPDoc for creating a package manual. And in the latter case, wouldn't it be enough to leave out the creation of the PDF version of the manual from CI tests?

Absolutely agree on the distinction.

I don't think any packages currently try to build a pdf version of their manual as part of their CI workflow? But the build-pkg action, and previously the gather_dependencies.py script in this repository, pick up texlive as a needed system package through the GAPDoc dependency regardless.

Do we know of any other package that really needs TeX in its tests? Even if, though: might be best to only install it for these few packages, explicitly?

I think Typeset and PackageManager both do, but only the latter relies on texlive being installed "transitively".

In this repository, we currently do this:

PackageDistro/tools/gather_dependencies.py

Lines 66 to 67 in 9eee3d6

	if not pkg in seen and (pkg != "gapdoc" or pkg_name == "packagemanager"):
	deps |= gather_dependencies(pkg, seen)

So that's more or less your suggestion, I think? In theory we will also install texlive for any packages declaring PackageManager as a dependency, but such packages currently do not exist.

I could make a similar change to the build-pkg action? Either by hardcoding packagemanager as an exception, or by adding a new input to determine whether to ignore GAPDoc's dependencies.

[fingolfin]

perhaps we should also simply ask Frank to revert the addition of texlive to GAPDoc's PackageInfo.g -- he wasn't thrilled about it anyway ;-).

Then no changes in build-pkg would be necessary; only the PackageManager CI would have to "manually" install texlive, which seems OK to me?