Run doctests with doctest, not cabal-doctest

[philderbeast]

Fixes #11493.

• Uses the recommended way of running doctests with the doctest tool, not the way using the experimental cabal-doctest external command.
• Runs cabal repl in quiet mode so that we only see the doctests summary output.
• Adds a -Wno-inconsistent-flags warning suppression that only seems to be triggered by Cabal-syntax.

Note

• Doctest content changes, I've split from this pull request.
  Moved to -XCPP doctest expectations for ghc-9.14 and above #11797.
• The doctest-% pattern rule, I've split from this pull request.
  Moved to Add a doctest-% pattern target #11799.

---

• Patches conform to the coding conventions.
• Is this a PR that fixes CI? If so, it will need to be backported to older cabal release branches (ask maintainers for directions).

[ulysses4ever]

there's a lot in here...

One thing I worry about is making the Makefile more complicated with generating targets ($(addprefix...). I prefer the previous setup, and I'd not want to maintain this sort of metaprogramming in Makefiles...

As for switching back to cabal repl, what's your motivation for that? I quite like the new way because the command line is much cleaner. Also, @sol would be welcome to opine on cabal doctest vs cabal repl -w doctest these days.

[philderbeast]

As for switching back to cabal repl, what's your motivation for that? I quite like the new way because the command line is much cleaner.

The motivation is fixing #11493. I agree, the cabal-doctest external command is cleaner but experimental and doesn't work for us with ghc-9.12 or ghc-9.14.

[philderbeast]

One thing I worry about is making the Makefile more complicated with generating targets ($(addprefix...). I prefer the previous setup, and I'd not want to maintain this sort of metaprogramming in Makefiles...

We already have variables with lists in the makefile:

cabal/Makefile

Lines 28 to 50 in e6ff0cc

	# NOTE: Keep this in sync with `.github/workflows/format.yml`.
	FORMAT_DIRS := \
	Cabal \
	Cabal-syntax \
	cabal-install \
	cabal-testsuite/src \
	cabal-validate \
	Cabal-tests/exes
	FORMAT_DIRS_TODO := \
	Cabal-QuickCheck \
	Cabal-described \
	Cabal-hooks \
	Cabal-tests \
	Cabal-tree-diff \
	bootstrap \
	buildinfo-reference-generator \
	cabal-benchmarks \
	cabal-dev-scripts \
	cabal-install-solver \
	cabal-testsuite/main \
	cabal-testsuite/static \
	solver-benchmarks

Instead, it'd be good to have a generic target e.g.

doctest-%:
    cd $* && $(DOCTEST)

-- doctest-% suggestion

I included your above suggestion but without the cd as there isn't a need to change directory.

I also added convenience doctest-PACKAGENAME targets to help speed things up when someone wants to use the makefile to run doctests on one package. I reuse DOCTEST_TARGETS in the recipe of doctest-PACKAGENAME and for the prerequisites of the doctest phony target.

DOCTEST_PACKAGES := \
  Cabal \
  Cabal-described \
  Cabal-syntax \
  cabal-install \
  cabal-install-solver \
  cabal-testsuite

DOCTEST_TARGETS := $(addprefix doctest-, $(DOCTEST_PACKAGES))

doctest-%: ## Run doctests for a specific package.
	$(DOCTEST) $*

doctest-PACKAGENAME: ## Run doctests for a single package (replace PACKAGENAME).
	@echo 'Please use one of the following targets:'
	@printf "%s\n" $(DOCTEST_TARGETS)

.PHONY: doctest
doctest: ## Run doctests in all packages.
doctest: $(DOCTEST_TARGETS)

[philderbeast]

I see doctest-0.25.0 was just released 2026-05-03. Maybe that fixes cabal-doctest?

[ulysses4ever]

Ah, got you. It was hard to discern given the slew of changes... We really should report it upstream: do you think you could do that?

[ulysses4ever]

I included your above suggestion but without the cd as there isn't a need to change directory.

I think one cd is much better than adding one $(addprefix...)

[sol]

@philderbeast what is the reason you pinned doctest to 0.24.3?

In general, I would prefer to stay with cabal doctest. If there are bugs, then I think the right approach is to fix these.

@ulysses4ever thanks for looping me in.

[philderbeast]

what is the reason you pinned doctest to 0.24.3

The doctest-0.25.0 release breaks the doctests as written in this PR with ghc-9.14.1. Until I work out why, I'm pinning it on this PR.

[philderbeast]

there's a lot in here...

@ulysses4ever I've split this pull request. Please see the note in the description of this pull request for details.

[philderbeast]

In general, I would prefer to stay with cabal doctest. If there are bugs, then I think the right approach is to fix these.

Me too @sol, and I would be very happy to switch back to cabal doctest when fixed but in the meantime, I at least, would like our doctests to work with ghc-9.12.* and ghc-9.14.*. I'm also available to help with getting cabal doctest to work on this repo for those compiler versions but I think we should do that work on a branch.

[philderbeast]

what is the reason you pinned doctest to 0.24.3?

@sol I've logged this as sol/doctest#487

[sol]

@philderbeast have you seen my comment at #11493 (comment).

[philderbeast]

I'm closing this in favour of #11929.