fix: use --group for RTD docs dependency install

The docs dependencies are in [dependency-groups], not
[project.optional-dependencies], so pip install '.[docs]' fails.
Use pip install --group docs to match the pattern from #2902.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Author: JacobCoffee

.readthedocs.yaml

@@ -10,6 +10,6 @@ build:
     python: "3"
 
   commands:
-      - python -m pip install '.[docs]'
+      - python -m pip install --group docs .
       - make -C docs html JOBS=$(nproc) BUILDDIR=_readthedocs
       - mv docs/_readthedocs _readthedocs