Skip to content

De-index old versions manually with robot meta tags#444

Closed
Yann-P wants to merge 1 commit into
napari:gh-pagesfrom
Yann-P:unindex
Closed

De-index old versions manually with robot meta tags#444
Yann-P wants to merge 1 commit into
napari:gh-pagesfrom
Yann-P:unindex

Conversation

@Yann-P

@Yann-P Yann-P commented May 22, 2026 •
edited
Loading

Copy link
Copy Markdown

napari/docs#364

This is a one-shot fix for versions prior to html_baseurl being added to napari/docs's config.py napari/docs#1029.

Script used

STABLE=$(readlink stable)
for d in */; do
  case "${d%/}" in stable|"$STABLE"|dev|_*) continue;; esac
  find "$d" -name '*.html' -exec sed -i '' -e 's|<head>|<head>\
    <meta name="robots" content="noindex,follow">|' {} +
done

Assisted-By: Claude (to generate the script). I checked that no stable docs contain nofollow (0.7.0).

@Yann-P
De-index old versions manually with robot meta tags
82df175 
napari/docs#364

This is a one-shot fix for versions prior to html_baseurl being added to
napari/docs config.py.

Script used
```bash
STABLE=$(readlink stable)
for d in */; do
  case "${d%/}" in stable|"$STABLE"|dev|_*) continue;; esac
  find "$d" -name '*.html' -exec sed -i '' -e 's|<head>|<head>\
    <meta name="robots" content="noindex,follow">|' {} +
done
```

Assisted-By: Claude
@Yann-P Yann-P mentioned this pull request May 22, 2026
Merged
@Yann-P

Yann-P commented May 22, 2026

Copy link
Copy Markdown
Author

I think this broke github, there are in fact 8294 files changed and not zero...

napari.github.io unindex % git show origin/gh-pages..82df17518f78d890feb452f1ca9663507893dddc --stat | tail -n 10
 tutorials/fundamentals/viewer.html                                       | 1 +
 tutorials/index.html                                                     | 1 +
 tutorials/processing/dask.html                                           | 1 +
 tutorials/processing/index.html                                          | 1 +
 tutorials/segmentation/annotate_segmentation.html                        | 1 +
 tutorials/segmentation/index.html                                        | 1 +
 tutorials/start_index.html                                               | 1 +
 tutorials/tracking/cell_tracking.html                                    | 1 +
 tutorials/tracking/index.html                                            | 1 +
 8294 files changed, 8294 insertions(+)
napari.github.io unindex %

@TimMonko

TimMonko commented May 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

@Yann-P thank you again for the full circle fix!
Can I ask why you/Claude used noindex,follow instead of noindex,nofollow
It also seems that follow is the default, so wouldn't need specificed.

I gotta agree with my college students on this one, searching is absolutely not leading me to anything useful ... just regurgitated AI slop websites or vague forum posts. Edit: YAY I think I found the actual documentation now.... https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag

In reading what I can it seems like nofollow will help in cases of forum posts that point to older pages, the search engines won't give "reputation" to the link

@Yann-P

Yann-P commented May 26, 2026

Copy link
Copy Markdown
Author

In my understanding "nofollow" tells Google to not trust these links / mark them as not controlled by us.

Noindex should be enough

@Yann-P

Yann-P commented May 27, 2026

Copy link
Copy Markdown
Author

As I wrote in napari/docs#1029, I think we should try to just merge the html_baseurl, publish a version, and see if this one is still needed.

If it is, I suggest you use the claude-generated script in the PR description locally and push instead of merging this PR that is already drifting out-of-date and that Github can't handle anyway because of its size.

Moreover I wonder if a robots.txt like this one would work if we want to hard-deindex all old versions.

https://github.com/ray-project/ray/blob/ba235570696d1525e74534be213d8d1a168ae28a/doc/source/robots.txt

@Yann-P

Yann-P commented May 27, 2026

Copy link
Copy Markdown
Author

OR we create a robots.txt for all existing versions of napari and let the canonical tag do its job for all future versions of napari

Disallow: /0.1
Disallow: /0.2
...
Disallow: /0.6.6

TimMonko added a commit to napari/docs that referenced this pull request May 27, 2026
@Yann-P @TimMonko
Add baseurl to favor stable versions in search engines (#1029)
cb36660 
# References and relevant issues
<!-- What relevant resources were used in the creation of this PR?
If this PR addresses an existing issue on the repo,
please link to that issue here as "Closes #(issue-number)".
If this PR adds docs for a napari PR please add a "Depends on <napari PR
link>" -->

#364

# Description
<!-- What does this pull request (PR) do? Does it add new content,
improve/fix existing
context, improve/fix workflow/documentation build/deployment or
something else?
<!-- If relevant, please include a screenshot or a screen capture in
your content
change: "An image is worth a thousand words!" -->
<!-- You can use https://www.cockos.com/licecap/ or similar to create
animations. -->
<!-- You can also see a preview of the documentation changes you are
submitting by
clicking on "Details" to the right of the "Check the rendered docs
here!" check on your PR.-->

Set html_baseurl which translates to canonical to point to the stable
version.

However, for this to be effective we need to set noindex as suggested on
#364, which is what I did in this
additional pull request:

napari/napari.github.io#444



<!-- Previewing the Documentation Build
When you submit this PR, jobs that preview the documentation will be
kicked off.
By default, they will use the `slimfast` build (`make` target), which is
fast, because
it doesn't build any content from outside the `docs` repository and
doesn't run notebook cells.
You can trigger other builds by commenting on the PR with:

@napari-bot make <target>

where <target> can be:
html : a full build, just like the deployment to napari.org
html-noplot : a full build, but without the gallery examples from
`napari/napari`
docs : only the content from `napari/docs`, with notebook code cells
executed
slimfast : the default, only the content from `napari/docs`, without
code cell execution
slimgallery : `slimfast`, but with the gallery examples from
`napari/napari` built
-->

<!-- Final Checklist
- If images included: I have added [alt
text](https://webaim.org/techniques/alttext/)
If workflow, documentation build or deployment change:
- My PR is the minimum possible work for the desired functionality
- I have commented my code, to let others know what it does
-->

Co-authored-by: Tim Monko <timmonko@gmail.com>
@TimMonko TimMonko mentioned this pull request May 27, 2026
Closed
@TimMonko TimMonko mentioned this pull request May 31, 2026
Closed
@TimMonko

TimMonko commented May 31, 2026

Copy link
Copy Markdown
Contributor

Closing in favor of waiting an and a tracking issue #445

@TimMonko TimMonko closed this May 31, 2026
@TimMonko TimMonko mentioned this pull request May 31, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Yann-P @TimMonko