Build site using Hugo instead of Jekyll (upgrade tracking issue)

[harding]

Since its inception, this site has always been built with Jekyll but Jekyll appears to be largely unmaintained; it's changelog indicates the last time a major feature was added was "4.0.0 / 2019-08-19". I don't we particularly need new features from the static site compiler, but as we consider adding new features to the site, I want to make sure we're building on a foundation that will continue to be supported for years to come.

Hugo appears to be actively maintained. In my tests, it also completes a whole site compile at least 10x faster than Jekyll. I've locally completed some preliminary conversion of our site to Hugo and I don't see anything that will prevent us from a complete conversion. This issue for for discussing any objections to a conversion as well as tracking its progress. The plan is:

• Work on converting the site in a local branch until near parity has been reached between the Jekyll and Hugo versions
• (In parallel with the above) Open PRs for changes to Jekyll content to simplify conversion
• When near-parity is reached:
  • update our CI infrastructure to begin building and testing both Jekyll and Hugo versions
  • Host a preview of the Hugo version for reviewers
• After a few weeks of parallel building and collecting reviews, switch live site to Hugo version (with an easy revert option in case of problems); continue building and testing the Jekyll version
• After a few weeks with Hugo running in production, drop support for the Jekyll version

Please let me know your thoughts on converting. Until we complete conversion, I suggest a moratorium on adding major new features to the site, with the exception of an update to the Compatibility Matrix (as work on that has already begun).

[murchandamus]

I like this approach. While it might be a bit more work to maintain the dual infrastructure until the switch-over, it should make the switch significantly smoother than some sort of one-time switch over. I don’t have much information on either Hugo or Jekyll, but generally running on an unmaintained stack seems like a source of future pain, so what you write sounds sensible to me. Also very excited not to wait a few minutes every time I want to preview a branch. :D

[kouloumos]

This is not an immediate action, but when/if you need to re-create the Topics index in Hugo native way, I might be able to help as we essentially mirrored the index for bitcointranscripts.

[harding]

@kouloumos thanks, that is helpful!

[harding]

The largest remaining challenge I have for the conversion to Hugo are the podcast pages. I want to propose a change to how we generate those, hopefully without creating too much extra work for @bitschmidty. First, here's an overview of how they work now.

• After an episode is recorded and post-processed, Mike adds tags like {% assign timestamp="20:08" %} to the corresponding newsletter file.
• Mike creates a post for the podcast episode, such as 2024-04-25-newsletter-recap.md. This references the corresponding newsletter
• When the site is generated, _plugins/recap_references_generator.rb looks at every podcast page and then:
  • Goes to its referenced newsletter page and builds a list of each timestamp reference, extracting the name of the section of the page (e.g. "News", "Notable code changes", etc.) and the item name (e.g. "Vulnerability disclosure foo", "Bitcoin Core n12345", etc.), while also replacing the {% assign timestamps %} with a link to an item name on the podcast page
  • Generates the top part of the podcast page with the section names, names of each item, links back to the newsletter page, looks for the names of items in the transcript content (in special formating) and (if it finds them) adds a link to them next to the items in the top of the page

This is a lot of nice automation (thanks @kouloumos !). The problem I'm having now is that Hugo doesn't natively provide anywhere near the same degree of content manipulation ability (AFAICT). I think it might be theoretically possible to replicate the above using Hugo's templating language, but it's far beyond my ability. Here's what I propose instead:

• After an episode is recorded and post-processed, Mike adds tags like {% include podlink name="Foo bar baz", link="#foo-bar-baz", timestamp="20:08" %} to the newsletter page ; this includes the timestamp like before but also the name that Mike wants to use for the item on the podcast page and the link to it on the Newsletter page. This will probably be a quick copy/paste from the newsletter page.
• After adding all the timestamps, Mike manually runs a script that builds a template for the podcast page by processing the newsletter page, extracting each section title and all the names and timestamps. This builds something that looks exactly like the current dynamically generated default podcast pages.
• After creating the podcast page, Mike adds a field to the newsletter page YAML indicating the URL of the podcast page, which turns all the {% include podlink %} callouts into the headphone icon links.
• After Mike adds the transcript with anchors corresponding to item names, he toggles a field in the podcast page YAML to True which will now show links at the top of the page to point to the specific content in the transcripts

I think that's only a medium amount of extra work each week for Mike but it eliminates the need for a plugin on the Jekyll side and makes implementing the podcast/transcript stuff in Hugo very straightforward.

A small potential upside of this is that the new method is a bit more flexible because it's entirely based on static content. If we have weird newsletters (like the year in review newsletters) or decide to do podcasts of blog posts or other stuff, or if anyone wants to manually edit the top part of a podcast page, any non-standard stuff can be addressed with manual editing of the source content.

Feedback appreciated!

[kouloumos]

I think it might be theoretically possible to replicate the above using Hugo's templating language,

Do you have a branch with the current state of the transition to Hugo? I can try to replicate the current logic.

[harding]

@kouloumos oh, that'd be neat! I was hoping to open a WIP PR soon anyway, so I'll ping you here as soon as I get that done.

[bitschmidty]

@kouloumos Thanks for taking a look at Hugo and replicating the current logic there.

@harding If replicating in Hugo doesnt work, Im up for this (more manual, more flexible) approach to keep the ball moving on Hugo.