Video support

[jimjam-slam]

I've added the first part of this work in 998617c and 485f235: videos that autoplay and loop. Really it's just a class to ensure that they run full-bleed; the looping and autoplay is done by the browser.

But as #132 discusses, we might want video to hold off starting until it's visible. So the next part is to trigger that manually wth JS.

I think it probably makes sense to add a shortcut, analogous to Quarto's video shortcode, to do this while inserting the necessary attributes (eg. preload) on the video tag. (In fact, we may end up borrowing their chortcode and tweaking it!)

The last part of this is the most complicated: image sequences that progress as you scroll through the container (although I'd love to know if you can do this with a traditional video!).

Most of the implementations of this I've seen (eg. this one on dev.to — it uses React, but the principle should apply with vanilla JS) use a canvas element: you preload the images by calling a function ASAP to download them, then call requestAnimationFrame() on scroll to update which image is showing.

A user might either want to use regenerated images or ones generated by a code block in the doc — but it isn't clear to me whether there's some special treatment we can give, for example, an R code block to say "use the images emitted from this code block for a scroll video".

That said, perhaps you could have:

a. a sticky block where the image glob or image path is specified as an attribute, and then
b. a trigger attribute that specifies how far through the sequence. Or perhaps a progress block... or perhaps both are viable options?

::::{.cr-section}

<!-- or maybe a shortcode makes sense for this sticky? -->
:::{#cr-images video-scroll-first="images/pic001.png" video-scroll-last="images/pic135.png"}
:::

Check out this awesome video [@cr-images]{video-scroll=1}

It's pretty rad [@cr-images]{video-scroll=20}

Now we're going quite fast [@cr-images]{video-scroll=80}

We've paused for a bit [@cr-images]{video-scroll=80}

Now we're finished [@cr-images]{video-scroll=120}

::::

Or maybe with a progress block (although no option to vary the speed here):

<!-- can we reuse focus-on and detect that it's scrolling? -->
:::{.progress-block focus-on="cr-images"}

Step 1

Step 2

Step 3

Step 4
:::

What do you think, @andrewpbray?

[andrewpbray]

Thanks for getting this started! Lots of fun stuff here.

Feature 1: Play videos with trigger

In terms of API, what do you think of:

Checkout this lovely video. [@cr-vid]{play-video="true"}

I'm inclined to avoid any mention of autoplay since that's less well-defined in this setting since the user is in control of when the video would become un-transparent. Not sure what the best attribute name is: play, play-vid, play-video? I can take a crack at the necessary JS for this later on today.

Feature 2: Control video progression with scroll

Seems simpler to use a progress block! Controlling the play speed could always be done on the video file itself in pre-processing, right? (i.e. removing frames).

I'm not quite sure what the best API for this is. I like what you have. There's also this:

:::{.progress-block scroll-video="cr-vid"}

Step 1

Step 2

Step 3

Step 4
:::

A few things:

• While we could infer what to do if a progress block is focusing on a video, might be better to make that explicit in the attribute name.
• Seems like we'll be able to do the png sequence and less clear about setting the frame of the video file. That'd be good to know for deciding if we should use separate attribute names for those two or combine them.
• Seeing scroll-video here makes me wonder if start-video is clearer for feature 1 (since scrolling is also a form a playing a video).

[jimjam-slam]

@andrewpbray Agree with you on Feature 1! autoplay is too confusing (and it's a browser attribute, so it ought to be avoided).

I agree that a progress block is a nice interface! While I do also agree that you can change the number of frames to tweak the output speed, I think at minimum it would be good for users to be able to pause the progression between a number of steps where they want to stop and discuss something in more detail. You could technically run duplicate frames off, but it'd be a big waste of bandwidth. Maybe something like:

:::{.progress-block scroll-video="cr-vid"}

Step 1

Step 2 [@cr-vid]{pause-video}

Step 3 [@cr-vid]{pause-video}

<!-- video resumes as you pass this step -->
Step 4

Step 5
:::

These sorts of pauses would be helpful for videos that sweep over an area (eg. as events unfold over time).

[andrewpbray]

Good point - that'd be some very useful functionality.

One thing I'm realizing: we have crProgressTrigger and crProgressBlock. Do we want an interface that would, in concept, work for both?

Scroll a video through a single narrative block

:::{#cr-myvid}
{{< video myvid.mov >}}
:::

Step 1. [@cr-myvid]{.scroll-video}

Scroll a video through a progress block

:::{#cr-myvid}
{{< video myvid.mov >}}
:::

:::{.scroll-video focus-on="cr-myvid"}

Step 1.

Step 2.

Step 3.
:::

This raises a few interface questions:

1. Should focus effects always be attributes or can they be classes in situations where this is no arguments/values that we need to pass?
2. How should we get the trigger and progress block version aligned? For the former, .scroll-video or scroll-video="true" seem to make sense since the ref to the sticky is right there. But then how should we pass the ref to the sticky in the progress block version? focus-on is one option but would actually mess things up: I think it'd put all three steps as separate Paras in the same narrative block without the padding. .progress-block scroll-video="cr-vid" would work but then we have a different interface than the single trigger scroll.
3. If we use .progress-block scroll-video="cr-vid", do we even need .progress-block? Seems like no. Seems to make sense to keep that one narrowly scoped to exposing that ojs variable.

[andrewpbray]

Maybe we wade into getting a working example of an image sequence and see how it turns out. That'll probably help clarify which interface makes the most sense. Looking back, my stuff above is all based on the general notion of a "video" but image sequences are a slightly different beast; there's not just one file to point to.

I love the {{< image-sequence start="file1.png" end="file15.png" >}} idea by the way! (or something in that neighborhood.

[andrewpbray]

@jimjam-slam Here's a first pass at implementing feature 1.

It's implemented generically so that play-video="true" and pause-video="true" both work, but really any available video method will work when passed as ___-video="true".

One thing to decide is question 1 above. Should these be classes instead of attributes? This should be determined I think just based on what seems like the best interface. The implementation is straightforward - change this JS function to reference classes instead of attributes and then modify the lua filter so that a trigger will pass both classes and attributes to its enclosing block (wrap_block currently only passes attributes).

While testing this out, I did bump into this: https://developer.chrome.com/blog/autoplay/. Essentially, if the user does nothing but scroll straight down, it might not execute play().

[andrewpbray]

I've been tinkering this morning with tying progressBlock.progress to video.currentTime; effectively skipping the video to an appropriate frame as the progressBlock increments. I haven't quite gotten it working yet but just discovered this:

https://scrollyvideo.js.org/

In order words, I've been trying the "traditional" method 3. What are your thoughts on using a dedicated library for this sort of thing? It looks like the author has been working is within the last 3 mo, which is a good sign. If these video standards change over time, sure would be nice if we could take advantage of his keeping things up to date.

[jimjam-slam]

I've gotta come back to your comments properly, sorry @andrewpbray! I had half an hour and banged out some rayshader code to generate a sample video.

[jimjam-slam]

One thing to decide is question 1 above. Should these be classes instead of attributes? This should be determined I think just based on what seems like the best interface. The implementation is straightforward - change this JS function to reference classes instead of attributes and then modify the lua filter so that a trigger will pass both classes and attributes to its enclosing block (wrap_block currently only passes attributes).

Usually boolean attributes in HTML work without the ="true" part, so a user could just add play-video or pause-video. That doesn't presently work, but I'm guessing that the current implementation explicitly checks in Lua for a value of true (rather than just the presence of the attribute key).

Okay, I've just deleted a big wall of text — turns out that apart from specific HTML boolean attributes (like the ones that actually go on video elements), where the mere presence of the boolean attribute makes it true, attributes are always supposed to have values. So maybe {play-video} is a bad idea and it should be a class (in fact, Pandoc aborts processing on a span or div when it encounters an attribute with no value, so we'd need to implement a Lua filter to make it work anyway!).

[jimjam-slam]

I've been tinkering this morning with tying progressBlock.progress to video.currentTime; effectively skipping the video to an appropriate frame as the progressBlock increments. I haven't quite gotten it working yet but just discovered this:

https://scrollyvideo.js.org/

In order words, I've been trying the "traditional" method 3. What are your thoughts on using a dedicated library for this sort of thing? It looks like the author has been working is within the last 3 mo, which is a good sign. If these video standards change over time, sure would be nice if we could take advantage of his keeping things up to date.

I'll try hooking this up now — if we can do it with an external lib, I'm down for that! It seems like it's handling a few tasks (ie. it's not just setting the progress as we probably would), so it probably makes sense to have a dependency handle it.

On a semi-related note (which I'll spin out to a separate issue later), it might be worth us incorporating a web bundler to track our external dependencies...

[jimjam-slam]

One other thing: I realised that when I set up the demo with the ship and tea videos, I wrote them inline:

![](tea.mp4){#cr-tea .scale-to-cover alt="One person pours tea into another person's tea cup in front of an open fire." autoplay="true" loop="true"}

As opposed to:

:::{#cr-tea}
![](tea.mp4){.scale-to-cover alt="One person pours tea into another person's tea cup in front of an open fire." autoplay="true" loop="true"}
:::

or:

:::{#cr-tea .scale-to-cover}
![](tea.mp4){alt="One person pours tea into another person's tea cup in front of an open fire." autoplay="true" loop="true"}
:::

The commit I've just pushed generalises the CSS a bit more so that it should work regardless of whether you put .scale-to-cover and the Closeread ID on the fenced div or directly on the video (or the ID on the div and the class on the element). It probably still needs some testing with respect to the figure problems we've encountered in the past.

[jimjam-slam]

I've managed to load the video with ScrollyVideo, but the default trackscroll: true doesn't work — I think because the demos on their page have the sticky video as a direct sibling of the scrolling cards, whereas we have them split up. I think we probably need to use the provided method .setVideoPercentage() in the relevant OJS scroller (keeping in mind that, as you proposed, we may want a progress block-based option!). Will try to keep at this tomorrow!

[jimjam-slam]

a40df27 has a single scrollytelling video working (I've hardcoded it straight into the .js file, so it'll break the rest of the site). So it seems like it works conceptually!

This demo just ties things to trigger progress, so when you scroll from one trigger to another, it goes from 100% to 0% and the video essentially rewinds quickly. I think for that case you'd probably want no interpolation at all (if, indeed, it is require usually!).

Generalising this means working out what the JS code has to track in order to potentially advance several videos. I think it's probably worth us talking through the API in person @andrewpbray before I jump into that!

[jimjam-slam]

Just summarising our discussion @andrewpbray — this was a candidate API we were looking at (but still discussing if either of us is unhappy with it!):

Non-scrolling videos

::::{.cr-section layout="overlay-left"}

You can tell all sorts of stories with videos.

Maybe you've gone on a journey.  @cr-ship

:::{#cr-ship .scale-to-cover}
![](ship.mp4){alt="A ship sails in a polar region" loop="true"}
:::

You can trigger a video to play using `[@cr-ship]{play-video="true"}`. [@cr-ship]{.play-video}

You can also pause it using `[@cr-ship]{pause-video="true"}`. [@cr-ship]{.pause-video}

Play the video again [@cr-ship]{.play-video}

The video keeps playing

One thing I'd noticed but might spin into a separate issue is that we never show a sticky until the first trigger hits. Videos are a case where you'd probably want the first sticky to be visible without having to hit a sticky first, but there might be others (in fact it might even be an appropriate default).

Scrolling videos

:::::{.cr-section}

::::{.progress-block scroll-video="cr-rayshader"}
Here's our rayshader video!

We can keep talking about it for a while... [@cr-rayshader]{.pause-video}

... and a while longer!
::::

:::{#cr-rayshader}
:::
:::::

Some considerations for scrolling video:

Right now this scrolling video is hardcoded in to the JS code, but in real use it needs to be specified in the QMD. Two ways we could do that:

1. It's specified in the progress block: scroll-video="cr-some_id" becomes scroll-video="rayshader.mp4" and the video sticky block is added programmatically by Lua instead of being supplied by the user

2. a video is supplied in the sticky just as with a non-scrolling video. If we detect scroll-video attribute and a video inside the corresponding ID in the Lua, we remove the video and decorate the cr-section with the video URL so that the JS code can hook it up.

We might want to consider how our choice in API impacts people's ability to transition from a scrolling video to some other type of sticky (is there any impact?).

[jimjam-slam]

Finally cracked onto this! f6fa1e8 gets the bones of it done.

It currently works with individual triggers but not progress blocks, and I've no idea why. I modified the Lua to ensure the progress block classes and attributes are retained (so they can be used as video scrolling triggers). I've mirrored the code for the trigger step progression for progress blocks. I've also given the scrolling video triggers IDs so we can progress the correct video when Scrollama fires an update.

The data looks the same to my eye in both scenarios, but the progress block one doesn't seem to initialise :( Would love to get your eyes on it once you're back, @andrewpbray!

[andrewpbray]

Finally getting to this today! Stay tuned...

[andrewpbray]

I was able to get this working by adding a @cr-rayshader-prog as a trigger on this first para. Something must be amiss in the lua for assigning getting line 132 to assign the .cr-active class to the video.

[andrewpbray]

also: i switched off the rayshader example while troubleshooting to save the video-making time when re-rendering.

[jimjam-slam]

Oh, amazing! I was tearing my hair out on that one. Yeah, freeze is nice, but it doesn't help if you're tweaking the actual QMD 😭

[jimjam-slam]

So turns out in the JS we were setting .cr-active on trigger enter but not on progress enter. So that's sorted now!

[jimjam-slam]

• Change scroll-video-"true" to .scroll-video (and .play-video, .pause-video)
• Look at causes of inconsistency in single trigger scrolls
• Progess block-based scroll: .scroll-video focus-on="cr-myvideo" on progress block (can we infer .progress-block from that too?)
• Address Andrew's other review feedback

[jimjam-slam]

I'm finally getting back into this, but I've been a little held up because it looks like macOS Tahoe might have broken XQuartz, which {rgl} (and hence {rayshader} use. I'm looking into some workarounds, but it's very likely that I'll just swap this example out for another (admittedly less exciting) example, like gganimate.

dmurdoch/rgl#488
XQuartz/XQuartz#438

[tylermorganwall]

As a note, the latest version of rayshader on github (tylermorganwall/rayshader@65c015c, v0.39.3) will simply forward all arguments from render_movie() to render_snapshot(), so you can set software_render = TRUE and it will render all the frames without the need for OpenGL/XQuartz.

[jimjam-slam]

Thanks very much, @tylermorganwall — I'll give this a crack and see if we can restore the rayshader example! I reckon it's a very good fit for scrollytelling 🥳

[jimjam-slam]

Thanks Tyler, can confirm this works great!

[jimjam-slam]

Okay, I've had another swing at this and it's working better now — I've reverted all the attributes to classes, gotten the trigger- and progress-based scrollies working, and restored the rayshader demo (I've set it to use the software rendered on a Mac but to use the faster X11 one otherwise).

I'd love to get some eyes on the changes, @andrewpbray, especially as I dipped into both the Lua and the JS a good bit. The Lua changes revolved around ensuring that classes go where they need to go on triggers (as attributes do) — I limited the transfer to our video-related classes rather than all classes, as I judge that usually if someone's adding their own class to a trigger, they probably intended to target just the content and not the entire (and invisible) trigger.

Still keen to tweak the easing and transition speed of the scrolls, as they're a bit jerkier than I'd like!

If you're interested in just comparing this week's changes to the state of the PR in September, here's that link:

f6fa1e8...29a7e13

Thanks!

[jimjam-slam]

(Need to add the R packages to the GHA too!)

[jimjam-slam]

I've added some polish to the demo now — I'd thought it was broken, but the feature actually works well! I'd been a bit stuck because of an issue with rayshader on macOS (documented in the demo with a workaround).

Slowing the frame rate of the video down paradoxically makes it work a lot better with scrolling: issues that I thought required tweaking the Scrollama interpolation settings might actually just be a matter of having a lower frame animation.

Also, the first version with individual triggers (as an example of what not to do) was also bad in terms of the Scrollama execution. That's gone now, and the progress block version works much better.

This might actually be fine to ship as is! I may not get back to widgets before we chat tomorrow morning, but I'm happy to see at least one major feature is finally locked down 😊

Still need to add the R packages to the GHA! ray shader in particular needs the dev version on Mac (though the CRAN version might be fine on GHA; I think the software_render arg just gets dropped on earlier versions).

[andrewpbray]

Hmm...I'm running into issues getting all the dependencies necessary for the rayshader video. Since this is just a demo of the video functionality, should we use an existing video and then save the rayshader for a different demo?