Skip to content

Commit 3414d03

Browse files
committed
docs: add explicit documentation for site-specific behaviours
We previously pointed users at the browsertrix-behaviors repo, but that repo didn't actually have substantive documentation about what they could do. Let's unify the docs here in one place along with the other places that users of the crawler are consulting.
1 parent 505f824 commit 3414d03

1 file changed

Lines changed: 61 additions & 3 deletions

File tree

docs/docs/user-guide/behaviors.md

Lines changed: 61 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,12 +16,66 @@ There is also a built-in 'main' behavior, which runs to completion (or until a t
1616

1717
## Site-Specific Behaviors
1818

19-
Browsertrix also comes with several 'site-specific' behaviors, which run only on specific sites. These behaviors will run instead of Autoscroll and will run until completion or timeout. Currently, site-specific behaviors include major social media sites.
19+
Browsertrix also comes with several 'site-specific' behaviors, which run only on specific sites. These behaviors will run instead of Autoscroll and will run until completion or timeout. Currently, site-specific behaviors include the following major social media sites:
2020

21-
Refer to [Browsertrix Behaviors](https://github.com/webrecorder/browsertrix-behaviors) for the latest list of site-specific behaviors.
21+
- Bluesky
22+
- Facebook
23+
- Instagram
24+
- Telegram
25+
- TikTok
26+
- Twitter/X
2227

2328
User-defined custom behaviors are also considered site-specific.
24-
29+
30+
### Bluesky
31+
32+
The Bluesky behavior loads additional JavaScript artifacts in order to ensure that Bluesky posts can be replayed with the user interface in any language.
33+
34+
### Facebook
35+
36+
The Facebook behavior is able to automate the following features:
37+
38+
- Viewing posts from the timeline for an individual, organization, or group page
39+
- Expanding the comment section in a post or photo in order to load additional comments
40+
- Browsing the photos grid and viewing full size versions of posts and their descriptions
41+
- Browsing all reels (videos) on a page
42+
43+
The Facebook behavior will use `addLink` to add single-page versions of posts encountered when crawling a profile to the crawl queue. For more information on `addLink` and scoping, [see below](#additional-links-from-behaviors).
44+
45+
### Instagram
46+
47+
The Instagram behavior is able to automate the following features:
48+
49+
- Viewing all photos/videos on a single post
50+
- Viewing all media from a profile
51+
- Viewing the currently-active story (requires a logged-in profile)
52+
- Viewing story highlights
53+
- Expanding the comment section in order to load additional comments
54+
55+
The Instagram behavior will use `addLink` to add stories and single-page versions of posts encountered when crawling a profile to the crawl queue. For more information on `addLink` and scoping, [see below](#additional-links-from-behaviors).
56+
57+
### Telegram
58+
59+
The Instagram behavior is intended to work with public Telegram channels. It supports the following features:
60+
61+
- Scrolling through all of the messages in a public channel
62+
- Loading external images
63+
64+
### TikTok
65+
66+
The TikTok behavior is designed to expand comment threads when viewing single videos.
67+
68+
### Twitter/X
69+
70+
The Facebook behavior is able to automate the following features:
71+
72+
- Viewing a timeline
73+
- Viewing a specific profile
74+
- Viewing single posts
75+
- Clicking through "view sensitive" buttons to ensure content is captured (requires a logged-in profile)
76+
- Viewing full-size images, audio and video
77+
- Viewing quoted posts
78+
2579
## Enabling Behaviors
2680

2781
To enable built-in behaviors, specify them via a comma-separated list passed to the `--behaviors` option. All behaviors except Autoclick are enabled by default, the equivalent of `--behaviors autoscroll,autoplay,autofetch,siteSpecific`. To enable only a single behavior, such as Autoscroll, use `--behaviors autoscroll`.
@@ -60,6 +114,10 @@ Custom behaviors are specified with the `--customBehaviors` flag, which can be r
60114
- A URL for a single behavior file to download. This should be a URL that the crawler has access to.
61115
- A URL for a git repository of the form `git+https://git.example.com/repo.git`, with optional query parameters `branch` (to specify a particular branch to use) and `path` (to specify a relative path to a directory within the git repository where the custom behaviors are located). This should be a git repo the crawler has access to without additional auth.
62116

117+
### Additional Links From Behaviors
118+
119+
Although behaviors can use `addLink` to add additional pages to the crawl queue, these pages are still subject to the crawl scope. This means that certain extra social media links may not be crawled. The `--alwaysAddBehaviorLinks` crawler flag can be used to instruct the crawler to always crawl these pages, even if they are outside the initial scope defined for the crawl. (In Browsertrix, this is controlled via the "Smart Scoping" setting in the user interface.)
120+
63121
### Examples
64122

65123
#### Local filepath (directory)

0 commit comments

Comments
 (0)