chore: test Academy exercises by honzajavorek · Pull Request #2097 · apify/apify-docs

honzajavorek · 2025-11-18T13:51:02Z

This is a proof of concept how we could test exercises in the academy:

Exercises target real world websites, so they can easily break without us knowing
We could test them e.g. weekly, so it's not too noisy
The solution should be as simple as possible, and capable running both JavaScript and Python
While working on the PoC I actually discovered one exercise which is broken 95% of time due to aggressive anti-scraping protections, so I changed it
The test can be executed with bats -r --print-output-on-failure .
Bats is a simple testing framework based on Bash, which allows to run arbitrary programs and evaluate their output - https://github.com/bats-core/bats-core Not sure yet how to get it inside the CI, but on macOS it's just brew install bats

Todo:

Note

Adds a monthly GitHub Action and Bats-based suite to run Academy JS/Python exercise solutions, embeds solutions into lessons, and updates docs/ignore files.

CI:
- Add monthly and manual workflow .github/workflows/test-academy.yml to run Academy exercises via Node (npm) and Python (uv).
Testing:
- Introduce Bats-based tests for Academy exercises: sources/academy/**/exercises/test.bats (JS & Python) executing solutions and asserting outputs.
- New npm script test:academy and dev dependency bats.
Docs/Academy content:
- Embed executable exercise solutions using CodeBlock + !!raw-loader across JS/Python lessons (04–12), replacing inline snippets.
- Update several exercises (e.g., switch example from AliExpress to LEGO) and add multiple new solution files (Cheerio/BeautifulSoup/Crawlee).
Repo:
- Update .gitignore to exclude exercise artifacts (storage, node_modules, package*.json, dataset.json).
- Document testing process in CONTRIBUTING.md (broken links check + Academy exercises CI).

^{Written by Cursor Bugbot for commit 118a550. Configure here.}

apify-service-account · 2025-11-21T16:38:21Z

Preview for this PR was built for commit 05099c1 and is ready at https://pr-2097.preview.docs.apify.com!

tomnosek · 2025-11-24T10:28:50Z

From my perspective, we should design them in a way that they run automatically and do not require manual start-up. Otherwise, we'll forget about it eventually.

honzajavorek · 2025-11-24T12:20:06Z

Sure, I'd make a GitHub Action, which runs like once a week or once a month (depends on our ability to fix the exercises, doesn't make sense to run them too often).

honzajavorek · 2025-11-24T12:20:47Z

(Creating such GitHub Action is a matter of a few lines and I'll add it to this PR)

apify-service-account · 2025-11-24T13:25:15Z

Preview for this PR was built for commit cf913593 and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account · 2025-11-24T13:43:39Z

Preview for this PR was built for commit cb4959b2 and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account · 2025-11-24T13:47:22Z

Preview for this PR was built for commit 5e78a4db and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account · 2025-11-24T13:51:35Z

Preview for this PR was built for commit 463457b9 and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account · 2025-11-24T13:57:30Z

Preview for this PR was built for commit 0214184a and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account · 2025-11-24T17:21:28Z

Preview for this PR was built for commit b930d1f and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account · 2025-11-25T09:31:41Z

Preview for this PR was built for commit 4fccbf45 and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account · 2025-11-25T12:02:32Z

Preview for this PR was built for commit 96ae391 and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account · 2025-11-25T12:06:49Z

Preview for this PR was built for commit 44d7b1a and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account · 2025-11-25T13:58:04Z

Preview for this PR was built for commit ddf09129 and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account · 2025-11-25T14:43:03Z

Preview for this PR was built for commit 6c5f05bb and is ready at https://pr-2097.preview.docs.apify.com!

apify-service-account · 2025-11-25T14:49:50Z

Preview for this PR was built for commit 118a550 and is ready at https://pr-2097.preview.docs.apify.com!

honzajavorek · 2025-11-25T14:49:51Z

I think this should be ready now. The Bats tests currently do fail, because there are failing exercises, so that's expected. But the testing infrastructure is solid and this PR is about the infrastructure. I've set the frequency of the tests to monthly and let's see. I didn't want to snowball this PR, so I recorded the failures and other issues separately and I'll work on them in subsequent PRs:

honzajavorek · 2025-11-25T14:51:31Z

    - Run `vale sync` to download styles
    - Configure exceptions in `accepts.txt`

+### Testing


@TC-MO Can you take a look at this README change, please? Does it make sense this way?

honzajavorek · 2025-11-25T14:54:56Z

 <Exercises />

-### Scrape AliExpress
+### Scrape LEGO


This is the only I fixed. After this one I decided fixing the exercises should be in separate PRs, not in this one: #2113

cursor

This PR is being reviewed by Cursor Bugbot

Details

Your team is on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle for each member of your team.

To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

Comment @cursor review or bugbot run to trigger another review on this PR

…atory comment

…ity of uv options

#2112 )

apify-service-account · 2026-01-14T08:02:39Z

Preview for this PR was built for commit 3bde6d6 and is ready at https://pr-2097.preview.docs.apify.com!

B4nan · 2026-01-14T14:28:07Z

+const crawler = new CheerioCrawler({
+  async requestHandler({ $, request, enqueueLinks, pushData }) {
+    if (request.label === 'DRIVER') {
+      const info = {};
+      for (const itemElement of $('.common-driver-info li').toArray()) {
+        const name = $(itemElement).find('span').text().trim();
+        const value = $(itemElement).find('h4').text().trim();
+        info[name] = value;
+      }
+
+      const detail = {};
+      for (const linkElement of $('.driver-detail--cta-group a').toArray()) {
+        const name = $(linkElement).find('p').text().trim();
+        const value = $(linkElement).find('h2').text().trim();
+        detail[name] = value;
+      }
+
+      const dob = info.DOB ?? '';
+      const [dobDay = '', dobMonth = '', dobYear = ''] = dob.split('/');
+
+      await pushData({
+        url: request.url,
+        name: $('h1').text().trim(),
+        team: detail.Team,
+        nationality: info.Nationality,
+        dob: dobYear && dobMonth && dobDay ? `${dobYear}-${dobMonth}-${dobDay}` : null,
+        instagram_url: $(".common-social-share a[href*='instagram']").attr('href') ?? null,
+      });
+    } else {
+      await enqueueLinks({ selector: '.teams-driver-item a', label: 'DRIVER' });
+    }
+  },
+});


this would be better done via router: https://crawlee.dev/js/api/core/class/Router

(up to you if you want to keep the old version, nothing wrong with that functionality-wise)

Suggested change

const crawler = new CheerioCrawler({

async requestHandler({ $, request, enqueueLinks, pushData }) {

if (request.label === 'DRIVER') {

const info = {};

for (const itemElement of $('.common-driver-info li').toArray()) {

const name = $(itemElement).find('span').text().trim();

const value = $(itemElement).find('h4').text().trim();

info[name] = value;

}

const detail = {};

for (const linkElement of $('.driver-detail--cta-group a').toArray()) {

const name = $(linkElement).find('p').text().trim();

const value = $(linkElement).find('h2').text().trim();

detail[name] = value;

}

const dob = info.DOB ?? '';

const [dobDay = '', dobMonth = '', dobYear = ''] = dob.split('/');

await pushData({

url: request.url,

name: $('h1').text().trim(),

team: detail.Team,

nationality: info.Nationality,

dob: dobYear && dobMonth && dobDay ? `${dobYear}-${dobMonth}-${dobDay}` : null,

instagram_url: $(".common-social-share a[href*='instagram']").attr('href') ?? null,

});

} else {

await enqueueLinks({ selector: '.teams-driver-item a', label: 'DRIVER' });

}

},

});

const crawler = new CheerioCrawler();

crawler.router.addDefaultHandler(async ({ enqueueLinks }) => {

await enqueueLinks({ selector: '.teams-driver-item a', label: 'DRIVER' });

});

crawler.router.addHandler('DRIVER', async ({ $, request, enqueueLinks, pushData }) => {

const info = {};

for (const itemElement of $('.common-driver-info li').toArray()) {

const name = $(itemElement).find('span').text().trim();

const value = $(itemElement).find('h4').text().trim();

info[name] = value;

}

const detail = {};

for (const linkElement of $('.driver-detail--cta-group a').toArray()) {

const name = $(linkElement).find('p').text().trim();

const value = $(linkElement).find('h2').text().trim();

detail[name] = value;

}

const dob = info.DOB ?? '';

const [dobDay = '', dobMonth = '', dobYear = ''] = dob.split('/');

await pushData({

url: request.url,

name: $('h1').text().trim(),

team: detail.Team,

nationality: info.Nationality,

dob: dobYear && dobMonth && dobDay ? `${dobYear}-${dobMonth}-${dobDay}` : null,

instagram_url: $(".common-social-share a[href*='instagram']").attr('href') ?? null,

});

});

Oh! This is cool! I was wondering if there's JS alternative to the project structure which in Python Crawlee can be achieved by the @crawler.router.handler(...) decorators. And – surprisingly – it has even the same name, router 🤦‍♂️

This needs to be done throughout the whole lesson, so I filed #2181 and will tackle it separately. Linking this code suggestion so that it's not lost! 📁

@B4nan

Fix #2181 Address #2097 (comment) by @B4nan

@B4nan

Fix #2181, address #2097 (comment) by @B4nan  --- > [!NOTE] > **Low Risk** > Low risk documentation/exercise refactor that changes sample code structure to the newer Crawlee `router` API without altering the scraping behavior. > > **Overview** > Refactors the Crawlee lesson (`12_framework.md`) to construct `CheerioCrawler` without an inline `requestHandler` and instead register routing via `crawler.router.addDefaultHandler()` plus labeled `addHandler()` functions (e.g. `DETAIL`, `IMDB_SEARCH`, `IMDB`). > > Updates the associated exercise solutions (`crawlee_f1_drivers.mjs`, `crawlee_netflix_ratings.mjs`) to match the router-based pattern, removing `request.label` branching and splitting listing/search/detail logic into dedicated handlers while keeping the same data extraction and dataset export steps. > > <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit 1aa2eb6. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup>

honzajavorek added the t-academy Issues related to Web Scraping and Apify academies. label Nov 18, 2025

honzajavorek changed the title ~~Test Academy exercises~~ chore: test Academy exercises Nov 18, 2025

honzajavorek force-pushed the honzajavorek/test-exercises branch from ebf76a7 to 05099c1 Compare November 21, 2025 16:35

honzajavorek force-pushed the honzajavorek/test-exercises branch from 0214184 to b930d1f Compare November 24, 2025 17:18

honzajavorek force-pushed the honzajavorek/test-exercises branch from 4fccbf4 to 96ae391 Compare November 25, 2025 11:59

honzajavorek mentioned this pull request Nov 25, 2025

Fix broken exercises #2113

Closed

7 tasks

honzajavorek force-pushed the honzajavorek/test-exercises branch from 6c5f05b to 118a550 Compare November 25, 2025 14:46

honzajavorek marked this pull request as ready for review November 25, 2025 14:50

honzajavorek requested a review from B4nan November 25, 2025 14:50

honzajavorek commented Nov 25, 2025

View reviewed changes

Comment thread package.json

honzajavorek commented Nov 25, 2025

View reviewed changes

cursor Bot reviewed Nov 25, 2025

View reviewed changes

honzajavorek added 20 commits January 14, 2026 08:55

chore: one does not simply npm install

390a890

style: make linter happy

44da821

chore: make sure there is no schedule until we merge this, add explan…

ab61f89

…atory comment

refactor: simplify the tests

9dbdc11

docs: document lychee and academy testing

9ca5758

refactor: make exercises testable

c6fceb3

fix: avoid the yes option, fix crawlee installation, improve readabil…

9c589f3

…ity of uv options

chore: make the tests more meaningful

bd72f32

chore: improve the JS test suite

d65a95e

style: make the code linter happy

820b1d4

style: condense and fix the solutions markup

f439180

chore: setup and teardown for Python

d9182fd

chore: fix the JS test suite not to rely on npx --package

5fcc7a2

fix: improve the Python test suite and fix solutions using Crawlee (see

1ad107f

#2112 )

style: fix markup

2f3cb5c

chore: fix typo

3d9bbba

chore: enable only as a cron

3239895

chore: run monthly

971da08

style: fix markup

1e413ef

fix: address bugbot comments

3bde6d6

honzajavorek force-pushed the honzajavorek/test-exercises branch from 0382a09 to 3bde6d6 Compare January 14, 2026 07:59

B4nan approved these changes Jan 14, 2026

View reviewed changes

This was referenced Jan 14, 2026

fix: rework broken Academy exercises #2180

Merged

Use Crawlee router in the framework lesson of the JS course #2181

Closed

honzajavorek merged commit b5f7f68 into master Jan 15, 2026
10 checks passed

honzajavorek deleted the honzajavorek/test-exercises branch January 15, 2026 08:34

honzajavorek added a commit that referenced this pull request Feb 27, 2026

refactor: use router

1aa2eb6

Fix #2181 Address #2097 (comment) by @B4nan

honzajavorek mentioned this pull request Feb 27, 2026

refactor: change Academy code examples to use Crawlee router #2293

Merged

Conversation

honzajavorek commented Nov 18, 2025 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

apify-service-account commented Nov 21, 2025

Uh oh!

tomnosek commented Nov 24, 2025

Uh oh!

honzajavorek commented Nov 24, 2025

Uh oh!

honzajavorek commented Nov 24, 2025

Uh oh!

apify-service-account commented Nov 24, 2025

Uh oh!

apify-service-account commented Nov 24, 2025

Uh oh!

apify-service-account commented Nov 24, 2025

Uh oh!

apify-service-account commented Nov 24, 2025

Uh oh!

apify-service-account commented Nov 24, 2025

Uh oh!

apify-service-account commented Nov 24, 2025

Uh oh!

apify-service-account commented Nov 25, 2025

Uh oh!

apify-service-account commented Nov 25, 2025

Uh oh!

apify-service-account commented Nov 25, 2025

Uh oh!

apify-service-account commented Nov 25, 2025

Uh oh!

apify-service-account commented Nov 25, 2025

Uh oh!

apify-service-account commented Nov 25, 2025

Uh oh!

honzajavorek commented Nov 25, 2025

Uh oh!

honzajavorek Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

honzajavorek Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

This PR is being reviewed by Cursor Bugbot

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

apify-service-account commented Jan 14, 2026

Uh oh!

B4nan Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

honzajavorek Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

honzajavorek commented Nov 18, 2025 •

edited by cursor Bot

Loading