finish the second lesson (no proofreading yet)

Author: honzajavorek

sources/academy/platform/scraping_with_apify_and_ai/02_developing_scraper_ai_agent.md

@@ -181,205 +181,148 @@ When Cursor opens the Actor's project folder, we'll see something similar to the
 
 ![Cursor ready](images/cursor-ready.webp)
 
-Now, finally, let's do some agentic coding!
+We can select files, and if we do so, we can browse and modify their content. The same as in the Web IDE. But as an addition, we now have an integrated AI agent which we can prompt and it'll do to the code at hand whatever we need.
+
+Finally, onto some agentic coding!
 
 ## Modifying code with Cursor
 
-:::note Course under construction
-This section hasn't been written yet. Come later, please!
-:::
+First, let's simplify how we can run the Actor. This will be our prompt:
 
-## Verifying changes
+```text
+Change the default input URL of the Actor
+to https://warehouse-theme-metal.myshopify.com/collections/sales
+```
 
-:::note Course under construction
-This section hasn't been written yet. Come later, please!
-:::
+After we submit the prompt, the Agent will start reading the code, planning, and working on completing the task. Before it runs commands, it'll ask us to approve them.
 
-## Pushing Actor to Apify
+![Cursor asking for approval to run a command](images/cursor-approve.webp)
 
-:::note Course under construction
-This section hasn't been written yet. Come later, please!
-:::
+When done, it'll print summary of its work and we'll be able to review all changes made.
 
-## Wrapping up
+![Cursor asking for a review of changes](images/cursor-review.webp)
 
-<!--
-
-## Creating an Actor
-
-Now let's use the Apify CLI to help us kick off a new Actor:
+We'll approve all changes and go to the command line to try out if the Actor now works as expected:
 
 ```text
-apify create warehouse-scraper
+apify run
 ```
 
-It starts a wizard where we can choose from various options. For each option, let's press <kbd>↵</kbd> to accept the default:
+We should see a scraper output like before, including the following line:
 
 ```text
-✔ Choose the programming language of your new Actor: JavaScript
-✔ Choose a template for your new Actor. You can check more information at https://apify.com/templates. Crawlee + Cheerio
-✔ Almost done! Last step is to install dependencies. Install dependencies
-
-...
-
-Success: ✅ Actor 'warehouse-scraper' created successfully!
+INFO  CheerioCrawler: Processing page: https://warehouse-theme-metal.myshopify.com/collections/sales
+```
 
-Next steps:
+That's our first successful change to the Actor with an AI agent! Without back-and-forth between the IDE and an AI chat like ChatGPT. Now before pushing this change back to Apify, let's do one more improvement to the scraper.
 
-cd "warehouse-scraper"
-apify run
+## Scraping prices
 
-💡 Tip: Use 'apify push' to deploy your Actor to the Apify platform
-📖 Docs: https://docs.apify.com/platform/actors/development
-🌱 Git repository initialized in 'warehouse-scraper'. You can now commit and push your Actor to Git.
-```
+In the previous lesson, we noticed that the prices in our resulting dataset are in a rather raw shape:
 
-Now that's a lot of output, but no worries, the important part is that we've successfully used a template to set up a new Actor project!
+| name | url | price |
+| --- | --- | --- |
+| JBL Flip 4 Waterproof Portable Bluetooth Speaker | https://warehouse-theme-metal.myshopify.com/products/jbl-flip-4-waterproof-portable-bluetooth-speaker | Sale price$74.95 |
+| Sony XBR-950G BRAVIA 4K HDR Ultra HD TV | https://warehouse-theme-metal.myshopify.com/products/sony-xbr-65x950g-65-class-64-5-diag-bravia-4k-hdr-ultra-hd-tv | Sale priceFrom $1,398.00 |
+| Sony SACS9 10" Active Subwoofer | https://warehouse-theme-metal.myshopify.com/products/sony-sacs9-10-inch-active-subwoofer | Sale price$158.00 |
 
-A new directory `warehouse-scraper` has been created for us, with a variety of files and directories inside. The output instructs us to go to this new project directory, so let's do it:
+Let's change that. We'll prompt the agent like this, with a clear example of what we want:
 
 ```text
-cd "warehouse-scraper"
+Change the code so that the Actor saves prices as numbers.
+Because some prices are "from", let's call the "price" field
+"minPrice" instead, as in minimum price. Example follows.
+
+Before:
+Sale price$74.95
+Sale priceFrom $1,398.00
+Sale price$158.00
+
+After:
+74.95
+1398.00
+158.00
 ```
 
-Now we can run commands which control this new project. We didn't change the template in any way though, so it won't scrape the Warehouse store for us yet.
-
-Out of the box, the template includes a sample Actor that walks through the [crawlee.dev](https://crawlee.dev/) website and downloads all its pages. This process is called _crawling_, and the sample Actor uses a crawling tool called Crawlee, so its documentation is chosen as a sample target website. Let's see if we can run it:
+When the agent is done, we'll approve the changes and verify in the command line that the Actor runs locally:
 
 ```text
 apify run
 ```
 
-If we see a flood of output mentioning something called `CheerioCrawler`, it means the template works and we can move on to editing its files so that it does what we want.
+It runs, that's nice! But looking at the output, we can't really verify what exactly gets scraped! When we're at it, let's change that with another prompt:
 
 ```text
-...
-INFO  CheerioCrawler: Starting the crawler.
-INFO  CheerioCrawler: enqueueing new URLs
-INFO  CheerioCrawler: Crawlee · Build reliable crawlers. Fast. {"url":"https://crawlee.dev/"}
-...
-INFO  CheerioCrawler: Finished! Total 107 requests: 107 succeeded, 0 failed. {"terminal":true}
+In the output of the scraper I want to see
+how the items being saved look like.
 ```
 
-We're done with commands for now, but do not close the Terminal or Command Prompt window yet, as we'll soon need it again.
-
-:::caution Debugging
-If we run into issues with the template wizard or the sample Actor, let's share this tutorial with [ChatGPT](https://chatgpt.com/), include the errors we saw, and ask for help debugging.
-:::
-
-## Scraping products
-
-Now we're ready to get our own scraper done. We'll open the `src` directory inside the Actor project and find a file called `main.js`.
-
-We'll open it in a _plain text editor_. Every operating system includes one: Notepad on Windows, TextEdit on macOS, and similar tools on Linux.
-
-:::danger Avoid rich text editors
-Let's not use a _rich text editor_, such as Microsoft Word. They're great for human-readable documents with rich formatting, but for code editing, we'll use either dedicated coding editors, or the simplest tool possible.
-:::
-
-In the editor, we can see JavaScript code. Let's select all the code and copy to our clipboard. Then we'll open a _new ChatGPT conversation_ and start with a prompt like this:
-
-```text
-I'm building an Apify Actor that will run on the Apify platform.
-I need to modify a sample template project so it downloads
-https://warehouse-theme-metal.myshopify.com/collections/sales,
-extracts all products in Sales, and returns data with
-the following information for each product:
-
-- Product name
-- Product detail page URL
-- Price
-
-Before the program ends, it should log how many products it collected.
-Code from main.js follows. Reply with a code block containing
-a new version of that file.
-```
-
-We'll use <kbd>Shift+↵</kbd> to add a few empty lines, then paste the code from our clipboard. After submitting, the AI chat should return a large code block with a new version of `main.js`. Copy it, go back to our text editor, and replace the original `main.js` content.
-
-:::info Code and colors
-Code is plain text. Some tools color it to make it easier to read, and ChatGPT does this by default. Plain text editors usually show code in black and white, and that's completely fine.
-:::
-
-When we're done, we must not forget to _save the change_ with <kbd>Ctrl+S</kbd> or, on macOS, <kbd>Cmd+S</kbd>. Now let's see if the new code works. To run the program, let's go back to Terminal (macOS/Linux) or PowerShell (Windows) and use Apify CLI again:
+We'll approve all changes and go to the command line again:
 
 ```text
 apify run
 ```
 
-If all goes well, the output should be similar to this:
+Now, the output of the scraper contains the actual items being scraped and we can verify we've been successful with changing the format of the prices (they appear at the very end of each line):
 
 ```text
-Run: npm run start
-
-> warehouse-scraper@0.0.1 start
-> node src/main.js
-
-INFO  System info {"apifyVersion":"3.6.0","apifyClientVersion":"2.22.2","crawleeVersion":"3.16.0","osType":"Darwin","nodeVersion":"v25.6.1"}
 ...
-INFO  CheerioCrawler: Starting the crawler.
 INFO  CheerioCrawler: Processing page: https://warehouse-theme-metal.myshopify.com/collections/sales
+INFO  CheerioCrawler: Saving dataset item {"name":"JBL Flip 4 Waterproof Portable Bluetooth Speaker","url":"https://warehouse-theme-metal.myshopify.com/products/jbl-flip-4-waterproof-portable-bluetooth-speaker","minPrice":74.95}
+INFO  CheerioCrawler: Saving dataset item {"name":"Sony XBR-950G BRAVIA 4K HDR Ultra HD TV","url":"https://warehouse-theme-metal.myshopify.com/products/sony-xbr-65x950g-65-class-64-5-diag-bravia-4k-hdr-ultra-hd-tv","minPrice":1398}
+INFO  CheerioCrawler: Saving dataset item {"name":"Sony SACS9 10\" Active Subwoofer","url":"https://warehouse-theme-metal.myshopify.com/products/sony-sacs9-10-inch-active-subwoofer","minPrice":158}
+INFO  CheerioCrawler: Saving dataset item {"name":"Sony PS-HX500 Hi-Res USB Turntable","url":"https://warehouse-theme-metal.myshopify.com/products/sony-ps-hx500-hi-res-usb-turntable","minPrice":398}
 ...
-INFO  CheerioCrawler: Finished!
-INFO  Total products collected: 24
 ```
 
-This output says `Total products collected: 24`. The Sales page displays 24 products per page and contains 50 products in total.
+Now let's push the changes back to Apify, so that our scheduled scraping happening on the platform can benefit from the improvements we've made locally on our computer.
+
+:::tip Automatically approving changes
 
-Depending on whether ChatGPT decided to walk through all pages or scrape just the first one, we might get 24 or more products. For now, any sign that it scraped products is good news.
+If you'll grow tired of approvals, you can enable _auto-keep_. Go to **Cursor** → **Settings…** → **Cursor Settings** → **Agents** → **Applying Changes** and turn off **Inline Diffs**.
 
-:::caution Debugging
-If our program crashes instead, let's copy the error message, send it to our ChatGPT conversation, and ask for a fix.
 :::
 
-## Exporting to CSV
+## Pushing Actor to Apify
 
-Our program likely works, but we haven't seen the data yet. Let's add a CSV export. CSV is a format most data apps can read, including Microsoft Excel, Google Sheets, and Apple Numbers. Let's continue our ChatGPT conversation with:
+To replace the Actor files living on the Apify platform with the ones we have locally, we can run the following command:
 
 ```text
-Before the program ends, I want it to export all data
-as "dataset.csv" in the current working directory.
+apify push
 ```
 
-ChatGPT should return a new code block with CSV export added. Let's replace `main.js` with that version and save our changes. Then let's run the scraper again:
+The command can take a while to finish, because it also immediately triggers a build. Once it's done, the new version of the Actor is ready to be ran. The output of the command ends with these two lines:
 
 ```text
-apify run
+...
+Actor detail https://console.apify.com/actors/EL7U7aNddXOzwEJ66
+Success: Actor was deployed to Apify cloud and built there.
 ```
 
-In the project directory, a new file called `dataset.csv` should emerge. We can use any of the programs mentioned earlier to check what's inside:
-
-| productName | productUrl | price |
-| --- | --- | --- |
-| JBL Flip 4 Waterproof Portable Bluetooth Speaker | https://warehouse-theme-metal.myshopify.com/products/jbl-flip-4-waterproof-portable-bluetooth-speaker | Sale price$74.95 |
-| Sony XBR-950G BRAVIA 4K HDR Ultra HD TV | https://warehouse-theme-metal.myshopify.com/products/sony-xbr-65x950g-65-class-64-5-diag-bravia-4k-hdr-ultra-hd-tv | Sale priceFrom $1,398.00 |
-| Sony SACS9 10" Active Subwoofer | https://warehouse-theme-metal.myshopify.com/products/sony-sacs9-10-inch-active-subwoofer | Sale price$158.00 |
+We'll follow the link to our browser and in the Apify interface, we'll click the **Start** button. Soon we should see items popping up in the **Output** section. For a full overwiew, let's switch to **All fields** again:
 
-…and so on. Looks good!
+![Modified Apify output](images/apify-output-modified.webp)
 
-Well, does it? If we look closely, the prices include extra text, which isn't ideal. We'll improve this in one of the next lessons. We'll also improve the workflow so we don't have to keep copying and pasting.
+We've done it, the prices save as numbers!
 
-Despite a few flaws, we've successfully created a first working prototype of a price-watching app with no coding knowledge. And with a bit of extra command-line work, we now have something we can deploy to a platform where it can run regularly and reliably. In the next lesson, we'll do exactly that.
+:::tip Specifying output schema
 
--->
+If we didn't want to always click on **All fields** to see full items, we need to specify an [output schema](https://docs.apify.com/platform/actors/development/actor-definition/output-schema) so that the platform knows what it can expect and how it should display it in the interface. With Cursor, such change is just a single prompt away:
 
-<!--
-Explaining benefits (delegation and independent work, AGENTS.md). Getting environment ready, learning the ropes with a GUI/TUI. Using the `apify` CLI to start a project. Creating a basic scraper which does what we need.
+```text
+Change the output schema of the Actor
+so that it represents the items being
+saved the best way in the Apify interface.
+```
 
-In lesson 3, students would try to make changes via ChatGPT and see that it gets tedious, which leads to introducing an agent-based IDE to work inside the template more comfortably.
+:::
 
-The lesson should use Cursor (or Google Antigravity). Only if it truly scales to zero as they claim and it is not required to have a paid account to try an agent. Minimal friction, just install – beats any other decision factors.
+## Wrapping up
 
-If the paragraph above turns out being a wrong direction, we should use VS Code and tell people to spend $10 to try Copilot. VS Code is mainstream. Paying for Copilot is the cheapest agent offering, and it's quite powerful.
--->
+We've been installing and setting up a lot, but once we got our environment ready, we could reap the benefits of fast changes to our scraper.
 
-<!--
-We'll choose [Cursor](https://cursor.com/), because it has a free plan and it's beginner-friendly.
+With a single prompt we tackled a significant change in how our app stores the prices. And we still didn't need to know any coding.
 
-#### Installing development environment
-Explaining benefits (delegation and independent work, AGENTS.md). Getting environment ready. Use https://docs.apify.com/platform/actors/development/quick-start/build-with-ai
-#### Scraping vendor names
-Learning the ropes with a GUI/TUI, prompting the agent to update the code so that it scrapes vendor names. Run the program again, get better results.
+To improve our project further, we ask the agent to perform a change, review and approve its work, then execute `apify run` in the command line to verify how it works, and finally `apify push` to upload our Actor files to Apify.
 
-Teaser: Explain why this is fragile. In the next lesson we'll learn how to develop features of the scraper in a robust way by first specifying them as documentation.
--->
+In the next lesson, we'll take a look at how we can develop our scraper by documenting how it should behave instead of prompting the AI agent feature by feature, without a track record of our intentions.

sources/academy/platform/scraping_with_apify_and_ai/03_docs_driven_prompting.md

@@ -6,6 +6,7 @@ unlisted: true
 ---
 
 <!--
+explain AGENTS.md
 Improving the README, e.g. input output. Pointing the agent to the README and turning the design to reality.
 -->