CONTRIBUTING.md

Development

Environment

To ensure smooth local development, we highly recommend installing Python 3.12 since the app has been written and tested for compatibility with this version.

We use Poetry for project management.

Dependencies

Use make command install-dev for installing all the necessary dependencies (Poetry and other Python packages specified in pyproject.toml):

make install-dev

Local execution

Prepare a storage/key_value_stores/default/INPUT.json file, here is an example:

{
  "startUrls": [{ "url": "https://crawlee.dev/" }],
  "maxCrawlingDepth": 1,
  "linkSelector": "a[href]",
  "linkPatterns": [".*crawlee\\.dev.*"],
  "proxyConfiguration": { "useApifyProxy": true },
  "pageFunction": "from typing import Any\nfrom bs4 import BeautifulSoup\n\n\ndef page_function(context: Context) -> Any:\n    soup = BeautifulSoup(context.response.content, \"html.parser\")\n    url = context.request[\"url\"]\n    title = soup.title.string if soup.title else None\n    return {\"url\": url, \"title\": title}\n"
}

Run the Actor using Apify CLI:

apify run --purge

Formatting

We use autopep8 and isort to automatically format the code to a common format. All tools are configured in the pyproject.toml.

Use make command format for formatting the code:

make format

Linting and type-checking

We use flake8 and many of its plugins (see pyproject.toml) for linting and mypy for type checking.

Use make command lint for running the linter:

make lint

Use make command type-check for running the type-checker:

make type-check

Documentation

We use the Google docstring format for documenting the code. We document every user-facing class or method.