diff --git a/episodes/16-verifying-code-style-linters.md b/episodes/16-verifying-code-style-linters.md index 462069cc6..09516ca74 100644 --- a/episodes/16-verifying-code-style-linters.md +++ b/episodes/16-verifying-code-style-linters.md @@ -27,7 +27,7 @@ We can also use tools, called [**code linters**](https://en.wikipedia.org/wiki/Lint_%28software%29), to identify consistency issues in a report-style. Linters analyse source code to identify and report on stylistic and even programming errors. -Let us look at a very well used one of these called `pylint`. +Let us look at a very well used one of these called `ruff`. First, let us ensure we are on the `style-fixes` branch once again. @@ -35,10 +35,11 @@ First, let us ensure we are on the `style-fixes` branch once again. $ git switch style-fixes ``` -Pylint is just a Python package so we can install it in our virtual environment using: +Ruff is written in Rust, which makes it very fast as compared to other Python linters. +It can be installed in our virtual environment using `pip`: ```bash -$ python3 -m pip install pylint +$ python3 -m pip install ruff ``` We should also update our `requirements.txt` with this new addition: @@ -47,18 +48,20 @@ We should also update our `requirements.txt` with this new addition: $ python3 -m pip freeze --exclude-editable > requirements.txt ``` -Pylint is a command-line tool that can help our code in many ways: +Ruff is a command-line tool that can help our code in many ways: - **Check PEP 8 compliance:** whilst in-IDE context-sensitive highlighting helps us stay consistent with PEP 8 as we write code, this tool provides a full report -- **Perform basic error detection:** Pylint can look for certain Python type errors +- **Perform basic error detection:** Ruff can look for certain Python type errors - **Check variable naming conventions**: - Pylint often goes beyond PEP 8 to include other common conventions, + Ruff can go beyond PEP 8 to include other common conventions, such as naming variables outside of functions in upper case - **Customisation**: you can specify which errors and conventions you wish to check for, and those you wish to ignore +- **Automatic fixes**: + Ruff supports automatic fixes for some lint errors. -Pylint can also identify **code smells**. +Ruff can also identify **code smells**. ::::::::::::::::::::::::::::::::::::::::: callout @@ -82,82 +85,126 @@ by Kent Beck and Martin Fowler in :::::::::::::::::::::::::::::::::::::::::::::::::: -Pylint recommendations are given as warnings or errors, -and Pylint also scores the code with an overall mark. +In addition, Ruff includes the functionality of a formatter: it can be used +to apply a stardardized format to Python files, so that the resulting layout +makes the code more consistent and readable. We will not cover Ruff's use as +a formatter here, but you can learn more about this topic from the +[Ruff documentation](https://docs.astral.sh/ruff/formatter/). + +Ruff recommendations are given as warnings or errors. We can look at a specific file (e.g. `inflammation-analysis.py`), or a package (e.g. `inflammation`). Let us look at our `inflammation` package and code inside it (namely `models.py` and `views.py`). From the project root do: ```bash -$ pylint inflammation +$ ruff check inflammation ``` You should see an output similar to the following: ```output -************* Module inflammation.models -inflammation/models.py:13:23: C0303: Trailing whitespace (trailing-whitespace) -inflammation/models.py:34:0: C0305: Trailing newlines (trailing-newlines) -************* Module inflammation.views -inflammation/views.py:4:0: W0611: Unused numpy imported as np (unused-import) - ------------------------------------------------------------------- -Your code has been rated at 8.50/10 (previous run: 8.50/10, +0.00) +F401 [*] `numpy` imported but unused + --> inflammation/views.py:4:17 + | +3 | from matplotlib import pyplot as plt +4 | import numpy as np + | ^^ + | +help: Remove unused import: `numpy` + +Found 1 error. +[*] 1 fixable with the `--fix` option. ``` Your own outputs of the above commands may vary depending on how you have implemented and fixed the code in previous exercises and the coding style you have used. -The five digit codes, such as `C0303`, are unique identifiers for warnings, -with the first character indicating the type of warning. -There are five different types of warnings that Pylint looks for, -and you can get a summary of them by doing: +The alphanumeric codes, such as `F401`, are unique identifiers for lint rules. +Ruff implements rules as derived by other tools and conventions - the starting +letter of the code refers to the tool or convention the rule is derived from. +To learn more about a lint rule, e.g. `F401`, you can run: + +```bash +$ ruff rule F401 +``` +Ruff will tell you that `F401`, as all other `F`-rules, are derived from the +[Pyflakes](https://pypi.org/project/pyflakes/) Python linter, and give you +examples, explanations and some reasoning on why the rule exists. +The full list of rules that Ruff supports is available +[as part of the Ruff documentation](https://docs.astral.sh/ruff/rules/). + +Note that by default Ruff does not check for all rules, but it enables +only a subset that is considered a reasonable choice to identify common errors. +You can enable a specific set of rules using the `--select` option. For instance, +try to include the following set of rules, which are derived from some of the most +popular tools, such as [pycodestyle](https://pypi.org/project/pycodestyle/) +(`E` rules) and [isort](https://pypi.org/project/isort/) (`I` rules): ```bash -$ pylint --long-help +$ ruff check --select E,F,I inflammation ``` -Near the end you'll see: +Ruff will identify more problems in the codebase: ```output - Output: - Using the default text output, the message format is : - MESSAGE_TYPE: LINE_NUM:[OBJECT:] MESSAGE - There are 5 kind of message types : - * (C) convention, for programming standard violation - * (R) refactor, for bad code smell - * (W) warning, for python specific problems - * (E) error, for probable bugs in the code - * (F) fatal, if an error occurred which prevented pylint from doing - further processing. +I001 [*] Import block is un-sorted or un-formatted + --> inflammation/views.py:3:1 + | +1 | """Module containing code for plotting inflammation data.""" +2 | +3 | / from matplotlib import pyplot as plt +4 | | import numpy as np + | |__________________^ + | +help: Organize imports + +F401 [*] `numpy` imported but unused + --> inflammation/views.py:4:17 + | +3 | from matplotlib import pyplot as plt +4 | import numpy as np + | ^^ + | +help: Remove unused import: `numpy` + +Found 2 errors. +[*] 2 fixable with the `--fix` option. ``` -So for an example of a Pylint Python-specific `warning`, -see the "W0611: Unused numpy imported as np (unused-import)" warning. - -It is important to note that while tools such as Pylint are great at giving you +It is important to note that while tools such as Ruff are great at giving you a starting point to consider how to improve your code, they will not find everything that may be wrong with it. -::::::::::::::::::::::::::::::::::::::::: callout +::::::::::::::::::::::::::::::::::::::: challenge -## How Does Pylint Calculate the Score? +## Exercise: Add Ruff configurations to the `pyproject.toml` file -The Python formula used is -(with the variables representing numbers of each type of infraction -and `statement` indicating the total number of statements): +You can define the Ruff configuration for a project by adding a section to the +`pyproject.toml` file. For instance, you can define the set of rules to be checked +for your codebase. Following [the Ruff documentation](https://docs.astral.sh/ruff/linter/), +add a section to the `pyproject.toml` to enable the `E`, `W`, `F`, `UP`, `A`, `B`, `SIM`, +and `I` rules for the project. Verify that the configuration is respected when running +`ruff` (without the `--select` option): ```bash -10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10) +$ ruff check inflammation +``` + +:::::::::::::::::::::::::::::::: solution + +Add the following section to the `pyproject.toml`: + +```toml +[tool.ruff.lint] +select = ["E", "W", "F", "UP", "A", "B", "SIM", "I"] ``` -For example, with a total of 31 statements of models.py and views.py, -with a count of the errors shown above, we get a score of 8.00. -Note whilst there is a maximum score of 10, given the formula, -there is no minimum score - it is quite possible to get a negative score! +Running `ruff check inflammation` should indeed show problems with some of +the `W` and `I` rules, which are not enabled with the default Ruff settings. +:::::::::::::::::::::::::::::::::::::::::: :::::::::::::::::::::::::::::::::::::::::::::::::: @@ -165,13 +212,15 @@ there is no minimum score - it is quite possible to get a negative score! ## Exercise: Further Improve Code Style of Our Project -Select and fix a few of the issues with our code that Pylint detected. -Make sure you do not break the rest of the code in the process and that the code still runs. -After making any changes, run Pylint again to verify you have resolved these issues. +Select and fix a few of the issues with our code that Ruff detected. +You can try using the Ruff's `--fix` command-line option to automatically fix +(some of) the issues. If you manually edit the code, make sure you do not break +the rest of the code in the process and that the code still runs. +After making any changes, run Ruff again to verify you have resolved these issues. :::::::::::::::::::::::::::::::::::::::::::::::::: -Make sure you commit and push `requirements.txt` +Make sure you commit and push `requirements.txt`, `pyproject.toml`, and any file with further code style improvements you did on to `style-fixes` branch and then merge all these changes into your development branch. @@ -184,8 +233,8 @@ with GitHub Actions - we will come back to automated linting in the episode on ["Diagnosing Issues and Improving Robustness"](24-diagnosing-issues-improving-robustness.md). ```bash -$ git add requirements.txt -$ git commit -m "Added Pylint library" +$ git add requirements.txt pyproject.toml +$ git commit -m "Added Ruff library" $ git push origin style-fixes $ git switch develop $ git merge style-fixes @@ -197,17 +246,16 @@ $ git push origin develop ## Optional Exercise: Improve Code Style of Your Other Python Projects If you have a Python project you are working on or you worked on in the past, -run it past Pylint to see what issues with your code are detected, if any. - +run it past Ruff to see what issues with your code are detected, if any. :::::::::::::::::::::::::::::::::::::::::::::::::: ::: challenge -## Optional Exercise: More on Pylint +## Optional Exercise: More on Ruff Checkout [this optional exercise](17-section1-optional-exercises.md) -to learn more about `pylint`. +to learn more about `ruff`. ::: diff --git a/episodes/17-section1-optional-exercises.md b/episodes/17-section1-optional-exercises.md index 428c19c6c..4ab36ca7e 100644 --- a/episodes/17-section1-optional-exercises.md +++ b/episodes/17-section1-optional-exercises.md @@ -48,7 +48,7 @@ Some suggestions to try: - [Sublime Text](https://www.sublimetext.com/) - [RStudio](https://posit.co/download/rstudio-desktop/) -The IDEs listed above are advanced source code editors capable of functioning as IDEs by manually installing plugins +The IDEs listed above are advanced source code editors capable of functioning as IDEs by manually installing plugins and add-ons for these tools to obtain more advanced features - such as support for a specific programming language or unit testing. What do you prefer, a lot of tooling out of the box or a lightweight editor with optional extensions? @@ -91,27 +91,38 @@ An open-source alternative is [mini-forge](https://github.com/conda-forge/minifo ::::::::::::::::::::::::::::::::::::::: challenge -## Exercise: Customise `pylint` +## Exercise: Customise `ruff` -Tell `pylint` to accept the maximum line length of 100 characters instead of the default 80. +Tell Ruff to accept the maximum line length of 100 characters instead of the default 79. -Hint: find out different ways in which you can configure `pylint` (e.g. via `pylint` command line interface or its configuration file). +Find out different ways in which you can configure `ruff` (e.g. via `ruff` command line interface or the project configuration file (`pyproject.toml`)). + +Hint: Note that the maximum line lenght will only be enforced if a rule on long-line violations (such as `E501`) is enabled. ::::::::::::::: solution ## Solution -### By passing an argument to `pylint` in the command line +### By passing an argument to Ruff in the command line -Specify the max line length as an argument: `pylint --max-line-length=100` +Specify the max line length as an argument, including all `E` rules: `ruff check --select E --line-length=100` ### Using a configuration file -You can create a file `.pylintrc` in the root of your project folder to overwrite `pylint` settings: +You can add a section to the `pyproject.toml` in the root of your project folder to overwrite Ruff settings: ``` -[FORMAT] -max-line-length=100 +... +[tool.ruff.lint] +select = [ + ... + "E", + ... +] + +[tool.ruff] +line-length = 100 +... ``` ::::::::::::::::::::::::: diff --git a/episodes/24-diagnosing-issues-improving-robustness.md b/episodes/24-diagnosing-issues-improving-robustness.md index 659765aab..479d6dc2b 100644 --- a/episodes/24-diagnosing-issues-improving-robustness.md +++ b/episodes/24-diagnosing-issues-improving-robustness.md @@ -189,7 +189,7 @@ Firstly, to make it easier to track what's going on, we can set up our IDE to ru ### Configuring the Test Framework -::: group-tab +::: group-tab ### PyCharm @@ -207,20 +207,20 @@ You can do this by: ### VS Code If you have not done so already, you will first need to configure the Pytest framework in VS Code. -Open the Test Explorer view (click on the test beaker icon on the VS Code Activity Bar on the left hand side). +Open the Test Explorer view (click on the test beaker icon on the VS Code Activity Bar on the left hand side). -You should see a `Configure Python Tests` button if a test framework is not enabled. +You should see a `Configure Python Tests` button if a test framework is not enabled. Clicking on it prompts you to select a test framework and a folder containing your tests (which in this project, is the `tests` subfolder). ![](fig/vscode-test-framework.png){alt='Setting up test framework in VS Code' .image-with-shadow width="1000px"} ![](fig/vscode-configure-pytest.png){alt='Setting up test framework in VS Code' .image-with-shadow width="1000px"} -Tests can be configured anytime by using the `Python: Configure Tests` command from the Command Palette -or by setting `python.testing.pytestEnabled` in the Settings editor or `settings.json` file (described in the VS Code Settings in the [episode on IDEs](/13-ides.md)). +Tests can be configured anytime by using the `Python: Configure Tests` command from the Command Palette +or by setting `python.testing.pytestEnabled` in the Settings editor or `settings.json` file (described in the VS Code Settings in the [episode on IDEs](/13-ides.md)). Each testing framework also has further specific configuration settings as described in the [Test configuration settings of the VS Code documentation for Python](https://code.visualstudio.com/docs/python/testing#_test-configuration-settings). -::: +::: ### Running the Tests @@ -360,7 +360,7 @@ Recall that the input `data` array we are using for the function is So the maximum inflammation for each patient should be `[3, 6, 9]`, whereas the debugger shows `[7, 8, 9]`. You can see that the latter corresponds exactly to the last column of `data`, and we can immediately conclude that we took the maximum along the wrong axis of `data`. -Now we have our answer, stop the debugging process. +Now we have our answer, stop the debugging process. ::: group-tab @@ -642,7 +642,7 @@ from inflammation.models import patient_normalise ]) def test_patient_normalise(test, expected, expect_raises): """Test normalisation works for arrays of one and positive integers.""" - + if expect_raises is not None: with pytest.raises(expect_raises): patient_normalise(np.array(test)) @@ -835,20 +835,28 @@ This approach is useful when explicitly checking the precondition is too costly. ## Improving Robustness with Automated Code Style Checks -Let us re-run Pylint over our project after having added some more code to it. +Let us re-run Ruff over our project after having added some more code to it. From the project root do: ```bash -$ pylint inflammation +$ ruff check inflammation ``` -You may see something like the following in Pylint's output: +You may see something like the following in Ruff's output: ```bash -************* Module inflammation.models -... -inflammation/models.py:60:4: W0622: Redefining built-in 'max' (redefined-builtin) -... +A001 Variable `max` is shadowing a Python builtin + --> inflammation/models.py:60:5 + | +58 | if np.any(data < 0): +59 | raise ValueError('inflammation values should be non-negative') +60 | max = np.nanmax(data, axis=1) + | ^^^ +61 | with np.errstate(invalid='ignore', divide='ignore'): +62 | normalised = data / max[:, np.newaxis] + | + +Found 1 error. ``` The above output indicates that by using the local variable called `max` @@ -869,78 +877,61 @@ push them to GitHub using our usual feature branch workflow. :::::::::::::::::::::::::::::::::::::::::::::::::: It may be hard to remember to run linter tools every now and then. -Luckily, we can now add this Pylint execution to our continuous integration builds +Luckily, we can now add this Ruff execution to our continuous integration builds as one of the extra tasks. -To add Pylint to our CI workflow, +To add Ruff to our CI workflow, we can add the following step to our `steps` in `.github/workflows/main.yml`: ```bash ... - - name: Check style with Pylint + - name: Check style with Ruff run: | - python3 -m pylint --fail-under=0 --reports=y inflammation + python3 -m ruff check --exit-zero --output-format=github inflammation ... ``` -Note we need to add `--fail-under=0` otherwise -the builds will fail if we do not get a 'perfect' score of 10! -This seems unlikely, so let us be more pessimistic. -We have also added `--reports=y` which will give us a more detailed report of the code analysis. +Note we need to add `--exit-zero` otherwise +the builds will fail if our codebase breaks any of the rule! +We have also added `--output-format=github` to format the output so that it can +more easily be visualized as part of the GitHub action logs. Then we can just add this to our repo and trigger a build: ```bash $ git add .github/workflows/main.yml -$ git commit -m "Add Pylint run to build" +$ git commit -m "Add Ruff run to build" $ git push origin test-suite ``` Then once complete, under the build(s) reports you should see -an entry with the output from Pylint as before, -but with an extended breakdown of the infractions by category -as well as other metrics for the code, -such as the number and line percentages of code, docstrings, comments, and empty lines. - -So we specified a score of 0 as a minimum which is very low. -If we decide as a team on a suitable minimum score for our codebase, -we can specify this instead. -There are also ways to specify specific style rules that shouldn't be broken -which will cause Pylint to fail, -which could be even more useful if we want to mandate a consistent style. - -We can specify overrides to Pylint's rules in a file called `.pylintrc` -which Pylint can helpfully generate for us. -In our repository root directory: - -```bash -$ pylint --generate-rcfile > .pylintrc -``` - -Looking at this file, you'll see it is already pre-populated. -No behaviour is currently changed from the default by generating this file, -but we can amend it to suit our team's coding style. +an entry with the output from Ruff as before, +but with a more concise format. + +We specified `--exit-zero` so that the builds will not fail even if some of +the linting rules are not followed. +If we decide as a team on a set of rules to be followed for our codebase, +we can remove this option. +With the agreed upon style rules that shouldn't be broken specified in the `pyproject.toml` file, +which will cause Ruff to fail, we can use the action to mandate a consistent style. + +We can also specify overrides to Ruff's rules in the `pyproject.toml` file. +For instance, we can ignore particular rules to suit our team's coding style. For example, a typical rule to customise - favoured by many projects - is the one involving line length. -You'll see it is set to 100, so let us set that to a more reasonable 120. -While we are at it, let us also set our `fail-under` in this file: +The default maximum line length as indicated by the +[Python PEP-8 coding conventions](https://peps.python.org/pep-0008/#maximum-line-length) +is 79 characters, so let us set that to a more reasonable 120: ```bash ... -# Specify a score threshold to be exceeded before program exits with error. -fail-under=0 -... -# Maximum number of characters on a single line. -max-line-length=120 +[tool.ruff] +line-length = 120 ... ``` -do not forget to remove the `--fail-under` argument to Pytest -in our GitHub Actions configuration file too, -since we do not need it anymore. - -Now when we run Pylint we will not be penalised for having a reasonable line length. -For some further hints and tips on how to approach using Pylint for a project, -see [this article](https://pythonspeed.com/articles/pylint/). +Now when we run Ruff we will not be penalised for having a reasonable line length. +For some further customization of Ruff for your project, see the +[Ruff Linter documentation](https://docs.astral.sh/ruff/linter/). ## Merging to `develop` Branch diff --git a/episodes/43-software-release.md b/episodes/43-software-release.md index 64a8c4901..cb4db02c7 100644 --- a/episodes/43-software-release.md +++ b/episodes/43-software-release.md @@ -187,7 +187,7 @@ Development dependencies are dependencies which are an essential part of your development process for a project, but are not required to run it. Common examples of developments dependencies are linters and test frameworks, -like `pylint` or `pytest`. +like `ruff` or `pytest`. When we add a dependency using Poetry, Poetry will add it to the list of dependencies in the `pyproject.toml` file, @@ -202,7 +202,7 @@ allowing us to distinguish between runtime and development dependencies. ```bash $ poetry add matplotlib numpy -$ poetry add --group dev pylint +$ poetry add --group dev ruff $ poetry install ```